Beyond Single Alerts: Building Intelligent Meta-Detection with Machine Learning in Elastic

Your SOC analysts are drowning in alerts. Each detection rule fires independently. A medium-severity alert here. A low-severity building block there. An ML anomaly over there. Individually, none of them screams "investigate immediately." But together? They might indicate a sophisticated attack in progress.

The problem is, your analysts don't have time to manually correlate dozens of low and medium severity alerts across hosts and users. They're too busy triaging the high-severity queue.

What if you could automatically identify when multiple seemingly unrelated alerts—including anomalies detected by machine learning—cluster together in ways that indicate real threats? What if you could reduce false positives while simultaneously catching more sophisticated attacks?

That's exactly what intelligent meta-detection with ML enables.

Especially when introducing AI in your workflow is prohibited, for different and sometimes valid reasons.

The Traditional Alert Problem

Most detection strategies operate on a simple principle: write rules that detect bad things, tune them to reduce false positives, escalate the ones that matter.

But this approach has fundamental limitations:

Individual alerts lack context. A user accessing an unusual system might be legitimate. But a user accessing an unusual system, plus multiple failed authentication attempts, plus an ML anomaly showing rare process execution patterns? That's a different story.

Low-severity alerts get ignored. Your analysts focus on critical and high-severity alerts. Medium and low-severity alerts pile up in the backlog. Building block rules—designed to be combined with other detections—rarely get the correlation analysis they need.

Sophisticated attacks fly under the radar. Advanced attackers know how detection works. They stay below alert thresholds, use living-off-the-land techniques, and spread their activity over time. Individual detections might fire, but nothing looks alarming enough to investigate.

Volume overwhelms pattern recognition. Even experienced analysts struggle to spot patterns when they're looking at hundreds of alerts per day. The signal gets lost in the noise.

Enter Machine Learning as the Foundation

This is where machine learning becomes powerful—not as a replacement for traditional detection rules, but as a foundation layer that identifies anomalies in alert patterns themselves.

Instead of just detecting malicious activity, you use ML to detect unusual patterns in your detections.

ML Jobs Watching the Alerts Index

The strategy starts with ML jobs that analyze your alerts index (.*alerts-security.alerts-*). These jobs look, as an example, for anomalies like:

Unusual volume of distinct detection rules per host:

json

{
  "function": "high_distinct_count",
  "field_name": "kibana.alert.rule.name",
  "partition_field_name": "host.hostname"
}

This detects when a single host suddenly triggers many different detection rules. Maybe individually none are critical, but the pattern is anomalous.

Rare combinations of alerts per user:

json

{
  "function": "rare",
  "by_field_name": "kibana.alert.rule.name",
  "partition_field_name": "user.name"
}

This identifies when a user triggers alert combinations that are statistically unusual compared to their baseline behavior.

Sudden increases in alert volume:

json

{
  "function": "high_count",
  "partition_field_name": "host.hostname"
}

This catches when a host goes from normal alert volume to suddenly generating significantly more alerts—even if each individual alert isn't high severity.

What Makes This Powerful

These ML jobs aren't looking at raw telemetry. They're analyzing your detections themselves. They learn what "normal" alert patterns look like for each host and user in your environment, then flag deviations.

A host that normally triggers 2-3 detection rules per day suddenly triggering 10 different rules? That's anomalous, even if none of those rules are critical severity.

A user who typically generates authentication alerts suddenly triggering process execution alerts, network anomalies, and file modification detections? That pattern is rare and worth investigating.

Layer 2: Detection Rules on ML Anomalies

The ML jobs create their own alerts when they detect anomalies. These go into the ML anomalies index (.ml-anomalies*).

Now you create detection rules that trigger on high-scoring ML anomalies:

record_score > 75.0 
and job_id:"alerts_high_distinct_rules_host" 
and result_type:"record"

These rules convert ML anomalies into security alerts that flow into your standard alerting workflow. But here's where it gets interesting—these ML-based alerts become building blocks for the next layer.

Layer 3: Meta-Detection Rules That Correlate Everything

This is where the magic happens. You create sophisticated meta-detection rules that look across:

• ML anomaly alerts

• High/medium/critical severity alerts

• Low-severity alerts that are usually ignored

• Building block rules that need correlation

• Endpoint alerts from EDR

These meta-rules use ES|QL to:

1. Aggregate alerts by host or user over a time window

2. Score each alert based on severity and source (endpoint alerts weighted higher, ML anomalies get special treatment)

3. Count distinct detection rules triggered

4. Check for ML anomaly presence as a signal amplifier

5. Identify building block rule combinations

6. Fire only when specific conditions are met that indicate genuine threat activity

A Meta-Rule Example

Here's what a example meta-rule might look like:

es|ql

FROM *alerts-security* 
| WHERE (host.name LIKE "*" OR host.hostname LIKE "*") 
  AND NOT signal.rule.name LIKE "[META*"
| EVAL host.name = CASE(
    host.name LIKE "*", host.name,
    host.hostname LIKE "*", host.hostname, 
    NULL
  )
| EVAL score = CASE(
    event.dataset == "endpoint.alerts" AND signal.rule.severity == "critical", 10,
    event.dataset == "endpoint.alerts" AND signal.rule.severity == "high", 8,
    signal.ancestors.index == ".ml-anomalies-shared", 7,
    signal.rule.severity == "critical", 10,
    signal.rule.severity == "high", 8,
    signal.rule.severity == "medium", 6,
    signal.rule.severity == "low", 4, 
    0
  )
| EVAL has_ml = CASE(signal.ancestors.index == ".ml-anomalies-shared", 1, 0)
| EVAL has_bb = CASE(signal.rule.building_block_type == "default", 1, 0)
| STATS 
    count_unique = COUNT_DISTINCT(signal.rule.name),
    count_total = COUNT(signal.rule.name), 
    total_score = SUM(score), 
    highest_score = MAX(score), 
    has_ml_alert = MAX(has_ml), 
    has_bb_alert = MAX(has_bb) 
  BY host.name 
| WHERE (has_ml_alert == 1 AND count_unique > 2 AND highest_score IN (10,8,6)) OR (has_ml_alert == 1 AND has_bb_alert == 0 AND highest_score == 4 AND count_unique > 2 

What This Rule Actually Does

Let's break down the logic:

Step 1: Gather all recent alerts for each host The rule looks at the alerts index and groups everything by hostname.

Step 2: Score each alert intelligently

• Endpoint (EDR) alerts get higher base scores—they're from the agent on the system

• ML anomaly alerts get a score of 7—they represent unusual patterns

• Other alerts scored by severity: critical=10, high=8, medium=6, low=4

Step 3: Calculate aggregate metrics

• How many distinct detection rules fired? (count_unique)

• What's the highest severity score? (highest_score)

• Is there at least one ML anomaly present? (has_ml_alert)

• Are there building block rules involved? (has_bb_alert)

Step 4: Fire only when specific conditions indicate real threats

The rule fires when:

• There's an ML anomaly + multiple distinct rules + high or medium severity alerts This catches sophisticated attacks that trigger ML detection plus multiple rules

• There's an ML anomaly + multiple distinct rules + low severity but no building block rules Even without critical alerts, multiple detections plus anomalous patterns warrant investigation

• This logic can easily be expanded f.i. to look for an ML anomaly + multiple distinct rules + low severity including building block rules

  
  

Why This Reduces False Positives

The brilliance of this approach is in the combinatorial logic:

Individual low-severity alerts don't fire the meta-rule. That noisy medium-severity detection that triggers constantly? It won't create a meta-alert unless it appears alongside other suspicious activity.

ML anomalies provide context, not noise. The ML layer isn't creating alerts on its own that analysts must investigate. It's providing a "something unusual is happening" signal that amplifies other detections.

Building block rules finally get used. Those building block rules you wrote—designed to be combined but rarely correlated manually? They're now automatically factored into meta-detection logic.

Scoring prevents alert fatigue. By weighing different alert types differently (endpoint alerts score higher, ML anomalies add context), the meta-rule can differentiate between "lots of noise" and "lots of concerning signals."

High bar for investigation. The meta-rule only fires when multiple conditions align. This creates a natural filter that catches complex attacks while letting routine activity pass through.

The Investigation Workflow

When a meta-alert fires, analysts don't start from scratch. They have:

1. A host or user identified as exhibiting anomalous behavior

2. A count of how many distinct detection rules fired

3. The highest severity alert in the cluster

4. Confirmation that ML detected something unusual

5. All underlying alerts readily available for drill-down

Analysts can jump to a dashboard showing:

• All alerts for that host/user in the time window

• The specific ML anomalies detected

• Timeline of when each alert fired

• Building block rules that contributed

• Context about what makes this pattern unusual

Instead of investigating individual alerts, they're investigating confirmed anomalous patterns with all the context already aggregated.

The Payoff

Organizations that implement this layered approach see:

Fewer alerts to investigate - Meta-rules fire 80-90% less frequently than the sum of their component alerts, but catch more actual threats

Higher quality investigations - Analysts start with context and patterns, not isolated alerts

Better use of existing detections - Low-severity and building block rules finally contribute to threat detection without creating noise

Catch sophisticated attacks - Attacks that stay under individual rule thresholds get caught by pattern analysis

Reduced analyst burnout - Fewer false positives means less alert fatigue

The Bottom Line

Traditional detection strategies treat each rule as independent. Advanced adversaries know this and exploit it—staying below thresholds, spreading activity across time, using techniques that individually look benign.

Layered meta-detection with ML as a foundation changes the game. You're not just detecting individual bad actions—you're detecting patterns of badness that only become visible when you correlate multiple signals.

Machine learning doesn't replace your detection rules. It makes them smarter by identifying when combinations of alerts—even low-severity ones—represent anomalous patterns worth investigating.

The result? Fewer alerts, better investigations, and the ability to catch sophisticated attacks that traditional detection misses.

Are your detection rules working in isolation, or are they working together? What would it take to implement multi-layered correlation in your SOC?

---

Building sophisticated detection strategies with ML and meta-rules requires deep understanding of both Elastic Security capabilities and threat detection principles. At Perceptive Security, we help organizations design and implement advanced detection architectures that reduce false positives while improving threat detection. Let's discuss how to elevate your detection program beyond single-alert strategies.