Somewhere deep in the forgotten minds of yesterday's engineer, a withering rule, sits, silent. Its purpose, unknown to the tribal knowledge of today. As you ponder this ancient puzzle's purpose, you are left bewildered. It passed the review. It hasn't changed recently. The query seems reasonable...but then why has the song of its people gone quiet.
_sourceCategory ="cloudtrail/*" "CreateUser" "iam.amazonaws.com"
| json "eventSource", "eventName","errorCode" as eventsource, eventname, errorCode nodrop
| where eventsource = "iam.amazonaws.com" and eventname = "CreateUser" and !(errorCode = "AccessDenied")This harrowing tale is not one of fantasy, but a commonplace among our industry.
The question: Is it working?
Unfortunately, you can’t tell from the rule and you can’t tell from the output.
Zero Results. That’s the only signal you have. Think about why that could happen?
There are at least five I think of immediately:
- Nobody has created a user in three months. The rule is working perfectly.
- The CloudTrail feed broke six weeks ago and nobody noticed.
- A parser change renamed
eventName,eventsource, orerrorCodeupstream. - The value was wrong the day it shipped…wrong case, wrong spelling, wrong format.
- A migration changed the
_sourceCategory, and this rule now points at nothing.
All five of these have the same result, but only one is a working detection…
So you do what everyone does. You paste the rule into the explorer and start deleting.
- Drop the
eventNameclause…anything? Good…data exists. - Put it back, then drop the
_sourceCategory. - Widen the query to thirty days.
- Try a bare keyword search for
CreateUserjust to see whether the string appears anywhere at all.
Fifteen minutes later you know what broke. You've also learned nothing you can write down, and you'll do the same thing next quarter for a different rule.
Those five explanations collapse into three causes:
- The query received the data and matched nothing.
- The query never received the data.
- The query received the data but could never have matched the behavior.
#1 is just a Tuesday. #2 and #3 are problematic, and the output looks identical either way.
I’ve seen this exact failure before, in a completely different field.
I came to detection engineering from data science, where I spent years building pipelines. The zero-result problem is well known in statistics, and one I’ve battled myself.
In data science you don’t ship a test without a positive control: a case you know must fire. If the control goes quiet, the instrument is broken, not the world. If a test can’t distinguish no effect from no data, it isn't a weak test, it isn't a test.
Detection rules have this control ready and waiting, we just throw it away by AND’ing it to the behavior filters:
Is the feed is aliveANDdoes the field existANDdoes the value match?
Detection engineering and Data Science are both about maintaining, cleaning, and analyzing a dataset to draw a conclusion about the trends and behaviors inside it.
How much of what exists for Data Science actually transfers? This is the first post in a series working through it: what a rule is really made of, the corpus it reads, the behavior it hunts, and the two places rules rot.
What does a query actually do?
Every query worth maintaining does two jobs, and they should be separated:
- Pick the corpus: which sources feed this rule
- Find the behavior/pattern: does anything in there match what I care about?
Aggregations, joins, thresholds, enrichment, window functions…all of these operate on the corpus to analyze the corpus.
Break the query into two parts and you have a positive control. The three causes stop looking alike:
Corpus returns nothing
- The query never received the data.
Behavior returns nothing
- The query received the data and matched nothing.
- The query received the data but could never have matched it.
The first cause is now answerable. The second and third are still two goblins in a trenchcoat, and no amount of splitting will separate them. A positive control proves the instrument is working. It does not prove you asked a sensible question.
Splitting these out gives us one provable scope of data, the conditions are true of every record from that source. Then a second part, behavior, is unprovable by nature, but that’s fine, as long as you know which half you’re dealing with.
Scope tells you data arrived. Behavior tells you the hunt found nothing.
What does this look like?
Scope: must always return rows:
_sourceCategory ="cloudtrail/*"
| json "eventSource" as eventsource nodrop
| where eventsource = "iam.amazonaws.com"Behavior: allowed to return zero:
| json "eventName","errorCode" as eventname, errorCode nodrop
| where eventname = "CreateUser"
| where !(errorCode = "AccessDenied") // suppression, not detection
The scope half now asserts two things out loud: the source category resolves and IAM records are arriving. That’s all it should do. Scope isn’t about finding the maximum number of conditions you can prove, it's about finding the minimum set of conditions that identifies the corpus. Schedule it. Alert on zero. You’ve now covered problems 2 and 5 from the top of this post.
The bare keywords "CreateUser" and "iam.amazonaws.com" are not a guarantee. A keyword search matches bytes anywhere in the raw message; nothing binds it to a field, so it cannot prove anything about the record's structure. CreateUser could appear in the policy document of a PutRolePolicy event that grants iam:CreateUser… same string, entirely different eventName.
Bare keywords are a nightmare for maintainability and guaranteed scoping; you will see them used to prune the index and make the query cheaper. Keep them out of the scope query, where a match on the wrong record means the heartbeat goes green on data that was never your corpus. A heartbeat that passes for the wrong reason is worse than no heartbeat at all.
Where this doesn’t save you
Liveness is not completeness. cloudtrail/* returning rows means the glob matches something. If three of your forty accounts stop shipping, scope still looks healthy and you still have a hole. Volume baselines are the next layer, not this one.
A rule can break without the corpus breaking. Referenced a field that does not exist, renamed upstream, misspelled the day it shipped or considered required when it's actually optional? All these will lead to a healthy corpus and a broken query. You catch that by validating the rule against the source's known schema. Schema validation against rules is one of the hardest problems we have worked on at Rilevera, which is why we started there.
In the original rule, nodrop is a declaration that treats these fields as optional. Leave it off and a missing field discards the record, which converts an optional field into a filter gate.
This isn’t a new idea
You’ll see this adopted or attempted in a number of ways. Sigma splits logsource: from detection:. YARA-L has events and condition. None of them, though, makes the scope executable: Sigma’s logsource is an abstract taxonomy, not a query you can schedule and alert on. That gap is the opportunity.
My claim is that we've built a system around rewarding 'it finds the thing' and not 'here's how you'd know if it stopped'. We ship rules that work, because there’s an alert queue on fire and a threat report due Friday. Then we inherit them, undocumented, from someone who left.
All this to say, the incentives need to shift.
Try this out for Yourself
Take a rule you have in production, and sort every condition into “picks the corpus” or “finds the behavior”.
It’s not always straightforward. If there are conditions you can’t confidently place into a bucket, that’s a condition where you aren’t able to identify the failure mode. It’s somewhere to look deeper. Think about how negations and exclusions impact a scope as well.
If your experiment landed cleanly, see what it returns. Run the conditions and find out whether anything comes back. You’ve got a positive control that will tell you whether or not your pipelines are working!
In the next post, I will talk about the one-sentence test that tells you which half a condition belongs to, and once you have it, that first half query becomes something you can schedule and alert on.
I would love to hear from you all, tell me which conditions you couldn’t classify easily.
