Shaken, Not Stirred
Any James Bond fans in the virtual crowd out there? Weird intro for a blog post about Detection Engineering, I know, but stick with me here. Whether you are or you aren't, I highly recommend checking out at least some of "007: First Light" on YouTube if you're looking for some time to kill. It has a very interesting and timely story that follows a young Bond as he unravels a mystery surrounding MI6, rogue agents, and AI. You read that right, AI.
Spoilers for "007: First Light" ahead - the central conflict revolves around MI6's use of a Quantum AI computer to prevent terrorist attacks, but along the way we learn the AI doesn't always get it right. Shocker, I know. The supplier has a quantum mirror computer that mirrors every prediction MI6 receives, so they can plant evidence whenever the AI misfires and keep up the illusion that it never does. It's an interesting premise for a story right now, and a scary one to sit with. What's even scarier is that I've seen something like this play out in real life, not to this extent, but hear me out.
When Science Fiction Meets Science Fact
I haven't personally experienced blatant corruption from an AI provider, nor have I seen a government agency act on false information handed to it by AI. What I have experienced is the underlying theme and cause of the events that play out in First Light: confirmation bias. Both in First Light and in real life, we're told AI is so advanced and so inevitable that it will shape the future. There's truth to that, but it also leads MI6 in the game, and real-world users outside of it, to blindly trust whatever output the AI hands them.
I won't say which tool it was, but I've watched a security tool use AI to map detection rules to MITRE ATT&CK techniques and get the mapping completely wrong. And that's the part that should worry you: it wasn't one rule with one bad mapping. It was the same classifier running against every rule in the program, which means whatever error rate goes uncaught on rule one is the error rate you're stuck with on rule five thousand. Aren't we supposed to be able to trust the companies building these tools? One would hope so, but I don't see many organizations budgeting the time to validate AI output rule by rule, so a lot of them end up trusting it blindly by default.
The Data Is the Key
Not everyone trusts AI blindly, though. I was in a meeting recently where a customer walked me through a solution they were building in their SIEM using AI, and how they'd caught it hallucinating fields that didn't exist in their logs almost immediately after the output came back. Rilevera doesn't blindly trust AI either, and I know that firsthand because I've personally done extensive testing and manual validation of our own products' AI classifiers and their output. It's tedious, but it's necessary, because AI output is only ever as good as the data behind it.
Here's the part that matters at scale: problems in a small SIEM don't disappear as the SIEM and the program around it grow. They scale right along with it. So whether you're using AI to classify existing rules, generate new detections, or even take response actions off the back of a detection firing, the accuracy and integrity of the underlying data isn't optional. It's the whole game. Get it wrong and you end up with detections referencing fields that don't exist, mappings that paint a false picture of your coverage, or a host getting isolated for no real reason. Get it right and the payoff compounds just as fast: you understand your coverage more quickly and deeply, you build new detections faster, and you respond at the same speed attackers are already moving.
Trust, But Verify
Bond eventually gets his answer, not by trusting the machine or rejecting it outright, but by going back to the source data and checking it himself. That's the boring part of the story that doesn't make the trailer, but it's the part that actually matters.
The same is true here. AI doesn't fail at scale because the model got worse. It fails at scale because the small data problems that were tolerable at 50 detections become invisible at 5,000, and being wrong and invisible is a lot more dangerous than just being wrong. A hallucinated field or a bad ATT&CK mapping is a nuisance when a human reviews every rule. It's a liability when AI is doing the mapping, the classification, and increasingly the response, across every client and every rule in your program. All of it faster than any analyst could keep up with.
Clean detection data isn't a nice-to-have on the way to AI operationalization. It's the precondition for it. Get it right, and AI becomes a force multiplier for your detection program. Get it wrong, and you're not scaling detection engineering, you're scaling confirmation bias.
At Rilevera, this is why we treat data validation as core to the platform rather than an afterthought bolted onto an AI feature. Not because AI is the enemy, but because it's only as trustworthy as the data we hand it.
