Raw Logs Get AI Triage Wrong. Every Time.

Everyone building an AI SOC right now is worried about the same thing: token cost. Agentic triage burns budget fast, and every vendor has a lever to pull - tighter prompts, cheaper models, less "effort." Fair enough. Run enough alerts through an agentic loop and the token bill becomes a line item.

But there's a question almost nobody asks before optimizing that bill: can the agent get the verdict right in the first place?

We tested it. We handed a security agent a real triage scenario as raw, unnormalized vendor logs — the way most environments actually generate them — and measured whether it reached the correct verdict. Nine independent runs. Zero correct.

What Was the Agent Actually Being Asked to Triage?

The scenario was deliberately realistic: a known-scanner IP brute-forces a domain controller, mints a cloud IAM key for a service account, and drops a malware loader - while four unrelated, benign IPs generate noise in the same window and must not be flagged. Eighty-two log lines, spread across four different vendor formats: Cisco ASA firewall syslog, Windows Security 4625 events, CrowdStrike-style EDR JSON, and AWS CloudTrail. Real formats, correlated events.

The agent got the verdict wrong in all nine runs, and turning the reasoning-effort dial up didn't change that. The failure mode was identical every time, which points at the data rather than variance.

Why Couldn't It Get a Single One Right?

Not because the model wasn't capable of the reasoning. It's because before the agent could even start reasoning about "malicious or benign," it first had to become a parser. Four incompatible log formats, no shared schema - so it wrote files to disk, tried jq, hand-rolled field extraction per format, then deduplicated and correlated eighty-two lines by hand. That took anywhere from 14 to 32 turns before a single triage judgment got made.

And even after all that work, the agent still didn't know whether that IP was a known scanner, or where that IAM key normally shows up in the environment — because raw logs don't carry that context. They just carry fields. So it guessed. Every time, it guessed wrong.

None of this shows up as an error. The agent completes its run, writes a verdict, and moves on. There's no exception thrown, no failed step in the transcript — just a wrong answer delivered with the same confidence as a right one would have been. That's the part that should make anyone measuring an AI SOC by uptime or completion rate nervous: the system looks healthy the entire time it's wrong.

Focusing on Cutting the Token Bill Only Makes it Worse

The single most effective lever for cutting tokens on raw logs is a tighter system prompt telling the agent to move faster and be less thorough. It works: roughly 69% less token spend in our runs. It also makes the agent more confidently wrong.

Cheaper and wrong isn't a tradeoff. It's just wrong, faster.

Data Changes the Answer

If an agent can't triage raw logs correctly at any reasoning effort, then the real optimization question isn't which model to run or how hard to trim the prompt. It's what has to be true about the data before the agent ever sees it.

We ran the same nine triages against normalized and enriched data and measured exactly where - and why - the answer changes. The full methodology, the cost breakdown, and the one architectural difference that actually moves the needle are in The Cost of a Correct Triage.