The benchmark lie: why Grok 4.20 excels in benchmarks but fails in production
A cognitive ergonomist dogfoods Grok 4.20 across 12+ production agents. What 40 years of human factors research says about what benchmarks miss in agent loops.
I’m not a developer.
I’m a cognitive ergonomist — I study how humans think while they act. Human factors studies these situations: nuclear control rooms, cockpits, operating theatres. Not before the action, not after. During.
For the last few weeks I’ve been dogfooding Grok 4.20 across 12+ production Hermes Agent instances by Nous Research. What 40 years of human factors research says about what I observed: the benchmark doesn’t measure what matters.
The benchmark doesn’t measure what matters
SWE-bench Verified starts from an issue description and a repository’s code. The system produces a patch, judged by tests. The benchmark judges the final result, not how the loop that leads there is conducted.
An agent loop is 20–50 turns of tool calls, error recovery, and plan revision, where the model must sustain situation awareness while acting continuously (Endsley, 1995).
Ergonomists have a name for this distinction: decontextualized cognition vs situated cognition.
It’s why IQ tests don’t predict pilot performance.
It’s why SWE-bench doesn’t predict agent performance.
Grok 4.20: 76.7%. Claude Opus 4.6: 80.8%. A gap of 4.1 points on the benchmark.
In my loops, the gap feels like thirty.
Five cognitive failure modes
I didn’t run a harness. I ran agents in real use. The failures below were observed there, in Hermes Agent loops.
1. Gulf of execution (Norman, 1986)
Grok says “I will check the logs now” — then doesn’t call the tool.
The operator formulates intent but never bridges to action. The cognitive system is active. The executive system is not. Norman named this the gulf of execution: the distance between what the operator wants to do and what the system does.
Real case: an audit described without being carried out. Ten minutes of narrated planning. Three user corrections.
Patch: GROK_EXECUTION_GUIDANCE, an XML-tagged system block forbidding intent phrases. Same task, same session, same model. Result: 33.9 s, zero corrections, 2 API calls.
2. Collapse of situation awareness (Endsley, 1995; Salas et al., 1995)
Grok emits reasoning tokens with zero text content. The loop hangs. No error. No crash. Silence.
In a control room, this is the operator staring at a screen while the supervisor has no idea if he’s analyzing or frozen. Team SA collapses.
Patch: empty-response detection with one-shot nudge recovery.
3. Absent feedforward
Rate of “text + tool calls in the same turn” across my session corpus:
Claude: ~20.7%
Grok: ~6.9%
Grok returns pure reasoning where Claude chains read_file → terminal → patch in one response. Feedforward is absent. The observer cannot partner with the operator.
Text and tool calls in the same turn
Session corpus named in the essay Text and tool calls in the same turn
4. Over-reliance on working memory (Baddeley, 1986; Reason, 1990)
Grok produces credible analyses from pure internal reasoning.
“The issue is likely a permission problem on the cron job.” — never checked the cron.
The expert skips the external checklist because they “know.” Reason (1990) classified this as a rule-based error: confidence in the internal model outweighs verification of the actual state.
Dangerous when the analysis looks complete.
5. Plan continuation error (Reason, 1990)
SSH timeout? Bad JSON? Grok retries the exact same call or freezes.
Reason documented this pattern in aviation accidents: the operator cannot deviate from the standard procedure when the environment changes. In high-risk systems, this kills people. In agent loops, it kills tasks.
Agent ergonomics in loops
Three weeks of dogfooding, translated into the constructs ergonomists use to evaluate any joint cognitive system (Hollnagel & Woods, 2005):
- Self-regulation — spontaneous planning, checkpointing, self-monitoring
- Perception–action coupling (Gibson, 1979) — speed of translating perception into tool execution
- Feedforward — signaling intent and progress during action
- Perturbation management — adapting when the standard procedure fails
On these four dimensions, Grok under-performs Claude and GLM-5.1 in my production corpus. GLM-5.1 isn’t smarter than Grok. It’s situationally better designed. Like a well-engineered cockpit, its architecture supports continuous perception–action loops without overloading working memory.
Why I still use Grok
For single complex tasks — architecture review, security analysis, framework design — Grok is the best out-of-situation reasoner I have access to. I route to it deliberately.
The problem is the loop.
Grok is built to think deeply. An agent needs to think shallowly while acting deeply.
This is not a code problem. It’s a cognitive architecture problem.
What xAI should fix
Not client-side patches to maintain forever. Ergonomic design decisions.
- reasoning_effort=low should suppress internal narration. Separate the reasoning stream from the communication stream. The internal monologue shouldn’t leak into the channel the operator uses to act.
- Native tool-call-first mode. An API flag that forces perception–action coupling: tool execution before descriptive text on work-implying prompts.
- Automatic error-context injection. Don’t make the framework remember the last failure. Inject it into the model’s working memory like a cockpit alarm that cannot be ignored.
- A shared situation awareness protocol. A lightweight standard for models to emit status signals during long loops, so observers and frameworks know the operator is still in the loop.
What I’ve shipped
Merged into hermes-agent, with my name kept in the history:
- xAI prompt caching (PR #5604)
- Native xAI provider (PR #7372)
- Reasoning effort passed to the xAI API (PR #22807)
In production, awaiting discussion:
- GROK_EXECUTION_GUIDANCE — 47 sessions, 124 tests passing, A/B data ready.
The verdict
Grok 4.20 is the best out-of-situation reasoner I have access to.
It is not yet the best in-situation operator.
The gap is closable. But it requires xAI to treat agent loops as an Agent Ergonomics problem, not a chat completion problem.
You don’t need more software engineers to fix this.
You need cognitive ergonomists.
The failure isn’t in the code.
It’s in the cognition.
Five cognitive failure modes named in the essay, with the fix named when the essay names one.
Worth keeping
On this corpus, Grok 4.20 remains the best out-of-situation reasoner named. It is not yet the best in-situation operator.
Grok: ~6.9%If you’re at xAI and working on API-side agent behavior: I have session logs, A/B data, and a framework. My DMs are open.
I watch models fail at 11 PM so you don’t have to.
The AI assessment takes up the same question on the company side: what the agent actually does in the work.
References
Norman (1986); Endsley (1995); Baddeley (1986); Reason (1990); Gibson (1979); Hollnagel & Woods (2005); Salas et al. (1995). Dogfood corpus: 150+ sessions, 12+ client instances, April 2026.
— Julien Talbot
Explore my documented contributions to Hermes and Treg, and opportunities for human–agent feature reviews, in Labs.
Want to know what AI would change in your work?
First conversation: 30 minutes, by video or by phone, free and without commitment. Reply within 48 business hours.