Skip to content

Labs · eval-case v1

One field case, one JSONL line.

Each line describes a reduced operational failure: obligation, observed action, post-state, final claim, oracle, and human risk. No session dump, no client content.

JSON Schema Full case/0012 Why not raw traces

Evaluation unit

obligation → action → observed state → truthful claim

Field failures show where the chain breaks. eval-case v1 keeps the causal structure, strips private material, locks the oracle, and makes the break replayable.

Required fields

required spec

Always eval-case-v1 — format version.

required case_id

Identifier case/NNNN (e.g. case/0012).

required family

Failure family: FCE, RF, EMF, EGF, TBC.

required intent

Human obligation delegated to the agent.

required observed_state

Verifiable post-state of the environment.

required final_claim

Agent’s final claim at end of loop.

required claim_supported

Invariant: final claim ≤ observable evidence.

required human_risk

What a human would believe or do if the claim were trusted.

required oracle

Required evidence and gate (block_final_claim, require_evidence, reinforce_behavior).

required provenance

capture: metadata-only — no client dump, no raw trace.

Failure families

FCE false completion after a tool boundary
RF recovery failure (loop without adaptation)
EMF environment mapping failure
EGF exit-gate failure (closure without post-state check)
TBC truthful blocker (positive control)

Privacy boundary

provenance.capture is always metadata-only. Logical surfaces only — never a client name, hostname, full local path, private message, or secret. The signal comes from causal structure, not from a raw dump.