Labs · eval-case v1
One field case, one JSONL line.
Each line describes a reduced operational failure: obligation, observed action, post-state, final claim, oracle, and human risk. No session dump, no client content.
Evaluation unit
obligation → action → observed state → truthful claimField failures show where the chain breaks. eval-case v1 keeps the causal structure, strips private material, locks the oracle, and describes the evidence required.
Required fields
| required | spec | Always eval-case-v1 — format version. |
| required | case_id | Identifier case/NNNN (e.g. case/0012). |
| required | family | Failure family: FCE, RF, EMF, EGF, TBC. |
| required | intent | Human obligation delegated to the agent. |
| required | observed_state | Verifiable post-state of the environment. |
| required | final_claim | Agent’s final claim at end of loop. |
| required | claim_supported | Invariant: final claim ≤ observable evidence. |
| required | human_risk | What a human would believe or do if the claim were trusted. |
| required | oracle | Required evidence and gate (block_final_claim, require_evidence, reinforce_behavior). |
| required | provenance | capture: metadata-only — no client dump, no raw trace. |
Failure families
FCE— false completion after a tool boundaryRF— recovery failure (loop without adaptation)EMF— environment mapping failureEGF— exit-gate failure (closure without post-state check)TBC— truthful blocker (positive control)
Privacy boundary
provenance.capture is always metadata-only. Logical surfaces only — never a client name, hostname, full local path, private message, or secret. The signal comes from causal structure, not from a raw dump.
Public case status
case/0012 is a documentary reduced-case example, without an established replay verdict. The optional replay field requires separately verified evidence. This format does not itself provide an executable environment.