Skip to main content
An evaluation in Checkfu is a client-side composition of ordinary resources:
There is no server-side EvalRun resource and no separate transcript. The Session log and Outcome verdict are the evidence.
This flow runs from a Checkfu source checkout. Part of this flow is behind a rollout fence; a deployment that has not enabled it refuses the request rather than running it. Confirm capability status before making availability part of your application’s contract.

​
Task directory format

checkfu eval task reads the open Harbor and Terminal-Bench lineage format:
A minimal task.toml is:
instruction.md becomes the Outcome task description. The compiler derives a rubric that asks the ordinary Outcome grader to judge the settled work against that instruction. The verifier command remains a bounded argv, never a shell string. Every selected task file must be a regular, fatal-UTF-8 file no larger than 1 MiB. Symlinks, FIFOs, directories in place of files, oversized files, and invalid byte streams are errors rather than reasons to skip a task.

​
Compile without creating a Session

The command validates the directory and prints the exact inputs an isolated graded Session needs. It does not create a Session, run the declared image, execute the verifier, or claim that a reward was observed. The current output keeps the task’s Environment intent and verifier contract explicit. A task Dockerfile may be preserved as authoring input, but Checkfu does not yet claim to build it into a qualified Environment as part of this command.

​
Run the currently supported experiment

The landed command creates one isolated Session for each discovered task under one selected Agent and Environment:
The command also accepts the [label=]<Agent ID> spelling, but only one --agent lane is currently admitted. Cross-Agent and cross-harness selection is explicitly refused while harness choice moves to final Session admission. --repeat retains its bounded parser for the future matrix shape, but the current control flow requires at least two Agent lanes for a value above 1 and separately refuses multiple lanes. The runnable posture today is therefore one Agent with the default repetition of 1. Do not present this command as a multi-lane benchmark. For each task, the CLI:
  1. creates a Session with { agent, environment_id } through the public Session API;
  2. posts user.define_outcome using the compiled description, rubric, and iteration bound;
  3. drives the task as ordinary Session input;
  4. follows the public Session event stream until settlement; and
  5. mines the retained event trace into a local report.
The Session freezes the resolved AgentVersion and Environment at admission. The report is a projection of that Session evidence, never a second execution authority.

​
What the report means

The human report contains:
  • Outcome verdict distribution;
  • mean revision iterations;
  • total and mean recorded tokens, tool calls, and active milliseconds;
  • failure-mode counts;
  • tool-usage counts; and
  • one per-Session summary with the verdict event sequence when present, or the number of events examined.
--json emits the same mined report as terminal-safe JSON, including the underlying per-Session summaries. A failed sample still produces evidence. Missing measurements remain missing and are never converted to zero. The process exits zero only when every mined Session verdict is satisfied. It cannot authorize an Agent change, mutate an AgentVersion, change an Environment, or become a source of Budget truth.

​
Verifier support is not yet the verdict authority

The task compiler preserves the verifier command, reward format, and pass threshold. The landed experiment path does not execute that task verifier or reconcile its reward into Outcome today. It uses the ordinary Outcome grader. The intended future mapping is:
  • reward at or above pass_threshold → satisfied;
  • lower reward → needs_revision until the Outcome iteration budget is exhausted;
  • unreadable or out-of-range reward → grading failure, never an invented verdict.
That integration must enter through the existing Outcome grading chokepoint. It must not create another evaluation store or bypass Usage and Budget enforcement.

​
Retention

Evaluation Sessions follow the Workspace retention posture. Under standard retention, authorized event content and requested deliverables may be retained according to policy. Under zero data retention, execution content is not durably stored, so analyses that require replayable content or durable deliverables must fail closed or operate only on the permitted live stream.