> ## Documentation Index
> Fetch the complete documentation index at: https://docs.checkfu.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate an Agent against a task set

> Compile open task directories into isolated Sessions and inspect Outcome evidence without creating a second evaluation runtime.

An evaluation in Checkfu is a client-side composition of ordinary resources:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
task directory
  → Environment selection
  → isolated Session with a frozen AgentVersion
  → Outcome definition
  → Session events + authoritative Usage
  → local report
```

There is no server-side EvalRun resource and no separate transcript. The Session log and Outcome verdict are the evidence.

<Warning>
  This flow runs from a Checkfu source checkout.
  Part of this flow is behind a rollout fence; a deployment that has not enabled it refuses the request rather than running it.
  Confirm [capability status](/getting-started/status) before making availability part of your application's contract.
</Warning>

## Task directory format

`checkfu eval task` reads the open Harbor and Terminal-Bench lineage format:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
my-task/
├── task.toml
├── instruction.md
└── tests/
    └── run.py
```

A minimal `task.toml` is:

```toml theme={"theme":{"light":"github-light","dark":"github-dark"}}
name = "sort-array"
max_iterations = 2

[environment]
image = "python:3.12-slim"

[verifier]
command = ["python", "/tests/run.py"]
reward_format = "reward_text"
pass_threshold = 1.0
```

`instruction.md` becomes the Outcome task description. The compiler derives a rubric that asks the ordinary Outcome grader to judge the settled work against that instruction. The verifier command remains a bounded argv, never a shell string.

Every selected task file must be a regular, fatal-UTF-8 file no larger than 1 MiB. Symlinks, FIFOs, directories in place of files, oversized files, and invalid byte streams are errors rather than reasons to skip a task.

## Compile without creating a Session

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
checkfu eval task ./my-task --json
```

The command validates the directory and prints the exact inputs an isolated graded Session needs. It does not create a Session, run the declared image, execute the verifier, or claim that a reward was observed.

The current output keeps the task's Environment intent and verifier contract explicit. A task Dockerfile may be preserved as authoring input, but Checkfu does not yet claim to build it into a qualified Environment as part of this command.

## Run the currently supported experiment

The landed command creates one isolated Session for each discovered task under one selected Agent and Environment:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
checkfu eval experiment ./tasks \
  --agent agent_... \
  --environment env_... \
  --json
```

The command also accepts the `[label=]<Agent ID>` spelling, but only one `--agent` lane is currently admitted. Cross-Agent and cross-harness selection is explicitly refused while harness choice moves to final Session admission.

`--repeat` retains its bounded parser for the future matrix shape, but the current control flow requires at least two Agent lanes for a value above 1 and separately refuses multiple lanes. The runnable posture today is therefore one Agent with the default repetition of 1. Do not present this command as a multi-lane benchmark.

For each task, the CLI:

1. creates a Session with `{ agent, environment_id }` through the public Session API;
2. posts `user.define_outcome` using the compiled description, rubric, and iteration bound;
3. drives the task as ordinary Session input;
4. follows the public Session event stream until settlement; and
5. mines the retained event trace into a local report.

The Session freezes the resolved AgentVersion and Environment at admission. The report is a projection of that Session evidence, never a second execution authority.

## What the report means

The human report contains:

* Outcome verdict distribution;
* mean revision iterations;
* total and mean recorded tokens, tool calls, and active milliseconds;
* failure-mode counts;
* tool-usage counts; and
* one per-Session summary with the verdict event sequence when present, or the number of events examined.

`--json` emits the same mined report as terminal-safe JSON, including the underlying per-Session summaries. A failed sample still produces evidence. Missing measurements remain missing and are never converted to zero.

The process exits zero only when every mined Session verdict is `satisfied`. It cannot authorize an Agent change, mutate an AgentVersion, change an Environment, or become a source of Budget truth.

## Verifier support is not yet the verdict authority

The task compiler preserves the verifier command, reward format, and pass threshold. The landed experiment path does **not** execute that task verifier or reconcile its reward into Outcome today. It uses the ordinary Outcome grader.

The intended future mapping is:

* reward at or above `pass_threshold` → `satisfied`;
* lower reward → `needs_revision` until the Outcome iteration budget is exhausted;
* unreadable or out-of-range reward → grading failure, never an invented verdict.

That integration must enter through the existing Outcome grading chokepoint. It must not create another evaluation store or bypass Usage and Budget enforcement.

## Retention

Evaluation Sessions follow the Workspace retention posture. Under standard retention, authorized event content and requested deliverables may be retained according to policy. Under zero data retention, execution content is not durably stored, so analyses that require replayable content or durable deliverables must fail closed or operate only on the permitted live stream.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.