Knowledge note

Harbor: Agent Evaluation Framework (Code Reading)

Repository-backed tour of Harbor's task packaging model, job/trial orchestration, agent/environment/verifier contracts, task distribution, and compile/exec workflows at pinned revision 97e65926410b.

Updated September 16, 2026Confidence: High

Harbor: Agent Evaluation Framework (Code Reading)

Repository: harbor-framework/harbor
Inspected revision: 97e65926410ba6b9cef15cd5d9bf9a0c700cef26 (main, 2026-08-04)

Related pages: Frameworks, Pier: Coding-Agent Evaluation Harness, DeepSWE: Long-Horizon Software Engineering Benchmark

TL;DR

What: Harbor is a framework for evaluating and optimizing agents and models inside container environments; the task package is the atomic portable unit. How: A JobConfig is planned into a JobPlan, fanned out through a TrialQueue into isolated Trial runs (single-step or multi-step), each combining a BaseAgent, a BaseEnvironment sandbox provider, and a Verifier, with every trial persisting its own evidence directory. The number: One task format runs from local Docker development through dozens of cloud sandbox providers, while job-level and per-agent concurrency plus retry backoff keep large runs tractable.

The Big Picture

The synthesized diagram below summarizes Harbor’s design philosophy and runtime architecture: portable task packages flow through distribution, orchestration, trial execution, and evidence collection.

Harbor design philosophy and runtime architecture

Synthesized explanation (editable source: harbor-architecture.drawio). The current runtime flow — configuration → planning → orchestration → execution → evidence — is shown below.

flowchart TB
    subgraph Input["1. Configuration and Distribution"]
        CFG["JobConfig<br/>(agents, datasets, tasks, metrics,<br/>retry, concurrency, n_attempts)"]
        DS["DatasetConfig<br/>(local / git repo / package / registry)"]
        TC["TaskClient<br/>downloads git & package tasks into cache"]
    end

    subgraph Plan["2. Planning"]
        JP["JobPlan.from_config<br/>resolves task configs, builds<br/>TrialConfig per task x agent x attempt,<br/>resolves metrics, caches tasks,<br/>writes JobLock"]
    end

    subgraph Orchestrate["3. Orchestration"]
        JOB["Job.create / Job.run"]
        TQ["TrialQueue.submit_batch<br/>job-level semaphore + per-agent<br/>pools, retry backoff"]
        TRIAL1["SingleStepTrial"]
        TRIAL2["MultiStepTrial"]
    end

    subgraph Execute["4. Execution primitives"]
        AGENT["BaseAgent<br/>setup() + run()"]
        ENV["BaseEnvironment<br/>start, exec, upload/download,<br/>network policy, healthcheck"]
        VER["Verifier.verify<br/>runs tests/, parses reward"]
    end

    subgraph Evidence["5. Evidence and results"]
        RES["TrialResult / JobResult<br/>result.json, logs, artifacts,<br/>metrics, pass@k"]
    end

    CFG --> DS
    DS --> JP
    TC --> JP
    JP --> JOB
    JOB --> TQ
    TQ --> TRIAL1
    TQ --> TRIAL2
    TRIAL1 --> AGENT
    TRIAL2 --> AGENT
    AGENT --> ENV
    ENV --> VER
    TRIAL1 --> VER
    TRIAL2 --> VER
    VER --> RES
    AGENT --> RES

Editable source: runtime-flow.mmd. The fuller layered view lives in the separate runtime-architecture.drawio and its render harbor-runtime-architecture.drawio.svg; the design rationale is in design-philosophy.drawio and its render harbor-design-philosophy.drawio.svg.

Why This Exists

Terminal-Bench was being reused for far more than one benchmark: custom evals, prompt optimization, RL rollouts, SFT trace generation, and CI-style agent testing. Each of those uses needs the same hard plumbing — reproducible task packaging, isolated runtimes, agent adapters, verifiers, artifact collection, and scaling — and doing it ad hoc per benchmark does not scale. Harbor exists to solve that systems problem as a framework rather than as a single benchmark runner.

The Landscape

Harbor’s lineage runs through Terminal-Bench (which it generalizes) and sits beside smaller Harbor-compatible harnesses and other software-engineering agent benchmarks in the knowledge base.

flowchart LR
    TB["Terminal-Bench<br/>(one benchmark)"] -->|"reused for custom evals, RL rollouts,<br/>SFT traces, CI-style testing"| HB["Harbor<br/>(agent evaluation framework)"]
    TB -->|"inspired"| PIER["Pier<br/>(coding-agent harness)"]
    HB -->|"shares task/trial model"| PIER
    SWEB["SWE-bench family<br/>(static patches)"] -->|"containerized runtime"| TB
    DSW["DeepSWE<br/>(long-horizon SE benchmark)"] -.->|"local-dataset execution"| PIER

Editable source: landscape.mmd.

The Core Idea

Harbor treats the task package as the atomic portable unit: one instruction, one container environment, one verifier, and an explicit runtime policy travel together. Everything else — jobs, trials, sandboxes, verifiers, distribution, compilation — is built around executing that unit repeatedly, reproducibly, and at scale.

Task Packaging: The Atomic Unit

The task directory is the contract every other layer consumes. Its canonical layout is defined by the Task class:

task/
├── instruction.md          # the prompt (canary markers stripped)
├── task.toml               # parsed into TaskConfig
├── environment/            # Dockerfile / docker-compose.yaml / etc.
├── solution/               # copied to /solution (optional)
└── tests/                  # test.{sh,ps1,cmd,bat} per target OS

Loading a task parses task.toml through the TaskConfig model, strips leading canary comment lines from the instruction, and validates that an OS-compatible test script exists. TaskConfig is the schema for everything declarative: [environment] (OS, Docker image, workdir, resource policy), [verifier] (mode, env, user), [agent] (user, concurrency, MCP servers), artifact collection rules, and — in newer tasks — [[steps]] for multi-step workloads with per-step verifiers and a multi_step_reward_strategy.

Distribution: Where Tasks Come From

A job’s task inputs are declared as JobConfig (plus DatasetConfig). DatasetConfig.get_task_configs() supports four source kinds:

  1. local — a directory of task folders, used directly.
  2. git repo — a git repository dataset resolved through the registry client.
  3. package — named org/name package tasks with content-hash refs.
  4. registry — a Harbor registry dataset identified by name/version.

Git and package tasks are materialized before trials run: TaskClient.download_tasks() groups by task-id type and downloads into a cache layout (<org>/<name>/<digest>/), or a flat layout in export mode, so every trial sees an immutable local snapshot.

Planning and Orchestration

JobPlan: resolve before you run

JobPlan.from_config() is the planning step: it resolves agent skills, expands task configs from explicit tasks and datasets, validates environment resource policies, resolves metrics per dataset, and caches remote tasks. JobPlan.build_trial_configs() then fans out one TrialConfig per task × agent × attempt. The plan also builds a JobLock that records the resolved run input, so retries and resumed jobs replay the same configuration.

Job: run and aggregate

Job.create() initializes a job directory, trial configs, and a TrialQueue; Job.run() writes the resolved config and job lock, runs the trial batch, then aggregates per-eval metrics and pass@k into a JobResult. The job surface is hook-driven (TrialEvent callbacks) so progress UIs and metric display plug in without touching trial internals.

TrialQueue: concurrency and retries

TrialQueue.submit_batch() returns one coroutine per trial. Execution is bounded by a job-level semaphore (n_concurrent_trials, default 4) plus optional per-agent pools, and _execute_trial_with_retries retries with exponential backoff on retryable exception types.

Trial Runtime

The base lifecycle

Trial.run() owns setup, hooks, result persistence, and teardown: _prepare() starts the agent environment, runs its healthcheck, injects skills, and runs agent setup; _finalize() stops the environment and writes result.json. Concrete subclasses own the workload shape. The agent phase in Trial._run_agent_phase() wraps the agent’s run() with a per-phase network policy, extra env, log context, and a timeout.

Single-step vs multi-step

SingleStepTrial._run() is the classic shape: one instruction, one agent run, artifact collection, then a shared or separate verifier. MultiStepTrial._run() executes sequential named steps, each with its own agent run and verifier, and combines per-step rewards into one trial-level reward (mean or last-step).

Execution Primitives

Agents

BaseAgent is the agent contract: setup(environment) prepares the agent and its tools inside the sandbox, and run(instruction, environment, context) executes it and populates an AgentContext. Class flags declare capabilities (SUPPORTS_ATIF for the trajectory format, SUPPORTS_RESUME, SUPPORTS_CONFIG, SUPPORTS_WINDOWS) so the trial runner can fail fast on mismatches. Agents receive MCP server configs and a skills directory from the task.

Environments

BaseEnvironment is the sandbox contract: start/stop, exec (BaseEnvironment.exec), upload/download (including tar-based filtered transfers), mounts, resource policy, healthchecks, and optional compose/sidecar services. Providers must advertise capabilities and enforce the task’s network policy exactly or reject the task before start. BaseEnvironment.set_network_policy() supports runtime policy switching when the provider advertises dynamic_network_policy. Concrete providers cover Docker, Daytona, Singularity, E2B, Modal, GKE, OpenShift, SkyPilot, EC2, Runloop, and more — one trial lifecycle across all of them.

Verifier

The default Verifier.verify() uploads the task’s test scripts into the environment, runs the OS-appropriate test command as the effective verifier user, downloads the verifier directory, and parses the reward from a JSON or text reward file. Verification can run in the shared agent sandbox or in a fresh separate verifier sandbox that receives only explicitly collected artifacts.

Derived Workflows: Compile and Exec

Two higher-level workflows build on the core runtime:

  • Compiler.compile() turns a CompileConfig (instructions × environments × verifiers) into concrete task directories, including generating auto-verifier test scripts that check required artifacts and JSON schemas.
  • Executor.execute() chains a map phase (compile + run a job) and an optional reduce phase (compile a reduce task from the map job’s artifacts + run it) for workflows like RL rollout generation.

Static vs Runtime Evidence

All claims above come from static code reading of the pinned checkout (97e65926410b, clean); runtime behavior was not executed. The trial lifecycle, environment provider behavior, and network-policy enforcement in particular are inferred from code and docstrings, not verified by running a trial. Consume the local checkout under external-repos/harbor/ for interactive navigation.

Limitations

  • Static reading only; no trial, environment, or verifier was executed at this revision.
  • The environment provider matrix (Docker, Daytona, E2B, Modal, GKE, …) was surveyed from source; provider-specific runtime quirks are not verified.
  • This reading reflects the codebase at 97e65926410b (2026-08-04); Harbor evolves quickly, so later revisions may differ.

Key Takeaways

  • The task package is the atomic portable unit: instruction + environment + verifier + policy travel together.
  • Planning (JobPlan) and orchestration (Job, TrialQueue) are separate from execution (Trial, BaseAgent, BaseEnvironment, Verifier).
  • Tasks can be local, git-repo, package, or registry sources, all materialized into an immutable cache before trials run.
  • Trials come in single-step and multi-step shapes; every trial persists its own evidence directory.
  • Compile and exec workflows generate tasks and map/reduce jobs on top of the same runtime.

One Thing to Remember

Harbor’s design philosophy is that the task is the atomic portable unit — jobs, trials, sandboxes, verifiers, and distribution all exist to execute that unit reproducibly and at scale.

Go Deeper