Terminal‑Bench tasks

Coding tasks on real codebases in the Harbor format, graded by hidden tests in a sealed verifier.

Available nowImproves: Terminal-Bench 4.0Format: Terminal-Bench 4.0 / Harbor

Request samples
Same-quality tasks in stock
500+Each with a container, task statement, reference solution and hidden tests
Median expert time estimate
3 h47% at 4 hours or more
GPT‑5.6 Sol successes in 5 attempts
1–⁠3 / 5280+ tasks in stock; Codex, max effort, agent offline
Programming languages
25
Stock by domain
  1. Scientific computing and data19.0%
  2. General software engineering16.8%
  3. Compilers and language tools16.6%
  4. Networking and distributed systems13.8%
  5. Databases and storage9.7%
  6. Security, forensics and reversing9.7%
  7. Systems, build and operations7.4%
  8. Graphics, media and front end4.0%
  9. Machine learning3.1%
Stock by language
  1. Python44.2%
  2. C++12.6%
  3. C8.6%
  4. JS / TS7.4%
  5. Rust6.4%
  6. Go4.1%
  7. Java4.0%
  8. R3.8%
  9. C#2.6%
  10. Other (15)6.2%

Other: Elixir, Kotlin, Zig, Scala, Julia, Ruby, Lua, OCaml, Fortran, Verilog and more

Stock by task type
  1. Features and extensions30.4%
  2. Bug fixes23.0%
  3. Migrations and upgrades21.9%
  4. New tools15.0%
  5. Investigation, forensics and reproduction7.1%
  6. Performance tuning and other2.6%
One unit
One Harbor task directory with its verifier and reference solution
Agent environment
Docker container with a real codebase
Reward
0 or 1, from hidden tests in a sealed verifier

Overview

Terminal-Bench tasks in the Harbor format, in stock and produced to order. Each puts an agent in a container with a real codebase and asks for a substantial change; hidden tests in a sealed verifier grade the result.

What you get

<task-name>/
├── task.toml       # limits, resources, output paths
├── instruction.md  # what the agent is told
├── environment/
│   └── Dockerfile  # agent image, codebase
├── solution/
│   └── solve.sh    # reference solution
└── tests/
    ├── Dockerfile  # verifier image
    ├── test.sh     # writes the reward
    └── ...         # hidden test cases
  • Sealed verifier. Only the output paths named in task.toml reach a fresh, offline container built from tests/.
  • Task notes. Each task explains its difficulty, solution and verification, with an expert time estimate, in task.toml or a README.md.

Samples

Three tasks picked to differ from each other: Go, Python and C++; databases, chemistry and robotics; two bug fixes and a new tool; expert estimates of 8, 10 and 12 hours. Each card ends with the task statement as the agent receives it.

Sample 1 of 3

Janus Datalog: AsOf snapshots under concurrent commits

  • Databases and concurrency
  • Go
  • Bug fix
  • Hard parts
  • Concurrent commits, race detector
  • Atomic, retryable failed commits
  • Every read honors the snapshot
  • Holds on in-memory and on-disk stores

Janus Datalog is a Datalog database written in Go. db.AsOf(id) should show the database exactly as a commit left it, but when one transaction buffers operations before another commits, they turn up in that other commit’s snapshot, and some read paths ignore the snapshot altogether. The agent gives commits a single publication order, makes a failed commit leave nothing visible and stay retryable, and makes every read path honor the snapshot.

Expert time estimate
8 h
Agent container
4 CPUs · 8 GB

Task statement

instruction.md

Janus Datalog (Go, in /app/janus) hands back an ElementID from Transaction.Commit(), and db.AsOf(id) is supposed to show the database as that commit left it. It doesn't. A transaction that buffered operations before another one committed shows up inside the earlier commit's view, and several reads on an AsOf handle ignore the basis. Fix it.

Commits need a single publication order. Bases increase along it, and a basis shows its own commit plus everything published before it, for good. Commits that don't overlap keep their real-time order, and while a commit is inside its store transaction, no commit started after it may return or become visible. Last-writer-wins, add-wins, unique-value ownership and RGA sibling order follow publication order, then buffering order. Set and Remove still compute their effects when called, against committed state, and those effects are what gets published. Vectors resolve as documented on ReconstructRGA; a removed element is skipped but keeps its place.

Store().Scan must hold exactly the published operations, once each, plus one :db/txInstant datom per commit. IDs rise in publication order and, inside a commit, in buffering order. The :db/txInstant datom comes last, with the basis as its ID, entity tx:<Lamport>, and the SetTime time as value when one was set. ElementID-typed values are data and are stored as given.

When BeginTx, StoreTx.Assert or StoreTx.Commit fails, a commit that could not be stored returns an error and leaves nothing visible. Later commits are unaffected, and the failed transaction stays open for more operations and another Commit, or a rollback. The tests inject failures through a wrapper passed as DatabaseOptions.Store; its failing StoreTx.Commit never reaches the wrapped transaction.

Commits that pile up behind a store transaction must share the next one instead of taking one each. The tests block a StoreTx.Commit, start twelve more commits, then release store commits in rounds, each round letting through whatever is waiting in StoreTx.Commit. All thirteen must have returned after four rounds.

This has to hold on memory-trees, memory and badger (also after reopening), with DisableCache and the algebra optimizer either way, for the latest database and for AsOf handles that are new, kept across later commits, or warmed by WarmCache. Reads go through Query (joins, not, or, count, get-else, pull), Matcher() (also after SetCache(nil)), Pull, PullMany, ResolveEntityAttributes, ResolveAllAttributes, ResolveAllAttributesMany, GetVectorNth and GetVectorLength; unique attributes, LookupByUnique included, are checked with the cache on. Concurrent commits run under the race detector. Closed transactions, rollback, validation errors and read-only handles work as before.

The tests use the existing exported API, take only your changes to .go files, go.mod and go.sum, and build offline, so don't add dependencies.

You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.

Sample 2 of 3

ChemCanvas: SMILES depiction with exact stereochemistry

  • Cheminformatics
  • Python
  • Bug fix
  • Hard parts
  • Exact InChI, stereo included
  • Stereo from wedges and 2D only
  • Legible 2D layout
  • Standard-library-only sandbox

ChemCanvas, a chemical drawing program, mangles molecules when it converts a SMILES string to an MDL molfile: a stereocenter either makes the import fail or ends up with the wrong configuration, isotope labels vanish, and a salt’s ions get joined by a bond. The agent repairs SMILES import, 2D layout and molfile export so that every molfile yields exactly the input’s standard InChI, stereo included, and still draws as a structure a chemist can read.

Expert time estimate
10 h
Agent container
2 CPUs · 4 GB

Task statement

instruction.md

ChemCanvas, the chemical drawing program checked out in /app/repo (…), does not keep the compound intact when it imports a SMILES string and saves it as an MDL molfile. Stereocentres crash the import or come out wrong, isotopes are lost, and dot-separated components such as the ions of a salt end up bonded to each other. Fix the SMILES import, the 2D layout and the molfile export so that Smiles().read_string(smiles) from /app/repo/chemcanvas/fileformat_smiles.py, followed by Molfile().generate_string(document) from /app/repo/chemcanvas/fileformat_molfile.py, returns one V2000 molfile with exactly the standard InChI of the input SMILES (as PubChem and RDKit compute it), stereo layers included. The InChI is computed from the molfile by IUPAC InChI 1.07.3 (inchi-1 on the PATH).

The result has to be a drawing a chemist can read. All z coordinates are 0 and the stereo must come from wedge and hash bonds and the 2D geometry alone: the InChI must not change when the atom parity flags are cleared. Measured in median bond lengths, every bond must be between 0.5 and 2 long and no two atoms that are not bonded may be closer than 0.25. The same holds for the coordinates of the imported document, which must keep the structure in ChemCanvas's own Molecule, Atom and Bond objects with isotopes in Atom.isotope. Every connected component keeps its heavy-atom composition and net charge, in the document and in the molfile, so a sodium carboxylate stays an ion pair even though standard InChI would not notice a covalent Na-O bond. Export works from the document's current state: changing an imported atom's isotope with Atom.set_property("Isotope Number", ...) changes the next export, and documents imported in the same process do not affect each other.

Only /app/repo is collected, without .git, and the verifier runs offline. It imports the code as /app/support/convert.py does, with its own copy of the PyQt5 stub, in a fresh Python 3.12 process per conversion that has the standard library only: no third-party packages, no ctypes, no network and a read-only filesystem. Each process gets 10 CPU seconds, 1 GiB of memory and 60 seconds of wall time; the isotope check imports two compounds and exports three times in one process. Four conversions run in parallel, and all of them together get 4500 seconds. Submissions that contain compiled code or symbolic links are rejected.

The hidden test set has 200 PubChem compounds, each written three ways (PubChem's SMILES, RDKit's canonical SMILES and a random RDKit SMILES), plus the cases in /app/data/regressions.csv. Most hidden compounds are stereochemically demanding: bridged and spiro ring systems, macrocycles, some with stereo double bonds in the ring, and molecules with eight or more stereocentres. A few carry isotopes or are salts. They contain only H, B, C, N, O, F, P, S, Cl, Br and I apart from isolated Li, Na and K ions, have at most 100 heavy atoms, have tetrahedral stereocentres only at carbon and have stereo double bonds only at C=C, C=N or N=N. /app/data/pubchem_dev.csv lists 100 more compounds of the same kind with their InChIs. Every conversion must pass.

You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.

Sample 3 of 3

g2o: a certifiable pose-graph solver

  • Robotics (SLAM)
  • C++
  • New tool
  • Hard parts
  • Checkable optimality certificate
  • Time and memory budget on large graphs
  • Clean offline rebuild
  • Existing certifiable code off-limits

g2o is a C++ graph-optimization library used as a SLAM back end. The agent adds a new tool that estimates 2D or 3D poses by solving the pose graph’s semidefinite relaxation and says whether its answer is certifiably the global optimum. A specification fixes the input, the output and the report, and every reported number is recomputed from the input and the poses.

Expert time estimate
12 h
Agent container
2 CPUs · 8 GB

Task statement

instruction.md

Add a certifiable pose-graph solver to the g2o checkout in /app/g2o (…). It is a new application, CMake target g2o_certified, that estimates SE(2) or SE(3) poses from a g2o file through the semidefinite relaxation of the chordal cost and reports whether its estimate is a certified global optimum. Its specification is /app/g2o/doc/g2o_certified.md.

The application has to build and run with

cmake -S /app/g2o -B /app/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build /app/build --target g2o_certified -j2
/app/build/bin/g2o_certified /app/input.g2o /app/poses.g2o /app/report.json

Only /app/g2o is graded. It is copied to a machine without network access that has the system packages this image had when the task started; anything you install will not be there. Compiled files, archives, .git directories and symbolic links that point outside the tree are deleted from the copy. An unprivileged user then runs the two cmake commands from scratch with other absolute source and build paths, and runs the tool on hidden graphs. Configuring may take 300 s, building 1800 s and each run of the tool 600 s, always with 2 CPUs and 4 GiB of memory.

The hidden graphs have from about 20 to 10,200 poses and up to 30,000 edges, and the relaxation is exact on some of them and not on others. Public benchmark graphs are in /app/benchmarks; rim.g2o and cubicle.g2o there contain information matrices that are not positive definite, so they are not valid input as shipped.

Do not copy, vendor, download or link existing certifiable-estimation code: SE-Sync and its Optimization library, CPL-SLAM, CPL-Sync, DPGO, and the certifiable and Shonan modules of GTSAM. The submitted sources are compared with these projects.

You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.

Customization

Including, but not limited to, the following; anything else is open for discussion.

  • Domain and stack. Field, programming language and kind of codebase.
  • Difficulty. Reference model, harness, effort level and pass-rate band.
  • Run conditions. Attempts per task, time limit, agent internet access.
  • Model runs. Trajectory, terminal log and verifier output of each attempt.
  • Volume and schedule. Number of tasks, split across fields, and delivery dates.

Tell us what your model needs to get better at.

We reply with samples and a specification.