概览
Harbor 格式的 Terminal-Bench 任务,有现货,也可按需生产。每道题给 Agent 一个装有真实代码库的容器,要求它完成一项较大的改动;结果由隔离的校验器用隐藏测试评分。
交付内容
<task-name>/
├── task.toml # 时限、资源、输出路径
├── instruction.md # 给 Agent 的题面
├── environment/
│ └── Dockerfile # Agent 镜像和代码库
├── solution/
│ └── solve.sh # 参考解
└── tests/
├── Dockerfile # 校验镜像
├── test.sh # 写入奖励值
└── ... # 隐藏测试用例
- 隔离校验:只有
task.toml中声明的输出路径会进入由tests/构建、不联网的全新容器。 - 题目说明:每道题都写明难点、解法和校验方式,并附专家用时估计,写在
task.toml或README.md里。
样例
3 道样题各有侧重:语言分别是 Go、Python 和 C++,领域分别是数据库、化学和机器人,两道修复缺陷、一道新增工具,专家用时估计依次为 8、10、12 小时。每张卡片最后附上 Agent 拿到的题面原文。
样例 1 / 3
Janus Datalog:并发提交下的 AsOf 快照
Janus Datalog 是用 Go 写的 Datalog 数据库。db.AsOf(id) 应当准确呈现某次提交完成时的数据库,实际却做不到:一个事务在另一个事务提交前缓冲的操作,会混进那次提交的快照;还有几条读路径完全不看快照。Agent 要为提交定下唯一的发布顺序,让失败的提交不留痕迹、可以重试,并让每条读路径都遵守快照。
- 专家用时估计
- 8 小时
- Agent 容器
- 4 核 · 8 GB
题面原文
Janus Datalog (Go, in /app/janus) hands back an ElementID from Transaction.Commit(), and db.AsOf(id) is supposed to show the database as that commit left it. It doesn't. A transaction that buffered operations before another one committed shows up inside the earlier commit's view, and several reads on an AsOf handle ignore the basis. Fix it.
Commits need a single publication order. Bases increase along it, and a basis shows its own commit plus everything published before it, for good. Commits that don't overlap keep their real-time order, and while a commit is inside its store transaction, no commit started after it may return or become visible. Last-writer-wins, add-wins, unique-value ownership and RGA sibling order follow publication order, then buffering order. Set and Remove still compute their effects when called, against committed state, and those effects are what gets published. Vectors resolve as documented on ReconstructRGA; a removed element is skipped but keeps its place.
Store().Scan must hold exactly the published operations, once each, plus one :db/txInstant datom per commit. IDs rise in publication order and, inside a commit, in buffering order. The :db/txInstant datom comes last, with the basis as its ID, entity tx:<Lamport>, and the SetTime time as value when one was set. ElementID-typed values are data and are stored as given.
When BeginTx, StoreTx.Assert or StoreTx.Commit fails, a commit that could not be stored returns an error and leaves nothing visible. Later commits are unaffected, and the failed transaction stays open for more operations and another Commit, or a rollback. The tests inject failures through a wrapper passed as DatabaseOptions.Store; its failing StoreTx.Commit never reaches the wrapped transaction.
Commits that pile up behind a store transaction must share the next one instead of taking one each. The tests block a StoreTx.Commit, start twelve more commits, then release store commits in rounds, each round letting through whatever is waiting in StoreTx.Commit. All thirteen must have returned after four rounds.
This has to hold on memory-trees, memory and badger (also after reopening), with DisableCache and the algebra optimizer either way, for the latest database and for AsOf handles that are new, kept across later commits, or warmed by WarmCache. Reads go through Query (joins, not, or, count, get-else, pull), Matcher() (also after SetCache(nil)), Pull, PullMany, ResolveEntityAttributes, ResolveAllAttributes, ResolveAllAttributesMany, GetVectorNth and GetVectorLength; unique attributes, LookupByUnique included, are checked with the cache on. Concurrent commits run under the race detector. Closed transactions, rollback, validation errors and read-only handles work as before.
The tests use the existing exported API, take only your changes to .go files, go.mod and go.sum, and build offline, so don't add dependencies.
You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.
样例 2 / 3
ChemCanvas:SMILES 转结构图,立体化学准确无误
化学结构绘图程序 ChemCanvas 把 SMILES 转成 MDL molfile 时会把分子弄坏:遇到立体中心,要么导入直接失败,要么构型出错;同位素标记会丢失,盐里的离子还会被连上化学键。Agent 要修好 SMILES 导入、2D 布局和 molfile 导出,让每个 molfile 算出的标准 InChI 都与输入的完全相同(立体层也不例外),画出的结构图也要让化学家看得懂。
- 专家用时估计
- 10 小时
- Agent 容器
- 2 核 · 4 GB
题面原文
ChemCanvas, the chemical drawing program checked out in /app/repo (…), does not keep the compound intact when it imports a SMILES string and saves it as an MDL molfile. Stereocentres crash the import or come out wrong, isotopes are lost, and dot-separated components such as the ions of a salt end up bonded to each other. Fix the SMILES import, the 2D layout and the molfile export so that Smiles().read_string(smiles) from /app/repo/chemcanvas/fileformat_smiles.py, followed by Molfile().generate_string(document) from /app/repo/chemcanvas/fileformat_molfile.py, returns one V2000 molfile with exactly the standard InChI of the input SMILES (as PubChem and RDKit compute it), stereo layers included. The InChI is computed from the molfile by IUPAC InChI 1.07.3 (inchi-1 on the PATH).
The result has to be a drawing a chemist can read. All z coordinates are 0 and the stereo must come from wedge and hash bonds and the 2D geometry alone: the InChI must not change when the atom parity flags are cleared. Measured in median bond lengths, every bond must be between 0.5 and 2 long and no two atoms that are not bonded may be closer than 0.25. The same holds for the coordinates of the imported document, which must keep the structure in ChemCanvas's own Molecule, Atom and Bond objects with isotopes in Atom.isotope. Every connected component keeps its heavy-atom composition and net charge, in the document and in the molfile, so a sodium carboxylate stays an ion pair even though standard InChI would not notice a covalent Na-O bond. Export works from the document's current state: changing an imported atom's isotope with Atom.set_property("Isotope Number", ...) changes the next export, and documents imported in the same process do not affect each other.
Only /app/repo is collected, without .git, and the verifier runs offline. It imports the code as /app/support/convert.py does, with its own copy of the PyQt5 stub, in a fresh Python 3.12 process per conversion that has the standard library only: no third-party packages, no ctypes, no network and a read-only filesystem. Each process gets 10 CPU seconds, 1 GiB of memory and 60 seconds of wall time; the isotope check imports two compounds and exports three times in one process. Four conversions run in parallel, and all of them together get 4500 seconds. Submissions that contain compiled code or symbolic links are rejected.
The hidden test set has 200 PubChem compounds, each written three ways (PubChem's SMILES, RDKit's canonical SMILES and a random RDKit SMILES), plus the cases in /app/data/regressions.csv. Most hidden compounds are stereochemically demanding: bridged and spiro ring systems, macrocycles, some with stereo double bonds in the ring, and molecules with eight or more stereocentres. A few carry isotopes or are salts. They contain only H, B, C, N, O, F, P, S, Cl, Br and I apart from isolated Li, Na and K ions, have at most 100 heavy atoms, have tetrahedral stereocentres only at carbon and have stereo double bonds only at C=C, C=N or N=N. /app/data/pubchem_dev.csv lists 100 more compounds of the same kind with their InChIs. Every conversion must pass.
You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.
样例 3 / 3
g2o:可认证全局最优的位姿图求解器
g2o 是常用作 SLAM 后端的 C++ 图优化库。Agent 要为它新增一个程序:求解位姿图的半正定松弛来估计 2D 或 3D 位姿,并说明这个结果能否被证明是全局最优。输入、输出和报告的格式都由一份规格说明规定,报告里的每个数值都会根据输入和输出的位姿重新计算核对。
- 专家用时估计
- 12 小时
- Agent 容器
- 2 核 · 8 GB
题面原文
Add a certifiable pose-graph solver to the g2o checkout in /app/g2o (…). It is a new application, CMake target g2o_certified, that estimates SE(2) or SE(3) poses from a g2o file through the semidefinite relaxation of the chordal cost and reports whether its estimate is a certified global optimum. Its specification is /app/g2o/doc/g2o_certified.md.
The application has to build and run with
cmake -S /app/g2o -B /app/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build /app/build --target g2o_certified -j2
/app/build/bin/g2o_certified /app/input.g2o /app/poses.g2o /app/report.json
Only /app/g2o is graded. It is copied to a machine without network access that has the system packages this image had when the task started; anything you install will not be there. Compiled files, archives, .git directories and symbolic links that point outside the tree are deleted from the copy. An unprivileged user then runs the two cmake commands from scratch with other absolute source and build paths, and runs the tool on hidden graphs. Configuring may take 300 s, building 1800 s and each run of the tool 600 s, always with 2 CPUs and 4 GiB of memory.
The hidden graphs have from about 20 to 10,200 poses and up to 30,000 edges, and the relaxation is exact on some of them and not on others. Public benchmark graphs are in /app/benchmarks; rim.g2o and cubicle.g2o there contain information matrices that are not positive definite, so they are not valid input as shipped.
Do not copy, vendor, download or link existing certifiable-estimation code: SE-Sync and its Optimization library, CPL-SLAM, CPL-Sync, DPGO, and the certifiable and Shonan modules of GTSAM. The submitted sources are compared with these projects.
You have 28800 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.
可定制
包括但不限于以下几项,其他需求都可以商量。
- 领域与技术栈:题目所属领域、编程语言和代码库类型。
- 难度:对标模型、Agent 框架、effort 档位和通过率区间。
- 运行条件:每题尝试次数、时限,以及 Agent 能否联网。
- 模型运行记录:每次尝试的轨迹、终端记录和校验输出。
- 数量和排期:题目数量、各领域占比和交付时间。