Quantifying infrastructure noise in agentic coding evals

생각 덩어리

인프라 설정이 벤치마크를 흔든다 — 최상위 모델 격차보다 큰 6%p

Infrastructure configuration can swing agentic coding benchmarks by several percentage points—sometimes more than the leaderboard gap between top models.

In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).

런타임은 수동적 컨테이너가 아니다 — 예산이 다르면 같은 시험이 아니다

Static benchmarks score a model's output directly—the runtime environment doesn't factor into the result. Agentic coding evals are different: models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns. The runtime is no longer a passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test.

바닥과 천장이 같으면 여유가 0 — 순간 스파이크가 OOM으로 죽인다

Our Kubernetes implementation treated the per-task resource specs as both a floor and a hard ceiling: each container was guaranteed the specified resources but killed the moment it exceeded them.

When these are set to the same value, there's zero headroom for transient spikes: a momentary memory fluctuation can OOM-kill a container that would otherwise have succeeded. To account for this, Terminal-Bench's leaderboard uses a different sandboxing provider, whose implementation is more lenient, allowing temporary overallocation without terminating the container in order to favor infrastructural stability.

여섯 구성 실험 — 인프라 에러 5.8%에서 0.5%로, 1x~3x는 노이즈 안

To quantify the effect of the scaffold, we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped. Everything else stayed constant: same Claude model, same harness, same task set.

This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped.

From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40). Most of the tasks that were crashing at 1x would have failed regardless—which is something that we observed in the data.

3배를 넘으면 성격이 바뀐다 — 자원이 성공을 직접 돕기 시작

Starting around 3x, however, this trend changes: success rates climb faster than infra errors decline.

The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. At uncapped resources, the total lift over 1x is +6 percentage points (p < 0.01).

한계가 측정 대상을 바꾼다 — 효율적 코드 대 무차별 대입

Above the 3x mark, however, additional resources start actively helping the agent solve problems it couldn't solve before, which shows that limits can actually change what the eval measures. Tight limits inadvertently reward very efficient strategies, while generous limits are more forgiving and reward agents that can better exploit all available resources.

An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones.

Under tight ones, the pod runs out of memory during installation, before the agent writes a single line of solution code. A leaner strategy exists (implementing the math from scratch using only the standard library), and some models do default to it. Others don't. Different models have different default approaches, and the resource configuration determines which of those approaches happen to succeed.

SWE-bench 교차검증 — 227문제·최대 5배 RAM, 방향 일치하나 +1.54%p로 작다

We also tested whether this pattern holds on evals outside Terminal-Bench by running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each. The same effect held, though the magnitude was smaller: Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x.

그 밖의 변수 — 시간 제한·시간대·API 지연, 흐려지는 능력/인프라 경계

In principle, every element of the evaluation setup can influence the final score, from the cluster health to the hardware specs, from the concurrency level to even egress bandwidth.

We have observed anecdotally, for instance, that pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents. ... the boundary between "model capability" and "infrastructure behavior" is blurrier than a single benchmark score suggests.

권고 — 보장 할당과 강제 종료 임계치를 분리해 명시하라

Given how container runtimes actually enforce resources—via a guaranteed allocation and a separate hard kill threshold—we recommend that evals specify both parameters per task, not a single pinned value. A single exact spec sets the guaranteed allocation equal to the kill threshold, leaving zero margin: the transient memory spikes we documented at 1x are enough to destabilize the eval.

The band between them should be calibrated so that scores at the floor and ceiling fall within noise of each other.

왜 중요한가 — 2점 리드가 더 좋은 하드웨어일 수도, 결국 더 큰 VM

As things stand today, a 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both.

Until resource methodology is standardized, our data suggests that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched.

A few-point lead might signal a real capability gap—or it might just be a bigger VM.