Harness design for long-running application development

생각 덩어리

두 문제, 한 구조 — GAN에서 빌린 생성자·평가자

Taking inspiration from Generative Adversarial Networks (GANs), I designed a multi-agent structure with a generator and evaluator agent.

To break through, I sought out novel AI engineering approaches that held across two quite different domains, one defined by subjective taste, the other by verifiable correctness and usability.

The final result was a three-agent architecture—planner, generator, and evaluator—that produced rich full-stack applications over multi-hour autonomous coding sessions.

Context anxiety — compaction이 아니라 리셋

Some models also exhibit "context anxiety," in which they begin wrapping up work prematurely as they approach what they believe is their context limit.

Context resets—clearing the context window entirely and starting a fresh agent, combined with a structured handoff that carries the previous agent's state and the next steps—addresses both these issues.

While compaction preserves continuity, it doesn't give the agent a clean slate, which means context anxiety can still persist.

In our earlier testing, we found Claude Sonnet 4.5 exhibited context anxiety strongly enough that compaction alone wasn't sufficient to enable strong long task performance, so context resets became essential to the harness design.

자기 평가는 늘 후하다 — 일하는 자와 심판하는 자의 분리

When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre.

Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue.

tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work, and once that external feedback exists, the generator has something concrete to iterate against.

"아름다운가" 대신 "원칙을 따랐는가" — 주관을 채점 가능하게

Absent any intervention, Claude normally gravitates toward safe, predictable layouts that are technically functional but visually unremarkable.

"Is this design beautiful?" is hard to answer consistently, but "does this follow our principles for good design?" gives Claude something concrete to grade against.

Originality: Is there evidence of custom decisions, or is this template layouts, library defaults, and AI-generated patterns? A human designer should recognize deliberate creative choices. Unmodified stock components—or telltale signs of AI generation like purple gradients over white cards—fail here.

The criteria explicitly penalized highly generic “AI slop” patterns, and by weighting design and originality more heavily it pushed the model toward more aesthetic risk-taking.

평가자에게 Playwright를 — 라이브 페이지를 직접 돌아다닌다

I gave the evaluator the Playwright MCP, which let it interact with the live page directly before scoring each criterion and writing a detailed critique.

I ran 5 to 15 iterations per generation, with each iteration typically pushing the generator in a more distinctive direction as it responded to the evaluator's critique.

Because the evaluator was actively navigating the page rather than scoring a static screenshot, each cycle took real wall-clock time. Full runs stretched up to four hours.

I also instructed the generator to make a strategic decision after each evaluation: refine the current direction if scores were trending well, or pivot to an entirely different aesthetic if the approach wasn't working.

기준의 문구가 결과물을 조형한다 — museum quality의 부작용

The wording of the criteria steered the generator in ways I didn't fully anticipate.

Including phrases like "the best designs are museum quality" pushed designs toward a particular visual convergence, suggesting that the prompting associated with the criteria directly shaped the character of the output.

Even on the first iteration, outputs were noticeably better than a baseline with no prompting at all, suggesting the criteria and associated language themselves steered the model away from generic defaults before any evaluator feedback led to further refinement.

Later implementations tended to be better as a whole, but I regularly saw cases where I preferred a middle iteration over the last one.

네덜란드 미술관 — 10번째 사이클의 창조적 도약

By the ninth iteration, it had produced a clean, dark-themed landing page for a fictional museum. The page was visually polished but largely in line with my expectations.

Then, on the tenth cycle, it scrapped the approach entirely and reimagined the site as a spatial experience: a 3D room with a checkered floor rendered in CSS perspective, artwork hung on the walls in free-form positions, and doorway-based navigation between gallery rooms instead of scroll or click.

It was the kind of creative leap that I hadn't seen before from a single-pass generation.

풀스택 3-agent — planner는 야심을, 경로는 모델에게

The generator-evaluator loop maps naturally onto the software development lifecycle, where code review and QA serve the same structural role as the design evaluator.

Opus 4.5 largely removed that behavior on its own, so I was able to drop context resets from this harness entirely.

I prompted it to be ambitious about scope and to stay focused on product context and high level technical design rather than detailed technical implementation.

It seemed smarter to constrain the agents on the deliverables to be produced and let them figure out the path as they worked.

Sprint contract — 코드 전에 '완료'를 협상

Before each sprint, the generator and evaluator negotiated a sprint contract: agreeing on what "done" looked like for that chunk of work before any code was written.

The generator proposed what it would build and how success would be verified, and the evaluator reviewed that proposal to make sure the generator was building the right thing. The two iterated until they agreed.

the evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states.

Communication was handled via files: one agent would write a file, another agent would read it and respond either within that file or with a new file that the previous agent would read in turn.

레트로 게임 메이커 — 20배 비싸지만 게임이 돌아간다

The harness was over 20x more expensive, but the difference in output quality was immediately apparent.

More to the point, the actual game was broken. My entities appeared on screen but nothing responded to input. Digging into the code revealed that the wiring between entity definitions and the game runtime was broken, with no surface indication of where.

This run started from the same one-sentence prompt, but the planner step expanded that prompt into a 16-feature spec spread across ten sprints.

The biggest difference was in play mode. I was actually able to move my entity and play the game.

Claude는 무른 QA — 로그를 읽고 프롬프트를 고치는 튜닝 루프

Out of the box, Claude is a poor QA agent. In early runs, I watched it identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway.

It also tended to test superficially, rather than probing edge cases, so more subtle bugs often slipped through.

The tuning loop was to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QAs prompt to solve for those issues.

하네스의 모든 부품은 가정이다 — 하나씩 떼어 검증

every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve.

Our blog post Building Effective Agents frames the underlying idea as "find the simplest solution possible, and only increase complexity when needed," and it's a pattern that shows up consistently for anyone maintaining an agent harness.

In my first attempt to simplify, I cut the harness back radically and tried a few creative new ideas, but I wasn't able to replicate the performance of the original. It also became difficult to tell which pieces of the harness design were actually load-bearing, and in what ways.

Based on that experience, I moved to a more methodical approach, removing one component at a time and reviewing what impact it had on the final result.

Opus 4.6 — sprint를 떼고, evaluator는 경계의 함수로

I started by removing the sprint construct entirely. ... Given the improvements in Opus 4.6, there was good reason to believe that the model could natively handle the job without this sort of decomposition.

Without the planner, the generator under-scoped: given the raw prompt, it would start building without first speccing its work, and end up creating a less feature-rich application than the planner did.

Tasks that used to need the evaluator's check to be implemented coherently were now often within what the generator handled well on its own, and for tasks within that boundary, the evaluator became unnecessary overhead. But for the parts of the build that were still at the edge of the generator’s capabilities, the evaluator continued to give real lift.

the evaluator is not a fixed yes-or-no decision. It is worth the cost when the task sits beyond what the current model does reliably solo.

DAW 실험 — QA가 잡는 last mile

The run was still lengthy and expensive, at about 4 hours and $124 in token costs.

The main failure point is Feature Completeness — while the app looks impressive and the AI integration works well, several core DAW features are display-only without interactive depth: clips can't be dragged/moved on the timeline, there are no instrument UI panels (synth knobs, drum pads), and no visual effect editors (EQ curves, compressor meters). These aren't edge cases — they're the core interactions that make a DAW usable, and the spec explicitly calls for them.

The generator was still liable to miss details or stub features when left to its own devices, and the QA still added value in catching those last mile issues for the generator to fix.

Additionally, Claude can’t actually hear, which made the QA feedback loop less effective with respect to musical taste.

하네스의 공간은 줄지 않는다 — 이동한다

In some cases, that will mean the scaffold surrounding the model matters less over time, and developers can wait for the next model and see certain problems solve themselves. On the other hand, the better the models get, the more space there is to develop harnesses that can achieve complex tasks beyond what the model can do at baseline.

when a new model lands, it is generally good practice to re-examine a harness, stripping away pieces that are no longer load-bearing to performance and adding new pieces to achieve greater capability that may not have been possible before.

my conviction is that the space of interesting harness combinations doesn't shrink as models improve. Instead, it moves, and the interesting work for AI engineers is to keep finding the next novel combination.