"It's Hard to Eval" Is a Product Smell

생각 덩어리

'평가하기 어렵다'는 제품의 냄새 — eval보다 설계가 먼저다

For the past 3 years, AI evals have been my professional focus. ... The most common objection I hear to evals is “our product is hard to eval”.

This objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification should come before building evals.

사례 1: 데이터 에이전트 — 답만 내놓는 설계의 실수

A common mistake when building AI data agents is to make the answer the only output, as illustrated below.

In the sketch above, the user has no way to verify the answer beyond redoing work.

데이터 사이언티스트의 검증법 — checkable artifacts를 넘겨라

A better design is to provide the user with checkable artifacts, informed by how a domain expert might validate the output.

Confirm the metric definition precisely. A number like net revenue can include or exclude things like returns and discounts.

Read the query. For an important number I look at the SQL to confirm it does what I think, and I tweak it and rerun to test my assumptions.

Note anything I could not verify. If a step has no trusted reference to check it against, I flag it instead of presenting it as settled.

progressive disclosure — 그리고 이것이 eval과 무슨 상관인가

There is progressive disclosure of details. The chat reply shows high value items like sources, assumptions, and issues. The user can optionally open an interactive notebook to see the full context.

But what does this have to do with evals? If you design your product for verification, annotation becomes less expensive and evals will have better signals to draw from. More importantly, you’ll provide your users with a better product.

사례 2: PE 수업안 빌더 — 검증된 플랜에서 출발해 diff만 리뷰

The founder asked me how to eval the lesson plans. I turned the question around: what does a teacher care about?

Therefore, a better design might start from vetted lesson plans that are actively used in schools. When the tool generates a plan, it shows which vetted plan it started from, who uses that plan, and a diff of what it changed for this teacher’s constraints.

Designing for this makes the product simpler to build. Instead of stuffing hundreds of examples into a prompt, the tool captures important dimensions, retrieves a close match, and adapts it. Automated evals now become tractable because there is less surface area to test.

사례 3: 산재 의료보고서 — 생성기가 아니라 research assistant로

The problem is the same as the other examples, but the stakes are higher. The only output is the report, and the doctor is the one accountable for it. To trust it they have to go back through the chart and confirm the facts and inferences themselves. That can take as long as writing the report from scratch, which defeats the point of the tool.

You might object that a fifty-page opinion is hard to verify. That is true, and the product should not pretend otherwise. Helping a doctor understand the evidence is arguably more valuable than the finished document. Therefore, I advised the founder to make the product work like a research assistant instead of a report generator.

패턴 일반화 — 검증을 위한 네 가지 질문과 provenance

Below are questions that can guide your product’s design for verification:

What does the user actually need to check?

What trusted thing can they compare it against?

Are there signals or heuristics that experts use to aid in verification?

What smaller units can they accept, edit, or reject?

A common thread across these examples is provenance. The fastest way to make an output checkable is to show where each part came from, with links to see more detail. Additionally, you can use progressive disclosure so these sources don’t overwhelm the user.

쉬워 보이는 코딩조차 — Cursor·Devin은 UI 변경을 영상으로 남긴다

These principles hold even when a product seems easy to eval. Coding is a good example: it is one of the most verifiable kinds of work, with tests, types, and diffs. Even so, some coding agents go the extra mile to make their work checkable. Cursor and Devin both record a short video of the UI changes they make, so you can confirm the work is right without reproducing it yourself.

새로운 건 없다 — 다만 AI 시대엔 검증이 병목이다

All of these ideas stem from well-established design principles. For example, watching an expert work to learn what they check before you build is called needfinding.

Even though these ideas are well established, a reminder is due in the age of AI. Before AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck. It is time to think about it more explicitly.