부서진 Evals를 그래도 쓰는 법 — 랩 숫자 불신·자체 벤치마크·hill climbing

생각 덩어리

비판이자 출구 — 대부분이 evals에 대해 틀렸고, 당신은 맞기를 바란다

the title of my talk today is evals are broken and you should use them anyway. And a lot of this talk is just like a straight-up critique of like the way we do evals these days. ... I kind of want to give you a way out of this.

Most people are wrong about evals. And I want you to be right about evals. I want you to use them. ... I want you to be able to build with them, interpret them, use evals in your own agentic flows, leverage them in any way sense that you can.

오답 진영 1: objective metrics — 대시보드의 비슷한 숫자는 동치가 아니다

there's two camps of wrong on evals. The first camp is a camp of objective metrics. ... there are people who would like look at a list this dashboard ... and they will interpret this as something akin to ... GPT 5.4 is effectively the same as like Gemini 3.1 Pro preview. Believe me, they're not the same.

there was this tweet this came out like just this morning and it was a critique of meta where meta came out and it was just like classic benchmark maxing. Just like we're doing best on the benchmark. Everything's great. And I assure you if you try a lot of these models like it just won't hold the test of like actual real-world evidence.

오답 진영 2: taste — vibes와 의인화, 진실은 그 중간

people in the taste camp are kind of like this. ... they think that it's like everything's about like vibes and you know, like it's like if you ask them like why do you like operas and they'll say things like I like talking to her. Like they'll anthropomorphize it.

I think the truth is somewhere in the middle that the emails are not the end all be all. They're also not completely useless. There are right ways to use them. There are wrong ways

세 단계 로드맵 — 남의 것을 빌리고, 내 에이전트를 개선하고, 직접 만든다

the first stage is like you can like leverage emails from other people. The second stage is to like use emails to improve your own agents. And then the third stage is like build actually build your own emails for specific use cases.

In the interest of time, I could personally talk about emails for hours, but like I can only talk about level one and two and I think those would be most helpful for most people in the audience.

휴리스틱 1 — 랩이 모델과 함께 내미는 숫자는 일단 불신

the first heuristic is that whenever a model apps comes out with like a number, just don't believe them. Just don't. Just like these are approximations. Coming back to the tweet like just like just don't believe the model app eval numbers.

Nikunj said that like a lot of like AI researchers and engineers routinely dismiss evals. They don't really think of it as like something that's like the numbers to be taken that seriously.

휴리스틱 2 — 최신은 따르되 earliest adopter는 되지 마라

if you notice like in the last like 2 years, every single couple months ... the frontier model is changing. And it's changing so fast. Like it's just like it's so hard to keep up with this stuff.

let the thing come out first. Let things settle on fire for like a couple weeks. And then if the thing still stands the test of time, I think at that point you should like do your model switch and like try something rather than like always trying to be on the cutting edge.

the people who have to always try the new model and try the new things like they'll be me. But like I do this for a living and you don't have to.

휴리스틱 3 — 아주 새롭고 아주 정밀한 eval만, 표준이 된 것은 이미 늙었다

the third heuristic for evals is that like you need to look for very very new and very precise evals. And the reason this is necessary is that like a lot of evals are like at this point they've become like standardized. They're actually kind of old. Like they're not useful for you.

this is a blog post from OpenAI where they straight up said as we bench verified no longer measures frontier coding capabilities.

it would have problems like solve the Fibonacci sequence. It would have problems like, you know, do matrix multiplication or something. And it's just like it doesn't apply to like real world software engineering.

Eval은 엔지니어링 문제이자 철학 문제 — 무한한 search space의 근사

you want to think of evals as like an engineering problem, but also as a philosophy problem, right? So, the engineering problem is obviously hard, but the philosophy problem is also very hard.

even then, coding problems have like an infinite search space. They can go in any direction. So, you want to build evals that like are somewhat more approximate representation of the actual thing that you're dealing with.

Cline의 결단 — '완전 무시'라는 업계 스탠스에서 자체 구축으로

this was the stance of the Codex team and a lot of other teams that we've talked to. ... just completely ignore them. They're completely unnecessary. Should probably probably wasting your time and it's just like I don't know who will be appeased by them.

I think we got to up the ante and we got to like we got to have some measure. We got to try emails. And if no one else is doing it, we'll do it ourselves. We'll build actual emails from scratch that would like actually test real-world programming problems of users.

we went through a lot of like a massive data sets of like people who had opted in to share their coding usage of client with us and we offered them money ... then spent a lot of time parsing through that ... just like completely cleaning it all up like doing a lot of like really hard manual labor trying to make like very decent problems

에이전트 채점은 다르다 — binary answer가 없는 trajectory 전체를 매긴다

a single turn email is very easy to do because it has like a binary answer. It has just like a very limited search space of what the answer could be. But when you're working with an agent, that can't be the case.

the agent like reads through files, searches through docs, installs the environment, sets things up, runs Python scripts, does all of that, and then in the end like runs some tests, and then maybe the whole thing works. ... did it actually work or did it not? Did it work, but like broke other things?

Terminal Bench와 Harbor — 89개 실전 문제, 격리 환경, 병목은 가장 느린 태스크

some very very awesome, smart, bright people from Stanford University came up with Terminal Bench ... they came up with like 89 coding problems, which are just like very approximate, decent representation of like real world programming problems.

some of these tasks would take up to like 30 to 40 minutes. And that's how you know they're legit because like ... the agent does a lot of things and just like runs into our code, and sometimes just goes crazy

you want to be able to give that the eval run an isolated environment like a virtual machine. In that virtual machine, it has the whole setup, it has the repo, it has everything. And then you install whatever agent you have, in our case client, cloud code, codex, whatever you want to use

Harbor basically lets you split up like the 89 tasks and then they all run in parallel. So, that way ... the limiting factor is basically the slowest task.

실패를 allocate하라 — trace를 읽는 또 하나의 에이전트, 그리고 작은 레버

you first do a run on the 89 task, you get a score, you evaluate all the failures.

you want to run like another agent which goes through the traces of all the failures. So, the trace would be like this massive file which has like every single LM call that the agent did and then be like, okay, this specific problem failed because it didn't run past. This failed because the retry tool was broken.

once you properly allocate those failures, you figure out, okay, these are the small levers that I can pull. If I pull those levers, like I can make like massive improvements to my AI agent.

테스트되는 것은 셋 — 모델, 하니스, 그리고 문제 자체

You are testing the model itself. ... you could have a horrible harness, you could have a horrible agent, but like the model just like overshoots so hard that just like you ... get a great score.

Anthropic's models could potentially work with the cursor, could potentially work with droid ... but for some reason it just seems to work so much better with the cloud code, right? And I think that that is like the testing the harness that like is the harness actually really leveraging the best of the model.

another problem is saying if you're solving stupid problems, it doesn't matter if you score 100% all the time.

43%에서 시작한 hill climbing — 개선의 세 구역, overfitting은 위험지대

we basically like had like this original score which was like much lower like 43%. We made changes to like CPU, we made changes to memory in front of the containers. We raised timeouts, we improved the thinking behavior.

Sometimes asking model to think more actually interferes with the quality of the response because it goes in like it gets like a stroke and it just like goes in like circles and it's like it'll just like I am a model. I am a model. It'll just like keep doing it for like ... 2,000 tokens.

the zone two is the most critical one where you actually do nuance improvements. And these nuance improvements are things like there are certain prompt engineering techniques that apply to entropic model families that just straight up would not apply to Codex model family. That would be very different from Gemini model family.

zone three is the danger zone where it's like you're straight up overfitting. So, you're overfitting in the sense of like you're just straight up cheating to get the highest score and then you can like make a tweet about it. Don't

결론 — 숫자와 vibe check, 둘 다 통과해야 한다

regardless of the kind of problem that you have, I want you to like find a benchmark and like build the eval and just like hill climb on it. So, hill climbing means that like you get a score and then you improve the score off your harness on the eval.

you got to both pass the vibe check. Like, does it actually feel good to use this product in this model? And at the same time, you also have like a very very very decent score, hopefully.

we were very decent on Anthropic model families, not so much on, say, Gemini model family ... when we started hill climbing, we learned that like, "Oh, if we support these models, we have this like entire swaths of people who love these models, and they can start using us."