Evals Skills for Coding Agents

생각 덩어리

evals-skills 공개 — 50+ 기업·4,000+ 수강생에서 본 실수의 방어막

Today, I’m publishing evals-skills, a set of skills for AI product evals. They guard against common mistakes I’ve seen helping 50+ companies and teaching 4,000+ students in our course.

코딩 에이전트를 evals에 투입한다

Coding agents now instrument applications, run experiments, analyze data, and build interfaces. I’ve been pointing them at evals.

OpenAI Harness Engineering의 증거 — 모델보다 에이전트 주변 인프라

They built a product entirely with Codex agents — three engineers, five months, ~1 million lines of code — and found that improving the infrastructure around the agent mattered more than improving the model. The agents queried traces to verify their own work.

Documentation tells the agent what to do. Telemetry tells it whether it worked. Evals tell it whether the output is good.

지루한 작업은 에이전트 몫 — 벤더 MCP 서버의 등장

All major eval vendors now ship an MCP server. The tedious parts: instrumenting your app, orchestrating experiments and building annotation tools now fall to coding agents.

플랫폼 접근 ≠ 일하는 법 — generic hallucination score의 함정

But an agent with an eval platform still needs to know what to do with it. Say a support bot tells a customer “your plan includes free returns” when it doesn’t. Another says “I’ve canceled your order” when nobody asked. Both are hallucinations, but one gets a fact wrong and the other makes up a user action. If you lump them together in a generic “hallucination score,” you’ll miss errors.

These skills fill the gaps. They complement the vendor MCP servers: those give your agent access to traces and experiments, these teach it what to do with them.

진입점은 eval-audit — 6개 영역 진단과 우선순위 처방

If you’re new to evals or inheriting an existing eval pipeline, start with eval-audit. It inspects your current setup (or lack of one), runs diagnostic checks across six areas, and produces a prioritized list of problems with next steps.

Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.

If you’re experienced with evals, you can skip the audit and pick the skill you need

범용 스킬은 출발점 — 결국 자기 스택·도메인·데이터로 직접 쓴다

These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.

각주 — product evals는 MMLU·HELM 류 벤치마크가 아니다

Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data.