스킬 미니멀리즘 — 에이전트 스킬 95%를 지우자 결과가 좋아졌다

생각 덩어리

"Hi, I'm the bottleneck" — 8개 언어 20여 개 repo, 그리고 매번 10분의 셋업 세금

Hi, I'm the bottleneck. I'm a DX engineer at WorkOS and I work on 20 plus repos across eight different languages.

I haven't written a line of code myself in probably eight months. I've gotten really good at just scaling that with agents and then reviewing what they do and then instructing them and getting the work done faster and better while still maintaining good quality.

for every one of those there's like this little bit of setup time that I'm doing each time, which is like giving it 10 minutes of my time to like set up and establish the problem. Let's look at this GitHub issue. Let's look at this linear ticket. Let's take a look at this Slack thread and figure out what's going on

that was a lot of my time just spent dealing with the agent, getting it basically the context that I already have, and then getting it to work on it from there.

개발자에게 가는 파이프라인은 이제 에이전트를 통과한다 — 두 방향의 AI native

while I said I'm a developer experience engineer, the developer is still the most important in my job, but increasingly the pipeline to get to that developer is through agents. And so, I see the agentic experience as being equally as important because that's how we're going to get in front of the developers.

So, there's two different ways I needed to go AI native and two different directions for that.

Case — 무엇을 던지든 증거 달린 PR까지 멈추지 않는 harness

I started building this project called case. This is a harness. If you've read Ryan ... Harness Engineering, it's that. Just kind of took those ideas and started building them.

basically I give it a GitHub issue, a PR, a Slack thread, a linear ticket, anything, and I could just point it at it and it could figure out the context that it needs and go. And then it wouldn't stop until it has a PR with evidence that it actually did what I asked it to ... most importantly it had to provide that evidence.

Claude skill에서 state machine으로 — 다섯 에이전트보다 사이사이의 gate

it originally started as a Claude skill because why not? I thought Claude could do anything. And it was working really well, but as it got more complex, the context drop became very real. It would just start forgetting things or skipping over tasks. And I would ask Claude, "Why did you do that?" It's like, "Oh yeah, you told me to do that. I decided not to."

I rebuilt it on top of Pi and using a TypeScript state machine to facilitate going through and stepping through these agents. So, it has five different agents in it, an implementer, a verifier, a reviewer, a closer, and a retro agent. And those are important, but they're not the most important thing. The most important piece of case is the gates in between that.

when we implement something, we can't move on to the reviewer until the verifier verifies it. And once the reviewer reviews it, if there's any issues, it has to send it back to the implementer

the next agent doesn't really matter. Proving that the work matters. Proving that what happened in each of these states is what matters.

에이전트는 거짓말한다 — touch 한 번의 위증, SHA-256 봉쇄

the agents, they would just lie to me all the time.

I just had it check for a dot case tested file. And if that file existed, great. It ran the tests, perfect. Well, it figured it out pretty fast. Claude would just touch that file and be like, "Yep, I ran the tests." Such a junior engineer, I swear.

actually take the test output and SHA-256 that and save that into the case tested file and then verify cryptographically, yes, you actually ran the tests. And really ... the main piece there is that I just made it easier to just do the work that I wanted it to do rather than lie about it.

it stopped lying not because I asked it very nicely, I made it prove it that it was going to actually do the work each time.

WorkOS CLI — 5분 설치, zero friction의 바깥 방향

on the outward side with the WorkOS CLI, this is a tool that our customers use. And it can do lots of things, but its kind of headlining feature is that it can install AuthKit for you.

With WorkOS install, it just goes and figures out what project you're in. "Oh, you're in a Next.js project. You're in a TanStack project. You're in a Ruby project. I'll figure that out. Oh, you've already got Auth0 set up? I can easily remove that and put in AuthKit and we'll be good." And it does it in less than 5 minutes.

If you don't have a WorkOS account, it will provision one for you that you can go claim later. So, there is zero friction to getting it set up.

나한테도 Claude한테도 맞아 보였던 코드 — start.ts의 암묵적 계약

it would be overly confident just like these models always are and say, "Yep, I did that."

it made some changes to a file called start.ts. That file ... has an implicit contract with TanStack. It ... has to export certain things. And we kind of messed that up. The code looked right to me. It looked right to Claude, but it did not look right to TanStack start. So, boom, it failed.

문서 전체를 스킬 10,000줄로 — 토큰을 늘리자 결과가 나빠졌다

I thought, "You know what? We have these great docs. I can just take our docs and generate some skills." So, I generated over 10,000 lines of skills that were all based on our docs.

I thought I was being really clever and awesome. And I generated this huge thing. And I even made some evals for it. I started making those and it would take me 68 minutes to run those scenarios. It was just crazy. And it would fail over and over and it would have these retries and get there eventually, but it was like a lot of work, a lot of tokens.

I had more tokens. I thought more tokens, great. That's way better. But it ended up producing worse results. And it was really the measurement there, the evals that were telling me, "Hey, this isn't right."

95% 삭제, 553줄의 gotchas — 성능은 오히려 올랐다

So I rewrote it by hand. And instead of focusing on covering comprehensively everything that we have in our docs, I was like, "Oh, I just have to cover some common gotchas for everything." So, for our entire docs, instead of having 10,000 lines of that, I have 553 lines of gotchas.

They ran faster, way smaller in terms of token count, only took 6 minutes per run, and I wasn't sending the models on these long goose chases ... It would stay focused on things. And so, by deleting 95% of that, the performance of it actually went up.

And I really only knew that because I measured it.

한 스킬은 로드 시 77% vs 미로드 97% — 측정 없이는 몰랐을 역효과

I ... had one skill in particular that I could see. And when I ran it with that skill and I gave it a task and said, "Hey, load this skill and then do this task." It got it correct 77% of the time. But if I asked it to do the same task without loading the skill, it was correct 97% of the time. So, I was actively making it worse, and I only knew about that because I was measuring it.

evals are super important when you're working with this non-deterministic code.

Claude makes it really easy now. They have ... a Claude skill that will do evals for you. And it'll even ... create like an HTML output of that and show you like side by side.

I just needed to trust that the model already knew how to code, and I just had to kind of gently nudge it in the right direction in some cases.

Enforce, don't instruct / Guide, don't prescribe — 지시 대신 강제, 백과사전 대신 지뢰 지도

you want to enforce things, don't instruct. The model can lie about it. It can decide ... not to do certain things because either it forgot about it, it got distracted with other things, but if you actually set up a pipeline where it has to enforce itself and prove to you that it did what you asked it to do, then you're going to have a better time for sure. And oftentimes with a lot less tokens.

you want to guide the model, don't prescribe it. So, don't just give it like, "Hey, here's a summary of all of my docs with like a whole bunch of information."

"Hey, when you're working in Next.js and you're in the proxy, you want to do this. If you're not in the proxy, you can't call redirects." That's a really big one that constantly comes up over and over and over.

Measure, don't assume — 코드를 보기 전에 Playwright 비디오부터

measure, don't ... assume that it works.

I still read all of the code that it generates to make sure that it's actually like code that I would be proud of shipping, but I'm not even going to waste my time looking at that code until it's proved to me that it did whatever I asked in a non-code way.

if it's working on a UI bug, I want it to use the Playwright CLI and record a video of itself doing something before and then doing it after the fix and showing me, "Hey, now it's fixed. It's working." And if it can prove that to me in those videos that it attaches to the PR, I'm way more inclined to look at that PR

If not, I just ask it to do it again. So, every failure became data for the next run.

실수를 고치지 말고 harness를 고쳐라 — retro agent와 memory 파일

if you are working on a harness and it is making mistakes, don't go fix the mistakes that it made, fix the harness so that it can fix the mistakes.

he talked about how their team would never work on the code itself. They would only work on the harness to fix the code itself. And I really took that to heart with Case, so I only work on Case itself to make sure that it's doing what I want.

the final piece of it is this retrospective agent. And all it does is it looks at what it did, and it goes in and looks at like the Claude and Codex transcripts like the JSONL files, and it pulls out information. Hey, was I running a lot of tools at the same time? Did I run the same tool request three times in a row without any changes to anything? Was I like getting in a doom loop there

Case keeps a whole bunch of memory files as markdown files ... I have a general memory file. If I'm working in Next.js, I have a Next.js memory file, a TanStack Start memory file, etc. ... So that it won't make a mistake and break the start.ts in TanStack Start again. It knows about that because it put it into its memory.

제품을 에이전트용으로 — 모델은 이미 코딩을 안다, landmine 위치만 알려라

figure out what the agents get reliably wrong about your product and focus on that. Don't focus on the product as a whole because it probably knows a lot about it a lot more than you think

Write down those gotchas, create skills around those. ... just remember that the models know how to code. They just need to know the intricacies of your product and where the landmines are in that.

Otherwise, you just might be adding noise and sending the model on wild goose chases.

think about those agents in the same way that you think about developers. What do they want to know? How can I make things better for them?

신뢰를 증거로 대체하라 — 일의 본질은 원래 시스템 구축이었다

you replace your trust with evidence. Never trust it. Always make it prove to you that it did something. If it ran the test, make it prove it. If it fixed a UI bug, it has to show it to you. Otherwise, don't waste your time on it.

enforce that with code not prompts. So, this is why I switched it to Pi and used a state machine to force it because I have full control over that state machine, and it's outside of the Pi or Claude deciding, "Should I do this or not?" No, you have to do it. I enforce that through that loop.

every failure becomes a system bug. Each time it messes up on something, that's a bug in the harness. Go fix the harness.

the practices that we have haven't really changed. Our job hasn't really changed. We've just kind of abstracted it a little bit. Your job was never really about writing code. It was always about building these systems, and now we just have a better abstraction to understand that.