에이전트는 또 하나의 사용자 계층이다 — Chrome DevTools MCP가 배운 에이전트 인터페이스 설계 네 가지 교훈

생각 덩어리

사람용 DevTools를 에이전트용으로 — 코딩 에이전트는 장님으로 날고 있었다

Chrome DevTools for humans is used by millions of web developers on a daily basis to debug web pages. It's directly built into Chrome and developers use it to debug web pages. Find errors, audit it, performance profile it, and so on and so on.

This is a purpose-built Chrome DevTools, but for agents. How surprising.

This is going to work in any MCP client and agent harness that is MCP capable. Doesn't really matter. That was Gemini CLI, also works in Cloud Code, Codex, Open Claw, doesn't matter.

We ship Chrome DevTools because we saw that coding agents were flying blind. So, like 1 and 1/2 years ago, they were very good with generating code, but they were not able to validate what they actually were doing, right? And that just sucked.

데이터를 퍼부으면 될 줄 알았다 — 5만 줄 JSON과 dump zone

But we assumed they are going to be fine if we throw a lot of data at them. Because I mean, they are machines, right? The thing is, we were wrong.

A trace file it has all the data about the performance profile. And this is a file like multiple megabytes of data, and this is like 50,000 lines of JSON, and we did throw that against common agent harness at that point ... this is too much data for an agent for a model to actually reason about, and it blew through the context window.

And if you have seen Matt's talk about the dump zone, you are moving the agent into the dump zone at that point.

책 전체를 읽히지 마라 — 맞는 문장을 가리키는 semantic summary

Our performance tracing endpoints, it can also return that for post-processing with other tools, but what it's really doing is it's returning markdown now and semantic summaries.

Well, essentially, we didn't force the agent to read the entire book, the trace, but instead we just pointed it at the right sentence, and this is the semantic summary.

에이전트는 별도의 사용자 계층 — 같은 의도, 다른 인지 병목

Agents are a different user class. ... they are kind of like a separate user segment.

The thing is, agents and humans, they share the intent, they share the goal, right? In our case, for example, both want to identify errors in a page and want to fix those errors. But they think differently. They have different cognitive bottlenecks, more or less.

For humans, it's a lot about visual complexity. So, humans are typically very visual creatures, and we need layout, we need color in order to find a signal.

Tokens per successful outcome — 인터페이스의 연비

Effectiveness is about does the agent complete the entire user journey? Is the functional intent actually fulfilled? Yes or no. And then there's efficiency, which unsurprisingly is about token cost, tool calls, duration.

In the end, what tokens per successful outcome tell you is the fuel efficiency of your interface, right?

Fuel efficiency is relatively worthless if you can't reach your destination. So, that's why it's called tokens per successful outcome and not token per outcome. So, make sure that you actually also measure effectiveness, right?

연비 비교는 user journey 안에서 — 불완전한 측정이 직감을 이긴다

It's going to be tremendously different between different user journeys and task classes. So, don't compare them globally, compare them within your user journey.

In Chrome DevTools, we have the user journey of web scraping, right? So, an agent going to website and extracting information. That's relatively cheap. But there's also user journeys that are more intricate like debugging a website, finding out why the responsive layout is not working. That thing is going to use more tokens, but that is fine because it's a much more intricate and more interactive session that's happening.

Measuring tokens per successful outcome is not straightforward. The thing is, but even an imperfect measurement is better than simply doing gut-driven decisions. And with that, at least you can do data-informed decisions.

토큰 버닝의 세 각도 — 니치 툴 숨기기, slim mode, CLI 파이프

In DevTools for Agents, we're addressing token burn from three different angles. First, there's tool categorization. So, very straightforward, we hide niche tools behind command line parameters. Like, for example, we have tools for Chrome extension debugging. And not everybody is developing Chrome extensions. So, why add it to the default context menu? There's no point in doing that.

What slim mode is doing is like pushing tool categorization to its limits. It's only exposing I think three different tools. Select page, navigate page, and evaluate script.

The less tools you expose, the less tools are also at the disposal for your agent harness, which means your agent might do extra turns to achieve the same goal. It might not actually have the right tools at the disposal to actually do something, like for example, getting network requests. You can't do that with evaluate script.

You can have your agent chain commands together to do post-processing. ... The accessibility tree is extracted with a grep command, and then the result, the ID of the control is being piped into a click command. And that is, of course, saving a lot of tokens ... because the model doesn't need to process all the tokens. The token post-processing is happening on your computer.

에러는 토큰을 태운다 — 마지막 한 문장이 만드는 self-heal

Efficiency is useless if your agent gets stuck. So, that brings us to error recovery.

Every time your agent encounters an error, it's going to cost you tokens because it needs to retry, it needs to understand what is happening and stuff like that. And that just sucks.

You should add useful error messages. That sounds obvious. For a lot of tools, it isn't.

We actually added the last sentence, and that enabled the agent to self-heal, which is super useful because then the agent doesn't need a human to actually fix the problems, but the agent can self-fix the problems.

Proactive detour와 troubleshooting skill — 학습 데이터를 거스르는 우회로

Beneath each of the agents, there's a model, and the model is being trained on certain data. And sometimes there are things where you want to counteract the training data. And that's what you can do with proactive detours.

We detour the agent for performance profiling to our start performance trace tool and not to the Lighthouse audit.

We have a skill that's called troubleshooting, and we see a lot of people have problems setting up the Chrome DevTools MCP server correctly, and that troubleshooting skill is then going to kick in and help the human and the agent to fix the setup issues. Again, enabling self-healing of the agent.

모놀리식 툴 하나에서 25개로 — 문제는 해결되지 않고 교환된다

Our initial design had one monolithic tool called debug webpage. ... And it was neat from an engineering perspective, but it didn't really work.

So, we decomposed it into 25 different tools. And did we solve the problem? ... Of course, it wasn't. Because we traded that problem off to another. And that was agents now had 25 tools at their disposal. How are you going to find out which one to use when?

Schema가 곧 에이전트의 UI — 97%의 description에 quality smell

According to this paper here, 97% of MCP tool descriptions have quality smells. And this matters because the schema is the UI for the agent. So, let's make the UI better.

And also, smaller models in particular are not that good with more descriptions because they get biased in using tools they shouldn't be using in the first place.

What we have in there as a description is used to find front-end performance issues and core web vitals, LCP, INP, CLS. Why is this relevant? LCP, INP, CLS are web performance metrics, and an agent is able to make the connection, "Oh, I'm going to use that tool if I need to improve page load, for example."

We are far from finished optimizing that because models and agent harnesses also keep on changing all the time, so it's kind of like an endless quest for minimum viable description.

Skill도 공짜 점심이 아니다 — 트레이드오프는 이동할 뿐 사라지지 않는다

You can supercharge all of that with skills. ... But again, there's a trade-off. They are not free lunch. If you pile in too many skills, you're going to shift the problem and run into the same problem again.

Agents are going to call your skills even if they shouldn't be calling them. Your context window size is going to increase and all that stuff. The trade-off is shifting, it's not disappearing.

마찰은 설계다 — allow 버튼을 없애달라는 요청을 거절한 이유

Okay, now we have optimized for cost, recovery, and discovery. Let's now talk a little bit about trust because you don't want to have a backdoor into the system.

Chrome DevTools for Agents has a feature called autoconnect, and it's kind of like it lets you, the human, using your coding agent, like Claude Code, share the screen with the agent, like, "Hey, I'm stuck here. Please help me debug that and fix the problem that I'm seeing here."

And users, of course, requested the feature, "Hey, why do I need to click allow all the time? I don't want to do that. Please remember my choice." And in a traditional user experience design, that would have been a clear win, right? Because it's just fixing the friction that you want to remove.

In a world where you are delegating away work to agents and automating away agents, you need to think about trust boundaries. And so, that's why we actually designed it so that friction is actually by design.

Lethal trifecta와 세 개의 tier — 툴은 공유해도 보안 모델은 공유하지 마라

There is at least three tiers that I'm thinking about in browsing agents more or less. You have tier one and that is the local development environment. In the local development environment, you have the human in loop and the human wants to grant access to the default Chrome profile to the data that you already have access to to the agent in a time bound manner.

Tier two is agents running in continuous integration environment. So, it's controlled environments, but they're separated away. At that point, you should be using data separation things like containers, of course, but also other things like separate Chrome profiles and stuff like that.

And third, there's agents with full internet access and that is YOLO mode essentially because every webpage out there is able to do some prompt check chain text to your agent.

Key point being is a local agent, tier one, and a browsing agent fleet, tier three, where you're research agents maybe, might share a tool like Chrome DevTools for agents, but they shouldn't share nothing else about your security model that you're having, right?

(중간에 Simon Willison의 lethal trifecta 블로그를 인용하며 — 인간 동의가 매번 필요한 이유는 tier 1에서 세 요소가 모두 한자리에 모이기 때문이라고 설명한다.)

에이전트는 우리의 다음 사용자 — 네 가지 takeaway

User experience is evolving to incorporate agent experience. An agent is just another type of user, segment of user, also with non-functional requirements. Efficiency, discoverability, security, stability, and so on and so on.

Measure fuel efficiency of the interface with tokens per successful outcome. Turn errors into recovery playbooks. Audit descriptions for intent and never compromise trust for convenience.

Agents are our next users. Let's help them help us.