컨텍스트가 많을수록 멍청해지는 에이전트 — U 커브, 전략적 컨텍스트 최적화, judge agent

생각 덩어리

결정론의 세계에서 비결정론의 세계로 — DevSecOps 출신의 시선

I have a background in dev sec ops. So I'm coming from an industry where everything was deterministic. The pipelines they run they crash. If they crash, we fix them ... to a place where we are doing agents where nothing is deterministic.

So in my last few years I have learned where and how agents fail, what are the learnings and today I'll be sharing some of my learnings with you.

에이전트 진화의 궤적 — 4K 정적 프롬프트, 그리고 멈출 줄 모르는 루프

it started with static prompts where it was a 4K context window and we tried to put whatever was important or whatever we deemed important and the AI models will process it and provide you with a result.

that means that it was on us to tell LLMs what they should look into. That means if we provide wrong inputs, we might not get proper results.

we started with agentic workflows. So we created an agent. We get them tools like search tool to go into search into documents and do something ... which again created kind of a loop where the tool does not know where to stop. It thinks like I need more inputs again going back and back. It's a loop

멀티에이전트의 함정 — 도구가 늘수록 충돌도 는다

nowadays multi-agents is becoming more popular. ... a security agent trying to figure security concerns, a review agent trying to review the tool, a coding agent trying to fix things. Now again, the more the tools, the more issues you have. Not every agent understands and they have clash in their understandings where you don't get into the results.

컨텍스트는 문제가 아니다 — U 커브, 가운데는 purge된다

context is not a problem. Day by day the models are coming where you can dump a lot of context, a lot of data. But does that make sure that the results you are getting is smart enough to give you everything or smart enough to decide what's important?

it takes the initial inputs you provide, it takes the last inputs, but the in between context is basically removed. ... This is like a U curve where some of the things from the start, some of the things from the end make sense but whatever you are providing in between, that is not taken up.

This is something which we are working on and we are actually benchmarking things. ... If we give something at the end as an input that is in focus but all between context like I have Jira I have MCPs can you look into that the LLMs try to get rid of those things and purge them

전략적 컨텍스트 최적화 — 컨텍스트 엔진은 바운서다, 그러나 600개 repo에서 무너진다

creating strategic solution for context optimization rather than dumping everything to the models and asking them to be smart enough to find out what is more important.

context engine is a buzz word like everybody wants to create context engine ... but context engine is like a bouncer right so your high-speed car is going and it acts as a bouncer and tells you this is more important

The problem is the indexing part takes moderate effort but the scaling is a challenge. Like if you start talking about 600 repositories or 700 repositories the mapping and the indexing starts to slow down and it becomes again unpredictable

계층 요약과 지식 그래프 — 선불 비용과 효용의 트레이드오프

hierarchal summarization where instead of creating or going through everything a summary is created for each file and folder so that when the agents try to find they can try to read the summary and see if that is more important to us or not

you need a lot of LLM processing. every time a file is created or changed some of the agents need to go and create a mapping for that.

knowledge graph is complex but it works wonder when you have logical dependencies. ... the initial input needed by the developer is very high. It takes a lot of time to create that. But if you have complex logics or you have dependencies on multiple repos that works wonder

반복 검색과 자기교정 — library card와 critic node

for most of the task if you're not a product company but if you're building agents for yourself or your processes iterative retrieval works really good because instead of even creating a summary it creates kind of an index. So it's like a library card which you give to your agents

There is also option of selfcorrection where you ask the LLM to do something and there is a critic node which looks and say if that is relevant to your initial goal or not. ... It takes a little bit more time because it adds a latency of running the agents and again but it does not require a lot of input initially from the developers

오케스트레이션 패러독스 — 문제를 푸는 대신 푸는 방법을 연구한다

It goes into a loop that instead of actually looking into solve the problem, they look for the method to solve the problem. They hop on from one method to one another method and most of the API tokens are wasted on finding a way to do it rather than doing it.

if you use Opus latest and greatest, they will try to see what is the best method to do it and challenging themselves again and again. Maybe not this, another way, another way. And it just goes into a loop of trying to do something rather than doing something.

80/20 하이브리드 — 연구는 자유롭게, 마무리는 하드 게이트로

What our teams are doing is giving the latest and the greatest models or giving the agents power to research 80% of the time. So you give them the goal and say okay try to do whatever you can but the 20% of the task where you need final validation you want summarization ... those are more hard gates for example if I get X results I want Y it's more deterministic

For some organization they do counter mechanism where after four or five counters you have to work with whatever was the last results. For some of them they have timeout counters that after 5 minutes whatever is the last tool or whatever is the last decision you work with that

for 80% usually you use high reasoning models latest and the greatest but you don't need a high reasoning model for the 20% ... they just need to see what was your goal what was the result you are trying to achieve

거대 단일 에이전트의 실패 — 4개를 시키면 2개는 실종된다

as the context grows, teams think okay we can do everything with one agent because the context window is quite great

when the agent is going forward it get overwhelmed with the inputs and again it tries to start losing what was the original task. So maybe you give four tasks to the agent and somewhere down the line it focuses on two tasks. So you get great results for the two task but the other two just get lost in the middle.

Mixture of Agents — 그리스 호텔에 암스테르담발 항공권, 그래서 judge agent

we have something called mixture of agents. ... instead of one big agent, we create issue expert agents. We create small small agents which are doing great in a specific task

I give an agent to find the best hotel. Another agent to find the best location. Another agent try to find best flights. But all three of them gives me different result. The hotel is in Greece. the flight is from Amsterdam to maybe Portugal and everything just doesn't make sense

there's a concept called a judge agent what it does it tries to get all the results and see if they can make sense together so now you are doing all the greatest things from different agent getting the best results from their part but a judge agent help us to combine these and make one sense out of it

Qodo의 PR 리뷰 아키텍처 — context collector가 나누고 judge가 거른다

as part of a PR review we have a context collector which actually goes and collect context from the PRs it could collect context from the context engine. it collect context from the tools but then it does not start working and giving you the reviews.

Once all these agents give us back, a judge agent actually looks for the results and say okay these are interesting enough but is it relevant to you? They can again go back in the context engine look into the PRs and see out of the 10 things which is provided by you how many of them actually make sense for your thing.

---

Q&A — 컨퍼런스 라이브 토크

스웜의 통신 — LangChain 위에서, 결과가 다음 prompt가 된다

In practice, how do you let the swarm communicate with each other? — we use longchain at the bottom and that is being used to communicate and build the infrastructure for different agents.

we try to get the results and create a prompt for the next agent and if it's multiple things again there is an agent just to collect the results and create a better prompt which is refined for the next agent

캘리브레이션 — PR 히스토리를 인덱싱해 컨텍스트에 두 번 주입

LLM does not know what is important for you or how do you work. ... they get input from healthcare industry they get from retail industry they get from finance industry and all of them can use same Java framework in different ways

we give them the PR history. So we index all your PR and see when was the last time something like this was identified and compare the current version

that again is transfer to the context twice. First is when we are actually giving context to the sub agents to find things for you and another time to the judge agent. So that when I get 15 different recommendations for a code review, my judge agent can look into what was there before, how did your reviewer commented, how your developers commented and based upon that decide if that is worth providing to your developers or not.

부분만 보는 에이전트 — 옛 코드리뷰의 전문가 분업과 같은 구조

you only share the specific information to each agent right and then basically each agent runs atomic autonomously ... And doesn't have a full picture

if you look into the older version of code reviews ... you used to have a senior engineer who knows your code who knows what kind of packages you're using ... Then you used to have a security person who used to see if you are providing all kinds of security, you are not hard-coding your APIs or you are not putting any SQL injection.

we have a web portal where architects can provide their guidelines ... compliance people can provide their guidelines and an agent can look into all those guidelines and say is it validated or not.

가중치의 운영 — 개발자의 수락·거절이 다음 추천의 무게를 바꾼다

every time your developer accepts a suggestion, it gets more weighted for the next one. If it does not accept the suggestion, it gets a less weight. So it's all about indexing and making sure those weights are managed somewhere.

there is something called bug fixes and there is something called rules so if you provide them as a rule it will get highlighted doesn't matter if you want it or not.

some people are used to hard-coding their API keys and I literally had a tough argument with the developer but this is how we do it. No this should not be the way.