Claude Code 위의 시맨틱 코드 검색 벤치마크 — embedding은 캐시된 컴퓨트

생각 덩어리

Turbopuffer — object storage 위에 first principles로 지은 검색 DB

we are a serverless full text and vector search database built from first principles on top of object storage. We serve some of the fastest growing AI companies in the world

Claude Code의 기본값 — semantic search를 버리고 grep을 택한 Boris

cloud code by default ... doesn't use semantic code search. There's actually a tweet from Boris. ... he's actually like the founding father of cloud code and he talks about how ... the early versions of cloud code did actually use semantic search with a local vector DB, but they just kind of found agentic search which is actually you know, grepping through your file system kind of worked better

그래도 Cursor는 인덱싱한다 — 24% 정확도 개선과 2.6%의 의미

one of our very first customer is Cursor does use semantic code search and does index code bases into Turbo Puffer and you may think like this seems like a lot of work and it kind of is, but the reason they do this is because they see real performance gains

performance gains they see with their composer model is like 24% increase in relative improvement in answer accuracy and I think it's like 12 and a half or 13% across all models

they perform an online AB test where they found that ... allowing or disallowing semantic code search led to like a 2.6% increase in code retention in large code bases and then 2.2% decrease in dissatisfied user requests

keep in mind that you know, out of 100 queries not all queries will ever really need semantic search or like benefit from it

Embedding은 캐시된 컴퓨트 — grep은 매 세션 같은 연산을 다시 한다

we think about how embeddings are cache compute

this is repeated on every session across you know, every agent running in for the same code base so that even if you're asking the same question or trying to do the same thing multiple times, you always have to do this compute again

you have this upfront cost where you have to chunk embed and index the code base, but then you would essentially have this cache of the semantic meaning

it's not a lot of savings in one time, but ... we're not really running one agent anymore. Like I'm running like three at one time.

Turbo Grep — chunk·embed·upload 세 동작의 CLI

we built a simple CLI tool for cloud code. We call it Turbo Grep. It's essentially ... just a simple way of using a tree splitter library to essentially parse through your code base, chunk it, embed it using the Voyage code model and then upload it to Turbo Puffer.

There is an open source library for the V1 version and then soon the V2 version will be open sourced.

it will call the T Puffer tool. Now it's called T Puffer search password reset token generator. This is in the Django repo. ... you can see that it gets the contents and then is able to kind of give the full explanation to the user.

Context Bench — 정답이 아니라 찾아가는 과정을 채점한다

the benchmark is not really testing whether or not the coding agent solved or didn't solve a problem. It tests when it in the process of solving the problem, did it find certain files? Did it find certain lines and ... certain symbols?

it's a human labeled data set of like in order to complete this task ... the agent should have looked at this file, these few lines and these few symbols in order to like actually complete the task well.

I tested with three conditions. Essentially raw cloud code out of the box. Then I tested it with cloud code with a max of 50 line reads at a time and then the same thing with a windowed reads with the T Puffer search tool.

why this like 50 line read thing limit? It's because it ... became really noisy really fast if it's just reading like long files. ... if it just reads a whole like a thousand line file, it doesn't really make a lot of difference in the numbers.

Precision 결과 — 세 번에 한 번 낭비가 여덟 번에 한 번으로

You can see the baseline it hit like 65% precision, 33% on line precision and 43% on symbol and it kind of goes up as we add windowed grep and then windowed grep plus semantic search got it to like 87% file precision for example.

cloud code by default like it's a really exploratory. It loves to read as much as it can and like try to read everything

cloud code one in every three file reads is actually just a completely wasted file and with windowed grep it was one in five reads was a irrelevant file and then with semantic search there's only one in eight files was a you know, quote unquote wasted or irrelevant file

Recall의 반전 — 모든 파일을 탐험하는 Claude Code가 file recall은 이긴다

Recall is essentially how many of the needed files did it find?

By default again, cloud code actually does win the file recall and again, part of this is because it just loves to explore every single file it can.

Then we have line recall where it drops a lot more. ... even though it did explore a lot, it also kind of explored the wrong things a lot

태스크가 도구를 고른다 — keyword 없는 인접 파일은 semantic, import 추적은 grep

if we break it down into where semantic search won you know, quote unquote won and where just like windowed grep won, we see some stark differences ... certain tasks requires different types of tools.

when semantic search won, it was really good at finding a lot of behavior adjacent files that didn't have the same kind of keywords.

grep won when ... the task was a lot of it like tracing through imports and if it ... was able to find the keywords in like the first or second tool call, it was able to like just keyword search through that and find the relevant files.

Cursor만큼 안 나온 이유 — 밖에서 붙인 도구와 학습에 들어간 도구의 차이

these numbers weren't as great as Cursor's because part of ... Claude code is it's built for just grepping. Like that's ... what Anthropic kind of focuses on.

we kind of like add as an extra tool and it's like, "Hey, like here's this cool tool. You probably should use it sometimes." But it's very hard for it to have a true understanding of when to use it, why to use it ... versus like for example Cursor's composer they understand this is a built-in tool that ... it knows when and how to use it and that's why they saw it's like 23 and 1/2 ... percent ... performance gain.

장기 승자 — billion 컨텍스트를 right million으로 줄이는 쪽

we think long-term winners were like kind of provide these lightweight tools to find the right context in various different ways. ... you can't just like grep through everything unfortunately in a file system.

the people that provide these like easy tools ... to shrink down these billion context windows into the right million ... will win in the long term.

---

Q&A — 현장 질의응답

Semantic search의 정의 — Voyage code 3 임베딩과 vector search

this was just ... doing vector search. It was just performing vector search using embedding it using Voyage's code model and then just embedding the ... query sentences or query tokens and just sending them back to ... Turbo puffer.

we are the vector database that powers companies like Cursor, Anthropic, Notion. So when you use something like Cursor ... you have by proxy used Turbo puffer. ... you may know us just not by name, I guess.

언제 잘 먹히나 — 주석이 곧 의미다

I think it works best when there's a lot of like comments on code ... because it kind of finds that semantic meaning.

those with like really good comments, for example, just like comments above the function, it's able to like really understand a lot more cuz you ... kind of give this context to the model and the embedding model.

the embedding it ... is not the hard part. It's like figuring out what meaning really is of that chunk.

Parent-child와 가짜 주석 — 코드를 사람의 질문으로 번역하는 법

In ... this case it was just simple ... just the code.

Cursor has their own embedding model, which I think kind of helps with this of like how do you translate code into more of like a human level query.

I think they actually do what you said like they create fake comments on top of ... your code and then embed the code with the comments.

Vector DB가 필요해지는 순간 — zero cost grep의 바깥, multiplayer와 multimodal

people love grepping because it's zero cost. Like if you're able to like download everything to your local file system and just like grep through it, like yeah, that works.

I think vector DBs are built for actually like multiplayer and like this ... complicated relationships between lots of data. For example, like a knowledge base like a Notion, like you can imagine kind of hard to like really grep through that really easily on your local machine.

You can't really grep through a video file. You can't grep through an audio file. You can't grep through an ... image file. Like maybe you can glob on the ... file name, ... but like get a true understanding of kind of multimodal data