매시간 스스로 개선되는 Lovable — 갇힘을 학습하는 Stack Overflow 루프와 에이전트 vent 도구

생각 덩어리

위성·입자물리·핵융합에서 온 물리학자 — 성배는 continuous learning at scale

I used to work with satellites, particle physics, and fusion reactors. So, I have a physics background. And today, I'm working at Lovable and working towards what is maybe the holy grail of AI engineering right now, which is continuous learning at scale.

So, you've probably all experienced working with an agent and having the feeling, "Why do I have to explain the same thing over and over again?" This is what we want to avoid. We want to have a mistake happen once and then never again.

vibe coding — 코드는 늘 거추장스러운 중간 레이어였다

I think we coined the term vibe coding. So, coding without actually looking at code.

The code was always just an annoying technical layer in between to create what we wanted. But ideally, you just say what you want, you see it, you test it, and you ship it.

We have some nice benefits with the Lovable platform, which is that people stick to one project for quite a long time. This chat can go on forever, essentially. ... We can actually get to know the user in quite detail.

Another exciting thing is that we're building for the 99% who can't code. ... we are trying to unlock software creation for everyone.

될 때까지 그냥 간다 — 하루 20만 프로젝트, 첫날 GitHub 밴

a reason that startups succeed with new technology is that they are a bit naive, and they don't really think about all the issues that can happen. They don't think too much about all the details in implementation. They just go until it works.

This is how I like to see our users. They just go until it works, and they're not sort of held back by old paradigms.

I joined Lovable around a year ago. At that time, we had a few thousand users. Now, we're creating over 200,000 projects per day. That's a significant percentage of all internet websites being created on Lovable.

My first day, GitHub banned us because we were creating too many ripples. And we've taken down a number of cloud providers along the way.

논테크니컬은 빨간 막힘에서 걸어나간다 — 다시 돌아오지 않는다

Technical personas generally have this amazing moment with AI where they are just building and building. You're accelerating 10 times, 100 times faster than you used to. But sometimes we hit some friction points, this yellow part, and you have to intervene ... And sometimes you even get stuck.

And non-technical persona is not really like this. They might prompt their way past the first friction, but as soon as this technical block happens, they generally walk away and they give up and they actually still never experience successful AI.

This is what we need to minimize. We can never get stuck to a point where a non-technical user cannot get past it.

But for the past year, we have been working on making sure none of these red bars happen. People can never get stuck.

'갇힘'의 정의와 분류 — 같은 요청 두 번이면 신호다

one clear signal of someone being stuck is that they ask for the same thing more than once. ... Maybe they're complaining about how something was implemented or that it failed very explicitly or they gave up on a session that they would otherwise have continued.

And we can have a LLM judge that is just looking at sessions and trying to look for these situations and they can flag it and say, "Hey, I think this user is stuck."

You can be stuck in a way where it's possible to solve. ... And you can get past it with the right prompting. ... And then there are tasks that are not solvable even if you say all the right things. Maybe we're just not supporting it for some reason.

There's no excuse to not improve on these things. If it is solvable, then it should work for everyone. And if it is just easy to do, we should just ship it.

Lovable판 Stack Overflow — 갇혔다 풀린 세션이 최고의 신호

We had an idea to build a Stack Overflow for Lovable. So, we basically learn from whenever someone is stuck and has an issue, we try to figure out the solution and give that to the agent.

The agent probably replies something like, "I fixed it. Maybe I quantized the animations to make it more performant." And it was lying. The agent has failed and it actually made things worse.

And eventually the agent might find the true solution. Which in this case happened to be that overlay text had individual gradients which made the animation super low performing. ... This we can flag and now we have a high signal sample of a problem that was high friction or not solved that was solved.

And the question we just ask is what should we have injected at the start of this query to jump straight into the solution so the next user does not experience this friction.

지식 항목의 생애 — 클러스터링, 에이전트 eval, 빈 주입 A/B

We don't want like a million ... Stack Overflow pages that are all talking about if you get this exact prompt then you should do this exact thing. It's not very helpful. We do some clustering.

And then we have an external reviewer, generally an agent and then maybe in some cases we have a human if we are uncertain, but in most cases it's actually just an agent that generates and runs a quick eval on this and sees if this did ... resolve the specific examples that we had in the set.

From that we get a full bank of ... Stack Overflow problems and solutions that's being continually updated. Whenever the model is working, we have a lightweight model that tries to inject that context when needed. If it detects there is an issue and there is an answer, it injects that context into the main agent.

And sometimes it detects that I should inject this, but we inject a blank. So, we don't actually send anything for a small sample of use cases. And this allows us to with very high signal review whether this solution was actually useful in production. ... It was more successful, then we show it more. And if it was less successful, we should show it less.

지식은 매시간 썩는다 — 리밸런싱과 폐기가 루프의 심장

And this loop is incredibly important. I cannot emphasize enough how important this step is. Because things are moving around this set of knowledge all the time. It gets stale whenever a new model is released. It gets stale whenever we change features. It gets stale incredibly quickly. So, very often, we have to rebalance this, but we also have to throw away a lot of this context.

And this allows us to really be at the frontier of what is solvable right now, but not have a lot of old deprecated knowledge that is giving context rot, and also in many cases just hampering the actual agent.

작동의 증거 — 갇힘 메시지 감소·deploy 증가·내부 모델 랭킹

the number of messages with fixing tent or people being stuck is dropping significantly. And we have also significant number of people that deploy more. This is one of our key metrics. That they actually finish a project. This is a very very strong signal that it has worked all the way through.

we also do internal ranking with this. So, it's really interesting to see how the models perform on this set of problems that we have collected. And with the Stack Overflow information, all models in the top of our ranking use this information. ... but it really really makes a significant boost in our internal ratings.

에이전트에게 vent 도구를 — 상사에게 불평하듯, Slack에 분풀이하듯

Well, if you were given a task and you just don't have the tools to do it, you would probably complain to your boss or go venting in Slack. So, we thought, why not do exactly that, but for the agent?

So, we basically are asking Lovable, "How are you doing? Can we help you in any way?" And we gave it an outlet to let out its frustrations.

What if we could let the Lovable agent give direct feedback to its creators? This sounds very scary, I have to say, and it sounds like an absolute insane idea. ... what's even more crazy is that it's kind of working.

매번 답을 강요하면 노이즈에 과적합한다 — 정말 답답할 때만 말하게 하라

you might have an external reviewer looking at the conversation and asking, "What could have been done better to reduce friction?" One problem with that is that you get a pretty low signal-to-noise ratio. Which you're basically forcing an answer on every iteration. And the reality is that most iterations actually just work pretty well. And you will overfit to noise.

it's prompted to only send feedback if it is really frustrated. And you can tune that balance until you get a lot of signal.

vent 도구의 명세 — 유저는 원인을 모르지만 에이전트는 안다

So, we gave our agent a vent tool. A way to complain to its creators.

It's a vent send feedback tool. You should use this when tooling docs or platform behavior material is low so it degrades your work. For example, missing or unsuitable tools, unclear tool names, parameters or schemas that are not matching what you were expecting, confusing or conflicting docs or instructions, broken or unexpected platform behavior, repeated failed attempts ... caused by environment limitations.

And our users generally don't know what the cause of a problem is. ... But the agent actually has a lot more context. It has literally been working on the issue, sometimes several turns. And it generally has a lot of context on this problem.

casting gymnastics와 non-breaking space — vent가 잡아낸 것들

It actually gets sent directly to our Slack. And the agent says something like, "I'm so annoyed at Frame Motion's TypeScript types for generating I think this is a cubic bezier. I just want to be able to send a list of four numbers. That should be fine. That's all I'm sending. I don't need all this casting gymnastics."

Another benefit of this setup is that it's very easy to understand as a human. It's very relatable that you're complaining about your workflow and that means that engineers sort of already have all this implicit context on how to alleviate the issue.

our copy tool was struggling with certain file names. ... we didn't even know that this tool was failing so regularly until the agent told us. ... And we got like 20 complaints about this in the first hour of launching this tool, which is crazy.

And we kept getting the reports. And then we realized that when you screenshot something in WhatsApp or on Mac, it enters a non-breaking space. Which we did not replace in our regex. And this kept complaining for various other special characters until we did solve it properly. And now this issue never happens again.

vent에서 PR까지 — 스파이크는 곧 인시던트다

Our head of product was very excited to see this. ... he was reading every single message. Now it's a bit more balanced and we actually have an agent that's monitoring removing duplicates and investigating and creating a PR for all these issues all the time.

And we're still at the point where devs are reviewing and then in many cases actually merging this to prod.

So, it turned out that this was actually a prime place to notice when our product was having an incident. And it actually gave a pretty good sense for what the problem was. It was complaining about right things in general.

인라인 지성의 값 — 닫히는 루프, 폰으로 받는 자동 PR 리뷰

you get the strong model intelligence versus an external reviewer. You generally don't want top frontier level intelligence to look through a lot of context. But if it's in line, it's very very cheap comparatively.

this was an example where the agent was actually giving meta feedback on the venting tool. It said it's too easy to send feedback and I can't pull it back. It was being ashamed of what it had sent to Slack.

it's actually the case now that I just get review requests on my phone from this automation to hey, review this PR that was completely automated. I look through it quickly and we can merge it.

And we're continuing to work to close this loop of detecting a shortcoming, merging a fix, and then continuously review and eval that.