에이전트가 직접 다루는 AI 인프라

생각 덩어리

쿠버네티스는 버스트 워크로드가 아니다 — 새 런타임의 출발점

It’s because you have to run them on Kubernetes. Kubernetes is hard to manage. It’s not built for burstiness and, custom images,

It has a terrible developer experience.

ChatGPT보다 1년 먼저 붙인 GPU — 범용 프리미티브라는 베팅

...this is a primitive that we can build a whole collection of products on, which are very verticalized... But we added GPUs to the product a year before ChatGPT came out.

We just didn’t think it would be that big of a deal.

compute-heavy라는 워크로드 형태 — 쿠버네티스가 못 맞추는 것

...a lot of the workloads that we were seeing were very. I wish there was a better word for it, but compute-heavy. Like, they need, one, like, need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for, like, slow scaling and, more for, like, web server use cases.

데코레이터에 인프라를 얹다 — YAML을 코드로 압축

...we really didn’t want people to spend, so much time, writing YAML, and it seemed like you could really condense the surface area of what you’re doing, put it in code so you can operate on it just like you operate on other code, and like build stuff that’s more expressive and dynamic.

...the nice thing about Modal is you can bring whatever code you have, and sure, the DSL is at the configuration layer for, what hardware you’re using, how you’re scaling things up, but you still own the code.

Developer Experience → Agent Experience — 같은 이점이 그대로 전이된다

We’ve changed our SDK team to think about agent experience instead of, developer experience and we think that the same benefits that apply for DX also apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that’s not even typed when it can make a couple of changes in a decorator and it gets this self-provisioning runtime of, being able to see its changes live in action?

아무도 코드를 안 본다 — 그래서 observability가 코드보다 중요하다

Yeah, people aren’t looking at code. one thing we still see is really important is observability.

...we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what’s going on and, make judgment calls and whatnot. and that’s I feel like, Maybe more important now than looking at the code itself.

Modal은 무엇을 위한 것인가 — AI 애플리케이션용 프리미티브

Modal is a cloud platform that’s built for, where we’ve built the primitives from scratch for AI applications. and right now it covers, inference, training, batch processing, and sandbox workloads.

...the differentiator for us is the, are the workloads that need specialized compute, need to scale up and down a lot. yeah, they’re, they’re, they’re just shaped differently.

2023년 5월의 샌드박스 — 스스로 도는 에이전트

Like, we built sandboxes in May of twenty-three before anyone was even knew this was gonna be a thing. And the first example we published was, we took smol developer

And put it in a loop, so the agent can iterate on itself.

...the models would just diverge after like ten iterations and not produce anything meaningful.

Elastic inference — 커스텀 모델과 지역별 오토스케일

our biggest use case is, elastic inference. And the thing we first found product market fit, with was inference for custom models. So we stayed away from the LLM space, and we were serving companies like Suno for audio, Runway for video, robotics, comp bio companies that train their own model elsewhere.

...all these companies deploy, lots of different models in different regions, and so the autoscaling problem becomes even harder because then you have to scale within a certain region, and those cycles are offset.

GPU 스냅샷 — 콜드스타트를 줄이는 상태 저장

...we’ve incorporated GPU snapshotting into the product so we can take the GPU state, like your torch.compile model, snapshot it, and the next cold start is way faster.

RL 롤아웃 = 10만 개 샌드박스 — 진짜 버스트는 여기서 나온다

I agree that agents are not that bursty. sandboxes are, except when you’re doing RL.

Like when you’re doing, rollouts, you sometimes need a hundred thousand sandboxes in your sandboxes.

Speculative decoding — accept length이 곱셈으로 앞선다

Speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, verify all of this, all the tokens are predicted.

...improving kernel will only give you like few percentage points of improvement, and, increasing accept length, literally is a multiplicative decrease

So there’s no drop in quality performance

DeFlash 오픈소스 — 프론티어급 성능을 모두에게

Recently, we, shared our work on DeFlash, which is a block-based, speculator, and we’ve open sourced, all of it. So, you can - By using open source DeFlash, you can get the same performance as you would with one of the proprietary providers.

Auto Endpoints — 코드를 안 건드려도 최적화된 추론

...sometimes people don’t wanna touch the code, and they wanna get started with an endpoint that works and has all the great performance and, scalability that Modal has... we give you the code, you can go run it yourself, and if you want, you can eject out into the full Modal experience... It’s it’s not a black box.

원 GPU 대비 델타 — 진짜 scale-to-zero와 tail latency

...with these endpoints, we are way more elastic, as you said, than, anyone else, and you have true scaling to zero. you have true, burstiness, and in practice, that matters a lot more to people than just finding, the GPU and, running Modal code on something.

...running production-grade inference is a hard infer problem.

...controlling things like tail latency and, making sure every, request is delivered at least once and whatnot.

17개 클라우드의 캐패시티 풀 — 데이터센터 없는 슈퍼클라우드

...we’ve built this capacity pool that spans, 17 cloud providers, so we’re, we’re very good at Running on various kinds of cloud capacity across the world

We don’t have our own data centers. We just run across a lot of neo clouds

...building our own reliability layer on top. so if the GPU falls off the bus or something happens, we user workloads are not affected

사설 IPv6 오버레이(I6PN) — 분산 학습을 위해 만든 프리미티브

But we have this thing called I6PN, which we haven’t talked about, which is this, like, overlay network using IPv6 addresses. so if Modal containers, within the same workspace, when this is enabled, can address each other using this private IPv6 address, and no one else can.

We built it because we needed it as a primitive for our distributed training product. so we have this other feature, which is you can add a decorator to a function, and you get a cluster of GPUs. and they have RDMA networking. so you can run a distributed training job, that’s truly serverless.

RDMA는 시스템 문제 — 메모리를 얼마나 잘 옮기나

Like, the way you move around your KV cache and how efficiently you can do it, how efficiently you move, your weights from your training GPUs to your inference GPUs in RL is there’s a lot of degrees of freedom, and it is a systems problem

When you run multi-node training on Modal, RDMA... it’s a way to bypass the TCP networking stack and, transfer, stuff much faster, between one node, to the other. And we have I think like 3 terabit per second, internal networking

Auto research — 모델 직관이 이끄는 하이퍼파라미터 스윕

...we have this one internal repo called auto inference, which essentially we’ve automated our own forward-deployed engineering efforts using, this harness, which is, the agent will just spin up a sweep of different things. It’ll even run like, NVIDIA inside profiler and it’ll like tweak configs and it’ll arrive the right thing.

...it’s it’s a hyperparameter sweep that’s guided by some model intuition, so it’s like much more efficient than, whatever other, sweep you would have.

Compute strategy — GPU 예약을 베팅하는 팀

There’s a lot of interesting financial questions of like what is the blend between one year and three-year reservations? how do we forecast our own capacity?... you also have to have an opinion on how the supply chain is gonna evolve, and then you have to like, take bets

하드 가드레일 — LLM 매개 권한에 회의적

Yeah, I’m, I’m skeptical of LLM media permission for stuff that is at the sandbox level because you do want hard boundaries.

Otherwise, someone can exfiltrate stuff.

...I think you always need hard guardrails when you want, And you can pair those with softer guardrails, right?

매니지드 에이전트 위의 전문 샌드박스 — 프로덕션 에이전트의 요구

...the manage agent thing is a great place to start if you’re starting out building an agent and, But then when you get to, building something more production grade... You need a lot more control over, your compute primitive... When you get to that point, you kinda want, a specialized sandbox provider, that gives you those things, and that’s the role that we are trying to play.

Modal Bench — AX를 벤치마크한다

...it’s the main shift for us has been, as I said, like, we built this, benchmark, Modal Bench, to see where agents are lacking

Literally add surface areas to a product if they’re reaching for something, like maybe this should just be a CLI.