Why Kubernetes Fails AI Agents

William Palmer

So... okay, imagine you're- you're building an AI agent, right? It's supposed to- to go out, write some Python, scrape a website, and run a terminal command. Sounds easy. But if you're- if you're thinking, "Oh, I'll just throw it in a Docker container on Kubernetes," you are... well, you're missing the whole picture. Seriously. It's- it's a massive security and architecture trap, and- and it's exactly what OpenAI had to solve for ChatGPT Work.

William Palmer

See, about a year or so ago, this- this engineer named Abhishek Bhardwaj, he- he was working on an open-source project called Arrakis. Yes, like the Dune planet. He was trying to figure out how to run these sandboxes safely. And- and Greg Brockman, co-founder of OpenAI, saw it and literally went, "Okay, we need this guy." Hired him to architect the cloud infra for ChatGPT Work. Because- because here's the thing... when an agent runs arbitrary code from the web, it's not just running in a sandbox. It- it shares the host OS kernel with other containers if you're just using standard Docker. If that agent gets tricked by a prompt injection, or- or it pulls down a malicious package... boom. Container breakout. It- it- it can escape right into your host system. It's an absolute security nightmare. I mean, you're basically handing a stranger the keys to your entire server room and- and hoping they don't look in the drawers.

William Palmer

But- but even if you ignore the security side, which... please don't do that. But if you do, Kubernetes still fails AI agents on a fundamental, state level. Think about it. An agent starts up, it spends five minutes downloading libraries, installing system packages, compiling some C++ code... and then it hits a tiny syntax error. Standard containers are- they're stateless. If it crashes, or- or if you need to backtrack because of an error, you have to- you have to spin up a whole new container from scratch. You lose everything. There's no native, instant way in Kubernetes to- to just roll back the RAM and the filesystem to where they were ten seconds ago without discarding all that work. It's- it's incredibly slow, and in the AI world, speed is... well, it's- it's the only thing that matters.

William Palmer

You know, everyone in the AI industry right now is- they're totally overtuned to compute. It's all "GPUs, GPUs, FLOPS, H100s, B200s." But they are completely ignoring the filesystem. The real bottleneck for agent execution isn't always the math—it's- it's storage, disk imaging, and network I/O. If you want to run these agents safely, you have to use MicroVMs. Technologies like AWS Firecracker. They boot in milliseconds, they use KVM for hardware-level isolation, so- so no shared kernels. Perfect, right?

William Palmer

Well, not quite. The problem is that when a MicroVM boots, it needs a disk image. A .img file. And if that image has all your development tools, it can easily be gigabytes. If you're pulling that from cloud storage, like- like AWS S3, every single time you boot... forget about it. The network latency alone kills you. It takes seconds, sometimes minutes, to transfer and mount. You- you can't have a snappy user experience if your agent has to wait for a network mount every time it wants to run a single line of bash.

William Palmer

This is where the engineering gets- gets really beautiful. To bypass that network delay, you have to use a combination of local NVMe caches, parallel S3 range requests—where you only grab the tiny chunk of the disk image you actually need right now—and Copy-on-Write, or CoW, memory overlays. So, instead of copying the whole disk, you mount a read-only base image, and you write any changes to a tiny, lightning-fast overlay layer. This lets you snapshot and- and restore the entire state of the machine—RAM and filesystems—in under a second. Sub-second backtracking! That means if an agent makes a mistake, the system can instantly rewind it to a known good state, like a- like a video game save state, enabling rapid reinforcement learning search.

William Palmer

It actually reminds me a lot of race-car engineering. I- I spend my weekends driving and- and working on track cars, and if you think about a Formula 1 pit stop, it's the exact same kind of mechanical orchestration. When the car pulls in, you don't rebuild the entire chassis. You don't swap out the engine or rebuild the suspension from scratch. That would be like rebooting a virtual machine or container from a cold start. No, the crew is ready. They hot-swap pre-cached wheels—bam, bam, bam—clean the vents, and send the car back out in under two seconds. That's what Copy-on-Write overlays do. You're keeping the base chassis intact, hot-swapping only the writeable filesystem layer that changed, and keeping the agent running at absolute peak speed. It's- it's pure mechanics, just done in silicon instead of steel. Alright, that's- that's a wrap on sandboxes. Talk soon.