Inside the 20x AI Cost Trap and Model Routing

William Palmer

You know, last week I was looking at telemetry from a customer who spent over two thousand dollars in a single afternoon... simply because their employees were asking frontier AI models to perform basic arithmetic. We are talking about running queries that literally add two numbers together or multiply two simple values, but firing them straight into massive, top tier frontier models.

William Palmer

Arvind Jain, the CEO of Glean, pointed this out recently. People are out there using the absolute most expensive intelligence on Earth as a glorified pocket calculator. And look, as a computer scientist, that just makes my skin crawl a little bit. It is the engineering equivalent of firing up an eight hundred horsepower twin turbo race engine... just to sit in line at a drive thru and idle at three miles per hour. It is pure compute waste.

William Palmer

And that waste is driving a absolute enterprise crisis right now. On paper, a new frontier model might cost two to four times as much per token as the previous generation. That sounds manageable until you look at real world user behavior. When you give knowledge workers access to autonomous reasoning tools, they do not just ask short questions anymore. They run long, multi step tasks that process thousands of tokens at a time. So that original four times price increase actually multiplies into a ten to twenty times jump in per user costs overnight. A company's monthly AI bill can go from fifty thousand dollars to a million dollars before anyone even realizes what happened.

William Palmer

Which brings us to model routing. Glean, which recently reached three hundred million dollars in annual recurring revenue and raised a hundred and fifty million dollars in Series F funding at a seven point two billion dollar valuation, has built its whole platform strategy around this exact problem. Tony Gentilcore, their engineering lead, noted that Glean averages around forty five cents per task compared to one dollar and eighty four cents on systems like Claude Cowork. That is a four times cost reduction achieved almost entirely through intelligent routing and prompt harness engineering.

William Palmer

And it is completely reshaping how big companies view open weight software. Just three or four months ago, there was this massive corporate resistance toward non US open weight models like Kimi K3 or Qwen 3.8 Max. Executives were hesitant, there was a real stigma around security and origin. But when CFOs saw their frontier API costs multiplying by twenty, that stigma practically evaporated overnight. Today, CIOs openly treat open weight models as mandatory infrastructure to keep their AI budgets from collapsing.

William Palmer

So how does dynamic model routing actually work under the hood without sacrificing response quality? Well, Glean uses an agentic search model called Waldo that sits right on top of the large language models. Think of Waldo as a context builder and gatekeeper. Before a query ever touches an expensive frontier LLM, Waldo breaks down the prompt, searches internal databases, checks user permissions, and gathers what Jain calls the raw materials for the job. And it does all of that harvesting without burning expensive LLM tokens.

William Palmer

This turns out to be huge because a smaller, cheaper open weight model loaded with pristine, precise internal context will routinely destroy a massive frontier model that is drowning in a mess of irrelevant documents. You do not need a multi billion parameter reasoning titan to summarize a status report if you have already clean gathered the exact three paragraphs that matter.

William Palmer

What gives Glean a massive advantage here is scale. They have company wide rollouts at places like Booking.com, and at Zillow they have eighty percent active adoption across seven thousand employees. That means Glean sits on an enormous stream of real world user interaction. They can see exactly when a user accepts an answer from a smaller model versus when that user feels unsatisfied and manually upgrades the query to a larger model. That live human feedback loop is absolute gold for training routing algorithms.

William Palmer

And behind the scenes, they run what I find to be the most fascinating piece of this whole stack: shadow evals. For a small sample of live enterprise traffic, Glean routes the user's query to one model, but simultaneously runs the exact same task in parallel on cheaper or alternative models in the background. Then, AI based judges evaluate the output quality side by side to see if the router made the right call. It is continuous, automated bench testing using real corporate workloads.

William Palmer

When you step back and look at the bigger picture, it raises a provocative question for the whole industry. If enterprise platforms act as meta harnesses that sit across ChatGPT, Claude, Gemini, and Grok, dynamically picking whichever model fits the split second cost profile... does that turn the world's greatest frontier AI labs into background commodity suppliers? If the end user only interacts with the router, the brand value shifts entirely to the harness. That is going to be the central battleground for software deployment over the next few years. Alright, that is the breakdown for today. Talk to you soon.