How Nubank Simulates AI and Kepler Stops Math Hallucinations

William Palmer

Imagine taking a corner at two hundred miles per hour. You do not just throw the car into the bend and hope for the best, right? You, uh, you rely on a simulation. You run the physics beforehand. Because if you do not, the cost of failure is, well, it is catastrophic. Now, try doing that with a customer service agent handling money for over one hundred and thirty five million people.

William Palmer

That is the exact high stakes arena where digital banking giant Nubank found themselves. Deploying customer facing AI in finance has traditionally been a complete compliance nightmare. I mean, one hallucinated promise about a credit limit or interest rate, and you are looking at massive regulatory fines, not to mention a complete breakdown of trust. So, for the longest time, banking AI was stuck in this slow lane of endless manual reviews.

William Palmer

But then, Aman, a Principal ML Engineer at Nubank, teamed up with Shreya Rajpal, the CEO of Snowglobe. And they decided to completely kill traditional live A B testing. Because, let us be honest, you cannot just test an unpredictable, non deterministic agent on live customers and hope it does not drive them off a cliff. Instead, they built this incredibly high fidelity offline simulation harness.

William Palmer

They created what they call synthetic personas. Picture this, the system generates thousands of highly grounded customer profiles. Like, uh, let us say, Maria trying to order a credit card. The simulation does not just give her a generic script. It generates her specific account history, her unique tone, her exact financial intent. And then, it unleashes the AI agent to interact with her in a fully sandboxed environment. When human experts compared these simulated conversations to real world traces, they found them to be eighty percent comparable. It is basically a flight simulator for financial bots.

William Palmer

And the results? Well, they are honestly mind blowing. Nubank managed to double its transactional NPS, that is their customer satisfaction score, for a core agent. And they cut their AI agent release cycle from weeks to under a day. Literally from weeks to under a day. But to really appreciate how they did this, we have to talk about the underlying computer science, which, um, is where things get really interesting.

William Palmer

In traditional machine learning, we dealt with flat, row based inputs and labels. It was simple. You feed in a row of data, you get a classification. But modern AI agents? They do not work like that. They operate in multi turn trajectories. It is a continuous loop of tool calls, API requests, state changes, and prompt adjustments. Tracing and evaluating these massive stateful feedback loops is incredibly complex. If the agent makes a wrong turn on step three of a ten step conversation, how do you catch that before it updates its model? That is the technical friction Nubank solved by shifting from static evaluation data to full, multi turn trajectory simulations.

William Palmer

But okay, what if the agent is not just chatting with Maria? What if it is doing deep financial analysis? What if it is calculating metrics that actual investors rely on? Well, then the stakes get even higher. And that brings us to Vinoo Ganesh, a former Palantir engineer who decided to study this exact problem. He surveyed one hundred and thirty seven financial firms, and he heard a resounding, universal answer. None of them trusted AI for serious financial research because of silent numerical hallucinations.

William Palmer

If an AI summarizes a paragraph slightly wrong, you might not notice. If it hallucinates a decimal point in an earnings report? You lose millions of dollars. So, Vinoo built Kepler with a very simple, almost radical design philosophy: do not let the AI do the math. Just, do not allow it.

William Palmer

Now, you might ask, wait, if the AI cannot do the math, then what is it actually doing? Well, Kepler completely decouples reasoning from arithmetic. They use Claude, Anthropic's model, strictly as a semantic translator. Claude's only job is to read a human's question, figure out what they are asking for, and map that question to a highly structured execution plan. The actual retrieval and math are handed off to deterministic, hard coded formula engines. The LLM handles the language; traditional, reliable code handles the numbers.

William Palmer

Let us walk through how this works in real life. Say you want to calculate a metric like inventory days outstanding over eight quarters. If you ask a generic chatbot, it might search the web, find some numbers, and perform the division itself, probably hallucinating somewhere along the way. In Kepler, Claude maps those variables, like cost of goods sold and average inventory, to a proprietary financial ontology. Then, Kepler's system pipes the structured formula directly into Excel. It builds the calculation deterministically and validates every single number back to the exact filing, page, and line item of the original SEC document with a single click.

William Palmer

But what I find most brilliant about this setup is how it handles ambiguity. In finance, terms are often overloaded. What one company calls operating revenue, another might categorize slightly differently. If Claude encounters an ambiguous definition, it does not guess. It is programmed to halt. It pauses the workflow and asks a human analyst for clarification. That self aware uncertainty, knowing exactly when to stop and raise its hand, is infinitely more valuable for financial risk management than any raw performance benchmark score. It is about building a system that is safe at any speed. Alright, that is it for today. Talk soon.