Why I Rebuilt My LLM Stack in Rust: Where Python Agents Choke Under Continuous Streaming

Every LLM framework demo looks the same. You import LangChain, wire up an agent with a couple of tools, ask it a question, and it answers. It's five lines and it works. I wrote a lot of those five-line scripts, and for a long time I believed the story they told: this is easy now, the framework has solved the hard parts.

Then I tried to keep one of those agents running. Not answer-and-exit — running. A process that sits there for hours, taking a stream of events, looping through tool calls, streaming tokens back to users. That is where the demo stopped being the truth.

For a while I had a neat explanation for what went wrong: Python leaks memory, the garbage collector can't keep up, and you end up restarting processes every few hours and calling it hygiene. I was going to write that post. Then I measured it, and the measurements corrected me. The symptom was real. The cause wasn't what I thought, and where it actually breaks is more interesting than "Python leaks."

So this post is three things: what I got wrong, what the benchmark actually shows about where a Python agent stack chokes, and why I still rebuilt the LangChain/LangGraph programming model in Rust — including what that cost.

The workload nobody demos

Here is the thing about an LLM application that a quick script never shows you: it is not a function that runs once. It is a conversation that never really ends.

An agent doesn't answer in one shot. It calls the model, gets back a request to use a tool, runs the tool, feeds the result back to the model, and streams an answer — and then the next event arrives and it does it again, carrying everything that came before. The defining shape of this work is back-and-forth communication between the model and your application, sustained over a long period of time, with a lot of state held live the entire way.

The agent loop, running: every event triggers a streamed tool call, a tool run, and a streamed answer, and the thread's state grows with every turn.

To do this well, the application wants to keep a lot in memory: the running conversation, tool outputs, retrieved chunks, the graph's checkpoints. That's not a bug in how you wrote it; that's the workload. The better you make the agent, the more it remembers — and the more tokens are flowing through the process at once.

A fair test: LangGraph vs Rust, streaming, continuously

I expected to find Python leaking memory. A quick framework-free test said otherwise (plain Python stayed flat), so I built a fair one that does what my agents actually do.

Two agents, identical work, same model endpoint:

Python: LangChain's create_agent (LangGraph 1.2) with ChatOpenAI, an InMemorySaver checkpointer, one thread per session, streaming with stream_mode="messages" — the setup most LangChain agents ship with today.
Rust: the same loop in plain tokio + reqwest: stream the model call, assemble the streamed tool call, run the tool, stream the answer.
The model: a local OpenAI-compatible server that streams at a realistic pace — 300 ms to the first token, then 50 tokens a second. It always calls a tool first (with the arguments arriving in pieces), then streams a 150–400-token answer.
The workload: each session is a long-lived thread fed a new ~500-character event about half a second after its previous reply. Every token is processed as it arrives. Inputs, tool outputs, and the model's decisions are byte-identical across both languages.
The matrix: 1, 10 and 100 concurrent sessions, with history either kept (the framework default) or trimmed to the last ~12 messages. Five minutes per run, memory and CPU sampled from outside each process.

One honest caveat up front: the Rust side is not Cognis yet. Cognis's agent can't stream tool calls today — I found that gap while building this and it's tracked in cognis#44. So this is Python-framework vs Rust-runtime, and I'll re-run it against Cognis when that lands.

The code, raw results, and charts are all in this gist. It needs nothing but cargo and uv — no API keys — so please run it and tell me where I'm wrong.

Where Python actually chokes: concurrency

With one session, there is no difference a user would notice. Time to first token was 0.81 s in LangGraph and 0.76 s in Rust; throughput was identical, because the model's pace is the bottleneck. The only visible cost is memory: about 90 MB for the Python process against about 5 MB for Rust.

With ten sessions, the cost shows up inside the runtime, but not yet for users. The event loop started running late — 24 ms at p99 against 3 ms in Rust — garbage-collection pauses reached 96 ms, and the process sat at about a third of a core.

With a hundred sessions, the Python process choked:

it sat at 95% of one core, while Rust handled the same load at 14%;
its event loop ran 242 ms late at p99, and 569 ms at worst — Rust: 3 ms and 23 ms;
time to first token was 1.34 s at the median and 2.72 s at p99, against 0.76 s and 0.77 s in Rust;
it completed about 10% fewer turns in the same five minutes.

Measured: p99 by number of concurrent streaming sessions per process (framework-default history, 5 minutes each). Lower is better.

The reason is mundane, and it's the thing I should have looked at first. asyncio is excellent at overlapping waiting — and most of an agent's life is waiting on the model. What it can't overlap is CPU work in your own process, because one event loop runs one thing at a time. Every streamed token passes through the framework: message-chunk merging, callbacks, stream plumbing, checkpoint writes. In this benchmark that came to about 0.28 ms of CPU per streamed token in LangGraph, against about 0.04 ms in Rust — roughly 7×.

That ratio is the whole story. At around 3,400 tokens a second, a single Python process runs out of core, and when it does, every stream in that process stalls together. The standard Python answer is more worker processes — which works, and multiplies the memory bill by the number of workers. Which brings me back to memory.

Memory does climb — just not for the reason I thought

With a hundred sessions and the framework defaults, the LangGraph process went from about 200 MB to about 530 MB in five minutes (averaged over the last 30 seconds — it swings by hundreds of megabytes as the collector and allocator hand memory back), peaking at 829 MB. The trend was roughly +90 MB a minute. Not because Python leaks, but because InMemorySaver keeps every checkpoint of every thread: every step of every conversation, retained so you can resume or time-travel. Trimming the conversation to its last dozen messages slowed that to about 16 MB a minute. It didn't stop it, because the old checkpoints were still there.

And a big, busy heap is exactly where garbage collection hurts. Over those five minutes the collector ran 8,224 times, paused the process for 26 seconds in total, and its longest single pause was 358 ms — mid-stream, with a hundred users waiting on tokens.

Measured: RSS of each agent process during a 5-minute run with 100 concurrent streaming sessions (mock model at 50 tok/s). Sampled once a second from outside the process.

Rust isn't magic here, and I want to be precise about that. With the same "keep everything" history, the Rust process grew too — about 8 MB a minute, from 13 MB to 49 MB. Ownership doesn't stop you from appending to a Vec forever. What's different is how much each message costs, the fact that nothing ever stops the world to reclaim it, and that when you trim, the memory is released on the spot: with trimmed history, Rust sat at 12–15 MB for the whole run with a hundred sessions.

So the honest version of my original complaint isn't "Python leaks." It's that a Python agent stack is heavy per message, keeps a lot by default, and pays for a big heap with collector pauses — and all three get worse precisely when you add concurrency. In production you'd swap InMemorySaver for a Postgres or Redis saver, which moves the checkpoint growth out of the process. It doesn't move the serialization cost out of the event loop.

Stringly-typed everything, and the failure that arrives after you've paid for it

The next problem is about when you find out something is wrong.

In the Python frameworks, the important shapes are dictionaries and strings. A tool's arguments are a dict. A message is a dict with a role string. Graph state is a TypedDict if you're disciplined, and a plain dict if you're in a hurry. None of it is checked until the code actually runs.

With an LLM in the loop, "until it runs" is a uniquely expensive place to discover a mistake. A typo in a state key or a tool that returns a slightly different shape than the next step expects doesn't fail at startup. It fails three tool-calls deep, after two model round-trips you already paid for in latency and tokens, in a branch that only executes for certain inputs. I spent real time chasing failures that a compiler would have refused to let me ship.

I wanted the schema of a tool, the variants of a message, and the transitions of a graph's state to be things the machine checks before I run anything — not hopeful conventions I keep in my head.

Where a shape mismatch surfaces: at runtime, three tool-calls deep and after tokens are already spent — versus at compile time, before anything runs.

Dependencies: not what I expected either

I went into this sure that the Python side would have far more dependencies. I measured that too, and I was wrong again. langchain plus langchain-openai installs 44 packages — LangChain 1.x already splits integrations into separate packages, so you no longer get every vector store for free. The Rust crate behind this benchmark — the agent and the mock server together — pulls in 94 crates. Rust's ecosystem is built from many small crates, and it shows.

What's different isn't the count. It's what you ship and when it can break. The Python agent ships as an interpreter plus 64 MB of site-packages, resolved at install time and checked at runtime. The Rust agent is resolved, compiled, and type-checked ahead of time into a single 2.5 MB binary. An environment that resolves on Monday and breaks on Friday isn't a failure mode a prebuilt binary has.

Measured: the Rust side pulls in more dependencies, not fewer. The difference is what ships and when it's resolved: one binary built and checked ahead of time, versus an environment assembled at install time.

Why Rust, and what Cognis actually is

None of this is a complaint about the ideas in LangChain and LangGraph. The ideas are good. Composable runnables, an agent loop, a stateful graph with checkpoints and time-travel — I wanted to keep all of that. What I wanted to change was the runtime underneath. So I rebuilt the programming model in Rust. The project is called Cognis, and here is what the benchmark says Rust actually buys:

More streams per process. Roughly 7× less CPU per streamed token means one process carries far more concurrent conversations before its runtime saturates — and tokio spreads them across every core instead of one.

No collector, and memory released where you can see it. Not "static" — keep everything and Rust grows too. But nothing ever pauses the process to reclaim memory, and a turn that leaves the window is freed right there. That's what kept the trimmed Rust run flat at a hundred sessions.

Types run end-to-end. The core abstraction is a typed Runnable<I, O>; tool schemas, message variants, and graph state transitions are all checked when I compile, not when a user hits the unlucky branch.

use std::sync::Arc;
use cognis::prelude::*;
use cognis::{AgentBuilder, Calculator};
use cognis_llm::Client;

#[tokio::main]
async fn main() -> Result<()> {
    let client = Client::from_env()?;

    let mut agent = AgentBuilder::new()
        .with_llm(client)
        .with_tool(Arc::new(Calculator::new()))
        .with_system_prompt("Use the calculator for any arithmetic. Always state the final answer.")
        .with_max_iterations(4)
        .build()?;

    let resp = agent.run(Message::human("What is 47 * 23?")).await?;
    println!("{}", resp.content);
    Ok(())
}

It's still five lines of intent. The difference is that the compiler has already read them.

The graph state is a real type with a real transition, not a dict I hope everyone spells the same way:

use cognis::prelude::*;

#[derive(Default, Clone, Debug)]
struct State { count: u32 }

#[derive(Default, Clone)]
struct Update { count: u32 }

impl GraphState for State {
    type Update = Update;
    fn apply(&mut self, u: Update) { self.count += u.count; }
}

You compile in only the integrations you use. Every provider, vector store, and checkpoint backend is behind a feature flag:

cognis = { version = "0.2", features = ["openai", "anthropic"] }

Streaming — with one gap I'm closing. Cognis's client streams tokens today, but its agent loop doesn't yet stream while it's calling tools, which is exactly the workload this post is about. That's cognis#44, and it's the next thing I'm building. When it lands, I'll re-run this benchmark against the real Cognis agent and publish the numbers, whichever way they go.

When to stay in Python, and when to move

The data doesn't say "Python bad." It says where the line is.

Stay in Python when you're prototyping, when each process handles a handful of concurrent conversations, when the model's speed dominates everything else, when the heavy lifting already happens in external services, or when you need a brand-new provider integration the day it launches. In this benchmark, at one session, you'd never know the difference.

Move to Rust — or at least a Rust core — when a single process has to carry dozens to hundreds of long-lived streams, when tail latency on streamed tokens matters, when memory per session is what drives your bill, when you need to embed agents in a native service, a CLI, or the edge, or when you're shipping a library other people will run in production.

What it cost, honestly

I don't want to pretend this was free, because it wasn't.

Rust asks more of you up front. The borrow checker has opinions, and the first few times you model an agent's shared, mutable, long-lived state you will argue with it and lose. That friction is real — the compiler is making you say out loud who owns what, which is the same discipline that lets memory be released the moment a turn falls out of the window. Worth it for me; not free.

The ecosystem is younger. Python has a library for every provider and every vector store the day they launch. In Rust you sometimes get there first yourself — and as cognis#44 shows, sometimes you find a gap in your own framework by trying to benchmark it.

And I want to be precise about what Cognis is and isn't: it is not a line-by-line port. It's the same mental model as LangChain and LangGraph — runnables, an agent loop, a Pregel-style stateful graph with checkpointers and interrupts — rebuilt to fit Rust's guarantees instead of fighting them. If you know those Python frameworks, you'll recognise the shape immediately.

Where this leaves me

I set out to prove that Python leaks memory. It doesn't — not in the way I claimed, and I'm glad I measured before I published. What the measurements showed is less dramatic and more useful: a Python agent stack spends about 7× more CPU per streamed token, keeps more per message, and pauses to collect a heap that grows with every conversation. At one session none of that matters. At a hundred concurrent streams in one process, it's the whole story.

That's the workload I care about — agents that run continuously, stream constantly, and can't be babysat — and it's why I rebuilt the stack in Rust. The benchmark is in the gist, and Cognis is at cognis.vasanth.xyz. If your numbers disagree with mine, I'd genuinely like to see them.