How Message Batching More Than Doubles Actor Model Throughput
If you’ve spent time optimizing actor models, you know the mailbox is the system’s heartbeat. I recently dug into a framework where draining an entire mailbox under a single lock — rather than popping one message at a time — combined with a message pool, took a two-actor pipeline from 5.5 to 13.6 million round-trips per second. That’s a 2.46× improvement.
Here’s the catch: this is entirely a throughput lever, not a latency one. And the one optimization that seemed the most obvious actually made things worse.
Let’s break down this case study in C++ and Rust.
Anatomy of the Bottleneck
In our baseline setup, each actor owns a blocking queue (a BQueue) that acts as its mailbox. The queue is a fixed-capacity ring that spills over into an unbounded deque if it fills up. It’s guarded by a single mutex and a condition variable, so idle actors sleep instead of spinning and burning CPU.
When we peeked under the hood of the original consumer loop, the per-message cost was hiding a lot of overhead:
while running {
let (env, _last) = cell.queue.pop(); // (1) lock the mailbox
let mut guard = cell.actor.lock(); // (2) lock the actor
guard.process_message(env.msg, &mut ctx); // dispatch + handler
}
For every single message, we were paying for:
A mailbox lock on
pop().A per-actor lock on dispatch.
A mailbox lock and
notify_one()from the sender pushing the message.A heap allocation (
new) and deallocation (delete).
The Fix: Batch Drain
Instead of this one-by-one dance, what if we grab everything at once? We updated the queue to pop the entire mailbox under one lock, take the actor lock once for the whole batch, and skip redundant wakeups.
pub fn pop_batch(&self, out: &mut Vec<T>) {
out.clear();
let mut g = self.inner.lock().unwrap();
loop {
if !g.ring.is_empty() || !g.overflow.is_empty() {
out.extend(g.ring.drain(..));
out.extend(g.overflow.drain(..)); // ring then overflow: FIFO
return;
}
g = self.cv.wait(g).unwrap();
}
}
On the sender side, we only signal the condition variable on the exact empty-to-non-empty transition to avoid redundant wakeups:
pub fn push(&self, x: T) {
let was_empty = { /* lock, note if empty, push */ };
if was_empty { self.cv.notify_one(); }
}
Now our consumer loop looks like this:
let mut batch = Vec::new();
while running {
cell.queue.pop_batch(&mut batch);
let mut guard = cell.actor.lock(); // one lock for the whole batch
for env in batch.drain(..) { guard.process_message(env.msg, &mut ctx); }
}
The Benchmark Results
To test this, we set up a ping-pong benchmark. Each side fires a burst of 10,000 messages before waiting, keeping both mailboxes deep (10,000,000 round-trips total per run).
Here’s how the numbers shook out in C++ (compiled with -O3 on Apple Silicon):
Rust matched the first step almost perfectly (5.56 → 9.00 M rt/s), which is a great sign that the mailbox architecture itself — not the language runtime — is dictating the performance.
Where the Speed Comes From
Batch Drain (1.63×): We eliminated two mailbox locks, two actor locks, and two notifies per round-trip. That saved us roughly 70 nanoseconds per trip.
Memory Pooling (another 1.50×): In the baseline, a message is allocated on one thread and freed on another. This cross-thread freeing absolutely destroys the standard allocator’s per-thread cache. By routing recycled blocks through a custom per-thread MemoryPool, the hot path never touches the global allocator.
One warning if you build your own pool: a naive single-global-mutex free-list is worse than no pool at all (we measured 0.5×), because that one lock, hit by both threads on every alloc and free, is worse than the allocator’s per-thread magazines. The magic is the thread-local cache, not the pool.
The “Obvious” Fix That Failed
We tried one more experiment: send_batch. If batching the consumer was good, why not have the sender build and push a whole burst of 10,000 messages under one lock?
It bombed. Throughput regressed from 13.60 down to 11.30 M rt/s.
Why? Because the sender was holding the lock while building all 10,000 messages, preventing the consumer from starting its work. It destroyed the producer/consumer pipelining. Fewer locks don’t matter if you ruin your concurrent overlap.
The Tale of Two Regimes: Latency vs. Throughput
You’ll often hear that async sends take microseconds (4–7 µs). Yet our batched benchmark runs in nanoseconds (74 ns). They’re both correct; they just measure opposite regimes dictated by queue depth.
The Latency Regime (empty mailbox): When arrivals are slow, the actor keeps up and goes to sleep. When a lone message finally arrives, you pay for two condition variable wakeups (~1–3 µs each). You have spare capacity, so throughput doesn’t matter. Batching can’t help here — there’s nothing to batch.
The Throughput Regime (backed-up mailbox): When arrivals outrun the actor, the mailbox fills up. The actor never sleeps, so wakeups drop to zero. Messages flow at tens of nanoseconds. Now drain rate is everything.
The beauty of the batch-drain approach is that it self-adapts. If the queue is 1-deep, it pops one message at essentially zero extra cost. If the queue is 10,000-deep, you get the 2.46× speedup. One code path perfectly handles both extremes.
This is also why batching isn’t the whole story: a bare single-producer/ single-consumer ring gets 20× from the same trick, because there the only cost is one cross-core cache-line bounce and amortizing it is transformational. The actor path spreads its cost across the lock, cross-thread allocation, dispatch, and the reply — so no single lever is 20×, because no single cost is 100% of the total.
Key Takeaways
Follow the locks. Batch drain works because removing lock overhead matters, but combining it with a thread-local memory pool is where the real magic happens.
Don’t ruin the overlap. Pipelining beats fewer locks. Don’t batch the sender if it starves the consumer’s ability to start working.
Queue depth dictates the metric. Batching is a throughput lever. It won’t speed up a lonely, isolated message in an empty queue.
Built into Kaspar
The framework I’ve been dissecting here is Kaspar, our high-frequency-trading actor system. Batch drain and the thread-local MemoryPool are how its mailboxes stay fast when the queues back up — and they’re a big part of why a saturated Kaspar pipeline moves messages in tens of nanoseconds instead of microseconds. The C++ and Rust implementations (BQueue, pop_batch, MemoryPool) live in the repo: github.com/vincent212/kaspar-hft (actors/cpp/ and actors/rust/).