Same Hardware, Five Times the Load

The first thing people reach for is a bigger box. The transaction rate climbs, the dashboards go red somewhere, and the instinct is to throw money at the machine. We didn't. kwil-db, the open-source distributed database behind TRUF.NETWORK, went from 100K to 500K+ transactions per day on the same commodity hardware. Five times the load, identical machines. None of it was clever. That's the part worth writing down.

Before I touched anything, I measured. This sounds obvious and almost nobody does it. The temptation is to read the code, form a theory, and start optimising the part that looks slow. The part that looks slow is almost never the part that is slow. So the first move was to instrument the path a transaction actually takes (through consensus, through execution, down into PostgreSQL) and put numbers on every hop. Not vibes. Numbers. Where does a request spend its time, and how much of that time is real work versus waiting?

When you do this honestly, the bottleneck is rarely where your pride wants it to be. I assumed the database was the wall. It wasn't, at least not first. The early ceiling was in how we moved work to the database at all: lots of small round-trips, each one cheap, all of them adding up. The machine wasn't out of capacity. It was out of patience, spending its time on coordination instead of computation. You can't fix that by buying more cores. More cores just wait faster.

You don't get to pick the bottleneck; you get to find it. And the only way to find it is to make it observable and then stand in front of the evidence even when the evidence is boring.

Batching and backpressure do the heavy lifting

The two changes that bought the most headroom were the least exciting things I shipped that year. The first was batching. When you have a stream of small operations hitting a database, the per-operation overhead (the round-trip, the parse, the bookkeeping) starts to dominate the actual work. Group them. Amortise the fixed cost across many items instead of paying it once per item. Say a thousand transactions arrive in a window: handling them as one batch instead of a thousand conversations changes the shape of the load entirely. The work didn't get smarter. It got grouped.

The second was backpressure. A database under sustained load fails in an ugly way if you let callers shove work in faster than it can drain. Queues grow, memory grows, latency grows, and then everything falls over at once, at the worst possible time. Backpressure is just the system being honest about its own limits: when the pipeline is full, you slow the intake instead of pretending you can absorb infinity. It feels like you're doing less. You're actually staying alive at load you'd otherwise collapse under.

Neither of these is a trick. They're old ideas, and that's exactly why I trust them. A few things I'd put on the boring-wins list:

• Batch the small stuff: amortise fixed per-operation cost instead of paying it on every item.
• Apply backpressure before the queue does it for you, violently.
• Cut round-trips before you cut algorithms. Coordination is often the real tax.
• Touch the hot path; ignore the cold one no matter how ugly it looks.

The cold-path point matters. There's always some gnarly piece of code that offends you aesthetically, and the urge to clean it up is strong. But if it runs once a minute, fixing it changes nothing. Discipline here means leaving ugly code alone because the profile says it doesn't matter. The work I'm proudest of from that period is the optimisations I didn't do.

A benchmark ends the argument

Every team has the same recurring fight: someone is sure the slowness is X, someone else is sure it's Y, and the discussion runs on conviction. Conviction is cheap. The thing that ended these arguments for us wasn't seniority or volume. It was a repeatable benchmark: a load you could run before and after a change and read the difference off the same scale every time. When you have that, the conversation stops being about who's most confident and starts being about what the number did.

This is also why I distrust the bigger-box answer. Scaling up the hardware is the move that looks like progress while teaching you nothing. The bottleneck is still there; you've just rented enough capacity to hide it for a quarter. Then the load grows again and you're back, paying more, having learned nothing in between. Doing the boring work first means that when you do eventually buy more machine, you're buying it for a system that already uses what it has well. I'd rather ship the boring solution with a benchmark than the clever one without.

Five times the load on the same hardware reads like a feat. It wasn't. It was measuring honestly, fixing the thing the profile pointed at, batching the small stuff, and refusing to let the queue grow unbounded. The unglamorous version of performance work is the version that holds. Buying a bigger box is the answer you give when you haven't looked.

Measure first. The bottleneck you can prove beats the one you're sure about, every time.