Boring on Purpose

The first question I ask about any piece of a distributed database is not how fast it is. It is: when this goes wrong at 3am, and it will, can someone who is not me understand what happened and fix it without paging me in Surabaya twelve hours out of sync with the rest of the team? That question rules out a lot of clever designs before they get written.

I'm the lead maintainer of kwil-db, the open-source database behind TRUF.NETWORK. It's written in Go and built on PostgreSQL. Neither of those choices is interesting on its own, and that is the point. A distributed database earns the right to be clever in exactly one place (the consensus and replication path that must never lose data) and nowhere else. Everywhere else, boring is a feature you ship on purpose.

Go and PostgreSQL are boring in the way that load-bearing things are boring. Go gives me a single static binary. There is no runtime to provision, no interpreter version to match, no dependency tree to reconcile on the host. You copy one file and run it. When a contributor or an operator wants to deploy a node, the deploy story fits in a sentence, and a deploy story that fits in a sentence is a deploy story that doesn't fail in novel ways. PostgreSQL is boring in a deeper sense: it's a database people already understand. Operators have run it for decades. They know how it behaves under pressure, how to read its logs, how to back it up, how it fails. Building on top of something the whole industry already has a mental model for means I inherit all of that operational knowledge for free. I'm not asking anyone to learn a new storage engine to trust the system with their data.

The hidden cost of cleverness is that it spends a budget you can't see until you're overdrawn. A clever data structure saves a few milliseconds and costs you a debugging session that no one can reason about at 3am. A bespoke storage format performs beautifully in the benchmark and then corrupts in a way no existing tool can inspect. In a stateless service you can afford that trade, because the worst case is a restart. In a database you cannot, because the worst case is silent data loss, and silent is the operative word. The system that loses your data and tells you is recoverable. The clever one that loses it quietly is the one that ends careers.

So my design constraint isn't elegance. It's predictability: the property that the system does the same thing under load that it does when idle, that its failure modes are the ones you already drew on the whiteboard, that the answer to "what happens if…" is never "we'd have to check."

Observability is the feature, not the dashboard

I treat observability as a first-class feature of the database, on the same shelf as durability. Not the dashboard. Anyone can bolt on a dashboard. I mean the deeper thing: every important state transition is something you can name, log, and reason about after the fact. When a node falls behind, I want the system to be able to tell you it has fallen behind, by how much, and why, before you've noticed in production. The goal is that the database explains itself.

This is also where my actual leverage as a maintainer comes from, and it's unglamorous. Across kwil-db I've shipped 27 releases, written over 1,100 commits, and reviewed pull requests from more than 30 external contributors. The reviews are where predictability is actually defended. Code review is the highest-leverage mentoring there is, and most of what I'm reviewing for is not correctness in the narrow sense. The tests usually catch that. I'm reviewing for the thing tests don't catch: is this change observable, is its failure mode legible, will the person who inherits this at 3am understand it. A clever PR that passes every test and can't be reasoned about under failure is a PR I'll push back on.

Some things I will not let cleverness near:

• The write path that has to be durable. It stays as close to PostgreSQL's own guarantees as possible, because those guarantees have been beaten on for thirty years and mine have not.
• The deploy surface. One binary, one understood database. Every extra moving part is a new way to be paged.
• Anything whose failure is silent. If a component can fail quietly, I'd rather it fail loudly and slightly less efficiently.

Fast, cheap, and honest

I hold the system to three things, in this order: fast, cheap, and honest. Honest is the one people skip, and it's the one that matters most. Honest means the system tells you the truth about its own state: about how far behind a follower is, about whether a write is actually durable, about what it's doing when it's slow. A fast, cheap system that lies to you about its state is worse than a slow one that doesn't, because you'll trust it right up until the moment it costs you everything.

The cheap part isn't an afterthought either, and it's where boring quietly pays off. The network grew from 100K to over 500K transactions per day on the same commodity hardware: no bigger machines, no exotic infrastructure. That headroom didn't come from one clever trick. It came from a hundred boring decisions that left the system predictable enough that we could find the real bottlenecks instead of fighting our own architecture. You can't optimize a system you can't reason about. Predictability is what makes performance work possible in the first place.

None of this is the kind of work that demos well. There's no screenshot for "this failure mode is legible" or "this deploy is one file." But the work I'm proudest of is unglamorous, and a database is exactly the place where unglamorous is the entire job. People are trusting it with the one thing they can't get back.

I'd rather ship the boring solution with a benchmark than the clever one without. Make it predictable first. Clever can wait until you've earned it. Most of the time, you'll find you didn't need it.