RaftKV

What it is

RaftKV is a small key-value store that stays consistent across a cluster of machines, even when some of them crash or the network loses messages. It implements the Raft consensus algorithm without a Raft library, in about 2,000 lines of Java: leader election, log replication, persistence and crash recovery, snapshots, linearizable reads with ReadIndex, and exactly-once client writes.

How it's tested

The core node only touches the network, the disk and the clock through three interfaces. In production those are gRPC, files and the system clock. In tests they are a simulated network, in-memory disks and a fake clock, so a whole 5-node cluster runs in one thread and every run can be replayed from a seed.

The randomized simulation runs 1,000 seeds with network partitions, up to 30% message loss, reordering and node restarts. After every event it checks the safety rules: at most one leader per term, no node ever applies a different command at the same index, and every acknowledged write is still readable at the end.

Testing the tests

To see whether the simulation actually catches bugs, I broke the code on purpose. Removing the election restriction fails on the very first seed. Removing the rule that a leader only commits entries from its own term was not caught in 1,000 random seeds, because it needs one exact interleaving to show up. So there is a separate test that replays Figure 8 of the Raft paper message by message, and it fails when that rule is removed.

Numbers, and what I'd change

On a 3-node Docker cluster on my laptop it handles about 152 writes per second, with a p50 of 105 ms and a p99 of 210 ms. When the leader is killed, the cluster accepts writes again after a median of 631 ms.

Throughput is limited by fsync. The leader syncs every write to disk separately, and one synced write inside Docker on that machine takes about 6.7 ms. Group commit, where several writes share one fsync, is the obvious next step.