Tutorial: Building with CleaveDB

Raft consensus

CleaveDB incorporates a Python-native implementation of the Raft Consensus Algorithm powered by pysyncobj. It transforms standalone CleaveDB instances into a high-availability, disaster-resilient distributed state machine with automated leader election and quorum-based write replication.

Starting a 3-node Raft cluster

To establish a high-availability cluster, launch multiple server instances specifying their client port, dedicated Raft consensus port, and list of peer addresses:

python cleavedb_server.py --port 8301 --raft-port 8311 --peers 127.0.0.1:8321,127.0.0.1:8331 --data node1

python cleavedb_server.py --port 8311 --raft-port 8321 --peers 127.0.0.1:8311,127.0.0.1:8331 --data node2

python cleavedb_server.py --port 8321 --raft-port 8331 --peers 127.0.0.1:8311,127.0.0.1:8321 --data node3

Upon launch, the nodes discover each other over their assigned --raft-port channels, initiate heartbeat exchanges, and hold an initial leader election.

Replicated write lifecycle

When a client issues a mutating write statement (such as POUR, CHANGE, or LINK):

1. Leader Interception: The elected Raft Leader intercepts the incoming command.
2. Log Append: The command is serialized and appended to the distributed Write-Ahead Log (WAL) maintained in journal.bin.
3. Quorum Consensus: The leader dispatches the entry to follower nodes. Once a majority quorum (e.g. 2 out of 3 nodes) acknowledges disk persistence, the entry is marked committed.
4. State Machine Application: The @replicated_sync decorated method executes the CleaveQL statement against the local Rust/C++ storage engine simultaneously across the cluster.

Zero-overhead read optimization

While write operations enforce quorum replication, read queries (such as FIND, SCOOP, and DISTILL) bypass the Raft consensus roundtrip entirely:

def execute_read(self, query: str, context: dict):
    # Reads don't need raft consensus — served directly from local engine
    interpreter = Interpreter(GLOBAL_DB, indexing_queue)
    return interpreter.execute(stmts)

Because every node maintains an up-to-date replicated state machine, any healthy node can serve analytical reads locally at full NVMe SSD and SIMD speeds without network overhead.

Automatic failover and disaster recovery

CleaveDB clusters are designed for unattended fault tolerance:

• Leader Crash: If the primary node experiences a crash or network partition, remaining followers detect missing heartbeats and automatically elect a new leader in milliseconds.
• Zero Data Loss: Committed transactions are guaranteed durable across the majority quorum.
• Node Rejoining: When a failed node recovers, it automatically replays missing log entries from dump.bin and synchronizes back to the cluster state.