Tutorial: Building with CleaveDB
Distributed Cluster — Overview
CleaveDB is designed from the ground up for high availability, zero-downtime fault tolerance, and horizontal query scaling. It combines a Python-native Raft consensus engine for replicated state machines with a standalone Go Distributed Coordinator for high-concurrency Scatter-Gather query orchestration.
Distributed Subsystems
1. Multi-Node Raft Consensus
Powered by pysyncobj in DistributedEngine. Appends writes to a distributed Write-Ahead Log (WAL), replicates across quorum, and elects leaders dynamically upon node failure with zero data loss.
2. Go Coordinator (coordinator.exe)
A standalone 5 MB pure Go binary (zero CGO) listening on port :8305. Dispatches concurrent Goroutines to query shards, deduplicates documents by global ID (gid), and merges streaming datasets.
3. CLUSTER Queries
Execute CLUSTER STATUS to inspect shard topology, or use CLUSTER SCOOP and CLUSTER FIND to fan out read queries across all active cluster nodes simultaneously.
4. SCATTER Writes
Broadcast or partition write commands across multiple cluster shards using SCATTER POUR and SCATTER CHANGE to scale storage capacity horizontally.
Port Topology
A standard multi-node CleaveDB cluster coordinates communication across several dedicated network ports:
| Service | Default Port | Protocol / Role |
|---|---|---|
| CleaveDB TCP | 8300 | Direct client connection & local query execution |
| CleaveDB WS | 8301 | WebSocket pub/sub events & real-time client feed |
| Go Coordinator | 8305 | High-concurrency Scatter-Gather orchestration & K-Way merge |
| Raft Consensus | 8311 | Internal peer-to-peer WAL replication & heartbeats |
Explore Cluster Guides
Select a guide below to master CleaveDB distributed topology and operations:
- Raft consensus: configure multi-node replication, quorum durability, and disaster failover.
- Go coordinator: architecture of
coordinator.exe, Goroutine scatter-gather, and Min-Heap merge. - CLUSTER queries: check cluster status and dispatch multi-shard reads via the coordinator.
- SCATTER writes: broadcast document creations and updates across the cluster topology.