Getting Started

Tutorial: Building with CleaveDB

Go coordinator

To achieve horizontal read scalability without placing coordinating strain on Python or Rust runtime threads, CleaveDB ships with a dedicated Go Distributed Coordinator (coordinator.exe). It is an ultra-fast, high-concurrency microservice specialized in Scatter-Gather query dispatching and streaming K-Way result merging.

Coordinator architecture

The coordinator sits between client interfaces (such as cleaveshell) and physical storage shards:

CleaveQLExample · Scatter-Gather topology

Client / Shell ──▶ Go Coordinator (:8305) ──▶ Concurrent Goroutines ──▶ [Shard 1, Shard 2, Shard N] ──▶ K-Way Min-Heap Merge

Incoming queries are accepted over TCP port 8305, fanned out concurrently to all configured shard instances, and reassembled before being returned to the client.

Key architectural features

Pure Go (Zero CGO)

Statically compiled with CGO_ENABLED=0 into a single standalone 5 MB binary. It requires no external C/C++ runtimes or DLLs, preventing conflicts with Python, PyO3, or AVX SIMD libraries.

Goroutine Scatter-Gather

Dispatches queries to dozens of cluster shards in parallel using lightweight Go routines bounded by context cancellation deadlines (context.WithTimeout).

Deduplication by Global ID (gid)

When shards return document arrays, the coordinator deduplicates records matching the same global document identifier (gid) and aggregates total document counts.

O(N log K) Min-Heap Merge

Implements a priority queue min-heap (coordinator/src/merge.go) that streams pre-sorted partitions from K shards without loading entire datasets into memory.

Automatic boot integration

When you run cleavedb_server.py, it automatically looks for coordinator.exe and launches it in the background on port 8305 with all active shard addresses:

CleaveQLExample · Server auto-launch snippet

# Automatically boot the Go Coordinator
coord_paths = ["coordinator.exe", os.path.join("coordinator", "coordinator.exe")]
go_coord_path = next((p for p in coord_paths if os.path.exists(p)), None)
if go_coord_path:
    coord_port = args.port + 4
    subprocess.Popen([go_coord_path, f"-port={coord_port}", f"-shards={shards_arg}"])

You do not need to manually manage separate coordinator processes during standard multi-node deployments.

Manual / standalone execution

In custom production or containerized environments, you can run the coordinator as an independent microservice:

CleaveQLExample · Launch standalone coordinator

coordinator.exe -port=8305 -shards="127.0.0.1:8300,192.168.1.10:8300,192.168.1.11:8300" -timeout=5

The flags specify the listening port (default 8305), a comma-separated list of backend CleaveDB TCP shard addresses, and the per-query timeout in seconds.

Health checks and ping telemetry

Verify that the coordinator is active and inspect its known shard topology by sending a JSON ping:

CleaveQLExample · Ping payload

{"action": "ping"}

The coordinator responds with its current operational state:

CleaveQLExample · Coordinator health response

{
  "status": "ok",
  "role": "coordinator",
  "shards": [
    "127.0.0.1:8300",
    "192.168.1.10:8300",
    "192.168.1.11:8300"
  ],
  "version": "4.0.0"
}