HNSW, from scratch
Malkov & Yashunin's Hierarchical Navigable Small World graph, coarse-grained RW-locked for correctness and verified under go test -race.
Vector search engine · written from scratch in Go
Not a wrapper around Qdrant.
The thing Qdrant is made of.
HNSW indexing, product quantization, a crash-safe WAL, mmap snapshots, multi-tenancy, and a Raft-coordinated distributed cluster: implemented and tested directly, then benchmarked head-to-head against a real Qdrant instance. Including where NuclaDB loses.
quickstart
$ curl -fsSL .../install.sh | sh$ nucladb-cli quickstartspinning up a throwaway local server…inserted 3 vectors, searched top-3 in 0.4ms→ keep using it: NUCLADB_ADDR=127.0.0.1:53211
Scroll
Playground · Runs in your browser
loadingThis is the real engine, compiled to WebAssembly.
Not a mock. internal/index/hnsw is the exact package the server runs, compiled unmodified to WASM and executing right here, in this tab. Insert random vectors, then search, and see the real graph timing and recall against a brute-force check computed alongside it.
dim=32 · cosine · M=16 · efConstruction=1000 vectors indexed
Insert
Insert a batch to build the index, then search it.
The query vector and its ground truth are generated fresh on every search, so recall reflects this exact corpus, not a cached number. Read the WASM entry point.
Product · Architecture
Most vector databases you see on GitHub wrap an existing engine.
NuclaDB goes the other direction. It is the internals a production vector engine is built from, implemented and tested directly, so the interesting engineering is in this repo, not imported from one.
Product quantization
Optional lossy compression: k-means++ codebooks per subspace, asymmetric distance computation, so the query vector is never itself quantized.
Crash-safe WAL
Every write is fsync'd before it's acknowledged. A custom binary record format with CRC32 checksums makes replay torn-write-safe.
mmap-backed snapshots
Atomic write-to-temp, rename-into-place snapshots, loaded via mmap so a dataset larger than RAM pages in instead of failing to start.
Real multi-tenancy
Every tenant gets an isolated graph, WAL, and snapshot on disk, plus an independent storage quota and QPS rate limit enforced before the engine.
Raft-coordinated cluster
Consistent-hash sharding, scatter-gather search routing, and async WAL-stream replication with automatic health-checked failover.
Architecture
One request, four layers.
Every stage below is a real package in the repo, linked to its source.
1
Client
gRPC :9090 · REST :8080
REST is a hand-written JSON layer over gRPC, not grpc-gateway: not worth vendoring the full googleapis proto tree for five routes.
internal/api/grpc · internal/api/gateway
2
Routing
engine.Store
Tenant routing, storage quotas, and QPS limits are enforced here before a request reaches an engine, opened lazily per tenant.
internal/engine.Store
3
Write path
WAL → HNSW graph → PQ (optional)
Every write fsyncs to the WAL before ack, then applies to the in-memory HNSW graph, with optional PQ compression after.
internal/storage/wal · internal/index/hnsw · internal/index/pq
4
Durability
mmap-backed snapshot
Atomic write-to-temp, rename-into-place snapshots let a restart skip WAL replay, and let a dataset larger than RAM page in via the OS.
internal/storage/segment
Benchmarks · Real numbers
We measured our own weaknesses too.
A real, reproducible, committed head-to-head against an actual Qdrant binary: 10,000 base vectors, 100 queries, dim=128, SIFT-small, measured over each system’s real network API. Not synthetic, not an in-process shortcut.
Build time
~350×
slower to build 10K vectors than Qdrant (43.9s vs 124ms): fsync-per-write with no batching, the correct-but-slow default for a WAL that means it.Why, in depth →
Memory, at every ef
<½×
NuclaDB’s RSS stays under half of Qdrant’s across every efSearch value tested (45.2–45.6 MB vs a flat 102.7 MB).
Product quantization
57.7%
recall@10 at a fixed 16× memory reduction, the near-worst-case config (no re-ranking, no IVF), measured as a clean read of quantization error alone.Why, in depth →
Queries per second, by efSearch
ef 10
NuclaDB
7432.4
Qdrant
3661.1
ef 20
NuclaDB
6298.9
Qdrant
4522.6
ef 50
NuclaDB
4932.9
Qdrant
5057.7
ef 100
NuclaDB
3675.8
Qdrant
4951.5
ef 200
NuclaDB
2367.5
Qdrant
5063.3
At low ef (10–20), NuclaDB’s QPS actually beats Qdrant’s outright: 7432 vs 3661 at ef=10.
View full data table (recall@10, RSS) ↓
| ef | NuclaDB recall@10 | Qdrant recall@10 | NuclaDB QPS | Qdrant QPS | NuclaDB RSS |
|---|---|---|---|---|---|
| 10 | 0.8960 | 1.0000 | 7432.4 | 3661.1 | 45.2 MB |
| 20 | 0.9620 | 1.0000 | 6298.9 | 4522.6 | 45.2 MB |
| 50 | 0.9970 | 1.0000 | 4932.9 | 5057.7 | 45.3 MB |
| 100 | 1.0000 | 1.0000 | 3675.8 | 4951.5 | 45.4 MB |
| 200 | 1.0000 | 1.0000 | 2367.5 | 5063.3 | 45.6 MB |
The benchmark caught a real bug in itself. Qdrant’s default full_scan_threshold (10,000 KB) sits above this dataset’s raw size (~5,120 KB): an out-of-the-box run would have silently compared HNSW against exact search, not HNSW against HNSW. Caught by noticing suspiciously perfect recall at every ef, then fixed by forcing the threshold down.
Engineering notes
Every design decision, with its measured cost.
Four write-ups on tradeoffs we made and what they actually cost, measured rather than assumed.
Distributed · Phase 2
A real cluster sits on top, benchmarked honestly.
Raft governs topology, never the write path itself, since running every vector write through consensus would be correct but dramatically slower. Replication is deliberately async, which buys latency at the cost of a real, measured failover window where an acknowledged write can be lost. Checked with Jepsen-style testing using the real porcupine checker, not just asserted.
4
shards benchmarked
22–42%
QPS cost vs. single-node
0.963
recall@10, ef=10, 4-shard
async
WAL-stream replication
/raft
Wraps hashicorp/raft to govern cluster metadata only: which nodes exist, which node leads each shard. Never touches a vector write directly.
/ring
Consistent hashing over a fixed shard count chosen at cluster creation, so shard identity, and therefore replication, is never a moving target.
/router
Insert/Delete hash a vector id (FNV-1a) to one shard. Search fans out to every shard concurrently and merges each shard's top-K into one ranked result.
/replication
A shard leader streams its WAL to followers over plain TCP, deliberately outside Raft, for write latency, with automatic full-snapshot catch-up.
/health
Only the current Raft leader probes liveness. Fast per-shard failover after a few missed probes; full eviction and rebalance after more.
Get started
One command. A real server. Your terminal.
install.sh installs both binaries and quickstart spins up a throwaway local server, runs a scripted demo, then leaves it running so you can keep poking at it.
install.sh
$ curl -fsSL .../install.sh | sh$ nucladb-cli quickstart$ export NUCLADB_ADDR=127.0.0.1:53211$ nucladb-cli insert -id=1 -vector=1,0,0,0 -meta=team=searchinserted id=1$ nucladb-cli search -vector=1,0,0,0 -top-k=51 score=0.000000 map[team:search]