A Reference Architecture for Building Vector Databases on S3

8 min read

This is a reference architecture for building a vector database on top of S3 — heavily inspired by how Turbopuffer works. S3 is the single durable source of truth, EC2 nodes provide stateless compute with DRAM and NVMe for caching and vector math, and every coordination artifact lives on object storage itself, so any node can serve any query.

The High-Level Components

• S3 — durable storage. All vectors live on S3 as immutable SSTable files, grouped by namespace prefixes. Write-ahead segments, the master manifest (lsm.json), and a compaction task queue (Q.json) sit alongside them. Nothing durable is ever stored anywhere else.
• EC2 nodes — cache and compute. Nodes are stateless: DRAM holds the active in-memory working set (centroid tree, fetched candidate blocks, SIMD scoring buffers), and the local NVMe SSD is a warm cache (a sequential FIFO ring buffer) for recently fetched blocks. Nodes can vanish at any time — the storage tiers are billed flat and accessed via native APIs, not cloud SDKs.
• lsm.json — the master manifest. A small JSON file at the root of each namespace on S3 containing the write boundary (LSN), the SSTable inventory, and the centroids. It is updated atomically with S3's conditional writes (CAS).
• Async background workers on each node. The same EC2 instances run background tasks pulled from Q.json: merging write-ahead segments into new SSTables (compressing vectors with RaBitQ) and keeping centroids up to date with SPFresh.

How Files Are Laid Out in S3

A database, table, or shard is called a namespace, and each namespace maps to a prefix (directory) on object storage:

s3://bucket-name/<namespace-prefix>/
  ├── lsm.json                # master manifest (CAS-managed)
  ├── Q.json                  # task queue: coordinates
  │                           #   compaction & indexing work
  ├── wal/                    # write-ahead log directory
  │    ├── <UUID_1>.bin       # uncompacted write batch 1
  │    └── <UUID_2>.bin       # uncompacted write batch 2
  └── sstables/               # compacted index files
       ├── <SSTable_UUID_A>.bin   # immutable SSTable
       └── <SSTable_UUID_B>.bin   # immutable SSTable

• lsm.json is the entry point for every read and write — the manifest described in detail below.
• Q.json is a shared task queue on object storage that coordinates background compaction and indexing across stateless worker nodes, with no separate consensus service required.
• wal/ holds incoming write batches as immutable segment files, ensuring writes are durable on S3 the moment they arrive — before any indexing has happened.
• sstables/ holds the compacted, immutable SSTables: the actual vector index.

What's Inside Each SSTable File

SSTables can be gigabytes in size, so their internal layout is designed around targeted byte-range fetching:

• Sparse index footer (last 32–64 KB): sits at the very end of the file. It contains the byte-offset map — which cluster/key-range lives at which byte offset, at which length.
• Data blocks (~128 KB each): fixed-size binary blocks holding the compressed vector data. Vectors are stored as RaBitQ-quantized $D$-bit binary strings (one bit per dimension) plus a small scalar corrector — with the full-precision floats stored alongside for exact re-ranking. The blocks are encoded in a zero-copy binary layout (e.g., rkyv), so bytes landing in memory can be cast directly onto native structs with no deserialization cost.

What lsm.json Contains — and Why Centroids Live There

lsm.json is the master namespace manifest. It holds three things:

1. The Log Sequence Number (LSN) boundary — the line between what has been compacted into SSTables and what still lives only in wal/ segments. Any query must replay writes above this boundary on top of its results to guarantee read-your-writes consistency.
2. The SSTable inventory — which SSTable file UUIDs exist, and which key ranges or vector clusters each one contains. This is what lets the engine prune away non-matching SSTables without downloading them.
3. The centroids — the center coordinates of every vector cluster, which SPFresh maintains as data shifts.

Why store centroids here, in one small JSON file, rather than inside the SSTables? Because it makes the whole system stateless and cheap to bootstrap: any node, on any query, can answer "which clusters are closest to this vector" by reading a few kilobytes from S3 — no node needs a local copy of the index, and no data blocks need to be touched at all for the pruning decision. And when SPFresh splits or merges clusters, the new centroid state is published to all nodes at once with a single atomic (CAS) update to lsm.json.

How a Write Flows Through the System

1. Routing: a load balancer uses consistent hashing on the namespace ID to send the write to a compute node.
2. Durability first: the node appends the write batch to S3 as a new immutable WAL segment file — wal/<UUID>.bin. The write is durable the instant this lands; no indexing has happened yet.
3. Task enqueue: the node records the pending indexing work in Q.json.
4. Background compaction: async workers on the same nodes pull tasks from Q.json. They merge WAL segments, sort the contents, compress vectors into RaBitQ bitstrings, and write out new immutable SSTables under sstables/ — as large sequential writes, in classic LSM-tree fashion.
5. Manifest commit: the worker updates lsm.json atomically via a conditional write (CAS): the LSN boundary moves forward, the SSTable inventory gains the new file, and — if cluster sizes drifted — SPFresh applies its split/merge decisions and the new centroids are published in the same commit.

How SPFresh Keeps Centroids Up to Date

Centroids drift out of date as data arrives and gets deleted. Rather than periodically re-clustering the entire dataset ("rebuilding the world"), async workers on each node run SPFresh continuously: when a cluster crosses its size threshold they split it with a local 2-means, when deletions hollow one out they merge it with its nearest neighbor, and boundary vectors get re-assigned to their true nearest centroid. All of this is local computation on a few blocks — and the resulting centroid updates are committed to lsm.json with a CAS write, instantly visible to every node.

How a Search Happens

The golden rule: never scan the database. A brute-force dot product across billions of vectors would mean downloading terabytes from S3. Instead the query prunes 95%+ of the dataset before reading any data blocks:

1. Read lsm.json (cheaply). The node checks the manifest's ETag with a conditional GET — if it matches the locally cached copy, nothing is downloaded at all. Otherwise it fetches the few-kilobyte manifest.
2. Pick the closest centroids. The engine computes the distance between the query vector and the centroids from lsm.json (for very large cluster counts, centroids are arranged in a hierarchical tree so this lookup is $O(\log K)$ instead of a linear scan). It selects the top 2–4 clusters — instantly eliminating over 95% of the database.
3. Resolve key ranges to files and offsets. The SSTable inventory in lsm.json says which SSTable UUIDs hold those clusters. The engine issues byte-range requests for the last 32–64 KB — the sparse index footer — of each target SSTable, which returns the exact byte offsets of the candidate data blocks.
4. Fetch only the candidate blocks. Armed with byte coordinates, the engine fires parallel range requests for the specific ~128 KB data blocks and streams them straight into DRAM (and the local NVMe cache). Zero-copy deserialization casts the buffers onto native structs with no parsing.
5. Score at SIMD speed. Vectors inside the blocks are RaBitQ bitstrings, so the CPU scores thousands of candidates per millisecond with bitwise SIMD AND + popcount — no floating-point dot products. RaBitQ's sharp error bounds discard candidates that mathematically cannot reach the top-$k$ before any further work.
6. Re-rank and merge. For the small remaining shortlist, the engine pulls the full-precision vectors stored alongside the bitstrings, computes exact distances, replays any unindexed WAL entries above the LSN boundary, and returns the final top-$k$.

The payoff of the footer-as-a-map design:

Over 99.9% less network transfer per query — the difference between an S3-backed database being theoretical and being practical.

Related reads

• S3 Data Fetching Techniques — the byte-range, block-size, and parallel-fetch primitives this architecture is built on
• RaBitQ — how vectors become $D$-bit strings with sharp error bounds
• SPFresh — incremental centroid splits and merges without re-clustering
• Building a Database on S3 — the consistency and CAS primitives that make lsm.json possible
• SSTables: The Sorted String Table — the immutable file format holding the index