Design a Vector Database (Semantic Search Infrastructure)

1. HNSW Index ⭐

HNSW is a widely used billion scale ANN approach based on multi layer graphs and greedy walks. Have the tuning knobs and the post filtering failure mode ready because they are common interview probes.

HNSW (Hierarchical Navigable Small World) is a widely used ANN algorithm in vector search systems. It builds a multi layer graph where upper layers enable coarse navigation and Layer 0 provides fine grained nearest neighbor search, delivering low latency on appropriately sized shards when benchmarked and tuned for the workload.

2. Filtered ANN (Pre filtering)

Pre filtering restricts HNSW traversal to authorized vectors, which is critical for access control enforcement in the Document QA Platform (RAG) and category scoped queries.

Metadata fields are indexed in an inverted index. The authorization layer validates the namespace independently, while the query planner evaluates metadata filters before ANN traversal and uses the resulting set as the relevance candidate boundary. For highly selective filters, direct candidate enumeration may be more efficient than graph traversal. For broader filters, the planner can use the pre filtered HNSW path.

3. Hybrid Search Pipeline

Hybrid search runs dense and sparse recall in parallel and fuses candidates with Reciprocal Rank Fusion, which is essential when queries contain part numbers, SKUs, or legal citations.

Hybrid Search Pipeline (used by RAG Platform and Recommendation System):

Stage 1: Parallel recall:
  Dense ANN (HNSW):     query_vector → Top-100 candidates    ~15ms
  Sparse BM25:          query_text  → Top-100 candidates    ~20ms

Stage 2: Fusion:
  Reciprocal Rank Fusion (RRF):
    score(d) = Σ 1/(k + rank_i(d))   where k=60, rank_i from each retriever
  Output: Top-50 unified candidates                              ~2ms

Stage 3: Optional rerank downstream in the RAG Platform:
  Cross-encoder: 50 → Top-10                                    ~150ms

When to use hybrid vs dense only:
  - SKU, serial number, legal citation: hybrid is typically preferred
  - Semantic Q&A, recommendation similarity: dense may be sufficient
  - Recommendation System candidate generation: dense ANN on item embeddings

Latency note:
  The ~15ms, ~20ms, ~2ms, and ~150ms figures are illustrative stage budgets, not universal guarantees.

4. Async Ingest Pipeline

High throughput writes cannot be synchronous, so Kafka buffering and batch index updates mirror the write pattern found in Live Likes Reactions.

High throughput writes (50K vectors/sec) enqueue to Kafka before being batch applied to HNSW segments by Index Builder workers. Stable vector IDs and idempotency keys make client retries safe, while per vector versions prevent stale updates from overwriting newer state. Large backfills return 202 Accepted with a job ID for status polling.

Ingest flow:
  1. POST /vectors/upsert → validate dimension, namespace quota
  2. Publish to Kafka: vector-upsert (partition by namespace_id)
  3. Index Builder consumer: batch 100ms window → apply to HNSW segment
  4. Flush segment to disk → replicate to followers
  5. Update metadata index (inverted index for filters)
  6. Vector searchable within ~5s (eventual consistency)

5. Delete, Compaction, and Index Lifecycle

Deletes and frequent updates create tombstones and stale graph entries, so the index needs background segment compaction and periodic rebuilds. Durable WAL records preserve ordering during recovery, while immutable segment snapshots provide faster restore points.

• Write tombstones with a monotonically increasing vector version.
• Exclude tombstoned vectors from ANN and metadata candidate sets immediately after the index version becomes visible.
• Compact segments when tombstone density or fragmentation crosses a configured threshold.
• Build replacement segments in the background, validate recall@10, then atomically switch the shard manifest.
• Retain tombstone metadata until all replicas and replay paths are beyond the delete version.

6. Multi-Tenant Namespace Isolation

API keys map to namespaces and namespaces map to shards, ensuring tenant isolation is enforced at every layer of the architecture.

Multi-Tenant Namespace Model:

  API Key → namespace_id (1:1 or 1:many for enterprise)
  namespace_id → shard assignment via a consistent hash ring with virtual nodes
  large namespace → optional multi shard placement by vector_id hash
  namespace_id → index config (dimension, metric, read replica count)

  Namespace quotas:
    - max_vectors: 10M (default soft limit with alerts at 80%, though enterprise tiers may raise it)
    - max_qps: 1000 queries/sec
    - max_upsert_qps: 500 vectors/sec

  Physical isolation tiers:
    Tier 1 (shared): Small tenants share shards (cost-efficient)
    Tier 2 (dedicated shard): Large tenants get their own shard
    Tier 3 (dedicated cluster): Enterprise SLA, isolated hardware

  Large-namespace routing:
    Namespace placement remains the authorization boundary.
    Vector-level sharding is used only when a namespace exceeds one shard's capacity.

  Live Likes Reactions uses a similar high write Kafka buffer pattern, and embedding math in Vector Embeddings and ANN Search uses a similar Kafka batch to index pattern.

7. Shard Router and Replication

Consistent hashing on namespace IDs routes normal tenants, while large namespaces can use explicit multi shard placement and high volume enterprise tenants can bypass shared pools through dedicated assignments.

• Consistent hash ring: shard_id = ring.lookup(hash(namespace_id))
• A placement directory stores dedicated shard overrides and multi shard ranges for namespaces that outgrow one shard.
• For a multi shard namespace, the query router fans the request out to all assigned shards, applies the same metadata and version visibility constraints, then performs a k way Top K merge before returning the final results.
• Each shard contains 1 leader for writes and 2 read replicas as the stated scenario topology. The query router selects replicas according to consistency, indexed version, and proximity requirements.
• Multi shard namespaces require parallel fan out to their assigned shard replicas, followed by a global Top K merge. The shard placement directory is the routing source for those assignments.
• Large tenants exceeding 10M vectors are an illustrative policy threshold for dedicated placement. Actual promotion should use vector count, QPS, memory, and noisy neighbor measurements.
• Write ahead logs per shard support crash recovery, paired with periodic HNSW segment snapshots to cloud object storage.