Design an Elasticsearch Search Cluster

Query Scatter and Gather

Inverted Index Internals

1. Cluster Node Roles ⭐

Next we walk through each component. Node roles and shard routing represent the essential core mechanics, while cluster coordination and JVM fielddata traps demonstrate staff level depth.

At scale, dedicating master and coordinating roles reduces the chance that data node garbage collection pauses destabilize cluster coordination.

• Master-eligible: Maintains cluster state including indices, mappings, and shard allocations. One active master coordinates cluster state changes through Elasticsearch cluster coordination. Three dedicated master eligible nodes provide a resilient voting configuration while keeping master duties separate from data workloads.
• Data: Stores shard data on local disk, using fast storage for hot tiers and lower cost storage for colder tiers. Executes indexing and shard local search. Keep heap at no more than about 50% of RAM and below the compressed ordinary object pointer threshold, which commonly means roughly 26 to 30 GB on a 64 GB host.
• Ingest: Runs pre-indexing ingestion pipelines before documents reach data shards. Offloads compute heavy grok parsing and GeoIP lookups from data nodes during 100K docs/sec ingestion bursts.
• Coordinating: Stateless entry points that handle client requests, scatter queries to relevant shards, and merge and sort result sets. They scale horizontally for 20K QPS without adding shard storage.

2. Sharding, Replicas, and Routing

Primary shard counts are fixed at index creation. Using _routing co locates tenant documents so tenant scoped queries can touch one shard instead of thousands, as detailed in Sharding and Partitioning.

Index creation (shard count immutable without reindex):
  PUT /products
  { "settings": { "number_of_shards": 32, "number_of_replicas": 1 } }

Document routing:
  shard_id = hash(_routing ?? _id) % number_of_shards

  POST /products/_doc/123?routing=tenant_acme
  → always lands on same shard for tenant_acme

Query with routing (single-shard):
  GET /products/_search?routing=tenant_acme
  { "query": { "match": { "title": "laptop" } } }

Replica reads: coordinating nodes normally use adaptive replica selection to choose among eligible shard copies using observed response time, service time, and search queue load.
For this HA baseline, keep at least 1 replica per primary. A zero replica index can still be valid for explicitly disposable or fully rebuildable data, but it removes protection against a primary node failure.

3. Bulk Indexing and Backpressure

Receiving HTTP 429 during bulk ingestion indicates thread pool saturation, meaning clients must apply exponential backoff rather than increasing worker concurrency.

POST /logs-write/_bulk
Content-Type: application/x-ndjson

{"index":{"_id":"evt_1","routing":"tenant_acme"}}
{"@timestamp":"2026-06-30T10:00:00Z","tenant_id":"tenant_acme","message":"order placed"}
{"index":{"_id":"evt_2","routing":"tenant_acme"}}
{"@timestamp":"2026-06-30T10:00:01Z","tenant_id":"tenant_acme","message":"payment captured"}

Response: 200 OK
{"errors":false,"took":42,"items":[...]}

Client best practices (100K docs/sec target):
  1. Batch 3K to 5K docs or 5 MB to 15 MB per bulk request
  2. Start with 8 to 16 parallel bulk workers per ingest pipeline and tune from measured thread pool pressure
  3. Set refresh_interval: 30s during backfill, then restore 1s after
  4. Use _routing for colocated tenant docs
  5. On 429 / EsRejectedExecutionException: exponential backoff (1s, 2s, 4s and so on)
  6. Monitor thread pools: bulk, write, search queue depth

Ingest node pipeline (optional):
  PUT _ingest/pipeline/enrich-geo
  { "processors": [ { "geoip": { "field": "ip" } }, { "set": { "field": "env", "value": "prod" } } ] }
  Normalizes documents before they reach the data node shard

Each data node has execution pools for indexing and search work. When capacity is exhausted, Elasticsearch can reject requests with HTTP 429. Clients should back off rather than simply adding more workers, because additional concurrency can deepen queue pressure and increase segment merge contention.

The write is acknowledged according to translog durability and active shard requirements, not according to search visibility. With the default translog durability, acknowledged operations are durably committed to the primary and allocated replicas before the request succeeds, while refresh separately controls when the document becomes searchable.

WRITE PATH VS SEARCH VISIBILITY:

1. Client sends _bulk request to the coordinating or ingest tier
2. Request is routed to the primary shard
3. Primary applies the operation and records it in the translog
4. Operation is replicated to allocated replica shards
5. With index.translog.durability = request (the default), the translog is fsynced and committed before the write request is acknowledged
6. A later refresh opens the new Lucene segment for search visibility

Key distinction:
  durability acknowledgement != search visibility
  translog durability protects acknowledged writes before the next Lucene commit
  refresh controls when those writes become visible to _search

Write availability:
  wait_for_active_shards controls how many shard copies must be active before the write proceeds
  default = 1, meaning the primary must be active

4. ILM Tiering (Hot, Warm, Cold, and Frozen)

Index Lifecycle Management can move aging indices through hot, warm, cold, and frozen phases. Searchable snapshots can materially reduce local storage requirements, and this policy introduces them in the frozen tier after cold retention. Actual savings depend on access patterns, replicas, snapshot storage, and licensing.

// ILM Policy: logs-hot-warm-cold-frozen

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d", "min_primary_shard_size": "10gb" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "read_only": {},
          "migrate": {},
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "read_only": {},
          "migrate": {},
          "set_priority": { "priority": 10 }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "s3-repo" }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Hot: NVMe, full read/write, frequent refresh, replicas for query scale
Warm: lower cost storage for read mostly indices, with force merge as a measured optimization. This design does not shrink the index because the hot tier already targets about 40 GB per primary, while a 50 GB shrink target would usually leave the primary count unchanged.
Cold: lower-cost read-only data nodes retain regular shards and replicas in this policy
Frozen: partially mounted searchable snapshots provide the lowest-cost searchable retention
Searchable snapshots require a deployment and license tier that supports the feature, such as the Enterprise license in current Elastic documentation

5. Cluster State and Split Brain Prevention

Deploy a small dedicated master tier and let modern Elasticsearch maintain the voting configuration and quorum automatically. Data nodes should not carry master responsibilities at this scale. The important operational rule is to avoid obsolete quorum settings and to treat cluster.initial_master_nodes as a bootstrap only setting.

Cluster State (master tier responsibility):

  Contents: index definitions, mappings, aliases, shard routing table,
            ILM policies, ingest pipelines, persistent cluster settings

  Update flow:
    1. Master receives create-index / mapping change / shard started event
    2. Validates (shard limits, mapping conflicts)
    3. Publishes new cluster state version to all nodes
    4. Data nodes apply allocation by moving or creating shards

  Cluster coordination and bootstrap:
    - 3 dedicated master eligible nodes for this cluster
    - Modern Elasticsearch manages the voting configuration and quorum automatically
    - cluster.initial_master_nodes is used only for first bootstrap of a brand-new cluster, then removed
    - Voting-only masters participate in elections without storing document data

  Dangerous operations:
    - Deleting an index without a verified snapshot
    - Closing an index and forgetting that reads and writes are blocked until it is reopened
    - Changing the type or incompatible mapping of an existing field, which requires reindexing
    - Allowing uncontrolled dynamic field creation, which can cause mapping explosion

6. JVM, Circuit Breakers, and fielddata vs doc_values

Enforce doc_values on keyword fields for aggregations and sorting, and strictly disallow fielddata on analyzed text fields at production scale.

• doc_values: Columnar on disk storage used for sorting, aggregations, and field value access. It is enabled by default for most field types, but not for text.
• fielddata: On heap field values used for operations such as aggregations on fields that do not have suitable doc_values. It is disabled by default on text fields. Enabling it for large analyzed text fields can create severe heap pressure, so use keyword subfields with doc_values where exact aggregation or sorting is required.
• Circuit breakers: Parent and child breakers prevent requests from reserving excessive memory. Circuit breaker rejections return HTTP 429. Keep modern defaults unless measurements justify a change, and fix sustained memory pressure by reducing query demand or scaling the cluster rather than raising limits blindly.
• Query cache: Caches eligible filter context query results at the node level using an LRU policy. Segment changes can invalidate cached results. A gateway or application cache can add short time to live caching for safe repeated tenant queries.

7. Relationship to Adjacent Systems

Understanding how Elasticsearch integrates with surrounding systems clarifies why it functions as a managed inverted index engine rather than a general crawler or vector database.

• Web Search Engine: Focuses on crawling, HTML document extraction, link graph indexing, and PageRank scoring to construct a search engine from scratch. In contrast, this design operates Elasticsearch and Lucene as a distributed search product.
• Log Aggregation & Search: Focuses on the end to end ingestion pipeline across Kafka and shippers into Elasticsearch. Log aggregation owns the ingestion topology, while this design governs cluster sizing, ILM tiering, and query performance for that sink.
• Vector Database & Semantic Search: Focuses on dense HNSW approximate nearest neighbor search. Elasticsearch supplies the complementary sparse BM25 retrieval leg for Reciprocal Rank Fusion, functioning alongside rather than replacing vector databases.