Design the Kubernetes Control Plane

Reconciliation Loop

Scheduler Pipeline

1. API Server + etcd ⭐

The API server, etcd, and scheduler pipeline form the interview core, while kubelet and CNI provide supporting operational depth unless the discussion focuses specifically on container networking.

Watch cache can serve cache eligible LIST and WATCH requests without reading etcd on the hot path, while APF prevents controller storms from starving kubelets.

• API server: Validates requests against the OpenAPI schema, runs the admission chain, encodes objects to JSON or Protobuf, and reads or writes etcd. Optimistic concurrency for applicable updates uses resourceVersion
• etcd: 5 member Raft cluster storing keys like /registry/pods/default/my-pod. Provides linearizable writes while watch streams propagate changes to the API server cache. Sensitive objects such as Secrets can be encrypted before persistence
• Watch cache: In memory index per resource type that can serve cache eligible LIST and WATCH requests without hitting etcd on hot paths
• List pagination: limit and continue token parameters are mandatory at scale by platform policy.resourceVersion=0 requests the "Any" consistency semantic for GET or LIST and can be served from the watch cache. A GET with no resourceVersion requests the most recent state, while LIST consistency depends on the resourceVersion and resourceVersionMatch combination. Clients must handle an expired continue token or stale watch resourceVersion by handling HTTP 410 Gone. For a stale watch, relist and then resume from the returned resourceVersion. For an expired paginated LIST token, follow the API server's 410 response semantics or restart the LIST when appropriate.
• APF: Classifies requests into configured priority levels and flow schemas, with fair queuing preventing controller or batch traffic from starving latency sensitive requests

2. Scheduler (filter + score + bind)

Filter infeasible nodes, score the remainder, and bind with optimistic concurrency, recognizing that bin packing vs spread is an intentional tuning trade off.

• Scheduling queue: ActiveQ for unscheduled pods, BackoffQ for failed attempts, and UnschedulableQ for pods that cannot currently be scheduled
• Filtering plugins: NodeResourcesFit, NodeAffinity, TaintToleration, VolumeBinding, and InterPodAffinity to eliminate infeasible nodes
• Scoring plugins: NodeResourcesBalancedAllocation, ImageLocality, and PodTopologySpread, where a weighted sum selects the best node
• Bin packing vs utilization spreading: MostAllocated packs workloads tightly for cost savings, whereas LeastAllocated favors lower node utilization. High availability across failure domains comes from topology spread constraints and related placement rules, with scheduler plugin weights tuned per workload
• Bind: Calls the Pod binding subresource to bind the Pod to the selected Node, with conflict handling when concurrent updates race with the decision

3. Controller Manager (reconciliation loops)

Deployment to ReplicaSet to Pod is the canonical reconciliation chain, driven by informers, workqueues, and rate limited backoff across every controller.

• Deployment controller: Manages ReplicaSets, rolling updates (configured via maxSurge and maxUnavailable), and revision history
• ReplicaSet controller: Maintains running pod counts matching the replicas specification
• Node controller: Monitors node heartbeats and conditions. After the node monitor grace period it can mark an unreachable Node unhealthy and add taints. Taint based eviction and related controller behavior then determine when Pods are evicted according to tolerations, eviction settings, and rate controls
• EndpointSlice controller: Maintains EndpointSlices from Service selectors and Pod state, including ready, serving, and terminating conditions
• Reconciliation pattern: Informers maintain watch streams and local caches, queuing events into workqueues with rate limited backoff for level triggered reconciliation

4. kubelet + CNI Networking

The kubelet synchronizes desired and running state. During Pod sandbox creation, the container runtime invokes CNI ADD, and EndpointSlice conditions reflect Pod readiness, serving, and termination state.

• Pod sync loop: Compares desired pods from the API server against running containers via CRI, creating, deleting, or updating state accordingly
• CRI: containerd pulls images and instantiates container runtimes within pod cgroups
• CNI: During Pod sandbox creation, the container runtime invokes CNI ADD so the plugin allocates an IP address and configures the network interfaces, with plugins such as Calico, Cilium, or AWS VPC CNI
• Service proxying: kube-proxy programs Service ClusterIP routing rules using iptables, IPVS, or nftables. eBPF based service proxying can instead be provided by an alternative such as Cilium
• DNS: CoreDNS uses Kubernetes Service and EndpointSlice state, plus related namespace and Pod state when configured for Pod records, to provide cluster DNS through the Service Discovery path
• Probes: Evaluates liveness for container restarts, readiness for EndpointSlice membership, and startup probes for slow initializations

5. Admission Webhooks + RBAC

Mutating webhooks can inject defaults or sidecars, while validating webhooks and admission policies enforce security requirements. With failurePolicy set to Fail, a webhook communication error can reject matching API requests until the dependency recovers.

CREATE Pod request flow through admission:

1. Authentication (x509 client cert / OIDC bearer token)
2. Authorization (RBAC: can user create Pods in namespace?)
3. Mutating admission phase (matching mutating webhooks run serially):
   - inject sidecar (service mesh)
   - set default resources / labels
   - apply other configured mutations
4. API field and schema validation:
   - OpenAPI based field and schema validation
5. Validating admission phase (matching validating webhooks are called in parallel, where any rejection rejects the request):
   - PodSecurity admission for namespace security standards
   - OPA/Gatekeeper policy (no :latest tag, require limits)
   - image signature verification (cosign) as an example validating policy
   - other configured validating webhooks or admission policies
6. Persist through the API server into etcd
   - resourceVersion is used for optimistic concurrency on applicable updates, not for ordinary object creation
7. Return 201 to client

failurePolicy:
  - Fail: reject request on webhook error (secure, availability risk)
  - Ignore: proceed if webhook down (available, policy gap)

Timeout: Kubernetes allows 1 to 30 seconds. This scenario tunes webhooks to 2 to 5 seconds, with asynchronous audit for slow policy checks

# RBAC example: developer can deploy in namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: team-payments
  name: deployer
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: team-payments
subjects:
  - kind: User
    name: alice@corp.com
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

6. Multi Tenant Isolation

Namespaces alone do not provide multi tenant isolation. ResourceQuota, LimitRange, NetworkPolicy, Pod Security Admission, and RBAC collectively define the tenant boundary.

Multi tenant isolation layers (shared cluster):

1. Namespace: logical boundary and name uniqueness per cluster
2. ResourceQuota: max pods, CPU, memory, PVCs per namespace
3. LimitRange: default, min, and max per-container resources
4. NetworkPolicy: L3/L4 rules. Create default deny ingress and egress policies, then add explicit allows
5. RBAC: Role and RoleBinding scoped to namespace
6. PodSecurityAdmission: restricted, baseline, and privileged levels
7. Node pools: dedicated nodes via taints (tenant=acme:NoSchedule). Admission policy should prevent tenants from selecting another tenant pool by granting arbitrary tolerations

Fleet level (1000 clusters):
  - One cluster per regulated tenant OR shared multi tenant with the controls above
  - Fleet RBAC maps each organization to its cluster list
  - Hub config sync for policy bundles across the fleet

7. Watch Fan Out at Scale

etcd delivers changes to the API server, which fans them out through its watch cache and watch path rather than sending updates directly to 15K clients. Bookmarks can reduce relist work on reconnect but are sent at server discretion.

Watch architecture at 500K pods / 15K subscribers:

API Server (per replica):
  - In memory watch cache (indexed by resource type + namespace)
  - Serves LIST from cache when resourceVersion within window
  - New watch: initialize from the API server watch cache when possible, then stream subsequent changes through the API server watch path

Subscriber breakdown (large cluster):
  - 10K kubelets (scenario count of logical watcher sources)
  - 2K controllers (Deployment, RS, Node, EndpointSlice, ...)
  - 2K operators (Prometheus, Istio, custom CRDs)
  - 1K kubectl and CI/CD platform long polls
  Note: one process can maintain multiple watch streams, so 15K is a scenario scale for logical watch streams, not a universal Kubernetes limit.

Reconnect storm mitigation:
  - Request resourceVersion bookmarks. A 5 min target is illustrative because bookmark delivery is not guaranteed at a fixed interval
  - APF: separate flow treatment for watch and mutating request traffic
  - Scenario quota: max 500 inflight watches per user identity

etcd does NOT fan out to 15K clients because the API server serves as the fan out hub

8. GKE Style Fleet Layer

One isolated control plane is dedicated to each customer cluster. The fleet Hub aggregates inventory and distributes policy bundles at scale.

• Cluster provisioning: Cluster API or a proprietary operator provisions control plane and node pools per tenant
• Fleet registration: Each cluster registers with the Hub, establishing a central inventory of 1,000 clusters
• Policy at scale: Distributes Config Sync bundles (such as OPA and NetworkPolicy templates) across all clusters
• Upgrades: Orchestrates surge upgrades while respecting the release-specific Kubernetes version skew policy. Control plane components normally match the kube-apiserver minor version and may be one minor version older. Kubelet and kube-proxy must not be newer than kube-apiserver and may be up to three minor versions older under the current policy. The exact supported range narrows when HA API servers themselves have version skew
• Observability: Aggregates control plane SLIs across the fleet using Observability and Distributed Tracing, accompanied by per cluster etcd dashboards