Data Serialization & Schema Evolution

Every RPC and event carries a serialization format — JSON, Protobuf, Avro. We pick based on schema evolution needs, payload size, and whether consumers can change independently.

Pick format by consumer diversity and volume. Public APIs → JSON + OpenAPI. Internal high-throughput pipes → Protobuf or Avro + schema registry. The hard part is not encoding — it is schema evolution without breaking old readers.

Serialization format choice affects compatibility, performance, and schema migration — plan evolution before production traffic. Three lenses:

Walk schema evolution as interview steps. Step 1 — Choose format: JSON for public APIs, Protobuf/Avro for internal high-throughput. Step 2 — Define schema: required fields, optional fields, defaults. Step 3 — Forward compat: new consumers read old data (add optional fields). Step 4 — Backward compat: old consumers read new data (never remove required fields). Step 5 — Registry: schema registry validates compatibility on publish (Confluent model).

Protobuf safe evolution example

// v1 — original
message OrderEvent {
  int32 order_id = 1;
  string status = 2;
}

// v2 — backward + forward compatible add
message OrderEvent {
  int32 order_id = 1;
  string status = 2;
  optional string customer_email = 3;  // new optional field
  reserved 4;                          // never reuse removed field numbers
  reserved "legacy_field";
}

JSON vs Protobuf vs Avro — schema enforcement, size, and evolution rules we compare:

• Format comparison — trade readability against wire efficiency and schema enforcement:

• Compatibility modes — registry enforces one of these on every schema registration:

• Breaking changes (avoid): remove field without reserved, change field type, rename field number, change field number assignment.
• Schema registry: central store (Confluent, AWS Glue); assigns monotonic schema IDs; runs compatibility check before accept; consumers fetch schema by ID at deserialize time.

Binary efficiency trades human readability and debugging ease — we articulate both:

JSON at the edge, binary on the bus is a common pattern: REST/GraphQL APIs expose JSON for client debuggability; internal Kafka topics use Avro/Protobuf for throughput. Concept #32 API Contract governs the JSON surface; this concept governs the pipeline bytes underneath.

Breaking schema changes, unknown field drops, and double serialization — we volunteer:

Kafka CDC pipelines, gRPC services, and event sourcing depend on schema evolution:

Discuss serialization when multiple services share an event stream or API over time:

Serialization connects to Kafka guarantees, API contracts, and event sourcing:

• #18 Kafka Architecture & Guarantees — Avro payloads, schema ID in message header, consumer deserialization flow.
• #32 API Contract & Integration Design — JSON/OpenAPI versioning, error envelopes, and public API evolution policy.
• #03 Network Protocols (HTTP/gRPC) — gRPC uses Protobuf on the wire; REST typically uses JSON.
• #19 Event Sourcing & CQRS — event schema versioning across replayed event logs.
• #32 Distributed Message Broker (problem) — schema-governed topic contracts between producer fleets.
• #102 CDC Pipeline (problem) — Debezium emits Avro/JSON change events with schema registry.

Avro Reader/Writer Schema Resolution

Avro separates writer schema (embedded in each message or referenced by ID) from reader schema (what the consumer expects). On deserialize, Avro resolves differences: missing fields get defaults; extra fields are skipped; field aliases map renames. This is why Avro dominates Kafka data lakes — Spark jobs written against schema v1 still read v5 data if evolution rules were followed.

Confluent Schema Registry Compatibility Levels

Protobuf vs JSON Size Example

// Same logical message:
JSON:   {"order_id":12345,"status":"SHIPPED","items":[{"sku":"ABC","qty":2}]}  → ~72 bytes
Proto:  [field tags + varint encoding]                                         → ~18 bytes

// At 100K events/sec:
JSON:   ~7.2 MB/s wire + parse CPU
Proto:  ~1.8 MB/s wire + faster zero-copy decode

JSON Schema Evolution (Weaker Guarantees)

Without a registry, JSON evolution relies on convention: only add optional keys, never remove keys consumers depend on, use null for absent vs missing distinction carefully. OpenAPI additionalProperties: false catches unexpected fields in strict mode but breaks forward compatibility. For public APIs (concept #32), version via /v1/ URL prefix when breaking changes are unavoidable.

CDC + Schema Registry Pipeline

Debezium captures row changes from Postgres/MySQL WAL, converts to Avro with a schema derived from table DDL. Schema registry stores one schema per table topic. Downstream Spark Structured Streaming reads by schema ID — column adds in the source DB propagate as compatible schema bumps without stopping the pipeline. Ties directly to concept #18 retention and consumer group offset management.

Interview Checklist

1. Name the format (JSON edge, Avro/Proto bus).
2. State compatibility mode (usually BACKWARD_TRANSITIVE for Kafka).
3. List safe changes: add optional field with default, add new message type.
4. List forbidden changes: retype, renumber, delete without reserved.
5. Mention schema registry CI gate before production deploy.