Stream Processing Basics – System Design Core Concept

What:

Real-time processing engines (Apache Flink, Spark Streaming) that aggregate and transform infinite message streams continuously.

Primary purpose:

Extracting analytical insights, metric counters, or fraud signals from event logs instantly as they occur.

Usually used for:

Real-time GPS coordinate ETAs, live payment fraud alerts, and sliding-window system health trackers.

How should I think about this inside system architectures?

🕰️ Event vs Processing Time

Event Time is when the update occurred on the client. Processing Time is when it reaches the server. Always process using Event Time.

🌊 Watermark advancement

A temporal anchor signal: 'We assume all events with Event Time < T have arrived.' Advance watermarks to close windows and emit states.

🗄️ Stateful RocksDB Cache

Store running aggregate counters (e.g. click counts) inside local RocksDB key-value stores to enable micro-second local state updates.

Window Strategy	Operational Mechanic	Primary Use Case
Tumbling (Fixed-size)	Non-overlapping time brackets (e.g. every 5 minutes: [10:00, 10:05], [10:05, 10:10]).	Hourly active user counts, summary billing aggregations.
Sliding (Overlapping)	Fixed duration evaluating at regular steps (e.g. 5-min window rolling every 1 min).	Real-time system CPU spike trackers, sliding-window rate limiters.
Session (Dynamic)	Tiers grouped by user inactivity thresholds (e.g. closes after 30 mins idle).	Analyzing website visitor navigation paths or user gaming sessions.

Benefit	Cost
Sub-Second Latencies (evaluates data logs on-the-fly as they arrive, enabling immediate alert actions)	Complex Stateful Storage (keeping active window states inside local RocksDB memories consumes significant hardware storage)
Out-of-Order Resiliency (using event-time watermarks guarantees late-arriving logs are merged into correct windows)	Memory State Accumulations (waiting too long for late events forces the system to preserve active window states in RAM)

Production System	Streaming Engine Selection	Architectural Rationale
Uber Ride ETA Calculator	Apache Flink (Streaming)	Continuous GPS coordinates telemetry streams must be aggregated inside tumbling windows to emit real-time driver ETA updates.
Financial Fraud Detector	Kafka Streams / Spark	Evaluating transactions within sliding 1-minute windows to spot rapid multi-charge card usage attempts.