What is Stream Processing?

What is Stream Processing?

Stream Processing is the method of continuously ingesting, processing, and analysing data as it is generated, rather than waiting to collect it in batches and process it later. In a stream processing system, data flows through a processing pipeline like water through a pipe — each record or event is handled as it arrives, producing results with minimal delay.

Traditional batch processing collects data over a period (an hour, a day, or longer), stores it, and then processes the entire batch at once. This approach works well for historical analysis and reporting but introduces delay. If a fraudulent transaction occurs at 9:00 AM, a nightly batch process will not detect it until that evening or the next morning.

Stream Processing eliminates this delay. The same fraudulent transaction is analysed within seconds or milliseconds of occurring, allowing the system to flag or block it immediately.

How Stream Processing Works

A stream processing system consists of several components:

1. Event sources (producers)

These are the systems that generate data events: web servers producing clickstream data, IoT sensors sending readings, payment gateways recording transactions, or mobile apps logging user actions. Each event is a small, self-contained record with a timestamp.

2. Message broker

A message broker (such as Apache Kafka, Amazon Kinesis, or Google Cloud Pub/Sub) acts as the central nervous system, receiving events from producers and delivering them reliably to consumers. The broker ensures that events are not lost and can be processed in order.

3. Stream processing engine

This is where the computation happens. Engines like Apache Flink, Apache Kafka Streams, Apache Spark Streaming, or cloud-managed services like AWS Kinesis Analytics process events as they arrive. Processing can include filtering, aggregating, joining streams, applying business rules, and running machine learning models.

4. Event consumers (sinks)

Processed results are delivered to downstream systems: databases for storage, dashboards for visualisation, alerting systems for notifications, or other applications for further action.

Common Stream Processing Patterns

Event filtering and routing: Selecting specific events based on criteria and directing them to appropriate handlers. For example, routing high-value orders to a priority fulfilment queue.

Windowed aggregation: Calculating metrics over a sliding time window, such as "total transactions in the last five minutes" or "average response time in the last hour." Windows can be tumbling (non-overlapping), sliding (overlapping), or session-based (grouped by user activity).

Complex event processing (CEP): Detecting patterns across multiple events, such as "three failed login attempts from different locations within ten minutes" which might indicate an account compromise.

Stream enrichment: Adding context to events by joining them with reference data. For example, enriching a transaction event with the customer's profile information and risk score.

Real-time machine learning inference: Applying trained models to incoming data for immediate predictions, such as determining whether a transaction is fraudulent or recommending a product to a browsing user.

Stream Processing in Southeast Asian Business Applications

Stream Processing is increasingly critical for businesses in Southeast Asia:

• E-commerce and marketplace platforms: Real-time inventory tracking, dynamic pricing, personalised recommendations, and fraud detection during flash sales and peak shopping events like 11.11, 12.12, and Ramadan sales.
• Financial services: Real-time transaction monitoring for fraud detection and anti-money laundering compliance, which is required by regulators across ASEAN.
• Logistics and delivery: Tracking shipments and delivery riders in real time, optimising routes based on current traffic and weather, and providing accurate ETAs to customers.
• Ride-hailing and mobility: Processing location data from millions of devices simultaneously to match riders with drivers, calculate dynamic pricing, and detect anomalies.
• IoT and manufacturing: Monitoring sensor data from factory equipment to detect failures before they cause downtime, a growing application in ASEAN's expanding manufacturing sector.

Stream Processing vs Batch Processing

The choice between stream and batch processing depends on your use case:

Factor	Batch Processing	Stream Processing
Latency	Minutes to hours	Milliseconds to seconds
Complexity	Simpler to implement	More complex architecture
Cost	Lower for large historical datasets	Higher per-event, but essential for time-sensitive use cases
Use cases	Reporting, training ML models, historical analysis	Fraud detection, real-time monitoring, live dashboards

Many organisations use both in a Lambda architecture (separate batch and stream pipelines) or a Kappa architecture (stream processing only, with batch as a special case of streaming).

Getting Started with Stream Processing

For organisations new to stream processing:

1. Identify time-sensitive use cases where batch processing is too slow. Focus on scenarios where faster processing directly translates to business value.
2. Start with managed services like Amazon Kinesis, Google Cloud Dataflow, or Azure Stream Analytics to avoid the operational complexity of managing your own streaming infrastructure.
3. Use Apache Kafka as your message broker if you need a flexible, vendor-neutral foundation.
4. Design for failure from the start. Streaming systems must handle network issues, processing failures, and out-of-order events gracefully.
5. Monitor latency, throughput, and backpressure continuously. Stream processing systems require different monitoring approaches than batch systems.