Streaming & Batch Processing

Discussed integration concepts through the lens of push versus pull communication, especially in microservice architectures.
Push-based communication involves Service A sending data directly to Service B, often resulting in more real-time and lower latency interactions (e.g., streaming, publish/subscribe, request-response models).
Pull-based communication involves the consuming system (Service B) reaching out to fetch data from Service A, typically on a schedule or as needed (e.g., batch processing, reading from a queue or file store).
Most integration patterns can be implemented using either push or pull approaches; the choice depends on specific system requirements.
Push methods tend to provide immediate updates and more direct efficiency, especially useful for scenarios needing real-time data flow.
Pull methods are often preferred for batch processing or when it’s more efficient to process data in larger groups at specific intervals.
The term “real-time” can be misleading—requirements should be clarified, as what is described as real-time may have a wider acceptable window than anticipated.
Choice between push and pull depends on real-time needs, latency tolerance, efficiency, and system durability requirements.

Provided an e-commerce scenario: storing invoices in Amazon S3 by the order service, then having the accounting service process invoices.
Push-based: accounting service is triggered immediately when a new invoice arrives.
Pull-based: accounting service runs periodically (e.g., once daily) to collect and process invoices in batch, increasing computational efficiency.
Determining which approach to implement depends on the desired update frequency and overall system optimization.

Transitioning from monolithic systems (with straightforward transactional integrity) to distributed microservices introduces new challenges.
Maintaining transactional integrity across microservice boundaries is much more complex than within a single-process application.

Introduced the concept of two-phase commit as a way to coordinate distributed transactions.
Used the example of Alice and Bob, where coordinated agreement and readiness are checked before committing work across services.
Phase 1: Participant (Bob) signals readiness to proceed.
Phase 2: If confirmed, actions are performed; if any participant signals failure, compensating actions may be needed to preserve integrity.
Two-phase commit helps coordinate and ensure agreement across distributed participants before a transaction is finalized.