Boosting Efficiency: Unleashing the Power of the Pub/Sub Model with AWS

Rohit Kukadiya |Updated on: July 31, 2024

• Challenges of REST API
• Pub/Sub API: A Scalable and Reliable Solution
• The Outcome: A Streamlined and Efficient Integration

At Tally, our custom CRM applications are the backbone of our business. But when we embarked on integrating these systems with COTS solutions, a significant challenge emerged: ensuring seamless interaction for real-time processing of high-volume commercial transactions.  We needed a system that could handle a large influx of requests quickly and efficiently. Initially, we leveraged the REST API for asynchronous communication. However, it has limitations, particularly when dealing with high-volume of request.

Challenges of REST API

• Scalability and Performance: It relies on synchronous communication, which can lead to performance bottlenecks. When a client makes a request, it must wait for the server to process the request and return a response. This can be inefficient for high-throughput systems or when dealing with large numbers of concurrent clients.
• Decoupling: It often creates tight coupling between clients and servers because clients need to know the exact endpoints and possibly the data structures expected by the server. Any change in the API can require clients to be updated.
• Handling of Event-Driven Architectures: Implementing event-driven architectures using REST can be complex and inefficient. It requires additional mechanisms like webhooks or long polling to handle events.
• Reliability: If a server goes down, clients will fail to get a response until the server is back up. Handling retries and failures requires additional logic on the client side.

Pub/Sub API: A Scalable and Reliable Solution

To address these limitations, consider leveraging the Pub/Sub model with AWS EventBridge. This approach offers a more efficient and scalable way for real-time communication across multiple systems.

Understanding the Pub/Sub Model

The Pub/Sub model revolves around two key elements: publishers and subscribers. Publishers send messages, while subscribers receive them. It operates on a "fire and forget" principle – the publisher transmits the message without waiting for a response, which is later received asynchronously through a subscriber in a separate thread. REST API waits for response and then sends other request where Publishers don't wait for responses, freeing them for other tasks, leading to a more responsive system overall.

AWS: Powering Your Pub/Sub Solution

Amazon Web Services (AWS) offers a robust Pub/Sub service called Amazon EventBridge. This serverless event bus acts as the central hub for message delivery, filtering, and routing.

Source: AWS

Example: When a customer place new order from system 1, System 1 publishes an event to a specific topic in EventBridge. This event contains details like customer information and order details. Meanwhile, your order processing system (System 2) subscribes to this dedicated topic. Upon receiving the published event, EventBridge steps in. It acts as a smart filter, removing unnecessary information like customer names or addresses. System 2 then receives only the crucial details it needs to process the order efficiently. This streamlined approach prevents System 2 from being overloaded with irrelevant data, ensuring smooth and rapid order fulfilment.

Here's How it Works

1. Publishers Send Events: Applications or services publish messages (events) to specific topics within EventBridge. These topics categorize messages based on their content or purpose.
2. Event Routing and Filtering: EventBridge allows filtering and transformation of events before routing them to their designated destinations. This ensures only relevant messages reach interested subscribers.
3. Subscriber Notification: Subscribers express interest in specific topics. When a published event aligns with a subscribed topic, EventBridge delivers the message asynchronously. This can trigger actions within the subscriber application.
4. Integration with Diverse Services: EventBridge seamlessly integrates with a wide range of AWS services, enabling you to connect applications and data sources readily. For instance, you can use API Gateway to trigger external API calls based on published events.

The Benefits: Unlocking Efficiency

• Scalability: Pub/Sub excels at handling large volumes of messages concurrently, eliminating bottlenecks and ensuring efficient processing even during peak loads.
• Asynchronous Processing: Improve overall system performance by delegating tasks without waiting for responses.
• Loose Coupling: It provides loose coupling since publishers and subscribers do not need to know about each other. They only need to agree on the message format and the topic. This decoupling enhances flexibility and maintainability.
• Improved User Experience: Eliminate delays and bottlenecks, leading to a more responsive and efficient system for your users.

Enhanced Reliability with Retry Mechanisms

Pub/Sub can be integrated with other AWS services like SQS (Simple Queue Service) to enhance reliability. SQS acts as a buffer for undelivered messages, allowing for automatic retries or manual intervention if necessary. You can also configure message lifespans to manage data persistence.

The Outcome: A Streamlined and Efficient Integration

The Pub/Sub model enabled us to connect our CRM applications with COTS solutions effectively. This created a strong and scalable solution that manages our expanding enterprise requirements seamlessly, preventing service interruptions. This approach transformed our integration process by ensuring real-time data synchronization and handling high volumes of requests simultaneously. Additionally, the retry mechanism automates the processing of failed transactions, ensuring system reliability.