Designing scalable Notification System

Question → What kind of Notifications are we talking about here?

Question → When do we have to send the Notifications ?

Answer → When some 2 customers does the transaction with the help of Payment-Gateway, then the Payment-Gateway has to send the timely notifications to both of the involved parties (i.e. Payer and Payee both) as shown below :-

Question → Why SLA is very important in this kind of Notification-System ?

Answer → Given the money is involved, maintaining a strong SLA is critical.

Question → Design the Base Version of Notification-System ?

Question → What is the main issue in the above design of Notification-System ?

Answer → Because the transactions (i.e. Notifications) are persisted in realtime (i.e. in Synchronous mode) to the database , therefore the p99 latency is not so good.

Question → What are the main challenges in scaling of this design of Notification-System ?

Question → How do we re-architect the design of Notification-System, so that it can handle the load & scale easily ?

Answer → We shall have to take-up multiple measures in order to re-architect the Notification-System :-

With this, we created multiple queues, in order to handle the different types of Notifications depending upon their priority-order.

Another measure is to reduce the DB-Bottleneck, by making the writes asynchronous :-

The overall design now looks like this :-

1.) Using Kinesis with AWS Lambda for Event-Driven Notifications → Here our use-case is to trigger an email, SMS, or push notification when an important event occurs.

• Ingest Events into Kinesis Data Streams → An application pushes event data (e.g., user activity, transactions, error logs) into Kinesis Data Streams.
• Trigger AWS Lambda on New Data → Configure a Lambda function to process new records from Kinesis in near real-time.
• Send Notifications via SNS or other services → Inside Lambda, publish messages to Amazon SNS (Simple Notification Service) to send email, SMS, or push notifications.

2.) Example Lambda Function (Python — Boto3):

import json
import boto3
sns_client = boto3.client('sns')
SNS_TOPIC_ARN = "arn:aws:sns:us-east-1:123456789012:MyTopic"
def lambda_handler(event, context):
    for record in event['Records']:
        payload = json.loads(record['kinesis']['data'])
        
        # Check condition to trigger notification
        if payload.get("alert_type") == "HIGH_PRIORITY":
            message = f"HIGH_PRIORITY Event Detected: {payload}"
            sns_client.publish(TopicArn=SNS_TOPIC_ARN, Message=message, Subject="Alert Notification")
    return {"statusCode": 200, "body": "Processed successfully"}

Question → What are different types of Observability metrics that we should have in this kind of system ?

Thanks for reading through this. See you in some other blog. Thank You.

References :-

• https://engineering.razorpay.com/how-razorpays-notification-service-handles-increasing-load-f787623a490f