One of the most valued fintech businesses in India is Razorpay, a payments service. The number of transactions on the system has been increasing exponentially as a result of the expansion in users and payment volume.
Razorpay's Notification service, a platform that handled all client notification needs for SMS, E-Mail, and webhooks, was one service that needed to be updated.
3. Scheduler nodes will check the MySQL databases for any notifications that were not sent out successfully and push them back to the SQS queue to be processed again.
Even this architecture can handle 2000 transactions per second then also Razorpay doesn't meet their SLAs with P99 latency that increases from 2 seconds to 4 seconds.
3. Unexpected Increases in Load - Load would increase unexpectedly for certain events/days and this would impact the notification platform.
In order to overcome these issues, the Razorpay team decide to :
Solutions: Queue
To ensure customer events are not affecting others, they use rate limiting on their APIs. Each queue, event, and customers have some configurable rate limit so that the message goes into separate queues.
As load increases, the worker/executor increases but DB is not elastic and hence input/output operations on DB become the bottleneck. Vertical scaling is costly and they can't do it forever.
Solution: Writing Database Asynchronously
the team decide to write database asynchronously with AWS Kinesis. Kinesis is a fully managed data streaming service offered by Amazon and it’s very commonly used with real-time big data processing applications. The worker nodes will now write the status for the notification messages to Kinesis rather than MySQL
When the webhook call is made, the worker node is blocked as it waits for the customer to respond. Some customer servers don’t respond quickly and this can affect overall system performance.
To solve this, engineers came up with the concept of Quality of Service for customers, where a customer with a delayed response time will have their webhook notifications get decreased priority for the next few minutes. Afterwards, Razorpay will re-check to see if the user’s servers are responding quickly.
Razorpay built a robust system around observability to ensure the system scale well with increased load.
They have dashboards and alerts in Grafana to detect any anomalies, monitor the system’s health, and analyze their logs. They also use distributed tracing tools to understand the behavior of the various components in their system.
Redis Insight RedisInsight is a powerful graphical user interface for Redis that runs on the web. It provides a graphical interface for managing and monitoring Redis databases, making it easier for developers, DevOps teams, and database administrators to work with Redis.
What is Load Balancing? To prevent any one device from becoming overloaded, load balancing is the process of dividing network traffic among several servers or resources. Optimizing resource utilization, increasing throughput, reducing response times, and avoiding overwhelming any single resource are the objectives of load balancing.