Evolution of a Backend for a Streaming Application

"Everything started one and a half years ago, when my company asked me to scale our streaming applications in a way for our international users. The problem was, the request was painful. Today I'm going to share how a team of two developers with no experience in AWS, led by me, changed completely our core services, where we removed a single point of failure, we increased availability and scalability of our applications."

"If you see at a high level, practically there is a worker that subscribes topics on Kafka, do some transformation, store it in the DB. We have the other component, that API behind GraphQL, that gets bombarded by our frontend. There is nothing wrong with this architecture, if made right. What I learned one and a half years ago is, when we are talking about technical debt, we are usually talking about code."

"This architecture didn't grow with the business. Servers were on fire. Database was crying. Every time there was a spike, everything crashed. The database was in a single node, no cache, nothing. It was painful. Data was completely inconsistent through services. This team of two, to our joy, we had six services that had no standards. This is where we say, we need to change the way of working."

"We moved to serverless. Not because serverless is cool, only because it lets us focus on what really matters, the code. During this one year and a half, we end up from active-active services, and a different type of multi-regional, active-passive, in different shades, depends on the severity of the services. This architecture pretty much solved all our problems, because serverless just scales. We leverage all the managed services from AWS. All the problems like availability, resiliency, are taken care of by AWS."

A streaming system used Kafka workers to transform data and store it in a database, with a GraphQL API serving frontend requests. The architecture initially worked but accumulated technical debt as the business grew, leading to servers and the database failing during spikes. The database ran as a single node with no caching, and data became inconsistent across services. A small team with no prior AWS experience changed the way of working and migrated to serverless to focus on code. The migration removed a single point of failure, increased availability and scalability, and used managed AWS services to handle resiliency and availability. The system evolved through active-active and multi-regional active-passive patterns depending on service severity.

Read at InfoQ