SparkPost is the fastest-growing cloud email API service, providing developers with robust email delivery so their applications and websites can send and receive email. SparkPost’s technology delivers over 25% of the world’s non-spam email. As you might expect, that means we generate a lot of data. Running a service has dramatically different requirements for scalability, reliability, and real-time performance. The tools and infrastructure that work well in the on-premises aren't always the ones best suited to the cloud. We knew we had to approach this from a different mindset than we might have in the on-prem past. When SparkPost shifted focus to cloud first my team started re-architecting problem areas to ensure better growth.

This talk will describe how my team transitioned from using single-node RabbitMQ message queues with clustered node.js ETL processes to embracing cloud technologies: Amazon Simple Queue Service(SQS), Elasticache(Redis backed), and reducing their node.js processes by 75% and transitioning to the “Omni ETL”, one ETL to rule them all. Moving to SQS gave us more breathing room when downstream services were slow or even down. The math is pretty simple: these changes mean fewer pages to our on-call engineers. Fewer pages = more sleep. More sleep = happier engineers. Happier engineers = a better product.

It was extremely cost effective because removing RabbitMQ allowed us to eliminate costly disk volumes. It also allowed us to be more aligned for auto-scaled, independent service tiers. The time to market was also fast because we took the best of what we already had and just solved the major pain points. We will also look at how we changed the engine of a 747 while in flight, and tips for you to do this too. Lastly we will share the outcome of this project and some lesson learned.