How PayPal Scaled to Billions of Transactions Daily Using Just 8VMs
Monday, August 15, 2016 at 8:56AM
Todd Hoff

How did Paypal take a billion hits a day system that might traditionally run on a 100s of VMs and shrink it down to run on 8 VMs, stay responsive even at 90% CPU, at transaction densities Paypal has never seen before, with jobs that take 1/10th the time, while reducing costs and allowing for much better organizational growth without growing the compute infrastructure accordingly? 

PayPal moved to an Actor model based on Akka. PayPal told their story here: squbs: A New, Reactive Way for PayPal to Build Applications. They open source squbs and you can find it here: squbs on GitHub.

The stateful service model still doesn't get enough consideration when projects are choosing a way of doing things. To learn more about stateful services there's an article, Making The Case For Building Scalable Stateful Services In The Modern Era, based on an great talk given by Caitie McCaffrey. And if that doesn't convince you here's WhatsApp, who used Erlang, an Akka competitor, to achieve incredible throughput: The WhatsApp Architecture Facebook Bought For $19 Billion.

I refer to the above articles because the PayPal article is short on architectural details. It's more about the factors the led the selection of Akka and the benefits they've achieved by moving to Akka. But it's a very valuable motivating example for doing something different than the status quo. 

What's wrong with services on lots of VMs approach?

Given the above forces PayPal wanted a system with the following characteristics:

It's clear PayPal wanted a thinner stack. They didn't want a stack with lots of layers and moving parts. Akka and state based systems in general are good for that as they collapse a good chunk of the stack down to one technology. PayPal chose Akka over Erlang because they have a lot of Java experience and Akka runs on Java. For many having to learn Erlang is a non-starter.

With Akka they could:

So of course PayPal immediately wrote their own framework on top of Akka, as one does, called squbs, rhymes with cubes, that creates a modular layer for building nano-services called “cubes”. Cubes are symmetric to other cubes, the interdependency between cubes are loose and symmetric, and only expose the messaging interface already provided in Akka.

The article brings up the difficulty of programmers adapting to the non-linear nature of Akka code, so you have to hire people that can be trained program in Akka/Scala.

Since most services do similar things--receive requests, make database calls to read/write the database, make other service calls, call a rule engine, fetch data from cache, write to cache--they were able to abstract that out using patterns like the Orchestrator Pattern and Perpetual Stream.

Squbs has become the standard for building Akka-based reactive applications at PayPal. So if you haven't considered stateful systems for your team, give them another look. It has worked for PayPal, Facebook, Uber, and Microsoft.

Related Articles

Article originally appeared on (http://highscalability.com/).
See website for complete article licensing information.