The WhatsApp Architecture Facebook Bought For $19 Billion
Wednesday, February 26, 2014 at 8:56AM
Todd Hoff in Example

Rick Reed in an upcoming talk in March titled That's 'Billion' with a 'B': Scaling to the next level at WhatsApp reveals some eye popping WhatsApp stats:

What has hundreds of nodes, thousands of cores, hundreds of terabytes of RAM, and hopes to serve the billions of smartphones that will soon be a reality around the globe? The Erlang/FreeBSD-based server infrastructure at WhatsApp. We've faced many challenges in meeting the ever-growing demand for our messaging services, but as we continue to push the envelope on size (>8000 cores) and speed (>70M Erlang messages per second) of our serving system.

But since we don’t have that talk yet, let’s take a look at a talk Rick Reed gave two years ago on WhatsApp: Scaling to Millions of Simultaneous Connections.

Having built a high performance messaging bus in C++ while at Yahoo, Rick Reed is not new to the world of high scalability architectures. The founders are also ex-Yahoo guys with not a little experience scaling systems. So WhatsApp comes by their scaling prowess honestly. And since they have a Big Hairy Audacious of Goal of being on every smartphone in the world, which could be as many as 5 billion phones in a few years, they’ll need to make the most of that experience.

Before we get to the facts, let’s digress for a moment on this absolutely fascinating conundrum: How can WhatsApp possibly be worth $19 billion to Facebook?

As a programmer if you ask me if WhatsApp is worth that much I’ll answer expletive no! It’s just sending stuff over a network. Get real. But I’m also the guy that thought we don’t need blogging platforms because how hard is it to remote login to your own server, edit the index.html file with vi, then write your post in HTML? It has taken quite a while for me to realize it’s not the code stupid, it’s getting all those users to love and use your product that is the hard part. You can’t buy love

What is it that makes WhatsApp so valuable? The technology? Ignore all those people who say they could write WhatsApp in a week with PHP. That’s simply not true. It is as we’ll see pretty cool technology. But certainly Facebook has sufficient chops to build WhatsApp if they wished.

Let’s look at features. We know WhatsApp is a no gimmicks (no ads, no gimmicks, no games) product with loyal users from across the world. It offers free texting in a cruel world where SMS charges can be abusive. As a sheltered American it has surprised me the most to see how many real people use WhatsApp to really stay in touch with family and friends. So when you get on WhatsApp it’s likely people you know are already on it, since everyone has a phone, which mitigates the empty social network problem. It is aggressively cross platform so everyone you know can use it and it will just work. It “just works” is a phrase often used. It is full featured (shared locations, video, audio, pictures, push-to-talk, voice-messages and photos, read receipt, group-chats, send messages via WiFi, and all can be done regardless of whether the recipient is online or not). It handles the display of native languages well. And using your cell number as identity and your contacts list as a social graph is diabolically simple. There’s no email verification, username and password, and no credit card number required. So it just works.

All impressive, but that can’t be worth $19 billion. Other products can compete on features.

Google wanted it is a possible reason. It’s a threat. It’s for the .99 cents a user. Facebook is just desperate. It’s for your phone book. It’s for the meta-data (even though WhatsApp keeps none).

It’s for the 450 million active users, with a user based growing at one million users a day, with a potential for a billion users. Facebook needs WhatApp for its next billion users. Certainly that must be part if it. And a cost of about $40 a user doesn’t seem unreasonable, especially with the bulk paid out in stock.  Facebook acquired Instagram for about $30 per user. A Twitter user is worth $110.

Benedict Evans makes a great case that Mobile is a 1+ trillion dollar business, WhatsApp is disrupting the SMS part of this industry, which globally has over $100 billion in revenue, by sending 18 billion SMS messages a day when the global SMS system only sends 20 billion SMS messages a day.  With a fundamental change in the transition from PCs to nearly universal smartphone adoption, the size of the opportunity is a much larger addressable market than where Facebook normally plays.

But Facebook has promised no ads and no interference, so where’s the win?

There’s the interesting development of business use over mobile. WhatsApp is used to create group conversations for project teams and venture capitalists carry out deal flow conversations over WhatsApp.

Instagram is used in Kuwait to sell sheep.

WeChat, a WhatsApp competitor, launched a taxi-cab hailing service in January. In the first month 21 million cabs were hailed.

With the future of e-commerce looking like it will be funneled through mobile messaging apps, it must be an e-commerce play?

It’s not just businesses using WhatsApp for applications that were once on the desktop or on the web. Police officers in Spain use WhatsApp to catch criminals. People in Italy use it to organize basketball games.

Commerce and other applications are jumping on to mobile for obvious reasons. Everyone has mobile and these messaging applications are powerful, free, and cheap to use. No longer do you need a desktop or a web application to get things done. A lot of functionality can be overlayed on a messaging app.

So messaging is a threat to Google and Facebook. The desktop is dead. The web is dying. Messaging + mobile is an entire ecosystem that sidesteps their channel. Messaging has become the center of engagement on mobile, not search, changing how things are found and the very nature of what applications will win the future. We are not just pre-pagerank, we are pre-web.

Facebook needs to get into this market or become irrelevant?

With the move to mobile we are seeing deportalization of Facebook. The desktop web interface for Facebook is a portal style interface providing access to all the features made available by the backend. It’s big, complicated, and creaky. Who really loves the Facebook UI?

When Facebook moved to mobile they tried the portal approach and it didn’t work. So they are going with a strategy of smaller, more focussed, purpose built apps. Mobile first! There’s only so much you can do on a small screen. On mobile it’s easier to go find a special app than it is to find a menu buried deep within a complicated portal style application.

But Facebook is going one step further. They are not only creating purpose built apps, they are providing multiple competing apps that provide similar functionality and these apps may not even share a backend infrastructure. We see this with Messenger and WhatsApp, Instagram and Facebook’s photo app. Paper is an alternate interface to Facebook that provides very limited functionality, but it does what it does very well.

Conway's law may be operating here. The idea that “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.” With a monolithic backend infrastructure we get a Borg-like portal design. The move to mobile frees the organization from this way of thinking. If apps can be built that provide a view of just a slice of the Facebook infrastructure then apps can be built that don’t use Facebook’s infrastructure at all. And if they don't need Facebook's infrastructure then they are free not to be built by Facebook at all. So exactly what is Facebook then?

Facebook CEO Mark Zuckerberg has his own take, saying in a keynote presentation at the Mobile World Congress that Facebook's acquisition of WhatsApp was closely related to the Internet.org vision:

The idea is to develop a group of basic internet services that would be free of charge to use — “a 911 for the internet." These could be a social networking service like Facebook, a messaging service, maybe search and other things like weather. Providing a bundle of these free of charge to users will work like a gateway drug of sorts — users who may be able to afford data services and phones these days just don’t see the point of why they would pay for those data services. This would give them some context for why they are important, and that will lead them to paying for more services like this — or so the hope goes.

This is the long play, which is a game that having a huge reservoir of valuable stock allows you to play. 

Have we reached a conclusion? I don’t think so. It’s such a stunning dollar amount with such tenuous apparent immediate rewards, that the long term play explanation actually does make some sense. We are still in the very early days of mobile. Nobody knows what the future will look like, so it pays not try to force the future to look like your past. Facebook seems to be doing just that.

But enough of this. How do you support 450 million active users with only 32 engineers? Let’s find out...

Sources

A warning here, we don’t know a lot about the WhatsApp over all architecture. Just bits and pieces gathered from various sources. Rick Reed’s main talk is about the optimization process used to get to 2 million connections a server while using Erlang, which is interesting, but it’s not a complete architecture talk.

Stats

These stats are generally for the current system, not the system we have a talk on. The talk on the current system will include more on hacks for data storage, messaging, meta-clustering, and more BEAM/OTP patches.

Platform

Backend

Frontend

Hardware

Product

General

The Quest for 2+ Million Connections Per Server

Tools and Techniques Used to Increase Scalability

Lessons

Related Articles

 

Article originally appeared on (http://highscalability.com/).
See website for complete article licensing information.