Strategy

So, Why is Twitter Really Not Using Cassandra to Store Tweets?

High Scalability

11 Jul 2010 — 3 min read

A firestorm of accusations circled around recently saying that Cassandra, the elected-by-major-adopters emperor of the NoSQL movement, has no clothes. It was said Twitter was dumping Cassandra; Reddit outages were linked to Cassandra; and even Facebook, Cassandra's cradle of birth, was said to have abandoned Cassandra. Shouts of NoSQL Fail! were heard in the streets. Much gloating followed. Is the emperor really naked? Casually dressed maybe, but not naked.

(Note: after this point the article contains a flow chart that is NSFW. Some people are very sensitive about cussing, so if that's you, please go back, don't read on. Danger! There are no nude pictures or anything, just some strong language. But this is my most favorite flow chart of all time, so it's worth it :-)

Is Twitter really abandoning Cassandra? Not according to Twitter, which came out with a post, Cassandra at Twitter Today, explaining that they are using Cassandra in production for geolocation and analytics. Twitter, however, will not be using Cassandra to store tweets. Why? Twitter’s Ryan King says: This is a change in strategy. Instead we’re going to continue to maintain our existing Mysql-based storage. We believe that this isn’t the time to make large scale migration to a new technology.

Twitter is busy fighting other fires and they don't have the time to retrofit something that is (more or less) working, namely their MySQL based tweet storage, with a completely new technology based on Cassandra.

This is the perfect opportunity to share a flow chart that I copied many years ago from a lone cubicle wall, deep in the heart of Dilbert-land. This may be something like the thought process Twitter went through in making their decision:

Flow Chart for Project Decision Making

I still get a laugh everytime I read it. It's so true. Every company has these decisions to make when deciding where to put resources. Should you build, buy, rebuild, expand, or hang on with your fingernails? The soul of an engineer says do it right and start over. The best business decision might be quite different.

Joel Spolsky once declared rewriting working code from scratch as on of the things you should never ever do. Remember Netscape? Many no longer do, but at one time they were the web. From those heights they were brought low says Joel, because they made the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.

Twitter may not want to make the same mistake. Does this mean Cassandra and NoSQL suck? No, I think it's just smart project planning. It's actually OK to have multiple platforms for different purposes.

Does Twitter work well enough though? Brad McCarty in Twitter implements more features; ignores its broken platform, makes a good case that Twitter should shore up its infrastructure before moving on. But the problems Twitter seems to be having are with facilities, not the core Tweet engine. Until recently Twitter's up-time has been pretty good. So fix that, keep the Tweet storage engine going, event if it's ugly, and start moving on with new competition crushing features. Seem reasonable?

Some of this is inevitable. Cassandra has been cloistered inside the loving arms of Facebook and dedicated early adopters. Only recently has Cassandra left home and had to enter a very complicated world where customers have a very wide variety of needs. It will take a while for Cassandra to mature, for substantial new features to be added, for existing features to be rearchitected, and for people to figure out what Cassandra is really good at. Until then there will problems. And for developers switching to a new tool chain there will be pain as they will have to learn, relearn, and overcome a lot of obstacles. This process will not be pretty. So we will hear about problems with Cassandra and every other product out there. That's just what it means to be new. Reddit seems to be in this zone.

Is Facebook really abandoning Cassandra? Nope. Facebook still has a 150 node Cassandra cluster used for Inbox Search, which supports close to 500M users, over, 150TB of data, and is growing rapidly everyday. Now, it would be interesting to see the rate of adoption of Cassandra into other groups. When a new feature is being implemented is Cassandra being selected internally? Is it still being developed? We don't know, but Facebook is still using Cassandra for a major feature.