Stuff The Internet Says On Scalability For August 24, 2012

It's HighScalability Time:

• 500 TB/day: Facebook data 
• Quotable Quotes:
  • Tumblr mobocracy: This is what migrating a billion cache objects into a new cache pool looks like
  • #hoti it is possible to build all to all wdm networks where there is a wavelength per core.
• A Generation Lost in the Bazaar. The problem is Cathedrals are multi-generation projects, built by master craftsman, using an empirical process, not engineering principles. There is no complete specification, it's iterative and adaptive.  How Bazaar. 
• Here's a unique look at how the new Digg 4 structured their architecture and development processes to scale down to 14 engineers from a high of 40 engineers for the old Digg. As the team shrunk very few new unit tests were written and the system tests were dumped. Using Thrift between components helped coordinate the team. A dark rollout mechanism allowed them to rollout and test features on a subset of users.  Development team used four horizontal teams (frontend, API, platform and infrastructure), and two vertical team (ads, analytics); practices allow them them to scale up to 40 engineers and back down to 14; Git, Gerrit, Puppet, Jenkins. 
• ServerFaults gets a serverlift, moving to SQL 2012. Always on Availability Groups provide: near realt-time replicas, no more copying backups for redundancy, spread reads across replicas, potential DR location. 
• Scaling Amazon’s DynamoDB to TreeCrunch Needs. Interesting technique to overcome "flipping" when using DynamoDB.  When making adjustments based on a metric it's easy to under or ever react, with DynamoDB's dynamic capacity management model it's no different. You need to be careful how you make adjustments and here's how you do it.
• OpenOnload is a high performance network stack that can produce application-to-application latencies as low as 3.5us.
• Scaling to Millions of Simultaneous Connections. Using what else: Erlang. Rick Reed shows the tools and techniques needed to reach nearly 3 million connections on a single server. Most of the problems fixed were centered around contention, some were in the OS, some were internal to BEAM, some were in the app. The app needed to partition correctly and avoid some common Erlang idioms. There's a whole list of of specific fixes that are quite interesting and should be generally helpful. Good work.
• Take a look at Facebook's new datacenter in Oregon. If you are a computer, it looks like a nice place to live. Otherwise a little sterile. I'd like to Design Stars take a crack at making it feel a little homier.
• The videos from Cassandra Summit 2012 are now available. If you are suffering from a dirth of summer TV shows then here is cornucopia of good mid-season replacements. 
• eBay cuts response times in half using HTTP caching : HTTP caching is not just for GET requests. By digesting the POST body, handling non-idempotent requests, and distinguishing between idempotent and non-idempotent requests, you can realize a substantial savings in round trips and bandwidth. For further savings, you can employ cache handshaking to send only the digest across the Internet.