Stuff The Internet Says On Scalability For October 4th, 2013

Hey, it's HighScalability time:

(Consumption as a Service: Amazon has 80 1.2 million square foot warehouses.)

• 100 megapixel cameras, taking 40 million pictures a second, creating of 1 petabyte of data every second:  Large Hadron Collider in Higgs search


• Quotable Quotes:
  • Tim Bell: You can't do transformation in transitional stages
  • @lleung: Also, we could really use a “high scalability” like blog for Enterprise IT to see the interesting stuff
  • xkcd: Functional programming combines the flexibility and power of abstract mathematics with the intuitive clarity of abstract mathematics.
  • @pbailis: That said, just because you can now build distributed transactions doesn't mean they'll be fast or work well during failures.


• Ever wonder how you can get those shoes and that tube of miracle cream from Amazon at the same time in the same box? Here's a fascinating look at The Amazon Warehouses chaotic storage system: Chaotic storage is like organized confusion. It’s an organic shelving system without permanent areas or sections. That means there is no area just for books, or a place just for televisions (like you might expect in a retail store layout). Every single shelf space inside an Amazon warehouse has a barcode. And every incoming product that requires storage is assigned a specific barcode that matches the shelf space in which it will be stored. The product’s characteristics and attributes are irrelevant. What’s important is the unique barcode associated with every product that enters the warehouse. In this storage system a wide variety of products can be found located next to each other. A necklace could be located beside a DVD and underneath a set of power tools. There are several key advantages to the chaotic storage system. First is flexibility. With chaotic storage, freed-up space can be refilled immediately. Second is simplicity. New employees don’t need to learn where types of products are located. They simply need to find the storage shelf within the warehouse. You don’t need to know what the product is, just where it is. Lastly is optimization.


• Learned at C2SV: Walmart is making a product genome, a list of every product in the world (which isn't really a genome BTW).


• Z Proxy. Good look at how bitly "leverage redundant datacenters, redundant code, and multi-tiered fallbacks in the quest for uptime. Z Proxy is essentially a self contained wrapper around S3, written in Go. To have an additional level of safety, the S3 key is the short link, but the value isn’t actually the long url itself. The S3 value is an HTML blob containing a meta refresh to the destination url. "


• When evaluating SSDs look at the Write Endurance Rating says Peter Zaitsev in Inexpensive SSDs for Database Workloads. For example, a SSD that if written at 450MB/sec only lasts two weeks so the performance won't matter that much. "If your writes are really high, measured in TBs a day or higher, you will be better off with more expensive enterprise storage drives that have a much higher endurance rating."


• libPhenom - an eventing framework for building high performance and high scalability systems in C (from Facebook).


• Inside the fall of BlackBerry: How the smartphone inventor failed to adapt. Wow, what a story. It's painful, especially when you remember how many lives are impacted. In all these situations people work so hard to make things work only to be lead into a dead end.


• Some good short Scaling Advice from MongoHQ (mostly general): Identify your data behavior; Refactor your schema to simplify queries; Separate hot and cold data; Don’t lean on mongodump; Check your gauges; Avoid queries causing page faults; Track and monitor slow queries; Buying time with hardware.


• Learning from Failure at Etsy. Couldn't agree more on A Blameless Post-Mortem. I would also add the Blameless Bug Report. Nothing is more corrosive than a culture that rejoices in the failure of others. Don't play the blame game.


• JVM memory management slower than C? My take is C can be deterministic. You can completely control memory allocation in your program so you can guarantee performance tolerances. With the JVM you twiddle command line arguments each time it disappoints you, which is often.


• Greg Linden with a fine set of October Quick Links.  As always a wide range of themes: security, tablets, games, and teaching.


• The great Jeremy Cole on The MySQL Ecosystem at Scale. It's 2013 and MySQL still running most of the web, including Twitter and Facebook and Google...Pretty fast usually...Robust storage...Easy to run and use...Replication usually works...Bad for ID generation...Bad for graph data...Replication lag for bust OLTP...use as-is for data sets <= 1.5TB...Build on top for missing features...use as a permanent backing store for larger datasets. Ecosystem of Oracle MySQL, Percona Server, MariaDB. And much more.


• Ilya Grigorik has released a new encyclopedic book on High Performance Browser Networking. A huge accomplishment and I hope to soon have an interview with Ilya. If you have any questions for him just let me know.


• Looks like a good example. OoVoo uses Riak to support cloud based chat history, rich interactive chat, group communication features, and an infinite retention policy to the tune of 85 million users worldwide, 2.5 million users generating an average of 300 million minutes of video every day, 1,000 chat messages per second, 40GB of data per day, and tens of terabytes of data. They give schema examples and their decisions like to use Bitcask for online backup purposes. Found challenging the lack of transactions, limited query language, and scale limited online backup.


• A Billion Rows per Second: Metaprogramming Python for Big Data, Excellent talk by Ville Tuulos on what you need to do to analyze all that data. His talk considers analytics, numerical data, not distributed search type problems or key-value lookups. It is faster and uses far fewer resources than Amazon's Redshift. And yes, it's coded in Python using Numba, a new NumPy aware dynamic Python compiler based on LLVM. You can do a lot on one machine using this metaprogramming approach. "You can use a high-level language to quickly implement domain-specific solutions that outperform generic solutions, regardless of the language they use."


• Why Tellybug moved from Cassandra to Amazon DynamoDB. DynamoDB has issues, but they liked its reliability, counters, predictable low latency, zero operations cost, and lower cost better than Cassandra. Interesting discussion of two of your major database options no matter which side you are on.


• I can just hear Nassim Taleb yelling beware the Black Swan in response to a Statistical Basis for Predicting Technological Progress: Our results show that technological progress is forecastable, with the square root of the logarithmic error growing linearly with the forecasting horizon at a typical rate of 2.5% per year. These results have implications for theories of technological change, and assessments of candidate technologies and policies for climate change mitigation.


• There Is More Consensus in Egalitarian Parliaments: This paper describes the design and implementation of Egalitarian Paxos (EPaxos), a new distributed consensus algorithm based on Paxos. EPaxos achieves three goals: (1) optimal commit latency in the wide-area when tolerating one and two failures, under realistic conditions; (2) uniform load balancing across all replicas (thus achieving high throughput); and (3) graceful performance degradation when replicas are slow or crash.


• Naiad: A Timely Dataflow System: Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations.


• Online, Asynchronous Schema Change in F1: We introduce a protocol for schema evolution in a globally distributed database management system with shared data, stateless servers, and no global membership.


• Your Server as a Function (Twitter): Building server software in a large-scale setting, where systems exhibit a high degree of concurrency and environmental variability, is a challenging task to even the most experienced programmer. Efficiency, safety, and robustness are paramount—goals which have traditionally conflicted with modularity, reusability, and flexibility. We describe three abstractions which combine to present a powerful programming model for building safe, modular, and efficient server software: Composable futures are used to relate concurrent, asynchronous actions; services and filters are specialized functions used for the modular composition of our complex server software