Stuff The Internet Says On Scalability For June 21, 2013

Hey, it's HighScalability time:

• 1 trillion messages/day: Facebook's pub-sub; 4ms: Google Fiber last mile latency
• Quotable Quotes:
  • @blueben: Just like general computing, Google observes that faster networks beget richer content rather than a faster user experience. #velocityconf
  • @unixdaemon: All of the big wins in performance come from removing work -- @brendangregg  #velocityconf
  • @RealGeneKim: "100ms slower API response resulted in lost millions $$ from our customers using our PayPal payment APIs" #velocityconf
  • @RealGeneKim: “In complex systems, the only way to find out you’ve gone over the line is to go over the line” #velocityconf
  • @RealGeneKim: “Twitter t.co link shortener built to reduce risk, but introduced fragility” #velocityconf
  • @odiszapc: It's win! #Cassandra read latency diagram after migrating to SSD. Min value of 0,05 millis is pretty awesome


• Even AdSense started small writes Jeff Dean: A little more than 10 years ago, I was part of the engineering team of five or six people that put together our initial AdSense product.  Two of my officemates had developed a machine learning system that could determine very fine-grained topics from arbitrary pieces of content (developed with some small amounts of systems-building advice from me).  We took that system and married it with a crawling system and our advertising system, to match up ads and page contents by fine-grained topics and AdSense was born.  This  project was great fun because we went from idea to launch in about 6 weeks, launching initially on message posts on Google Groups, and then on a few partners before a broader launch a few months later. 


• Google brain teasers don't work. Neither does GPA. Or test scores. Or interview scores. Well no kidding. But I wonder if you lose social cohesion if you remove the frat house like initiation process that the hiring gauntlet gives? You may not be finding the best, but you may be creating a bonded band of brothers ready to sacrifice themselves for each other? 


• What happens when network partitions form in distributed databases? How is it handled? Jepsen: Testing the Partition Tolerance of PostgreSQL, Redis, MongoDB and Riak with an awesome test of various failure scenarios. Lessons: clients matter, the are not objective observers; there are several levels of transaction consistency; avoid systems which assume node state consensus implies data consistency; distrust systems which rely on the system time; bugs you can fix. Designs are much harder to re-evaluate; Choose the right design for your problem space; Preventing write loss in databases like MongoDB requires a significant latency tradeoff.


• Netflix continues as the vision, document, code production leader in Cloud Native architecture. Just a few of their recent posts: Isthmus - Resiliency against ELB outages, Announcing Ice: Cloud Spend and Usage Analytics, Announcing Zuul: Edge Service in the Cloud, What is Hystrix?


• BigData still needs its Newton. Johannes Kepler data mined Tycho Brahe's observational data and came up with the three laws of planetary motion. But it was Newton that derived the same laws from first principles using simple models and in the process created a real modern looking science, fulling Bacon's vision. Or have we found a new method? Francis Bacon laid out in Novum Organum a new method for discovery, experimental science, saying nothing new can be found using the same old ways. Aristotle was great, but we need to go beyond the old wisdom. Are we at that same point today where a new enlightenment is necessary or is the current meta method eternal?


• Williams Edwards writes a wonderful, slow lingering tale of Scaling my Server through a series of rewrites to create a system that effortlessly maintains millions and millions of active users. The article is packed with goodness, but the result was to write in Java, use a central shardable server to push messages to devices, scale proportional to active users not total users, keep live data in RAM persist in a database, version the DB, use application level message ACKs, use DataInput/OutputStreams, use a faster JSON parser, there's a  limit to the number of TLS connections can terminate, Java data types have a lot of overhead which limits the number you can keep in RAM, Java hashtables are wasteful. Good discussion on Reddit.


• Do we need AWS clones or do we simply need viable alternatives? Twitterers react.


• Excellent basic article on setting up Nginx for Developers: An Introduction. 


• Ilya Grigorik reports Google+ switched their CSS download from XHR to a regular link tag and saw a 4x speedup at the median: using declarative markup like <link> enables the browser to do appropriate resource prioritization.


• Apica did a thorough test of Google Compute Engine using RightScale scaling up a 3-tier architecture to 330,000 page views per minute from 200,000 concurrent users, maxing out at 42 servers on GCE during the peak load. Nothing really dramatic to report, which I guess is the point.


• imgur on their HBase and EC2 setup. Some interesting findings: instance stores outperform EBS volumes when it comes to sequential I/O, while EBS drives are better at random access patterns when using provisioned IOPS of 200 and above; EBS is expensive so they went with instance storage; m1.xlarge instances provide a good balance of resources; light map-reduce jobs are run on HBase and heavy jobs are on EMR; consider a AZ as a rack for configuration purposes; Cross-AZ traffic response time were unaffected and latency increased by a negligible amount, but could be expensive. 


• Sneakernet for the win! Offline Disk Import for Google Cloud Storage - ship your HDDs to Google. It's always amazing how long bulk data loading takes. It's the crazy uncle in the basement problem nobody talks about.


• Stacey Higginbotham with a good article on peering, a little understood "feature" of the Internets: Peering pressure: The secret battle to control the future of the internet. Dumb pipes aren't money makers so there's always a move to charge more somewhere in the network. Which is why we in the US have the best network connectivity in the world...nobody said ever.


• VMs aren't the only way to share machine resources. There is another. A container is an alternative lighter weight approach. Well explained by David Strauss in Containers—Not Virtual Machines—Are the Future Cloud: Containers have to run the same kernel as the host, but they can optionally run a completely different package tree or distribution. Containers are fairly transparent to the host system. The kernel manages memory and filesystem access the same way as if it were running on the host system. Containers obtain access to resources from the host over normal userland/IPC facilities.


• Awesome article on How FPGAs work, and why you'll buy one with a good discussion on Hacker News. Yes, it's still a complicated hardware addition with deployment issues and an unfamiliar tool chain, but the win: FPGAs evolved from a circuit implementation platform into a software delivery platform.


• HyperLevelDB improves LevelDB's performance by 4X by adding parallelism, improving compaction, thread-safe log, and a thread-safe memtable.


• Kent Dodds has my new favorite acronym: The MEAN stack - MongoDB, ExpressJS, AngularJS, node.js.


• Wonderful article on the OpenStack Grizzly Architecture, describing “a massively scalable cloud operating system.” Lots of details and well written and organized. Open Stack has seven core components: Compute, Object Storage, Identity, Dashboard, Block Storage, Network and Image Service.


• Revolution in Storage (RICON East 2013). James Hughes with an interesting analysis of storage trends: flash will not take over disks for quite a while; object stores are replacing distributed file systems; storage will be fronted with a key/value API; log structured storage is the future.


• Back to basics learning: Building a Modern Computer from First Principles. Amazing work.


• Thinking of adding some of that machine learning to your system? An Intelligent Approach to Image Classification By Color. They make it almost look easy. Also, Weekend Reads and Videos on Deep Learning.


• The 7 principles of successful open source communities: The heart of an open source project is the community. The prosperity  correlates  with  the  ability of  the  members  to form  a  passionate,  respectful,  healthy  and  focused  community.


• Improving availability in distributed systems with failure informers: This paper describes Pigeon, a failure reporting service designed to enable high availability in the applications that use it. Pigeon exposes a new abstraction, called a failure informer, which allows applications to take informed, application-specific recovery actions, and which encapsulates uncertainty, allowing applications to proceed safely in the presence of doubt.


• An NSA Big Graph Experiment.