Stuff The Internet Says On Scalability For November 22th, 2013

Hey, it's HighScalability time:

Test your sense of scale. Is this image of something microscopic or macroscopic? Find out.

• 26,496: number of cores in Amazon's supercomputer; 128 billion: WDC's huge web graph publicly available; 400 million: Snapchat more photos than Facebook per day; 300,000: Microsoft's servers for Xbox; 1 million: MessageMe user growth in one week
• Quotable Quotes:
  • Jony Ive: I feel that ideas are very fragile, so you have to be tender when they are in development. I realized that if he pissed on this, it would be so sad because I know it was so important.
  • @BenedictEvans: 100m users is the new 1m users. $4bn is the new $400m. 30 staff is the new 300 staff.
  • @postwait: Knowing the latency of every I/O on every spindle in every machine in every rack... over all of time. Done & eye-opening. Thanks @circonus.
  • @mrb_bk: Things that hurt: unbounded memory growth. Things that help: concurrency control.
  • @sogrady: the latest version of chaos monkey described by @adrianco at re:invent is positively sadistic. most weaponizable software ever?
  • Joe Landman: More efficient, faster units, pay for themselves very quickly when compared to inspecific designs that make fine mail servers, but terrible storage controllers, or network hubs, or … The argument makes sense at scale and at small scale.
  • Sync: In other words, a dumb rule (majority rule) running on a smart architecture (a small world) achieved performances that broke the world.


• Jamestown was a startup established in ~1617. Lethal conditions soon required hiring more employees. The hiring bonus used to induce people to sign on was greater independence and self rule. The first American pivot.


• Everyone hates cubicles, yet they are still the best packing algorithm for humans in rectangular spaces. And if we lose productivity to noise and other cube farm diseases, does that really matter? Research: Cubicles Are the Absolute Worst


• My how the world has changed. In 1996 The Oracle of Bacon, a web site for automatically computing the shortest possible chain of costars between Kevin Bacon and any other film actor, was selected by Time magazine as a top 10 web site. At its height the site was bludgeoned with as many as 20,000 hits a day.


• You can't compete with a entrenched technology like the cloud by doing more or better cloud. You need to take a leap. Here's an example of what a leap looks like. +PeerServer:  a peer-to-peer client server using WebRTC, where your browser acts as a server for other browsers across WebRTC peer-to-peer data channels. 


• Sync: Life depends on nonlinearity. In any situation where the whole is not equal to the sum of the parts, where things are cooperating or competing, not just adding up their separate contributions, you can be sure that nonlinearity is present. Biology uses it everywhere. Our nervous system is built from nonlinear components. The laws of ecology (to the extent we know them) are nonlinear. Combination therapy for AIDS patients—drug cocktails—are effective precisely because the immune response and the viral population dynamics are both nonlinear; the three drugs in combination are much more potent than the sum of the three of them taken separately. And human psychology is absolutely nonlinear. If you listen to your two favorite songs at the same time, you won’t get double the pleasure.


• You've setup Cassandra to be an infinite sink for all your data. Now what? John Berryman has a couple of excellent articles: Estimating Ad Conversion Rates Using Cassandra and Using Cassandra To Build A Naive Bayes Classifier Of Users Based Upon Behavior. 


• Why POP Chose API First Development: makes it easier to provide an API later, separate front-end from back-end, makes it easier to scale later, works on all platforms, liberates developers because they can just follow the spec, widgetize your frontend into a series of independent asynchronous requests.


• Salesforce is making a different kind of cloud, one that's made of mini-me's, where some of their biggest customers can have their own pods where they can offer their own cloud services to internal customers. These are called Superpods. One customer with their own Superpod is the US government. Superpod's run in Salesforce's datacenter and are not on-premise. 


• Zach Holman (GitHub) with an introspective look at the purpose of being in a corporate structure. Is it just to cache out? And if you don't cache out and grow up a bit, what changes? Lots. But in the end it's your company that should change, not your values.


• Craig Kerstiens is a great Postgres resource. 


• So often good configuration equals good performance. Here's how HackerEarth configured Apache and mod_wsgi to scale their Python/Django application. Spiked traffic led to spiked resource use. Solutions: remove unnecessary modules, use Apache MPM worker, KeepAlive off, go Daemon Mode, tweak mpm-worker configuration, and check configuration. Good advice.


• Though my Chrome performance often sucks, Google says SPDY and HTTP/2 have lowered latency for their highly tuned apps by between 20% and 44%.


• HubSpot details how they deploy 300 times a day. Though that seems a little OCD, it's a good how-to. Only small teams on each part of the product; reduce shared code with small focussed libraries; build automation; unshipped code is dying code; send customers lots of notifications for all the changes; test automation; plug into Jira and other systems like HipChat; ability to push configuration to servers using Zookeeper; use metrics; and a focus on individual responsibility. One of the comments asks the million dollar question: what about data model changes? Needs a better answer. 


• Patrick McFadin has collected a few Must-See Cassandra Summit EU Presentations. If you want a seamless migration between MySQL and Cassandra then the The Cassandra Experience at Orange may be of interest. And you just may be able to mix batch and real-time.


• If you have a lot of tiny nanoantennas sitting around then maybe you should create a hologram: The metasurface, thousands of V-shaped nanoantennas formed into an ultrathin gold foil, could make possible "planar photonics" devices and optical switches small enough to be integrated into computer chips for information processing, sensing and telecommunications.


• If you couldn't attend Strata London 2013 Lars Garshol has written up an experience report just for you. Airbnb used BigData to find most interface changes made no difference. In case you thought all this sensor talk is BS there are actual examples. Crossrail uses hundreds of thousands of sensors that measure ground movement up and down to detect sinking. Hong Kong uses a similar system. Bayesian statistics can be used to perform magic. The insight that "problems can be very different in a business or conceptual sense, but often the underlying mathematics can be the same." Bandit algorithms "provide an optimal solution to the tradeoff between exploring unproven user interface alternatives and playing it safe by always reusing the alternative that's done best so far." The takeaway: Big Data analysis is not about tools and products, but about maths.


• Bitly with a story of going from sad to slightly less sad network configuration for Hadoop. Networking: Using Linux Traffic Control for Fun and Profit Loss Prevention. Good look at some complicated tooling.


• StorageMojo on Ideal workload for enterprise arrays?: Flash arrays – and other flash-enabled storage – are similarly disruptive. When you look at all the inventive effort to make enterprise arrays not look like the big, clumsy and overpriced dinosaurs they are, it only underscores how ready customers are for something better.


• In the democracy is the worst form of governement except for everything else department...An Empirical Evaluation of TCP Performance in Online Games: TCP is unwieldy and inappropriate for MMORPGs. This is due to four distinctive characteristics of MMORPG traffic: 1) tiny packets, 2) low packet rate, 3) application-limited traffic generation, and 4) bi-directional traffic. 


• MimicDB: An Isometric Key-Value Store for S3. Nice drop-in cache system for S3.


• LevelUP: a simple key/value data store built by Google, inspired by BigTable. It's used in Google Chrome and many other products. LevelGraph is built on the uber-fast key-value store LevelDB through the powerful LevelUp library. You can use it inside your node.js application or in any IndexedDB-powered Browser.


• Next time you feel the deep driving need to rewrite TCP on UDP again, take a look at UDT: a reliable UDP based application level data transport protocol for distributed data intensive applications over wide area high-speed networks. The new protocol can transfer data at a much higher speed than TCP does. UDT is also a highly configurable framework that can accommodate various congestion control algorithms.


• Certainly something like this could now be done with careful coding without resorting to an FPGA, but if you are thinking about taking the next performance step then Achieving 10Gbps line-rate key-value stores with FPGAs is a good look at what it takes. 


• If your code embeds monitoring performance tracking then take a look at very advanced LatencyUtils: useful utilities for tracking latencies. Especially in common in-process recording scenarios, which can exhibit significant coordinated omission sensitivity without proper handling.