Stuff The Internet Says On Scalability For January 1st, 2016

Hey, Happy New Year, it's HighScalability time:

River system? Vascular system? Nope. It's a map showing how all roads really lead to Rome.If you like Stuff The Internet Says On Scalability then please consider supporting me on Patreon.

• 71: mentions of innovation by the Chinese Communist Party; 60.5%: of all burglaries involve forcible entry; 280,000-squarefoot: Amazon's fulfillment center in India capable of shipping 2 million items; 11 billion: habitable earth like planets in the goldilocks zone in just our galaxy; 800: people working on the iPhone's camera (how about the app store?); 3.3 million: who knew there were so many Hello Kitty fans?; 26 petabytes: size of League of Legends' data warehouse; 


• Quotable Quotes:
  • George Torwell: Tor is Peace / Prism is Slavery / Internet is Strength
  • @SciencePorn: Mr Claus will eat 150 BILLION calories and visit 5,556 houses per second this Christmas Eve.
  • @SciencePorn: Blue Whale's heart is so big, a small child can swim through the veins.
  • @BenedictEvans: There are close to 4bn people on earth with a phone (depending on your assumptions). Will go to at least 5bn. So these issues will grow.
  • @JoeSondow: "In real life you won't always have a calculator with you." — math teachers in the 80s
  • James Hamilton: This is all possible due to the latencies we see with EC2 Enhanced networking. Within an availability zone, round-trip times are now tens of microseconds, which make it feasible to propose and commit transactions to multiple resilient nodes in less than a millisecond.
  • Benedict Evans: The mobile ecosystem, now, is heading towards perhaps 10x the scale of the PC industry, and mobile is not just a new thing or a big thing, but that new generation, whose scale makes it the new centre of gravity of the tech industry. Almost everything else will orbit around it. 
  • Ruth Williams: Bacteria growing in an unchanging environment continue to adapt indefinitely.
  • @Raju: Not one venture-backed news aggregator has yet shown a Sustainable Business Model
  • @joeerl: + choose accurate names + favor beauty over performance + design minimal essential API's + document the unobvious
  • @shibuyashadows: There is no such thing as a full-node anymore. Now there are two types: Mining Nodes Economic Nodes. Both sets are now semi-centralized on the network, are heavily inter-dependent and represent the majority of the active Bitcoin users.
  • @TheEconomist: In 1972 a man with a degree aged 25-34 earned 22% more than a man without. Today, it's 70%
  • Dr. David Miller~ We are in the age of Howard Hughes. People make their fortune elsewhere and spend it on space. 
  • Credit for CRISPR: Part of that oversimplification is rooted in the fact that most modern life-science researchers aren’t working to uncover broad biological truths. These days the major discoveries lie waiting in the details
  • @BenedictEvans: Idle observation: Facebook will almost certainly book more revenue in 2015 than the entire internet ad industry made up until to 2000
  • Eric Clemmons: Ultimately, the problem is that by choosing React (and inherently JSX), you’ve unwittingly opted into a confusing nest of build tools, boilerplate, linters, & time-sinks to deal with before you ever get to create anything.
  • Kyle Russell: Why do I need such a powerful PC for VR? Immersive VR experiences are 7x more demanding than PC gaming.
  • @josevalim: The system that manages rate limits for Pinterest written in Elixir with a 90% response time of 800 microseconds.
  • catnaroek: The normal distribution is important because it arises naturally when the preconditions of the central limit theorem hold. But you still have to use your brain - you can't unquestioningly assume that any random variable (or sample or whatever) you will stumble upon will be approximately normally distributed.
  • Dominic Chambers: Now, if you consider the server-side immutable state atom to be a materialized view of the historic events received by a server, you can see that we've already got something very close to a Samza style database, but without the event persistence.
  • Joscha Bach: In my view, the 20th century’s most important addition to understanding the world is not positivist science, computer technology, spaceflight, or the foundational theories of physics. It is the notion of computation. Computation, at its core, and as informally described as possible, is very simple: every observation yields a set of discernible differences.


• The New Yorker is picking up on the Winner Takes All theme that's been developing, I guess that makes it an official meme. What's missing from their analysis is that users are attracted to the eventual winners because they provide a superior customer experience. Magical algorithms are in support of experience. As long as a product doesn't fail at providing that experience there's little reason to switch after being small networked into a choice. You might think many many products could find purchase along the long tail, but in low friction markets that doesn't seem to be the case. Other choices become invisible and what's invisible starves to death.


• I wonder how long it took to get to the 1 billionth horse ride? Uber Hits One Billionth Ride in 5.5 years.


• Let's say you are a frog that has been in a warming pot for the last 15 years, what would you have missed? Robert Scoble has put together quite a list. 15 years ago there was no: Facebook, YouTube, Twitter, Google+, Quora, Uber, Lyft, iPhone, iPads, iPod, Android, HDTV, self driving cars, Waze, Google Maps, Spotify. Soundcloud, WordPress, Wechat, Flipkart, AirBnb, Flipboard, LinkedIn, AngelList, Techcrunch, Google Glass, Y Combinator, Techstars, Geekdom, AWS, OpenStack, Azure, Kindle, Tesla, and a lot more.


• He who controls the algorithm reaps the rewards. Kansas is now the 5th state where lottery prizes may have been fixed.


• What Is The Power Grid? A stunning 60% of generated energy is lost before it can be consumed, which is why I like my power grids like my databases: distributed and shared nothing.


• Nobody expects a 2000% increase in traffic, yet that's what Steam experienced because of a DoS attack. Update on Christmas Issues. Interestingly it was the pressure of responding to the attack that forced an error: "During the second wave of this attack, a second caching configuration was deployed that incorrectly cached web traffic for authenticated users. This configuration error resulted in some users seeing Steam Store responses which were generated for other users." It's way easier said than done, but errors can be minimized by practicing these kind of scenarios in advance.


• It's possible to think of the cloud as a type of fractional reserve banking system for compute power. Switzerland to vote on banning banks from creating money.


• Bill Gates on the conservation of ineptness. An Interview with Bill Gates on the Future of Energy: “Yes, the government will be somewhat inept,” he said brusquely, swatting aside one objection as a trivial statement of the obvious. “But the private sector is in general inept. How many companies do venture capitalists invest in that go poorly? By far most of them.


• Dilbert is wise. LOL: It depends on if we have enough bandwidth to growth-hack the analytics.


• I wonder how the economics change when these noisy suckers start being disappeared from the sky? The Economics of Drone Delivery: Tasha Keeney suggests that Prime Air could cost Amazon only 88 cents per delivery. If Amazon charged customers $1 per delivery, Keeney estimates, the company could earn a 50% return on its investment in drone infrastructure while offering same-day delivery that is significantly cheaper than current alternatives.


• Nice example. Rails5, ActionCable, Redux, and React: Walking through an example chat application. Lots of code with explanations. Does it lead to javascript UI framework Nirvana? Only if you don't want it too. 


• The WhatsApp solution for slowing down growth? Charge money. 


• Math ∩ Programming is the name of the blog and it's not false advertising, as you can deterministically determine for yourself in Load Balancing and the Power of Hashing: But to understand why hash tables are so applicable, you should have at least a fuzzy understanding of the math that goes into it, which is surprisingly unrelated to the actual act of hashing. Instead it’s the guarantees that a “random enough” hash provides that makes it so useful. The basic intuition is that if you have an algorithm that works well assuming the input data is completely random, then you can probably get a good guarantee by preprocessing the input by hashing.


• If you like a little Myth Busters combined with the Get Off My Lawn Guy you might like Metalworker Is Sick Of Hearing That Jet Fuel Can't Melt Steel Beams. They do bend, easily.


• Developers drove the decision to move from Scala to Go at CrowdStrike. Moving a team from Scala to Golang. Why? Increased developer productivity, fast build times, small binaries, one file, built in formatting, great tooling, built in test framework, profilers, a nice concurrency model, a wider pool of backgrounds that can be hired from; must faster developer onboarding.


• See what happens when you install a back door? When Ebola Is Cured but Not Over: Certain parts of the body — the eye, the testes, the central nervous system — are “immune-privileged” sites, meaning they are off limits to the body’s natural defenses, the immune system. But if a virus like Ebola manages to invade those sites, immune privilege can backfire.


• Videos from 32C3 (Chaos Computer Club) are now available. With talks with titles like "Breaking Honeypots for Fun and Profit" it's a must for a well secured New Year.


• Like coffee and code for programmers? Spiders Ingest Nanotubes, Then Weave Silk Reinforced with Carbon.


• A wonderful and far ranging Interview with Jesper Louis Andersen about Erlang, Haskell, OCaml, Go, Idris, the JVM, software and protocol design — PART II. 


• Compass rearchitected their product and for their stack they chose: PostgreSQL, RabbitMQ, Sinatra, AngularJS. In a twist they found they didn't like the schemaless approach afterall, it was a pain point. Sinatra gave them the clear separation between web and client.


• David Karapetyan with a good list of basic patterns he's noticed come up over and over again while working on build/CI/deployment related things: pipeline; hashed cache; retries and fallbacks; pipeline in a loop; 


• As predicted in Revelations, Facebook's Instant Articles is the death of the Internet. Why Is Facebook Engagement Suddenly Dropping?: Over that period, Facebook referral traffic (on desktop, not mobile) to BuzzFeed fell 41 percent. For Fox News, the drop was 48 percent. For the Huffington Post, it was an astonishing 60 percent.


• A computer with a terabyte of RAM is not that far away. Samsung’s Colossal 128GB DIMM: This is eight times the density of the largest broadly-available DIMM and rivals the full capacity of mainstream SSDs.


• An Awesome List of YouTube channels.


• This article, Moores law hits the roof, generated 801 comments on reddit, by saying: The software industry has to cut down on the resource-hungry development tools and multilayer software and develop less feature-bloated products that take longer time to develop, but use fewer hardware resources and run faster on the small low-power portable devices. If the commercial software industry fails to change its course now, it will quite likely lose market shares to the slimmer open source products.


• Host-Independent PCIe Compute: Where We're Going, We Don't Need Nodes: For some compute workloads, all you need is more coprocessor cards. Storage and memory might be decentralized, and adding in hosts creates cost and complexity - a host that seamlessly has access to 20 coprocessors is easier to handle than 20 hosts with one each. This is the goal of EXTOLL as part of the DEEP (Dynamical Exascale Entry Platform) Project.


• Better high availability: MySQL and Percona XtraDB Cluster with good application design: My advice: design your application with the possible read-only partial outage in mind, and test how the application works in that mode during it’s development life cycle. I think it will pay off greatly and increase the perception of availability of your product. As an example, check out some of the big open source projects’ implementation of this concept, like MediaWiki or Drupal (and also some commercial products).


• YOW! 2015 videos are now available. 


• 5 AWS mistakes you should avoid: managing infrastructure manually; not using Auto Scaling Groups; not analyzing metrics in CloudWatch; ignoring Trusted Advisor; underutilizing virtual machines.


• Not what you might expect. Bitcoin Runs On Altruism: Listen up. There is a dirty secret in the Bitcoin world. All the core devs know it, all the academics know it, and it is commonly discussed among the people who actually understand the protocol, so it's time the cult followers are clued in as well. Namely: Bitcoin relies heavily on altruism. The system as we know it today would just not work were it not for the kindness of strangers.


• Benchmarking Streaming Computation Engines at Yahoo!: Due to a lack of real-world streaming benchmarks, we developed one to compare Apache Flink, Apache Storm and Apache Spark Streaming. Storm 0.10.0, 0.11.0-SNAPSHOT and Flink 0.10.1 show sub- second latencies at relatively high throughputs with Storm having the lowest 99th percentile latency. Spark streaming 1.5.1 supports high throughputs, but at a relatively higher latency. 


• The democratization and multiplying effects of technology, like when guns took the skill out  of shooting a bow and made it possible to field large armies of relatively unstrained warriors. US Air Force allows enlisted ranks to fly drones. What does the future of war look like now?


• How do you accelerate a WordPress-based application by up to 400x? Microcaching with NGINX: Microcaching is a caching technique whereby content is cached for a very short period of time, perhaps as little as 1  second. This effectively means that updates to the site are delayed by no more than a second, which in many cases is perfectly acceptable.


• A sobering appraisal. WebSockets, caution required! There are 7 well described reasons to avoid WebSockets. Overall, it's not the future: In a world that already had HTTP/2 it is very unlikely we would see WebSockets being ratified as it covers that vast majority of use cases WebSockets solves. What should you do?: Consider avoiding WebSockets altogether and only enabling the following 3 protocols provided they fallback cleanly: long-polling with chunked encoding, long-polling and polling.


• How the Computer Revolution Changed U.S. Cities: The key to success, it seems, came down to whether or not residents of a given metro area possessed “abstract” skills capable of complementing computer technology, such as problem-solving, analytical reasoning, and complex communication. Metros with an abstract knowledge-base prospered, report University of Oxford researchers Thor Berger and Carl Benedikt Frey in Regional Science and Urban Economics. Those with more “routine” workforces didn’t.


• Did someone say papers? Murat has published his Distributed Systems Seminar's reading list for Spring 2016. Feed your brain. 


• This is an epic discussion. Breaking and Entering: Lose the Lock While Embracing Concurrency: To summarize, we’ve seen why subscription matching is often a major concern for message-oriented middleware and why it’s frequently a bottleneck. Concurrency is crucial for high-performance systems, and we’ve looked at how we can achieve concurrency without relying on locks while framing it within the context of linearizability. Compare-and-swap is a fundamental tool used to implement lock-free data structures, but it’s important to be conscious of the pitfalls. We introduce invariants to protect data consistency. The Ctrie is a great example of how to do this and was foundational in our lock-free subscription-matching implementation. Finally, we validated our work by showing that lock-free data structures scale dramatically better with multithreaded workloads under contention.


• Quite a fine origin story. How I Created A $350 Million Software Company Knowing Nothing About Software. Where you start is not where you end up, if you listen and adapt. 


• Facebook improves software RAID with a write-ahead log. But what doesn't a write-ahead log improve? The win: The main advantage of a write-ahead log is improved reliability. That means the array should work in extreme scenarios — for example, I/O errors, disk loss, crash recovery with disk loss, and so on. Existing Linux RAID 5 implementation can handle these scenarios for RAID disks, and we add corresponding support for the log disk. Our current practice is to prevent further data corruption if an error like this happens. If the log disk is lost, for example, the RAID 5 array will forcibly enter a read-only mode. Our use of the write-ahead log fixes the write hole and rebuild performance issues.


• Against all odds, print books are on the rise again in the US. A surprise, but ebooks could be so much more than they are. They are gruel compared to a 12 course tasting menu that's possible with multimedia. Why isn't Amazon moving the industry forward instead of keeping us in the paper ages?


• News Flash: What we learned about SSDs in 2015: "Finally, the unpredictable latency of SSD-based arrays - often called all-flash arrays - is gaining mind share. The problem: if there are too many writes for an SSD to keep up with, reads have to wait for writes to complete - which can be many milliseconds. Reads taking as long as writes? That's not the performance customers think they are buying." scurvy disagrees: "This is completely false in a properly designed server system. Use the deadline scheduler with SSD's so that reads aren't starved from bulk I/O operations. This is fairly common knowledge. Also, if you're throwing too much I/O load at any storage system, things are going to slow down. This should not be a surprise. SSD's are sorta magical (Artur), but they're not pure magic. They can't fix everything." Great discussion on HackerNews. And good news, Seagate says Hard Disk Drives Set to Stay Relevant for 20 Years.


• Combine your love of sports, programming, and big data to make millions. Sound good? Saahil Sud has made more than $3.5 million this year playing fantasy sports: Professionals such as Sud play a high-volume game powered by software, using computers to find promising players and to submit hundreds of rosters for a single contest, eliminating what might have been hours of laborious work.


• Surprise! Google CDN Alpha already outperforming competitors in Europe: if we look at our measurements in Europe… Surprise! Amazingly low latencies...On the other hand, in the US we could observe a notoriously high latency and not very fast download speeds...From the current state of development, we have observed that Google Cloud CDN isn’t going full speed yet.


• Are ML and Statistics Complementary?: Deep learning’s successes can be explained by three factors:
  1. Scaling up computation (e.g. by using GPUs). 2. Collecting and processing massive amounts of data. 3. Using models with massive amounts of parameters, even if they are unidentifiable and
  uninterpretable.


• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin: We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system


• Kanaloa: a set of work dispatchers implemented using Akka actors. These dispatchers sit in front of your service and dispatch received work to them. They make your service more resilient through the following means: Auto scaling; Back pressure control; Circuit breaker; Real-time monitoring.


• GoshawkDB: A distributed, transactional, fault-tolerant object store.


• If you've ever been part of a fault tree analysis it's as brutal a process as it is useful. Here's how NASA does it. Fault Tree Analysis (FTA): Concepts and Applications. Prepare to be impressed.


• samjabrahams/tensorflow-white-pages-notes: Annotated notes and summaries of the TensorFlow white paper, along with SVG figures and links to documentation.


• BBC Digital Media Distribution: How we improved throughput by 4x: "Varnish soon became a bottleneck in the caching server architecture. It only managed to serve 4 Gbps during load tests. This post investigates why internal Varnish architecture limited a caching server to 4 Gbps, and why replacing Varnish with nginx increased output to 20 Gbps." What makes this article excellent is it shows you why: General poor performance due to Varnish's usage of memory mapped files and the large size difference between RAM and backing store; Lock contention within the Linux kernel when evicting large amounts of pages, exacerbated by the use of memory mapped files.


• A Critical Review of Recurrent Neural Networks for Sequence Learning: In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models.


• Perfect Consistent Hashing: We analyse the desired properties and present an algorithm that perfectly achieves them without resorting to any random distributions. The algorithm is simple and adds to our understanding of what is necessary to create a consistent hash function.


• Neon: Node + Rust: Neon is an API for writing fast, crash-free native Node modules in Rust.


• Algorithm hacking 201. Explaining and Harnessing Adversarial Examples: We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets