Stuff The Internet Says On Scalability For June 24th, 2016

Hey, it's HighScalability time:

A complete and accurate demonstration of the internals of a software system.If you like this sort of Stuff then please support me on Patreon.

• 79: podcasts for developers; 100 million: daily voice calls made on WhatsApp; 2,000; cars Tesla builds each week; 2078 lbs: weight it takes to burst an exercise ball; 500 million: Instagram users; > 100M: hours watched per day on Netflix; 400 PPM: Antarctica’s CO2 Level; 2.5 PB: New Relic SSD storage; 


• Quotable Quotes:
  • Alan Kay: The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
  • @jaykreps: Actually, yes: distributed systems are hard, but getting 100+ engineers to work productively on one app is harder.
  • @adrianco: All in 2016: Serverless Architecture: AWS Lambda, Codeless Architecture: Mendix, Architectureless Architecture: SaaS
  • @AhmetAlpBalkan: "That's MS SQL Server running on Ubuntu on Docker Swarm on Docker Datacenter on @Azure" @markrussinovich #dockercon
  • Erik Darling: EVERYTHING’S EASY WHEN IT’S WORKING
  • @blueben: Bold claim by @brianl: Most of the best tech of the last 10 years has come out of work at Google. #VelocityConf
  • Joe: there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. Because it really never works like that.
  • Eran Hammer: The web is the present and it’s a f*cking mess. Deal with it.
  • @etherealmind: If you believe in DevOps/NetOps you have to believe that leaving Europe is a difficult but better course of action. Small, fast & fluid
  • Sanfilippo: Redis is currently not good for data problems where write safety is very important. One of the main design sacrifices Redis makes in order to provide many good things is data retention. It has best effort consistency and it has a configurable level of write safety, but it’s optimized for use cases where most of the time, you have your data, but in cases of large incidents you can lose a little bit of it. 
  • David Smith: The best time you are ever going to have to make a new app is when there's a new iOS update. Go through the diffs. Go through the What's New? Find something that couldn't be possible before and make an app around that. 
  • @DanielEricLee: There was a timezone bug so I wrote a test and then the test failed because the CI runs in a different timezone and then I became a farmer
  • @jasongorman: Reminds me of someone I know who led a dev team who built something that won an award. None of team invited to awards bash. Only snr mgmt.
  • David Robinson: My advice to graduate students: create public artifacts
  • @cdixon: Because distributed clusters of commodity machines are more powerful.
  • @wmf: 128-port TORs have outrun the compute capacity of most racks, so putting two mini-TORs in 1U is a great idea.
  • msravi: I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibility. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.
  • Mobile Breakfast: What is clear that we will get rid of the classic circuit-switched technology and move to all IP networks fairly soon in the US.
  • Douglas Rushkoff: I think they all come down to how you can optimize your business or the economy for flow over growth; for the circulation of money rather than the extraction of money.
  • Alan Hastings~ [Species] that go into synchrony may be more subject to extinction because a single driver can trigger a collapse
  • @TechCrunch: More cars than phones were connected to cell service in Q1 http://tcrn.ch/28MLtmt  by @kristenhg
  • @docker: "Nobody cares about #containers, it's the application that matters!" - @solomonstre, CTO @Docker #DockerCon
  • @cmeik: The most commercially successful NoSQL database: Lotus Notes.
  • Brittany Fleit: behavior-based push results in open rates 800 percent higher than immediate blasts. Personalizing a message based on individual actions garners much more engagement.


• Dilbert, of course, nails it on AI.


• How will hackers be stopped from using Return-oriented programming (ROP) to execute privilege escalation attacks? ROP was created when "clever hackers realized that they could successively invoke snippets of code at the end of existing subroutines as 'proxies' to accomplish the work they needed done." Randomizing memory locations didn't stop them so Intel created new hardware magic called Control-flow Enforcement Technology. Intel added a new "ENDBRANCH" instruction and created a new "shadow stack". It turns out the immense power and beauty of the stack in von neumann architectures is also a great weakness. The story of life. Steve Gibson with an inspiring deep dive on CET in Security Now 565. 


• Does nature favor cool computations?: "There is a massive pressure in natural selection, which nobody has modeled before, for organisms to try to perform computations that are as noisy as possible in light of this trade-off,” says Wolpert. “You only want to be precise where it’s fitness-relevant." Seems in the same spirit as Ask For Forgiveness Programming - Or How We'll Program 1000 Cores.


• Philosophy is nice, but at some point you have to earn a living. What happened to the good ol' Unix philosophy? The docker command used to be about containers, not service and network scaling in the cloud: Docker does follow the Unix philosophy: it's just that it's a unix-like collection of tools rather than a single tool...Docker's trying to lock in on enterprise customers.This has nothing to do with programming, and everything to do with encouraging lock in. I've lost a lot of faith in Docker...You can either have a bunch of different tools that do their own little thing and do them well and need a bunch of perl scripts and sysadmins on call to actually do anything useful, or you can have one large tool that does one meaningful thing and, by virtue of being one tool, does it well.


• I often find myself when programming hesitating to create new types. Instead I'll just use string, int, or float properties and I tell myself just use them appropriately. Sure. This almost always leads to problems. The reason I think is illustrated in the differences between Go and Chess. Go is more complex because the rules are simpler, which means almost counter-intuitively complexity increases quickly. For Chess there are 20 possible opening moves. For Go the first player has 361 possible moves. This greater degree of choice continues throughout the game. Go maps to the scenario using base types everywhere. Using a string that can be anything instead of a Date that would be more specific and a Birthday that would be even more specific. Chess maps to the using types scenario. Types reduce the number of moves your program can make, which reduces complexity, which makes it easier your program easier understand. This is why Chess was solved long ago by computers and Go only recently. So Go create types and simplify your life.


• Still hard to beat custom. Custom Processor Speeds Up Robot Motion Planning by Factor of 1,000: “Robot Motion Planning on a Chip,” in which they describe how they can speed up motion planning by three orders of magnitude while using 20 times less power. How? Rather than using general purpose CPUs and GPUs, they instead developed a custom processor that can run collision checking across an entire 3D grid all at once.


• 5 Myths About 5G: 5G will be a “hot spot” system; 5G will require substantial investment; 5G will replace 4G; 5G will require more spectrum; or 5G, everything will need something new. 


• Sports and BigData are the new power couple. Professional Sports Get Makeover From Big Data: analysts [can] see the complete X-Y-Z coordinates of every player with unprecedented detail. The tags blink 25 times per second and deliver data in 120 milliseconds...If you’re not spending $250K and having two or three people dedicated to [Big Data analytics] full time, you’re probably too light on it...The single most talked about use of data in sports is actually fan retention and fan base building. Chris Collinsworth with a (dystopian?) vision of the future: data will reimagine training in sports. He speculated to the New Yorker that in ten years, “there will be almost no contact during the week. Pre-season will be eliminated.” Instead, virtual reality will be developed and used to get players building up muscle memory.


• Here's a recap of Facebook's Data @Scale, June 2016 event. Dropbox talks about their Exabyte Storage System; Facebook talks about Presto Raptor: MPP Shared-Nothing Database on Flash; and more from Qumulo, Twitter, Google, and Microsoft.


• Parallel programming made easy: a new chip design they call Swarm, which should make parallel programs not only much more efficient but easier to write, too...In simulations, the researchers compared Swarm versions of six common algorithms with the best existing parallel versions, which had been individually engineered by seasoned software developers. The Swarm versions were between three and 18 times as fast, but they generally required only one-tenth as much code...What distinguishes Swarm from other multicore chips is that it has extra circuitry for handling that type of prioritization. It time-stamps tasks according to their priorities and begins working on the highest-priority tasks in parallel. Higher-priority tasks may engender their own lower-priority tasks, but Swarm slots those into its queue of tasks automatically.


• What’s the right way to setup your infrastructure? Here's how Segment sets up AWS. They've open sourced a set of Terraform modules for configuring production infrastructure with AWS. The Stack comes with: an auto-scaling group of instances to run your services; a multi-az VPC with different subnets for availability; self-managed services run via docker and ECS; an ELB and ECS definition for each service; docker logs that populate in CloudWatch; a bastion node for manual SSH access; automatic ELB logging to S3. Seems pretty clear and well explained.


• Brett Slatkin with Links from PyCon 2016 in Portland. Quite a variety, lots on deep learning and statistics.


• Is that a datacenter in your closet? Greg Ferro asks if you really need a datacenter anymore when he infrastructure needs of most companies can fit in a single rack. Upgrade Your Data Centre To A Closet. Of course you will need closets in at least two geographic regions, just to be safe.


• Doug Thompson with a fine lament on how Apple squandered their lead with iBeacon. Google is killing it with their Physical Web ecosystem. 


• How Google Is Remaking Itself for “Machine Learning First”: For many years, machine learning was considered a specialty, limited to an elite few. That era is over, as recent results indicate that machine learning, powered by “neural nets” that emulate the way a biological brain operates, is the true path towards imbuing computers with the powers of humans, and in some cases, super humans. Google is committed to expanding that elite within its walls, with the hope of making it the norm. 


• Lessons fom New Relic at scale: 1: Nothing lasts forever; 2: Run experiments; 2a: Experiments don't always work, for example horizontally scaling also horizontally scaled failure points due to synchronous calls in the datapipeline, a new async system built on Kafka was successful; 3: Good architecture supports change, using APIs to incrementally change implementation, using dark deploys; 4: Technology choices have cultural impact, a message drive architecture decoupled teams and let them run on their own; 5: autonomy requires responsibility, skin in the game, teams own their own deploys and support: 6: Knowledge equals power, need deep visibility into systems.


• The brain needs a new search algorithm as more data accumulates over time. Get on that please. Memory Study Reveals Fascinating Explanation For ‘Senior Moments’: Recordings of electrical brain activity suggested that older people were making more effort to try and reconstruct their memories...While trying to remember, their brains would spend more time going back in time in an attempt to piece together what was previously seen.


• The Hidden Dividends of Microservices: #1: Permissionless Innovation; 2: Enable Failure; #3: Disrupt Trust; #4: You Build It, You Own It; #5: Accelerate Deprecations;  #6: End Centralized Metadata; #7: Concentrate the Pain; #8: Test Differently.


• If you can imagine it Amazon can stick it in AWS. What is Docker Datacenter? an end-to-end platform for agile application development and management, enabling organizations to deploy a CaaS (containers as a service) solution, both on-premises and in the cloud. With the AWS Quick Start: Docker Datacenter, we are providing customers with a tried and tested reference architecture to deploy on AWS.


• Clocks or Cores? Choose One: It is clear to see that as core counts go up, the base clock frequency goes down. The highest clocks are found in the single-socket 1600 series of chips, where you have from 4 to 8 cores, and clock frequencies up to 3.70 GHz for the 4-core variants...The most extreme scale-out chip (the E5-2699) on the right gives you 22 cores (that’s 44 threads, an insane amount!) but “just” 2.2 GHz in base clock frequency. If I were to spend the same amount of power on a 4-core E5-2637, it would run at 3.5 GHz base clock, more than 50% higher. The aggregate performance of the scale-out chips is clearly much higher. If you just multiply core count by clock frequency it is a factor of more than 3, but for a workload that does lots of independent parallel processing, 


• It's a sad day when you have to apologize for building tools in Perl. A Perl toolchain for building micro-services at scale. Good story of moving from an icky monolith to a beautiful microservices architecture using a Perl toolkit of Pinto, Carton, Minilla, Perlbrew, and Mojolicious. Which is exactly what Perl is great at. No justification necessary.


• ServelessConf slides are now available.


• Twitter hosted an event to explain all the work they've done on their infrastructure. Lots to learn from. Overview of the Twitter Cloud Platform: Compute: Twitter Cloud Platform: Compute powers over 95% of all stateless services at Twitter. It is built atop of open source technologies including Apache Mesos, Apache Aurora, and a suite of internal services that address both operator and user needs. The platform has grown from managing a few hundred containers to over 100,000 containers across tens of thousands of hosts. 


• stickfigure: By this definition, we've been running "serverless" on Google App Engine for most of a decade. * We don't monitor how many instances are running and don't really care. Our "functions" are http endpoints. GAE spins up or down instances to meet the load. Our interface with "instances" is just the size of the bill at the end of the month.* Async task processing is also just an http endpoint function. We even have a little bit of syntactic sugar so it looks like we're just adding functions to a queue. * We have no ops or devops staff. We just deploy code that implements functions (http endpoints). * Persistence is also scaled by Google; there's no database server, just an API to read/write data that scales infinitely (or at least, scales with linearly our bill, not limited by the dataset).


• A great list and meditation drawn from data collected over 13 years of experience. 18 Lessons From 13 Years of Tricky Bugs. Henrik Warne splits his lessons into coding, testing and debugging. You'll no doubt recognize yourself. I certainly do. Event ordering, silent failures, changing assumptions, all resonate. 


• Is this a classic having your cake and eating too wash? The Whole Truth About Embedding of Dependencies in Microservices: The property of being "independently deployable" is the hallmark of a properly designed microservice...It is quite obvious that you will have very hard time embedding entire Cassandra, Oracle or ElasticSearch clusters in each and every microservice you develop. Especially if you are far down the Microservices journey and possibly have hundreds of microservices. This is just not doable. Neither is it necessary...The most important question we need to ask, when deciding embedding dependencies vs. "expecting" traits in an environment is: will our decision increase or decrease mobility?


• A nicely contained recap of DockerCon 2016. Here's another Report from DockerCon 2016. 


• The harrowing story of Netflix's migration of their Oracle based billing system to, you guessed it, an AWS hosted services based architecture. Data was split into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence all relevant data was migrated to MYSQL. 


• Just because something is secure in theory doesn't mean it is in practice. Phil Daian with a wonderful explanation of the $150 million Ethereum exploit. Analysis of the DAO exploit:  The attacker was analyzing DAO.sol, and noticed that the 'splitDAO' function was vulnerable to the recursive send pattern we've described above: this function updates user balances and totals at the end, so if we can get any of the function calls before this happens to call splitDAO again, we get the infinite recursion that can be used to move as many funds as we want.


• Includes code examples. Scaling A Websocket Application With RabbitMQ: Socket.io is extremely powerful when it comes to communicating between the browser and a server in real-time. However, the problem of scaling quickly arises with the situations of very high numbers of clients or the need to implement load balancing. This problem can be easily and effectively addressed with RabbitMQ.


• Moving from Graphite to InfluxDB: Overall we have been very happy with InfluxDB and don’t miss Graphite at all. Alerting directly to PagerDuty from Kapacitor is much nicer than using custom Nagios checks although it did take us a while to get to grips with the TICKscript syntax and debugging is not easy. Their approach: Challenge complexity and simplify; clean up code; purge data; Build tooling to be resilient and compliant; Test with a clean and limited dataset first; Decouple user facing flows to shield customer experience from downtimes or other migration impacts; Moving a database needs its own strategic planning. 


• If you think you understand Java's Memory Model, well, LOL. Close Encounters of The Java Memory Model Kind.


• Hard to imagine this can work, using neural learning in branch prediction. Research proves the improbable can be made possible: Jiménez has developed many algorithms for performing branch prediction, which allows the processor to predict whether a branch instruction will cause a change in the flow of control in a program. Most of his algorithms are based on neural learning – the same form of information processing method neural networks such as the biological nervous system use.


• Speaker slides are available for the DevOps & Web Performance, O'Reilly Velocity Conference.


• Most people probably don't even realize analog computers exist, but they did and may once again. Oh the irony if Skynet is analog. Analog computing returns: “‘Digital’ is almost synonymous with ‘computer’ today, but that’s actually kind of a shame,” says Adrian Sampson, an assistant professor of computer science at Cornell University. “Everybody knows that analog hardware can be incredibly efficient — if we could use it productively. 


• Nice slide deck (120 slides!) on using ELK: Moose-ively scaling your log system.


• Golomb-coded sets: smaller than Bloom filters: GCS is well suit in situations where to want to minimize the memory occupation and you can afford a slightly higher computation time, compared to a Bloom filter. Google Chromium, for instance, uses it to keep a local (client) set of SSL CRL; they prefer lower memory occupation because it is specifically important in constrained scenarios (e.g.: mobile), and they can afford the structure to be a little bit slower than Bloom filter since it’s still much faster than a SSL handshake (double network roundtrip).


• Concrete Problems in AI Safety:  In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk


• localytics/humidifier: allows you to build AWS CloudFormation (CFN) templates programmatically. CFN stacks and resources are represented as Ruby objects with accessors for all their supported properties. 


• Tiered Replication: A Cost-effective Alternative to Full Cluster Geo-replication: We present Tiered Replication, a replication scheme that splits the cluster into a primary and backup tier. The first two replicas are stored on the primary tier and are used to recover data in the case of independent node failures, while the third replica is stored on the backup tier and is used to protect against correlated failures. The key insight of our paper is that, since the third replicas are rarely read, we can place the backup tier on separate physical infrastructure or a remote location without affecting performance. 


• A 5.8 pJ/Op 115 Billion Ops/sec, to 1.78 Trillion Ops/sec 32nm 1000-Processor Array: 1000 programmable processors and 12 independent memory modules capable of simultaneously servicing both data and instruction requests are integrated onto a 32nm PD-SOI CMOS device. At 1.1 V, processors operate up to an average of 1.78 GHz yielding a maximum total chip computation rate of 1.78 trillion instructions/sec. At 0.84 V, 1000 cores execute 1 trillion instructions/sec while dissipating 13.1 W.


• eBay/fabio:  a fast, modern, zero-conf load balancing HTTP(S) router for deploying microservices managed by consul.


• Ambry: LinkedIn’s Scalable Geo-Distributed Object Store: Ambry has been running in LinkedIn’s production environment for the past 2 years, serving up to 10K requests per second across more than 400 million users. Our experimental evaluation reveals that Ambry offers high efficiency (utilizing up
  to 88% of the network bandwidth), low latency (less than 50 ms latency for a 1 MB object), and load balancing (improving imbalance of request rate among disks by 8x-10x).


• Quick links. Get your Quick links here. Greg Linden has Quick links.