Stuff The Internet Says On Scalability For February 19th, 2016

JPL is firing up their Exoplanet Travel Bureau . Reserve your space now.If you like this sort of Stuff then please consider offering your support on Patreon.

• 200K : msgs send per second through iMessage; 750 million : xactions per week in App and iTunes store; 11 million : Apple Music subscribers; .7c : speed of light in silicon; 1.125Tpbs : fastest ever data transmission; 360TB : Superman memory crystal stores data forever;  $1bn : Uber’s yearly cost for market share in China;


• Quotable Quotes:
  • Joseph Bradley : “Here is the takeaway. Blockchains must be massively more scalable than the current tech that supports Bitcoin. We start scaling slowly or quickly. And if we choose the latter, it will “require fundamental protocol redesign.”
  • @sigfpe : Nobody knows how to “program” DNA. They just copy-and-paste bits from other organisms. A bit like how most code is built from stackoverflow.
  • @evankirstel : Slack now has 2.3 million daily active users, 675,000 paid seats, and 280 apps in its directory
  • Jonas Luster : Money spoiled blogging. Why? Because people moved from doing great things for money and then talking about them on their free blogs, to people doing nothing but talking on their monetized blogs. 
  • @mattocko : What’s vilely hypocritical re Koch’s latest dirty dealings (vs electric cars) is they enjoy 100x subsidies for oilco 
  • Greg Ferro : In my view, the most common architectural flaw made by network engineers is that the data centre has a single network. I believe that the correct perspective is that any “network” is a “network of networks”.
  • @antirez : @Nick_Craver I’ve always this nice feeling that you manage to run a top-traffic site with 1/100 of the hardware. It’s like a reality check…
  • @levie : With Apple, Amazon, and Netflix now producing their own shows, Box will be accepting script submissions for enterprise software dramas.
  • Fred George : We had 50 IT professionals, 25+ titles and zero people understanding the project they were working on…
  • @jasongorman : “They asked for a bridge, but I know what they really needed was (another) reusable civil engineering framework”
  • @aplokhotnyuk : @cbrisket @giltene @netty_project @eBay Neutrino is highly available… We have measured upwards of 300+ requests per second on a 2-core VM.
  • jsmthrowaway : You don’t even need unikernels, and as much as I loathe myself for saying it, I find myself agreeing with a few of Cantrill’s points regarding unikernels in prod. Not all of them, and I think it’s worth exploring, but there’s a spectrum here: on one end, unikernel app containers, and on the other full jails. The Google approach with minimal containers that still act Unixy and Posixy but carry very little distribution overhead is somewhere around 0.1 on the spectrum.
  • @Beaker : This is why I call these “Internet-scale monoculture vulnerabilities.” FFS. 


• We never considered the possibility Skynet may just be stupid. The NSA’s SKYNET program may be killing thousands of innocent people : At root, this is a story about the problems that occur in the absence [of] adversarial peer review. NSA and GCHQ cut corners in their machine-learning approach, and no one called them on it, and they deployed it, and it kills people. But is also a microcosm of the spy services’ culture of secrecy and the way that the lack of peer review turns into missteps.


• A  buffer overflow exploit in glibc  remained undetected for 8 years. How Bazaar. Also,  Linux kernel bug delivers corrupt TCP/IP data to Mesos, Kubernetes, Docker containers .


• How Uber Engineering Evaluated JSON Encoding and Compression Algorithms to Put the Squeeze on Trip Data . They tested a whole bunch of different compression approaches. A whole bunch. Their goal was to find a solution that both yielded a small size and a short time to encode and decode. The conclusion: “MessagePack with zlib. We felt this was the best choice for our Python-based, sharded datastore with no strict schema enforcement (Schemaless). We only discovered this combination because we took a disciplined approach to test a wide range of protocols and algorithm combinations on real data and production hardware. First lesson learned: when in doubt, invest in benchmarking.” Result: A 1 TB disk will now last almost a year (347 days), compared to a month (30 days) without compression. We now have enough space to last over 30 years compared to just under 1 year, thanks to putting the squeeze on the data.


• That feeling when you try to show your Grandma how to use the TV remote.  When you’re house sitting for millennials and ask how the lights work . This is funny and a little sad, not much has changed in 30 years. 


• If you are IBM and your insatiable demon child needs feeding, what do you do? You buy companies for their data.  Why IBM Just Bought Billions of Medical Images for Watson to Look At : Merge’s data set contains some 30 billion images, which is crucial to IBM because its plans for Watson rely on a technology, called deep learning, that trains a computer by feeding it large amounts of data.


• Only two weeks notice? To be a developer is to continually have your heart broken.  Kimono Labs Acquisition Leaves Devs Hanging .


• Here’s how Facebook was able to  ship #friendsday videos , not a simple task when scaling to hundreds of millions of people. They started with an extensive load testing, efficiency improvement, and capacity planning process. This feature ran on a shared infrastructure so it had to play nice with others. A throttling mechanism allowed the plug to be pulled if it used too many resources. To figure out what videos to include they used their advanced image-understanding technology. The rollout was staggered. Here’s how you know you are Facebook: “We  looked at our peak traffic on every continent and tried to alleviate the load during peak time.” These “videos can go viral immediately and have global traffic pouring in” managing the rollout is not a minor consideration. They used React for web and React Native for both iOS and Android to generate the interface for the feature in less than a month.


• Google crafts custom networking CPU with parallel computing links : Google engineers are actively working on developing code for the hardware, which is excellent at accelerating memcache workloads. Memcache stores objects in huge pools of memory distributed across thousands of servers, and is a crucial component in many large-scale services.


• If you are looking for an overall approach to building a project then you might like  Zalando Tech’s Rules of Play : As part of Radical Agility, implemented by Zalando’s technology team in March 2015, we have adopted this set of “Rules of Play”: microservices, API First, REST, Cloud, Software as a Service (SaaS). This document focuses primarily on services, as the principles for interoperating between services are quite mature and stable. 


• Velocity Conference Santa Clara 2015: Video Compilation . 


• Formula One Racing Can Drive CFD Innovation Faster . You may know that Formula One Racing puts a lot of restrictions on how racing cars are constructed, but they also put IT on a budget: “the current FIA rules restrict the capacity of CFD modeling systems to 25 teraflops.” An incredible evolutionary environment for selecting for efficiency. “The cluster that Sahara Force India has, for example, has specialized Interlagos chips that have been tuned specifically to get the most performance out of CFD applications within a 25 teraflops performance ceiling. This cluster has 28 pre-processing and post-processing nodes, and another 432 solver nodes to run the core fluid dynamics applications across more than 7,000 cores in the system. The cluster has 400 TB of storage, and the nodes and the storage are all linked together by 10 Gb/sec Ethernet switches. (Sahara Force India can’t talk specifically about this iron in terms of brands.)”


• Riot Games continues the story of how they make League of Legends work on a hostile internet.  FIXING THE INTERNET FOR REAL TIME APPLICATIONS: PART II . Their task? Rebuild part of the Internet: “But where Google is focusing on the cable running to your house, we’re more interested in creating a network like the backbone companies. This would allow us to pick up player traffic as close to the player as possible and then put it on our network—a network with many fewer routers (and other devices) that drop packets and degrade the player experience.” How?: “the endeavor took us over a year to complete. Connecting our servers to ISPs required us to first purchase routers, lease space to place the routers in ten of the largest internet hubs in the US (commonly referred as colocation spaces, more on that later), lease dark fiber or optical wave services to connect those routers, and finally peer with all the ISPs we could.” Result: we’ve actually created one of the fastest networks on the continent.


• Good things happen when you squash spikes from the VM’s Garbage Collection.  Fast C10M: MigratoryData running on Zing JVM achieves near 1 Gbps messaging to 10 million concurrent users with 15 millisecond consistent latency . avg: 61ms to 15ms; 99th percentile: 585ms to 25ms; max: 1700ms to 126ms (computed on 1.8 billion messages) 


• Are you planning on world wide product roll out? Here are some lessons from Netflix’s experience: Recommending for the World . Most interesting is the interaction between licensing and recommendations and search:  “Our recommendation models rely heavily on learning patterns from play data, particularly involving co-occurrence or sequences of plays between videos. In particular, many algorithms assume that when something was not played it is a (weak) signal that someone may not like a video, because they chose not to play it. However, in this particular scenario we will never observe any members who played both “Equilibrium” and “The Matrix”. A basic recommendation model would then learn that these two movies do not appeal to the same kinds of people just because the audiences were constrained to be different.” The advantage is they’ve now have more data so they’ve been able to: “discover worldwide communities of interest so that we can make better recommendations.”


• [AMA] We are the Go contributors: ask us anything! . Go forth and read these over 600 questions and comments on Go 1.6.


• Using API Simulation to Build Microservices Alongside a Java Monolith . sbov : We already do this in our monolith. All business operations are declared on interfaces, which is used to codegen a client & http request handler. Normal app code uses the codegen’d client, and we have a REST api which runs the codegen’d http request handler (and underlying business logic that we write by hand). This gives us certain kinds of flexibility that we use. That said, it’s still a monolith. Even if we split each call onto a different server ultimately they will be relying on a shared database schema residing on the same, singular MySQL server.


• Systemantics . The total amount of anergy in the universe is fixed. The author defined anergy as the energy required to bring about a change. This as meant as a tongue-in-cheek analog of the law of conservation of energy. One of the problems that a system creates is that it becomes an entity onto itself that not only persists but expands and encroaches on areas that were beyond the original system’s purview.


• The fall… and rise and rise and rise of chat networks .  digi_owl : I think it was in part technical. Those networks were built when you were either online or offline. but with the phones you were kinda both. Note that the likes of Whatsapp only really happened after Apple introduced their push notification service. Thus allowing apps to behave comparably with SMS. Before that you basically had to keep the data connection going 24/7 for the likes of AIM to make sense on mobile.


• Should you optimize? Here’s a calculator . Are you going to save more than 412 hours of instance time? Then spend an hour optimizing. Otherwise, maybe wait.


• Put a ring buffer on it. Nice article on  Ring buffers / circular buffers : You can imagine a ring buffer as a circular block of memory that cycles around like this (the numbers represent the indices into the block of memory, which is treated as an array of bytes).


• In the age of whitebox this and merchant silicon that it might help to know a little about ASICs. Here’s an excellent talk by Dave Zacks on what ASICs are all about:  Cisco Enterprise ASICs Discussion . It’s a good bet next gen 1Tbps network routers won’t be using merchant silicon.



• Baron Schwartz with another great free book:  Best Practices For Architecting Highly Monitorable Applications : This ebook collects the experience of a variety of experienced architects and combines it with what customers have taught us about monitoring at VividCortex. My hope is that you will be able to apply the best practices in this book to avoid pitfalls later, and create a highly monitorable architecture for your application, so you can get excellent visibility with minimal cost and effort.

• This is clever. Not as good as Comcast turning all our routers into open wifi access points, but clever.  Windows 10 uses your bandwidth to send other people updates . They are mixing in a little Skype magic by using their customer’s computers as part of a peer to peer network to deliver updates to other customers. The new new dictatorial Microsoft has an empathy problem. Not everyone has fast connections and lots of bandwidth to give, for free, to Microsoft. 


• Backblaze gives their  Hard Drive Reliability Review for 2015 . 8TB drives are a thing. 4TB drives are cheaper than 6TB drives for now. HGST seems to be the most reliable. 


• Migrate GAE Datastore to SQL using Golang. GAE wins on scalability, but usability suffers. Loading data into memory to perform your own joins and nested queries gets old. So here’s how you move out of GAE to Google Cloud SQL. 


• And here’s  Migrating from App Engine to Kubernetes : So, now that my app is running on Container Engine, I have effectively the same functionality as I did on App Engine. The performance different is negligible (about 2ms faster than App Engine), and now instead of falling under the App Engine free tier, I’m paying about $15 a month to run a single g1-small Kubernetes node. This isn’t really better in any way except that I’m no longer restricted to the App Engine APIs and I can scale up very quickly. As I said at the beginning, this was mostly a learning experience, but overall it feels like a more flexible way of running an app. 


• A clear walk through on  Scaling from 1 to 100K+ connected devices with IoT on Bluemix .


• A wonderful talk by Alex Russell on The Future of the Web Platf orm: Does It Have One? ~ It’s not the technology that decides where we are going. It has nothing to do weather Java is garbage collected. It has nothing to with one having a more elegant threading model. Or that jit is more portable. All these are details that don’t matter to the end experience of users when deciding what users are going to use next. How do they want to compete? Chrome supports: Add to Home Screen, Service workers, push notifications, ability to work offline, ability to use manifests.


• More Chinese Mobile UI Trends : Today, China is not yet making the operating systems, frameworks, programming languages, or open standards that the rest of the industry is using. Yet when I think about it, neither is most of Silicon Valley…But it’s about to get a lot weirder. While China is an interesting case, we’ve still got places like India and large parts of Africa that are still far behind. Global internet penetration is still low. And as more people come online in those places, the apps and OSes they use may in no way resemble what we use now.  


• Yes it is. Awesome Elasticsearch : A curated list of the most important and useful resources about elasticsearch: articles, videos, blogs, tips and tricks, use cases. All about Elasticsearch!


• Is There An App Divide? : In short, rather than trying to export mobile apps to established economies, aspiring app developers should turn their attention to selling physical goods through mobile apps and websites within their home markets. In such markets mobile phones are already the preferred consumption device, giving them a solid base of consumers.


• A huge sucky part of any data warehouse is integrating in all the different data sources. It’s no different for streaming solutions.  Kafka Connect : a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Also,  Evolution of the Netflix Data Pipeline . 


• Codedeploy Notifications as a Service from Gilt: our codedeploy notifications service gets deployments directly from Codedeploy and relies on a number of different channels – e.g., SNS, SES, Newrelic, Hipchat – for sending out deployment notifications. These channels are implemented and plugged in as we need though, so not really part of the core of our service. Dynamo DB is used for persisting registrations


• Measuring Docker IO overhead : I’m not going to show the final numbers or charts, as the results are identical for all docker modes and for the bare metal case. So I can confidently say there is NO IO overhead for any docker data volume pattern described above.


• Brave New Geek takes on  Benchmarking Message Queue Latency : I think these results show how much we can miss by not looking beyond the 99th percentile. In almost all cases, everything looks pretty good up to that point, but after that things can get really bad…I think the key takeaway is to consider your expected load in production, benchmark configurations around that, determine your allowable service levels, and iterate or provision more resources until you’re within those limits. 


• Robots aren’t like any other technology, so projecting from the past is faulty.  Technology has created more jobs than it has destroyed, says 140 years of data : The dominant trend is of contracting employment in agriculture and manufacturing being more than offset by rapid growth in the caring, creative, technology and business services sectors,” they write. “Machines will take on more repetitive and laborious tasks, but seem no closer to eliminating the need for human labour than at any time in the last 150 years. 


• If you are looking for a high level comparison of features from various cloud platforms then  Cloud Computing Platform might be of interest. 


• A useful   6-Part Guide to NGINX Application Performance Optimization that covers Time Spent Analysis; Reverse Proxy Buffering; Load Balancing; Throughput; Connection Optimizations; Content Caching, Relay, and Micro-Caching.


• The Morning Paper looks at How Complex Systems Fail . The paper consists of 18 observations: Complex systems are intrinsically hazardous, which drives over time the creation of defense mechanisms against those hazards. (Things can go wrong, and we build up mechanisms to try and prevent that from happening).


• Pinterest is open-sourcing Teletraan , their code deployment system: it deploys hundreds of Pinterest internal services, supports tens of thousands hosts, and has been running in production for over a year. It empowers Pinterest Engineers to deliver their code to pinners fast and safe. 


• Stemming the Flood – How to Rate Limit an API : One of the most common and easy ways to do so is to use internal caching on the server…As another implementation, Redis utilizes rate limit patterns…Simply, this limits the interactions with the server as in the other examples, but also creates a dynamic counter for each connected IP, expiring every 10 seconds, and tracking the utility rates of the connected IPs.


• Design Principles for Scaling Multi-core OLTP Under High Contention : In this paper we identify two prevalent design principles that limit the multi-core scalability of many (but not all) transactional database systems on contended workloads: the multi-purpose nature of execution threads in these systems, and the lack of advanced planning of data access