hot links

Stuff The Internet Says On Scalability For September 28th, 2018

High Scalability

High Scalability

— 17 min read

Hey, it's HighScalability time:

@danielbryantuk: "A LAMP stack is a good thing. Never inflict a distributed system on yourself unless you have too..." @mipsytipsy #CloudNativeLondon

Do you like this sort of Stuff? Please support me on Patreon and you'll get 100 free cloud credits in heaven. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

  • $2 billion: Pokémon GO revenue since launch; 10: say happy birthday to StackOverflow; $148 million: Uber data breach fine; 75%: streaming music industry revenue in the US;  5.2 TB: Fastly peak per second traffic; 10 billion: Ethereum requests per day; 01%: DNS resolution issues when the KSK rolls; 15B: projected gaming community views on Reddit; £4.1bn: saved by UK Government's Digital Transformation Journey; 10X: Core ML model runs  faster on the A12 processor; 4 million: cores managed by Open Stack at Yahoo; 1PB: Azure's data box; 21 million: US Apple music subscribers; .675: Curry's league leading true shooting percentage;  $3 trillion: taxes collected by IRS; 61,000: network of Mayan structures discovered using lidar; 90%: China's percentage of $4.2 billion increase in pure-play foundry market; 

  • Quotable Quotes:
    • WhatsApp cofounder: I am a sellout. I acknowledge that.
    • MrTonyD: I was writing production code over 30 years ago (C, OS, database). It is much worse to be a software developer now. It used to be a very high autonomy job - where you were trusted to figure out your work process and usually given lots of freedom to dynamically define many of your deliverables (within reason). I remember when I first read about Agile - I looked at the practices and thought "I've done most of those." But when our nightly builds broke it was no big deal, we would just fix them when we got around to it (as opposed to managers now assigning blame and calling in people on weekends to "fix" it). And if things weren't going well then we might have daily brief meetings for a couple of weeks. But now there are managers who insist on daily standups irregardless of their actual business necessity. I could go on and on. There is a reason why I'm not a practicing programmer anymore - even though I love to code.
    • Tiger Woods: "This was different," Woods said, acknowledging that the scene on the final hole was unlike anything he experienced in his previous 79 career victories. "I guess it’s different now because the art of clapping is gone. You can’t clap with a cell phone in your hand, so people yell."
    • @tomgara: "Slate makes more money from a single article that gets 50,000 page views on its site than it does from the 6 million page views it receives on Apple News in an average month"
    • Steve Case: We are seeing the beginnings of a slowing [in Silicon Valley] of what has been a brain drain the last 20 years. It’s not just watching where the capital flows, it’s watching where the talent flows. And the sense that you have to be here or you can’t play is going to start diminishing.
    • Martin Sústrik: Philosophers, by and large, tend to be architecture astronauts. Programmers' insight is that architecture astronauts fail. Or, maybe, they can succeed as in getting comfy job at IBM, but their designs don't translate into useful products. What else? After decades of pain, we have finally concluded that hierarchies of concepts don't work. That's not an obvious statement at all. After all, nobody gets fired from creating a hierarchy. 
    • vl: I have a hilarious story about this from Google: I wanted second 30" monitor, so I filed a ticket. They sent me long email listing reasons why I shouldn't get a second monitor, including (numbers are approximate, employee count from 2013 or so) "If every googler gets an extra monitor, in a year it would be equivalent to driving Toyota Camry for 18000 miles."
    • @dhh: The iPhone XS is faster than an iMac Pro on the Speedometer 2.0 JavaScript benchmark. It's the fastest device I've ever tested. Insane 45% jump over the iPhone 8/X chip. How does Apple do it?! 
    • @davidcrespo: how is this different from making entries in a traditional database how is this different from making entries in a traditional database how is this different from making entries in a traditional database how is this different from making entries in a traditional database  re:@marshallk: Wow: Walmart will *require* food suppliers to upload data to IBM's blockchain in order to trace quality issues through the supply chain. They say issues that take 7 days to track today can be tracked in 2.2 seconds with the system. 
    • @DrQz: Somewhat more revealing USL curves that go BEYOND the performance measurements (also corrects previous legend): • XDP-1 saturates single NIC at ~4 cores • DPDK scales "infinitely" but reaches NIC satn at ~13 cores • XDP-2 peaks before satn at ~15 cores due to non-zero β term.
    • Mark Szulyovszky: Would I have thought 3 years ago that we end up building a mobile tech stack on Microsoft’s & Facebook’s open source code, wishing that Google’s and Apple’s clunky and slow middle layer would disappear? No way.
    • Jeff Barr: We’re [AWS] not stopping at 12 TiB, and are planning to launch instances with 18 TiB and 24 TiB of memory in 2019.
    • Robert Gutman: The hyperscale leasing has been through the roof. 
    • Joanne Itow: These high volume semiconductor products are enjoying growing end market applications but the growth is being challenged by capacity shortages and rising material costs. Companies are choosing different strategies to address these conditions resulting in a variety of fluctuations in the market.
    • Kevin Fogarty: the market for deep-learning chipsets will rise from $1.6 billion in 2017 to $66.3 billion by 2025, which includes CPUs, GPUs, FPGAs, ASICs, SoC accelerators, and other chipsets.
    • Vadim Tkachenko: the message I am trying to delivery is that MyRocks shows the better throughput on limited cloud resources, but also it is more cost efficient comparing to InnoDB
    • Erich Sanchack:  We anticipate that, over time, air-cooled systems will be phased out and there will be a return to water. We are actively transitioning air cooling to water across our portfolio, at both the storage and chip levels, and using them to cool individual racks of up to 90kWh.
    • Shaun Nichols: US JEDI military cloud network is so high-tech, bidders will have to submit their proposals by hand, on DVD
    • Oren Eini: So no, I utterly reject the idea that partitions are rare events in any meaningful manner. Sure, they are rare, but a million to one event? We can do million packets per second. That means that something that is incredibly rare can still happen multiple times a day. In practice, you need to be aware that your software will be running in a partition, and that you will need a way to handle that.
    • DanNeely: CPU cores have been a minority of the die area on most consumer facing CPUs for a long time. For complex SoC's this has always been the case because you've got so many not-CPU things packed in. It's also been the case in the PC world since the CPU ate the north bridge and brought the dram controllers and IGPU onboard. 
    • Hamza Shaban: Google’s former chief executive predicts that the global Internet will split in two within the next decade, bifurcating into a Web led by the United States and another helmed by China with fewer freedoms and greater censorship
    • Steve Kerr: One of the things that I learned from Phil was how important it was being funny watching game film, editing stuff in from movies. Nobody I had ever played for had ever done that, and, to me, that was such an effective way of getting a message across. When you could tie together the point you’re trying to make on the basketball floor with a humorous message coming from a movie — when the message is clear and it carries over to what you’re trying to teach — you’re not having to either kiss up to the player or criticize them.
    • Jeff Dean: George Dahl and others in our research group worked with some chemists at different universities and took the simulator they were using, trained a neural net to do it. And now they can't distinguish the accuracy versus the original, sort of, HPC-style simulator, but the tool is 300,000 times faster. And any time you take your tool, and you make it 300,000 times faster, that just transforms how you would do science. Right? You can imagine, I'm going to go make coffee. Let me screen 100 million molecules, and then when I come back, I'll look at the 10,000 that are most interesting.
    • Brian Bailey: I just saw a worrying figure from Semico. They predict that SoC design costs will only rise by only 1.7% by 2023. If that is true, then I hope engineers suddenly became a lot more productive, because that figure suggests to me that the market is not yet looking at the future aggressively enough.
    • Jeff Dean: And I think there's huge potential for the thousands, and thousands, and thousands of heuristics in computer systems to really mostly be learned and adapt to how they're actually being used.
    • w8rbt: Yep. No one telling me no anymore and I can write Lambas to replace cron jobs, use RDS to replace DBs, use S3 and Glacier to replace Storage, etc. Fargate is awesome too. No gate keepers nor bureaucracy just code and git repos. That's why AWS is so awesome. And, I can show exactly how much everything costs. As well as work to reduce those costs. AWS added to a small, skilled dev team is a huge multiplier.
    • Ev Williams: Roughly speaking, there are two engineering cultures in Silicon Valley, which you could describe as hackers and engineers. And obviously Google is engineers. Engineers generally have computer science degrees and higher. They study the fundamentals. Hackers just want to make stuff work, and it’s not about doing it right necessarily. Facebook was kind of famously built by hackers, but they’re not the hippie hackers. Hippie hackers are a particular strain of hackers
    • DSHR: The legal system is gradually building trust in the evidentiary value of the Wayback Machine by requiring personal testimony from the people operating the underlying processes, and exposing these witnesses to cross-examination. Systems based on blockchains would have to undergo a similar process; those using public, permissionless blockchains would have difficulty doing so since it would not be possible to obtain testimony from "the people operating the underlying processes". In the case of ARCHANGEL, which is intended to use a permissioned blockchain, testimony would be required from both the operators of the network nodes and from the archives generating the hashes to be injected into the chain. Mere public confidence would, and should not, be sufficient.
    • spullara: It sounds very similar to the Rails frontend I helped replace at Twitter — no business logic was thrown away, it was done without any loss in application fidelity. We did get approximately 10x fewer servers with 10x less latency after porting it to the JVM as a result. However, without the latency improvement, I don't think we should have done it. Fewer servers just isn't as important as the developer time necessary to do the change as you just pointed out. Just as using the cloud to simplify things and reduce the amount of effort spent on infrastructure is the main driver of adoption. There is clearly a cross-over point though where you decide it is worth it. The CTOs you are speaking of are making that choice and it probably isn't a silly excuse.
    • Jeff Dean: So the TPU v3, which Sundar announced it at Google I/O in May 2018, essentially got water cooling. So a board has four of these chips on it, and the water cooling kind of goes to the surface of these chips and takes excess heat away. And the largest footprint deployment, we call these things pods, the TPU v2 pods were essentially made up of 64 of those devices, 256 TPU v2 chips. And the pod scale for the TPU v3 is much larger. So we have about eight times as much computation in the TPU v3 pod as we did in the TPU v2 pod.

  • Gather around my friends. Listen to ancient words of wisdom. What are some things that only someone who has been programming 20-50 years would know?: Everything in software development has already been invented. People just keep rediscovering it and pretending they invented it...Don’t trust the compiler. Don’t trust the tools. Don’t trust the documentation. Don’t trust yourself...We don’t need any more computer languages...Maintaining code is harder than writing it...You have been taught to program as though memory, processor time, and network bandwidth are all free and infinite. It isn’t, it isn’t, and it isn’t...You’re going to forget what your code does in a few months...Beware of programmers who speak in absolutes. Along the same lines is this oldy but goody: Notes on Distributed Systems for Young Bloods. Good comments on reddit and on HN. 

  • What happens when the AIs get tired of humans stealing their jobs? Skynet. AI Company Accused of Using Humans to Fake Its AI.

  • The challenge with health checks — striking the balance between failure detection and reaction. Patterns for Resilient Architecture — Part 3: There are two types of failures that can impact a server instance: local or host failures...A major issue with health checks occurs when server instances make the wrong local decision about their health — and all of the other hosts in the fleet make that same wrong decision simultaneously — causing cascading failures throughout adjacent services. To prevent such a case, some load balancers have the ability to act smart: When individual server instances within a fleet fail health checks, the load balancer stops sending them traffic, but when server instances fail health checks at the same time, the load balancer fails open, letting traffic flow through all of them

  • Videos from Altitude San Francisco 2018 are now available. You might like BBR congestion control: a new model-based approach or Scaling Ethereum to 10B requests per day. 

  • Yes, building your own is almost always cheaper—if you can. Why building your own Deep Learning Computer is 10x cheaper than AWS.  Good discussion on HN. scosman: Missing 1 important point: ML workflows are super chunky. Some days we want to train 10 models in parallel, each on a server with 8 or 16 GPUs. Most days we're building data sets or evaluating work, and need zero. When it comes to inference, sometime you wanna ramp up thousands of boxes for a backfill, sometimes you need a few to keep up with streaming load. Trying to do either of these on in-house hardware would require buying way too much hardware which would sit idle most of the time, or seriously hamper our workflow/productivity.

  • Cloudflare's mission is to help build a better Internet. Google's mission is to organize all the world's information. It's interesting to see how aiming at different levels of the stack causes different decisions to me made. Introducing the Bandwidth Alliance: sharing the benefits of interconnected networks. What is Cloudflare offering? Free egress: So today — on the eve of Cloudflare's 8th birthday — we're excited to announce the Bandwidth Alliance: a group of forward-thinking cloud and networking companies that are committed to providing the best and most cost-efficient experience for our mutual customers...We are proud to announce the following cloud providers and hosting companies who have joined together with us in committing to reduced or zero bandwidth rates for mutual customers...To give you some sense, we estimate that current Cloudflare customers could save nearly $50 million in data transfer fees per year from hosting with a Bandwidth Alliance member as these programs come online. Also, Building With Workers KV, a Fast Distributed Key-Value Store.

  • Another win for Postcentralization. AWS wants to put their whitebox NAS in your datacenter. New – AWS Storage Gateway Hardware Appliance. Dog not included. 

  • Interested in a new stack? Two years of Elixir at The Outline: We chose Elixir and Phoenix as the foundation of our website after being attracted to its concurrency model, reliability, and ergonomics...90ms is a 90th percentile response time on The Outline...We got this performance out of the box, without really any tuning or fine-grained optimizations...Elixir is so fast that we haven’t had much need for CDN or service level caching...Before reaching for Vue.js or Svelte, consider going old school and rendering your HTML on the server; you might be delighted...My favorite part about Elixir documentation is the link right back to the source code. The way I usually read documentation is by trying to understand it through the text, and the if something isn’t clear I click the link to the source code and follow the code directly...Phoenix Channels are such a great abstraction over WebSockets...The Community is wonderful!...Stack traces are not always the best...Sometimes tests throw random warnings that I don’t know how to trace...Working with Ecto Associations are still hard...You usually don’t need a GenServer!

  • Looking for datasets? Try Google Dataset Search. 

  • Scaling ETL: How data pipelines evolve as your business grows: Our new architecture extracts data into a data lake on Amazon’s S3. Then, our scheduler — Azkaban — transforms the data for analysis. Azkaban both populates the raw data lake as well as our data warehouse, for which we use Hive. We have an excellent team of data engineers that works closely with our business users to come up with a defined set of transformations to present as our standard business object. This allows us to decouple the source ingestion from the business level transformations. Allowing one set of engineering to concentrate on optimizing the flow from our production system into Hive tables and another set of engineers working on optimizing the data for the business. Also this means that as services are spun up on our excellent microservice architecture the access to source data is almost immediately available. This allow for insights into the data before engineers can transform it. Mitigating the downsides of the traditional ETL where business users would have to wait week or even days before measuring a new piece of the business.

  • Ben Kehoe, Cloud Robotics Research Scientist, Discusses Serverless @iRobot: Cloud-enabled Roombas talk to the cloud via the IoT Gateway (which is MQTT) and are able to perform large file uploads using mutually authenticated certificates signed via an iRobot Certificate Authority. The entire system is event-driven with lambda being used to perform actions based on the events that occur...Serverless is all about the total cost of ownership. It’s not just about development time, but across on areas that need to support operating the environment...iRobot uses Red/Black deployments to have a completely separate stack when deploying. In addition, they flatten (or inline) their function calls on deployment. Both of these techniques are used as cost optimization techniques to prevent lambdas calling lambdas when not needed.

  • We don't hear all that much about Alibaba. Here's a good article Comparing Serverless offerings from Alibaba Cloud and AWS. Features are comparable if not favoring Alibaba. Available services are dominated by AWS. 

  • A firmware bug on one of the IRS’s high-availability storage arrays caused an 11 hour outage at the IRS. There was a known fix, but because of internal rules about running code in production for 450 machine weeks at other sites before it can be installed at the IRS, the fix couldn't be installed. A sensible rule. The real problem was the SPoF. Review of the System Failure That Led to the Tax Day Outage: The root cause of the Tax Day outage was a cache overflow issue causing repeated warmstarts and had deployed a preventive script on the storage device that will monitor and correct the issue if the condition reoccurs...The Internal Revenue Service (IRS) relies extensively on information technology systems to annually collect more than $3 trillion in taxes, distribute more than $400 billion in refunds, and carry out its mission of providing service to America’s taxpayers in meeting their tax obligations. In Fiscal Year1 2017, the IRS processed more than 187 million Federal tax returns. Of those, over 150 million were individual returns, with over 87 percent of these individual returns filed electronically. During Calendar Year 2018, the IRS expects to receive approximately 153.7 million individual income tax returns, with more than 89 percent of those filed electronically...this new environment provides an agile solution for rapid deployment of continuous improvements, optimized data storage and retrieval, and flexibility to adapt to IRS growth. The IRS environment consists of 7.5 petabytes of disk space in various configurations in data centers and other facilities across the country...Unisys follows an internal company policy that requires microcode bundles to have, at a minimum, 450 machine weeks in a production environment prior to installation on IRS equipment...The Tier 1 storage architecture consists of seven high-speed, high-availability storage arrays with an average age of less than three years. Presently, three of the storage arrays, located at the Martinsburg Computing Center, host a portion of core tax processing data. However, there are no automatic failovers or built-in redundancies for these three storage arrays. Additionally, not all of the applications and systems data hosted in the Martinsburg Computing Center are fully replicated to the Memphis Computing Center.  

  • Wonder if Turing ever read The Sandman, a short story written in 1816 about a man who falls in deeply in love with an automaton? Long ago people realized we humans are not all that good at telling us from them. Or does it say more about men who prefer to fall in love with a reflection of themselves?

  • Old Tech Sounds: hand mixer; 10 key; stareoscope; bolex.

  • Time to think of each house an edge availability zone in which all IoT stuff runs? Honeywell’s 'smart' thermostats had a big server outage and a key feature stopped working entirely — and customers were furious. I pulled our smart thermostat long ago for just his reason. When it's 105 degrees outside and your local device doesn't work because of remote problem, you get steamed. There's no reason for this code to run in a datacenter. I could run on a box in a house that's administered remotely.

  • Very thorough. Lots of insights. An Introduction to Serverless Microservices. 

  • An advantage Microsoft has with VSCode is they can build first class tooling into an already excellent program. Have to wonder why Apple is letting all that backend iOS revenue slip through their digital fingers? Serverless with Microsoft Azure Functions: Using a combination of the local runtime and the extension, you can create a function project, add some functions, test and debug them locally, and then publish them to Azure — within just a few clicks or commands...Azure Function Bindings offers developers a simple way to integrate with a bunch of services...One of the really nice features is that ability to configure our bindings right alongside our function code in a function.json file...With Azure Functions, this feature comes out of the box. You just need to provide your function a HTTP Trigger, and then you can invoke it straight from the web...Function Proxies provide developers the ability to have a lightweight API gateway — right within your Function App.

  • Dropbox search is the opposite of Google, each customer is like their own internet. The result is a different approach. Architecture of Nautilus, the new Dropbox search engine: Nautilus consists of two mostly-independent sub-systems: indexing and serving...We generate “offline” builds of the search index on a regular basis (every 3 days, on average)...As users interact with files and each other, such as editing files or sharing them with other users, we generate “index mutations” that are then applied to both the live index and to a persistent document store, in near real-time...We group a number of namespaces into a “partition,” which is the logical unit over which we store, index, and serve the data. We use a partitioning scheme that allows us to easily repartition namespaces in the future, as our needs change...For most documents, we rely on Apache Tika to transform the original document into a canonical HTML representation, which then gets parsed in order to extract a list of “tokens” (i.e. words) and their “attributes”...The underlying technology has extra custom built layers that provide access control and data encryption. It contains one row per file, with each column containing the output from a particular extractor. One significant advantage of this schema is that we can easily update multiple columns on a row in parallel without worrying about changes from one extractor interfering with those from others...The main advantage of using an ML-based solution for ranking is that we can use a large number of signals, as well as deal with new signals automatically.

  • Great explanation with nifty graphics. Accurately measuring layout on the web: it’s important to understand how the HTML5 event loop works, and to place performance marks at the appropriate points in the component rendering lifecycle. This can help avoid any mistaken conclusions about what’s “slower” or “faster” based on an incomplete view of the pipeline, and ensure that style and layout costs are accounted for.

  • Why is software hard? Look at what it takes just to input an address. How Etsy Localizes Addresses. 

  • The EwoK microkernel: a microkernel targeting micro-controllers and embedded systems. It aims to bring an efficient hardening of embedded devices with a reduced impact on the device performances. EwoK has been designed to host complex drivers in userspace.

  • facebookincubator/LogDevice:  scalable and fault tolerant distributed log system. While a file-system stores and serves data organized as files, a log system stores and delivers data organized as logs. The log can be viewed as a record-oriented, append-only, and trimmable file.

Read more