Stuff The Internet Says On Scalability For April 27th, 2018

Hey, it's HighScalability time:

Did ancient Egyptians invent Wi-Fi? @sherifhanna

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• $17,500: price to give up Google search; 51.8,31.2,18.79: % using AWS, Azure, Google for IoT; 400: items per second shipped by peak Amazon; 43%: music revenues came from streaming; 800%: boost in downloads from apps featured by the Apple App Store; 45: average age of startup founder; 


• Quotable Quotes:
  • Broad Band: By the mid-twentieth century, computing was so much considered a woman’s job that when computing machines came along, evolving alongside and largely independently from their human counterparts, mathematicians would guesstimate their horsepower by invoking “girl-years,” and describe units of machine labor as equivalent to one “kilogirl.”
  • thegayngler: Most engineers would not hire themselves. That has been apparent to me for awhile now. I’m not sure why they expect people to be to be better than they were when they were hired. I don’t expect engineers to be better than me. I have but one qualification. Can they do the job? Are they strong enough that I can guide them into the position I need them at if it is required.
  • Where Wizards Stay Up Late: Heart liked working with small, tightly knit groups composed of very bright people. He believed that individual productivity and talent varied not by factors of two or three, but by factors of ten or a hundred. Because Heart had a knack for spotting engineers who could make things happen, the groups he had supervised at Lincoln tended to be unusually productive.
  • @AllenDowney: Can someone explain why, if you write an idea in math notation, that's "theory", which provides deep understanding of the math "behind" it, but if you write the same idea in a programming language, it's just hacking? This bizarre prejudice is the bane of my professional life.
  • @kwchang: 'The internet went from a democratizing free space to having power very centralized; crypto decentralization is a reaction to that' - @starkness #Angels #cryptointro
  • Lynn Langit: To me, containers are the new VMs. All this frenzy about containers, and more specifically container management systems — look, somebody has to manage the things. I want to pay the cloud providers to do it so I don’t have to.
  • @kellabyte: Many-core servers are a huge problem. We have no idea how to write software to actually use the hardware properly. For example, in Go, there’s no IO library that can go faster than 10GbE.
  • Markus Winand: Don’t say relational database when referring to SQL databases. SQL is really more than just relational.
  • Catalin Cimpanu: A loud sound emitted by a fire suppression system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe.
  • dmoy: My mother in law is already paranoid, warning us not to talk about certain topics with her. Scary stuff. I should point out that my MIL is a completely apolitical person who worked in the Chinese government for her entire career, spending most of that time just helping poor people, without a shred of corruption. She has nothing to worry about, doesn't care about politics, and even she's paranoid about this.
  • vkjv: +1 for Neon! The best part about Rust is that it changes this question to "what language AND Rust?" Node + Rust is a great experience.
  • Errata Security: Ozzie pretends the [crypto backdoor] problem is political, that he's created a solution that appeases both sides. He hasn't. He's solved the problem we already know how to solve. He's ignored all the problems we struggle with, the problems we claim make secure backdoors essentially impossible.
  • @brendangregg: context switching & CPU cache invalidations are both CPU utilization. I'd look for single-threaded/single-CPU bottlenecks first, then off-CPU analysis. 
  • @swardley: In other words, if I'm right, then somewhere in Amazon and China is a group of Deng Xiaoping's playing havok with the world. I've noticed a set of misdirection plays which ... well ... if it's luck, I find that difficult. Why does it matter? Well a properly formed ecosystem gives advantages of efficiency, customer focus and innovation ... if you know how and where to play it. Learning context will take such simple gameplay to a whole new level of deadly.
  • @PeterGleick: Do people understand the magnitude of this? This is the power equivalent of 10 giant nuclear plants brought on line in three months.
  • @jessitron: The speed of light is not in a config file. @KevlinHenney Don’t put something out on the network if you can avoid it. #GOTOchgo
  • @jimbojsb: CTO’s Prayer: God, grant me engineers to build the things I can’t buy, budget to buy the things I can’t build, and the wisdom to know the difference.
  • Brendan Burns: As we move into the future, I’m fully convinced that future Kubernetes clusters will contain a mix of containers running on dedicated machines as well as bursting into serverless infrastructure.
  • @davetron5000: My company went from 0 to IPO in five years and have had 50-75% remote devs the entire time. Never touched a whiteboard in that time. The world is what you make it.
  • Richard Hamming: And I started asking, ``What are the important problems of your field?'' And after a week or so, ``What important problems are you working on?'' And after some more time I came in one day and said, ``If what you are doing is not important, and if you don't think it is going to lead to something important, why are you at Bell Labs working on it?'' I wasn't welcomed after that; I had to find somebody else to eat with! That was in the spring.
  • Lolimaster: Unless you work moving huge chunks of data (editing 4k for example) a lot there's no point going NVME over the Crucial MX500 sata.
  • @clemensv: I've talked to a lot of distributed systems engineers (who build cloud-scale stuff) from across the industry about blockchain. While most platform folks I talked to are perfectly happy to help with frameworks that help selling product or services, and some even rode the crypto wave to make some hay, I have a hard time finding people in the distributed systems platform community who believe that blockchain is even remotely as significant as the hype wants to make us believe.
  • Quirky: Outsiders are important to innovation; they often operate in fields where they are highly motivated to solve problems in which they are personally invested. They often look at problems in different ways from those who are well indoctrinated in the field, and they may question (or ignore) assumptions that specialists take for granted.
  • dis-sys: When the $ spent on importing chips is more than the amount spent on importing oil, you know full well that that there is no real alternative but to invest heavily into the semiconductor industry. In 2016, $227 billion worth of chips were imported by China [1].
  • shadowtree: Nice thing is you can use this in an offensive manner too. Japanese forums have been raided by Chinese trolls in the past. Now the Japanese just post a statement about Tienanmen square and poof, no more Chinese attendance, as the Great Firewall starts blocking...Interesting times.
  • kllrnohj: Except shared data doesn't give you a little bit better performance, it gives you massively better performance. Or, in some cases, it's the only way to get usable performance at all. Now what you could do is break objects down into annotated types. Consider immutable vs. mutable in combination with thread-unsafe, thread-compatible, and thread-safe. Immutable data that's not thread-unsafe you can share freely across threads, all is well. L2/L3 caches are happy. Mutable that's thread-safe can similarly be shared at will. Then you can force that thread-compatible objects be wrapped & accessed only from a Mutex or transfered between threads as part of a move operation. Rust gives you the tools to do all of this, and indeed does some of it, but as part of the steep learning curve of the ownership model.
  • Michael Cohn: The IRS got the system back up and running by 5:00 p.m. when it sent an email to tax professionals. It was down for about 11 hours. The problem seems to have originated with a piece of hardware running the IRS’s master file system that stores taxpayer information. The system was running assembly language, computer code programmed for specific hardware.
  • Errata Security: Some of the actual lessons should be things like how Stuxnet crossed air gaps, how Wannacry spread through flat Windows networking, how Heartbleed comes from technical debt, and how Shamoon furthers state aims by causing damage. But this article doesn't cover the technical lessons. Instead, it thinks the lesson should be the moral lesson, that we should take these things more seriously. But that's stupid. It's the sort of lesson people teach you that know nothing about the topic. When you have nothing of value to contribute to a topic you can always take the moral high road and criticize everyone for being morally weak for not taking it more seriously. 
  • ChuckMcM: Ah a full stack failure. That is when you decide to change (or implement) everything in the full stack from raw material to consumer in one go. As a "go to market" strategy it is usually fatal. In the technology business it is a company that makes their own CPU, their own operating system, and then opens their own retail stores for selling their computers with their own software. Plastic Logic failed this way when instead of just marketing their screens they tried to build an entire reader (screen/case/os/etc).
  • Corey Quinn: AWS billing is getting much more understandable, with the advent of such things as free Reserved Instance recommendations, the release of the Cost Explorer API and the rise of serverless technologies. For their part, Google's GCP and Microsoft's Azure learned from the early billing stumbles of AWS, and as a result, both have much more understandable cost structures...In return for this ridiculous pile of complexity, you get something rather special—the ability to spin up resources on-demand, for as little time as you need them, and pay only for the things that you use. 


• At least it wasn't a configuration error. New code changes hammered the database which caused all hell to break lose. As usual, just figuring what the problem was was the biggest problem. Fortnite with an excellent POSTMORTEM OF SERVICE OUTAGE 4/11/2018 - 4/12/2018.
  • The Mongo database that backs our account service experienced severe performance degradation, which in turn caused our account service to be unavailable...We had several call pattern changes in Fortnite 3.5 that slowly, over time, impacted database cache utilization...A specific change in a call pattern caused us to reach new cache limits and cache evictions, which resulted in increased I/O to the point of performance degradation. Our short connection timeouts to Mongo resulted in rapid connection churn and when the database performance became degraded, the number of connections to the database from the application spiked rapidly. These spikes caused us to hit maximum configured connection limits. When maximum connections were reached, even intra-replset communication failed causing nodes to drop in and out of a healthy state. These connection storms caused a cascading failure across our Mongo replica set.
  • They moved to MongoDB v3.4 to try and take advantage of many of its performance improvements. They corrected a code path that pinned specific queries to the primary node. Other specific tunings allowed them to increase performance to the point of recovery and mitigate the circumstances that had resulted in cascading failures with the data store. They also removed some newly introduced excesive database calls. The moved DB update calls to Redis to take pressure of the database. They improved caching to remove more database load. 
  • What's next? Correct our I/O monitoring. Put in alarming on early indicators of DB cache exhaustion. Improve the speed of adding and replacing replica sets nodes. Continue with optimizing DB access patterns. Working with the MongoDB team on defining an optimal data sharding strategy. Develop a runbook to test our cold start process during high traffic periods can improve our recovery time. Improve our analysis of API calls per release. Continue working with the MongoDB team on database performance and reliability.
  • Didn't see a mention of improving testing. Seems like this kind of failure should have been caught during testing. Maybe record different traffic scenarios and replay them during system testing?


• If you've ever wanted a detailed explanation of how Redis Cluster works then Scaling a High-traffic Rate Limiting Stack With Redis Cluster is for you. Stripe runs many hundreds of thousands of operations per second on one node. The verdict: Philosophically, there’s a lot to like about Redis Cluster’ design – simple, yet powerful. Especially when it comes to distributed systems, many implementations are exceedingly complicated, and that level of complexity can be catastrophic when encountering a tricky edge in production. Redis Cluster is scalable, and yet with few enough moving parts that even a layperson like myself can wrap their head around what it’s doing. Its design doc is comprehensive, but also approachable. In the months since setting it up, I haven’t touched it again even once, despite the considerable load its under every second of the day. This is a rare quality in production systems, and not found even amongst some of my other favorites like Postgres. We need more building blocks like Redis that do what they’re supposed to, then get out of the way.


• How do you define senior engineer? Malte Ubl has a thoughtful answer: What I think being senior means is that I’d be able to solve almost every problem that somebody might throw at me. I know my tools, I know my domain. And the other important part of that job is that I make the junior engineers eventually be senior engineers. I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem.


• Build it, ship it, it works. Torii with good Lessons Learned — A Year Of Going “Fully Serverless” In Production. Their goal was to go no-ops path as much as possible. They like Serverless because it lets them sleep at night. Application is divided into three parts: Static websites, Background jobs, API Server. Static websites: Netlify to host these static assets on a CDN, no Nginx/Apache servers needed. API Server: using serverless leads to automatic scalability, high availability and reduces costs dramatically. It also makes things simpler since there are less moving parts: no servers, no load balancers, no auto-scaling groups. An entire Node.js app and package it as a single AWS Lambda function. An API Gateway routes all traffic to it and the Node.js app sees it as a regular HTTP request. Deployments: each deployment creates a new version of the Lambda. Background jobs: scheduled to run by CloudWatch or as a response to events in our system.


• Ever wonder why all the early internet standards like SMTP were federated/distributed/decentralised? I have. Now we centralize everything. When I read Where Wizards Stay Up Late the answer was obvious: there was no centralization. There weren't datacenters full of computers that could provide centralized services. The internet was made of lone computers connected over phone lines. So the only protocols that could be created were federated protocols. Nothing else could have worked.


• Hosting Notebooks for 100,000 Users. Quantopia runs an investment algorithm marketplace on Jupyter Notebook. The problem they have is upyter Notebook is inherently stateful so they can't scale horizontally in the usual stateless way. What they did is clever. They implemented a drop-in replacement for the default FileContentsManager, because state was stored in the file system, and retargeted it to store notebooks in PostgreSQL database. Files are stored in binary blobs in the database. Gives them separate namespace per user, multiple checkpoints per Notebook, and security at rest to protect IP. Have over 450GB of Notebooks. Storing big binary files was not a problem. PostgreSQL handled it perfectly. Most significant issue was the usual running out of database connections using pgbouncer.


• The deepest reason why modern JavaScript frameworks exist: Keeping the UI in sync with the state is hard.


• You know how a few rotten apples spoil the barrel? Maybe a few good automated drivers can improve traffic. Study shows first computer-driven cars will have noticeable impact on rush hour: 
  • According to research from the Schulich School of Engineering, even a few automated vehicles in the mix will influence all traffic for the better, and a smattering of cars controlled by computers will result in faster, smoother commutes for all motorists, no matter who — or what — is driving. “Basically, automated vehicles influence traffic with their good behaviour, because they maintain the ideal distance and speed, and that reflects on the traffic around them by influencing the driving habits of other vehicles in their vicinity,” explains Omid Ebadi, who studied the impact of automated vehicles for his master’s thesis in civil engineering.
  • Have to say this sounds like wishful broken windows type thinking to me, but...
  • In a fixed bottleneck scenario like a collision, equipping only 10 per cent of the vehicles with AACC made a significant difference.
  • An AACC-equipped vehicle can react must faster than a human to the vehicle ahead, needing only 1.5 seconds headway, compared to  2.7 seconds for manually-driven cars, meaning tighter traffic and better utilization of the road capacity.
  • The shockwave caused by a slow-moving car can be almost entirely suppressed by introducing the AACC system in all cars, and vastly reduced when 50 per cent or more cars are AACC-equipped.


• Serverless vs Containers: Comparing your Application Deployment Option. I think people are making this too complicated. If you are building a scalable service then containers on some sort of fabric are great. If you're able to simply consume other scalable services then serverless is the way to go, even if it's on your own on-premise cluster. And can we please just get over the name? Yes, servers are still be used behind the scenes. Who cares?


• The more coupled your services are, the more they are together in failures. Good series on How to build Resilience in large scale Distributed Systems. Covers basic terminology, including Faults vs Failures. Then it goes on to discuss resiliency patterns. Pattern[0] = nocode. Pattern[1] = Timeouts. Pattern[2] = Retries. Pattern[3] = Fallbacks. Pattern [4] = Circuit Breakers. Pattern[5] = Resiliency Testing. Pattern[6] = Rate-limiting and Throttling. Pattern[7] = Bulk-heading. Pattern[8] = Queuing. Pattern[9] = Monitoring and alerting. Pattern[10] = Canary releases. 


• We're always being programmed. We're always being nudged. We're always part of some A/B test. That's the nature of design. What matters is if it's being weaponized for evil or not. Ex-Facebook Executive: “You Don’t Realize It But You Are Being Programmed”. This link is no longer valid. Person said the origin source is Chamath Palihapitiya, Founder and CEO Social Capital, on Money as an Instrument of Change. I haven't listened to the whole thing yet, so I'm not sure if that's true, but the start is very entertaining and insightful.


• It's the wild Westworld out there folks. BGP leaks and cryptocurrencies. 


• How does Stack Overflow do pagination so fast? (good comment thread): We have a custom built .NET application called the Tag Engine which holds post IDs as well as metadata. Think of it as an inverted index that you can use to look up a post ID by its data (such as creation date, tags, score, etc)...So, we take the result set of post IDs and query the SQL database for them...we sort the resulting page of posts again in memory.


• Why aren’t IoT devices more secure? Nearly 22% say they don’t even have security on their to-do list. Many design best practices that raise security levels aren’t followed. Read the rest of the survey, the reason IoT is so insecure is because they do next to nothing to make it secure. Heck, they barely use bug tracking or version control. We may need laws to make the world safe again.


• Didn't really expect big improvements in latency, but bandwidth improvements are still huge. No “Material Difference Between 5G & LTE”: Eric Xu, current Huawei Chairman, concludes "consumers would find no 'material difference between 5G & LTE'...Andrus Anders and Roberto Viola at the EU, as well as Jessica Rosenworcel & Ajit Pai in the U.S., are still lying to themselves and too proud to face their errors...Latency:  Ericsson has promised LTE latency of 9 ms in 2018. AT&T's 5G latency is 9-11 ms...Speed: 4G LTE 2018 is hundreds of megabits, peaking over a gigabit. 90% of 5G on the way is midband, the same hundreds of megabits. Only 10-20% of the first few years will be millimeter wave, often a gigabit. Applications: 5G's main application around the world will be more capacity, a good thing for telcos. 


• Considering a SD-WAN? Here's a good conversation. How does this sub really feel about SD-WAN? Smells_Like_Vinegar: I saved 8000 a month by cancelling my MPLS. Getting a raise for that. The next week, the fiber link (that used to hold the MPLS) went unstable and none of my users at other sites had any idea. They'd have been down under the previous setup. Went from 20 meg to 100 and 150 on two links. All meshed up. Better quality for a lot less $. Easy sell for me. dastylinrastan: The problem with "going SD-WAN" is that it's a nonsense term that's too broad, like "going virtual" or "going cloud". However, it's generally accepted that most of the time it means implementing a method that applies your routing flows using a method other than a traditional routing protocol (BGP/OSPF/etc.) and using L4-L7 information in the decisions.


• Do you need to make a secure IoT device? Microsoft has something that may help. Yes, Microsoft. It's called Azure Sphere. It's a comprehensive stack ranging from a custom chip (MediaTek MT3620), custom Linux OS, software stack, to cloud service. Want something like Apple's Secure Enclave? Azure Sphere has it. Introducing Microsoft Azure Sphere: Secure and power the intelligent edge. The Seven Properties of Highly Secure Devices. And it's completely open and freely available without licensing. What's the power envelope? Didn't see that mentioned.


• What’s New in MySQL 8.0 (article)? A lot. JSON Extended syntax. SQL Window functions. Histograms. Data dictionary in InnoDB. Invisible indexes. Better defaults. InnoDB is significantly better at Read/Write workloads. 


• Something that would have required a large team with $10 million in funding can now be done by a lone programmer in an afternoon. Build a serverless multi-region, active-active backend solution in an hour: The solution is built using DynamoDB Global Tables, AWS Lambda, regional API Gateway, and Route53 routing policies.


• Good to know Google is protecting themselves from hiring people who know too much. Google's "Director of Engineering" Hiring Test. 


• Yah, that's kind of boring. Go's New Brand. 


• SCALING DEDICATED GAME SERVERS WITH KUBERNETES: PART 4 – SCALING DOWN: We’ve successfully containerised our game servers, scaled them up as demand increases, and now scaled our Kubernetes cluster down, so we don’t have to pay for underutilised machinery — all powered by the APIs and capabilities that Kubernetes makes available out of the box. While it would take more work to turn this into a production level system, you can already see how to take advantage of the many building blocks available to you.