hot links

Stuff The Internet Says On Scalability For September 21st, 2018

High Scalability

High Scalability

— 18 min read

Wake up! It's HighScalability time:

Like sparkle from a fairy godmother's wand a swarm of 300 autonomous burner drones take flight. (mnn)

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

  • 60: Marchetti's constant—minutes humans allocate to commuting over history; 300 billion: eBay data queries processed each day;  $1: 1GHz ARM Cortex-A8 processor capable of running Linux; 20M:Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset; 1 billion: Lyft rides; 600+: engineers @ Pinterest (+175B Pins to +250M people); 50%: workplace tasks handled by machines by 2025; 28: years of Hubble Space Imagery on AWS; 500 petabytes: eBay data footprint; 10 million: Gmail spam messages prevented per minute; 5 trillion: Apple's A12 operations per second; $220 billion: investment in new fabs and lines; 43%: Americans who something on Netflix each day; 

  • Quotable Quotes:
    • Linus Torvalds: This week people in our community confronted me about my lifetime of not understanding emotions.  My flippant attacks in emails have been both unprofessional and uncalled for.  Especially at times when I made it personal.  In my quest for a better patch, this made sense to me. I know now this was not OK and I am truly sorry.
    • @jessfraz: To those in my mentions. This "foundation" which has tax breaks... Is paying people who don't contribute to the kernel 400k a year. Would you like to rethink your "they build the kernel" comments.
    • Aria Bracci: Using a previously established, peer-reviewed technique, the team conducted more than half a million data traffic tests across 161 countries. From this data, the team found that internet service providers are “giving a fixed amount of bandwidth—typically something in the range of one and a half megabits per second to four megabits per second—to video traffic, but they don’t impose these limits on other network traffic.”
    • @mjpt777: "Patterson indicated that rewriting Python into C gets you a 50 times speedup in performance".
    • @JoeEmison: The idea that any kind of planning will necessarily lead to overengineering is part of the anti-planning developer culture that misunderstands the agile manifesto.
    • Marie Hicks: The dynamic continues to this day. Silicon Valley reaps enormous profits at the expense of the majority of users, and calls it progress. But technology’s alignment with actual progress has a long and uneven history, and its effects are rarely straightforward or fully foreseen. Real progress isn’t synonymous with building another app—it involves recognizing the problems in our society and confronting the uncomfortable fact that technology is a tool for wielding power over people. Too often, those who already hold power, those who are least able to recognize the flaws in our current systems, are the ones who decide our technological future.
    • @0xmchow: I compiled a very short list to illustrate how long we've been playing this "cyber security skills/workforce shortage" game for [link] …. This game has been going on for a long time and is becoming "boy who cried wolf."
    • Strachey: It has long been my personal view that the separation of practical and theoretical work is artificial and injurious. Much of the practical work done in computing, both in software and in hardware design, is unsound and clumsy because the people who do it have not any clear understanding of the fundamental design principles of their work. Most of the abstract mathematical and theoretical work is sterile because it has no point of contact with real computing.
    • @EmilyGorcenski: Excuse me, I have been a government software contractor and I assure you this is not the largest man-made waterfall
    • Nikita: You don’t have to be a genius to write fast programs. There’s no magic trick. The only thing required is not building on top of a huge pile of crap that modern toolchain is.
    • David Patterson: We are now a factor of 15 behind where we should be if Moore’s Law were still operative. We are in the post–Moore’s Law era...single program performance only grew 3 percent, so it’s doubling every 20 years. If you are just sitting there waiting for chips to get faster, you are going to have to wait a long time...Revolutionary new hardware architectures and new software languages, tailored to dealing with specific kinds of computing problems, are just waiting to be developed. There are Turing Awards waiting to be picked up if people would just work on these things.
    • Vint Cerf: differential traceability. The ability to trace bad actors to bring them to justice seems to me an important goal in a civilized society. The tension with privacy protection leads to the idea that only under appropriate conditions can privacy be violated. By way of example, consider license plates on cars. They are usually arbitrary identifiers and special authority is needed to match them with the car owners ... This is an example of differential traceability; the police department has the authority to demand ownership information from the Department of Motor Vehicles that issues the license plates. Ordinary citizens do not have this authority.
    • Dr Denis Bauer: We started out with one infrastructure [for genetic analysis] and then tweaked it to get better performance and do the analysis we wanted to do. I strongly believe that once you go serverless you never go back.
    • Herb Sutter: Perhaps the most important thing we can do for C++ at this point in its evolution is to make sure we preserve its core strengths while also directing its evolution in ways that make it simpler to use. That is my own opinion at least, and so this talk starts with a perspective question: What “is C++,” really? The language continues to evolve and change; as it does so, how can we be sure we’re picking C++ evolutionary improvements that not only don’t lose its “C++-ic” qualities, but make it a better C++ than ever?
    • S. Brent Rodriguez-Plate: As philosophers like Andy Clark have argued, “our tools are not just external props and aids, but they are deep and integral parts of the problem-solving systems we now identify as human intelligence.”
    • Mark Ames: The secret wage-theft agreements between Apple, Google, Intel, Adobe, Intuit, and Pixar (now owned by Disney) are described in court papers obtained by PandoDaily as "an overarching conspiracy" in violation of the Sherman Antitrust Act and the Clayton Antitrust Act, and at times it reads like something lifted straight out of the robber baron era that produced those laws. 
    • nemothekid: In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases: 1. The host database is written in a GC’d language and experiences a pathological GC pause. 2. The Virtual machine is migrated and experiences a pathological pause 3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?) 4. AWS retires your instance and you need to reboot your VM.
    • @GossiTheDog: I should point out with an organisation that size [Equifax], which is effectively a series of large different businesses linked by a patent (and WAN) it is difficult to know what is going on. But if you have passwords in plaintext in CSV files things are probably not to policy.
    • @mipsytipsy: Anyway.  What you need to build an elite engineering team is not lots of elite engineers, but a team that is willing to iterate on their tools and processes. Care about user experience.  Double down on what works.  And choose good tools to help you get there.
    • @QuinnyPig: No, AWS. They’re @harrys. They didn’t “reduce” their operations cost, they shaved it.
    • Ezra Callahan: The single greatest growth mechanism ever was photo tagging. It shaped all of the rest of the product decisions that got made. It was the first time that there was a real fundamental change to how people used Facebook, the pivotal moment when the mind-set of Facebook changes and the idea for News Feed starts to germinate and there is now a reason to see how this expands beyond college.
    • @dougtoppin: That moment when it is Fri evening, you push your serverless application model to AWS CodeCommit, CodePipeline triggers CodeBuild, your new stuff deploys to Lambda and API Gateway, your Postman tests successfully run and you run out before something goes Kablooey! AWS SAM is cool
    • Wired: But a big California farmers’ lobbying group just blithely signed away farmers’ right to access or modify the source code of any farm equipment software. As an organization representing 2.5 million California agriculture jobs, the California Farm Bureau gave up the right to purchase repair parts without going through a dealer. Farmers can’t change engine settings, can’t retrofit old equipment with new features, and can’t modify their tractors to meet new environmental standards on their own. Worse, the lobbyists are calling it a victory.
    • Charles Schwab: The trade deficit is being used as a yardstick, as a rationale for moving into this trade war or trade skirmish, as we’ve been calling it. Yet at the same time, tax cuts have helped to boost the economy, boost consumer spending, which, in turn, will widen out the trade deficit, because we simply buy more imported goods. Yet a widening trade deficit may be used as a rationale for more tariffs. 
    • @mipsytips: The most effective way to structure your instrumentation, so you get the maximum bang for your buck, is to emit a single arbitrarily wide event per request per service hop. We're talking wiiiide.  We usually see 200-500 dimensions in a mature app.  But just one write.
    • @kchoudhu: Super important distsys thread. I implemented fat logging my last job for a pipeline handling millions of financial transactions a day. L3 support requests went down 82% after people lower down in the support hierarchy gained god's eye visibility of single events.Kamil Choudhury added,
    • @codinghorror: I need to find time to blog this, but TL;DR we've gotten *insane* value for the dollar out of our public vulnerability bug bounty program, so much so that it has completely changed my mind on the matter
    • @brianleroux: Avoiding "vendor lock in" with AWS by building your own functions as a service on a container platform with a cloud provider that has *at best* 1/4th the revenue/customers/scale. Bonus: you also get to avoid 100% utilization!
    • SoulMan: My company is was full on into GCP and after a year of terrible support management suddenly decides to go all in into Azure. We use every managed service from GCP including BigQuery which is nearly impossible to find a replacement of in Azure. Appengine needs to be converted to Azure Kubernetis Service. So its not a technical decision completely, just that company signed a bigger deal with Microsoft and wants to pay a single bill. Based on feedback from other teams company feels Azure support is way more professional when it comes to production outage. Now all of 3 years work sing Google services needs to be re-written in "cloud agnostic" way.
    • sgt101: Lock-in occurs when you no longer can generate the capex to resolve the opex drain that a vendor squeezing you has created. If any vendor realises that you can raise the capex to get off them then they will raise the opex demand to the point of financial viability to maximise their returns and to ensure future returns (because by doing this they underline their control)
    • Spotify: Read requests to Cassandra are cached on a memcached cluster. (For reference, you can find Folsom, an open source version of our client here.) The p99 request latency to our memcached cluster is between 2 and 4 milliseconds, while the request latency to our Cassandra cluster is around 10 milliseconds. When serving traffic without cache, Cassandra request latency grows to a p99 of about 15 milliseconds. Thus, having a memcached cluster for caching lowers our p99 by about 10 milliseconds.
    • eslaught: One of the central insights of Legion was to [explicitly reify the notion of data partitioning](http://legion.stanford.edu/pdfs/oopsla2013.pdf) in the programming model. Sequoia didn't need this because it simply knew the inputs and outputs to every task. Legion couldn't afford to pay the $O(N \operatorname{log} N)$ cost to perform this analysis at runtime. By lifting partitioning into a first-class primitive, we were able to radically reduce the runtime cost of this analysis by leveraging what the user already know about their data partitioning. Originally this required users to declare certain properties of their partitions, such as disjointness, in order to make things efficient. Eventually we discovered a [partitioning sublanguage](http://legion.stanford.edu/pdfs/dpl2016.pdf) which allowed us to statically discover most of these important properties, and to provide many static guarantees even in the presence of data-dependent behavior.

  • When programmers become disillusioned with their profession, as the thoughtful inevitably do, the result is often an epic high quality rant. We got another one. Welcome to the club. 
    • Software disenchantment: So I want to call it out: where we are today is bullshit. As engineers, we can, and should, and will do better. We can have better tools, we can build better apps, faster, more predictable, more reliable, using fewer resources (orders of magnitude fewer!). We need to understand deeply what are we doing and why. We need to deliver: reliably, predictably, with topmost quality. We can—and should–take pride in our work. Not just “given what we had…”—no buts!
    • Most experienced programmers will emphatically nod their head in agreement with much of this post. The problem with the problem of programming is the same as the problem of evil—we as humans can't do better over the long run. We are what we are. What we need to do is change the game, not just play the game better. What Will Programming Look Like In The Future?

  • NewSQL database systems are failing to guarantee consistency, and I blame Spanner. Why? You have to read through an excellent description of consistency before arriving at the reason, but in essence the reason is Spanner is too good: Spanner is a beautiful and innovative system. It was also invented by Google and widely used there. Either because of the former or latter (or both), it has been extremely influential, and many systems (e.g., CockroachDB and YugaByte) have been inspired by the architectural decisions by Spanner. Unfortunately,  these derivative systems are software-only, which implies that they have inherited only the software innovations without the hardware and infrastructure upon which Spanner relies at Google. In light of Spanner’s decision to have separate consensus protocols per shard, software-only derivatives are extremely dangerous. Like Spanner, these systems rely on real-world time in order to enforce consistency --- CockroachDB on HLC (hybrid logical clocks) and YugaByte on Hybrid Time. Like Spanner, these systems rely on knowing the maximum clock skew across servers in order to avoid consistency violations. But unlike Spanner, these systems lack hardware and infrastructure support for minimizing and measuring clock skew uncertainty.

  • Notes Overheard at Distributed Data Summit 2018: Being able to implement this low-level thread-per-core architecture brought significant advantages to Scylla. In one use case, it allowed the contraction from 120 Cassandra i3.2xl nodes to just three (3) i3.16xl nodes for Scylla. Such a reduction in nodes maintained the exact same number of cores, yet required significantly lower administrative overhead.

  • Pinterest made the transition from a batch system to a real-time system. Here's the story, not in great detail, but there are pins of interest. Building a dynamic and responsive Pinterest: As we can see from the previous sections, the new systems must achieve low long-tail latency with big fanout (sometimes all shard fanout), and some systems are CPU intensive (e.g., Scorpion, Pixie and RealPin). We chose to adopt C++11, FBThrift, Folly and RocksDB to build these systems...We built a prefix-based AZ-aware traffic routing library that minimizes cross-AZ traffic and supports all possible routing patterns (e.g., single shard, multiple shard and all shard fanout)...We optimized the in-memory LRU caching module to achieve zero-copy; i.e., cached data is fed into ML models without any data copying. Batch scoring was implemented to allow GCC to better utilize SIMD instructions. Decision trees in GBDT models are compacted and carefully laid out in memory to achieve better CPU cache hit rates. Thundering herds on cache miss are avoided by object level synchronization...Data stored in Aperture is bucketed along the time dimension. We use a frozen data format for old data which is immutable and suitable for fast read access.

  • Facebook's 2018 @Scale Conference Recap. Titles include: Inside NVIDIA’s AI infrastructure for self-driving cars; A golden age for computer architecture; Presto: Fast SQL on everything; Distributed AI with Ray; Scaling concurrency bug detection with the Infer static analyzer. Also, David Patterson Says It’s Time for New Computer Architectures and Software Languages. 

  • Are benchmark suites correct guides to performance? Usually the answer to that question is short and dismissive. Laurence Tratt in Why Aren’t More Users More Happy With Our VMs? Part 2 takes the time to give a thoughtful and reflective answer, taking us through a tour of various benchmarking problems. His conclusion: our benchmarking and our benchmarks have misled us. In a perfect example of Goodhart's Law, we've too often optimised for metrics that aren't representative of what we really care about...A big question is whether fixing existing VMs is practical or not. I must admit that the current signs (including some work that I've been involved in) aren't hugely positive in that regard

  • Is the multi-cloud cure worse than the lock-in disease? Multi-Cloud Is a Trap. 

  • That dystopic world you were worried about, the one where everything is watching you, judging you, rating you, charging you for every contract violation—it's here. Strap on the Fitbit: John Hancock to sell only interactive life insurance. Good discussion on @Techmeme. @JeremiahLee: "I led Fitbit’s API for 4 years. I predicted insurance companies might raise rates to incentivize healthy behavior discounts using surveillance. An exec (also no longer at Fitbit) blocked my attempt to explicitly prohibit this use case. My tin foil hat is now more fashionable." @katecrawford: "We saw this coming, and here it is. Endless trapdoors ahead: data inaccuracies, intentional gaming, constant intimate surveillance 24/7,  data breaches that will be infinitely worse, &c..." China is leading the way. Leave no dark corner: "China is building a digital dictatorship to exert control over its 1.4 billion citizens. For some, “social credit” will bring privileges — for others, punishment." Even Apple likes this approach. Apple creates an iTunes device trust score based on your calls and emails.

  • Eager and lazy evaluation are both bad for performance. What should you do instead? Idle Until Urgent: After spending a lot of time thinking about this problem, I realized that the evaluation strategy I really wanted was one where my code would initially be deferred to idle periods but then run immediately as soon as it’s needed. In other words: idle-until-urgent. Idle-until-urgent sidesteps most of the downsides I described in the previous section. In the worst case, it has the exact same performance characteristics as lazy evaluation, and in the best case it doesn’t block interactivity at all because execution happens during idle periods. Result:  I have multiple, much shorter tasks. In fact, the longest one is now only 37ms! A really important point to emphasize here is that the same amount of work is being done as before, it’s just now spread out over multiple tasks and run during idle periods.

  • Do the job better and cheaper. How Talroo Used AWS Fargate to Improve their Application Scaling. Talroo would like to write code all day and not manage infrastructure. In their old solution they had to worry about auto-scaling and segmenting application pieces within the cluser. With Fargate they can set and forget. Can write task definitions, assign them to services in a cluster, and scale them independently while still keeping organizational benefit of keeping work in a cluster. Hard to mix workloads in an EC2 cluster with the exact same instance types in the cluster. Result: went from 6 hours to .5 hour to process 100,000 jobs with drops in throughput processing. 22% drop in EC2 costs. 40% drop in EC2 instance count. 

  • Etsy did not turn to an artisinal cloud provider, they went with GCP. How is it going? No details, but We’re on (Google) cloud nine: In less than a year since beginning our journey to the cloud, we’re excited to share that we’ve successfully migrated our site and mobile apps to GCP.

  • Most know about Redis as a datastructure database, you might not know it has more advanced features like RedisGraph and Redis CRDTs. Good overview in On Redis. Q&A with Yiftach Shoolman,Founder and CTO of Redis Labs. The RedisGraph module, based on GraphBLAS engine, shows significant performance advantages over existing graph databases. And CRDTs in an Active-Active configuration keep databases sync'ed over a Multi-region WAN.

  • An interesting case where systems are purposefully built to not play nicely with others as way of staying under the radar. The F-22 and the F-35 Are Struggling to Talk to Each Other:  In developing the F-22—and later, the F-35—designers needed to preserve the jets’ stealth against rapidly evolving adversaries. Standard radio emissions would reveal their locations, which meant devising ways the low-observable fighters could talk to each other without giving away their position. Both jets have what are called “low probability of detection/interecept” communications gear to stay hidden...As a result, Air Force combat communications can become a kind of Tower of Babel. While Raptors can receive over the Link 16 network—the standard across US and NATO aircraft—it can’t transmit over the system. Instead, it uses the F-22-only Intra-Flight Data Link (IFDL). F-35s can transmit on Link 16 to fourth gen jets and talk among themselves using the stealthy Multifunction Advanced Data Link (MADL), a capability the Air Force had planned to install on the F-22 but canceled because of cost about five years ago

  • Videos from ElixirConf 2018 are now available.

  • After Falling in love with Rust, Bryan Cantrill wrote an epic love poem for the ages. In it he proclaimed: "I have believed (and continue to believe) that we are living in a Golden Age of software, one that will produce artifacts that will endure for generations. Of course, it can be hard to hold such heady thoughts when we seem to be up to our armpits in vendored flotsam, flooded by sloppy abstractions hastily implemented. Among current languages, only Rust seems to share this aspiration for permanence, with a perspective that is decidedly larger than itself." Why do I love thee? Thy error handling is beautiful. Thy macros are incredible. Thy format! is a pleasure and include_str! is a godsend. Thy serde is stunningly good. I love your tuples. Thy integrated testing is terrific. Thy community is amazing. Thy performance rips. And I look forward to many years of implementing with you.

  • Videos from ClojuTRE & Small FP Conf 2018 are now available. 

  • I looked it up, "replatform" is actually a word. eBay Replatforming to Kubernetes, Envoy and Kafka: As part of a three-year effort to replatform and modernise their backend infrastructure, eBay has recently announced that they are building their own custom-designed servers "built by eBay, for eBay"...The eBay engineering team are making changes across the entire technology stack, including both physical and logical layers, because they believe that all of the layers are intertwined in some way...For the physical foundation, eBay is a Point of Presence (PoP) strategy, and is decentralising their cluster of US-based data centers towards an "edge computing approach". This will enable them to "create a faster, more consistent user experience, saving 600-800 milliseconds of load time"...In the data layer, eBay has created more "customized models". Using open source technologies, the team has built "NuData", a fault tolerant, geo-distributed object and data store...they have used open source to build an in-house "AI engine" that can be shared across all of their teams, with the goal of "increasing productivity, collaboration and training". Their AI engine has already accelerated the production of new features, such as computer vision, Image Search and sharing on social media platforms.

  • Hopefully support by a company like Cloudflare will help drive distributed web adoption. Cloudflare goes InterPlanetary - Introducing Cloudflare’s IPFS Gateway: At the most basic level, you can access any of the billions of files stored on IPFS from your browser. But that’s not the only cool thing you can do. Using Cloudflare’s gateway, you can also build a website that’s hosted entirely on IPFS, but still available to your users at a custom domain name. Plus, we’ll issue any website connected to our gateway a free SSL certificate, ensuring that each website connected to Cloudflare's gateway is secure from snooping and manipulation. jgrahamc: That's the whole point. We think the Distributed Web is cool and coming and we want to accelerate that happening. Hence the time and effort to make this happen. steven_pack: That was one of the drivers. We liked the core technology, but could see there was a significant barrier to actually using it. Being able to point your domain at IPFS content and then providing that (ssl secured) site to the masses is a big step forward. MrEldritch: This is also, in my opinion, the biggest objection to IPFS - that it really doesn't necessarily lead to any kind of true decentralized hosting unless someone else has decided to pin your files.

  • A fun way to learn concurrency. Go 1.11 Rocket Tutorial: you’ll build a highly concurrent web server which reads SpaceX images from the Unsplash API and applies seam carving on those images.

  • Enabling ‘internet of photonic things’ with miniature sensors: Washington University in St. Louis researchers have for the first time recorded environmental data using a wireless photonic sensor resonator with a whispering-gallery-mode (WGM) architecture. The photonic sensors recorded data under two scenarios, including a real-time measurement of air temperature over 12 hours, and an aerial mapping of temperature distribution using a sensor on a drone. Both measurements also included a commercial thermometer with a Bluetooth connection for comparison purposes. The sensors belong to a category called WGM resonators because they function like the whispering gallery in St. Paul's Cathedral in London, where a person on one side of the dome can hear a message spoken to the wall by someone on the other side. In addition, the sensor resonates at a light frequency and also at vibrational or mechanical frequencies. Washington University in St. Louis researcher Lan Yang says, "The successful demonstrations show the potential applications of our wireless WGM sensor in the IoT," which include magnetic, acoustic, environmental, and medical sensing, she says.

  • Juniper: A Functional Reactive Programming Language for the Arduino: This paper presents the design and implementation of Juniper: a functional reactive programming language (FRP) targeting the Arduino and related microcontroller systems. Juniper provides a number of high level features, including parametric polymorphic functions, anonymous functions, automatic memory management, and immutable data structures. Also included is a standard library which offers many useful FRP signal processing functions. Juniper is translated to standard C++ and compiled with the existing Arduino development tools, allowing Juniper programs to fit on resource-constrained devices, and enabling seamless interoperability with existing C++ libraries for these devices

  • dennybritz/reinforcement-learning: This repository provides code, exercises and solutions for popular Reinforcement Learning algorithms. These are meant to serve as a learning tool to complement the theoretical materials from Reinforcement Learning: An Introduction (2nd Edition); David Silver's Reinforcement Learning Course

  • yahoo/HaloDB (article): a fast, simple, embedded, persistent key-value storage engine written in Java. HaloDB is suitable for large datasets and IO bound workloads, and handles high throughput for both reads and writes at submillisecond latencies.

  • CrimsonDB: decides autonomously how to change its core design to adjust to workload, hardware and other parameters. It can assume arbitrary shapes in between the core design classes of log, LSM-tree, B-tree, and their hybrids. It offers optimal reads and writes given a memory budget and an application workload.

Read more