hot links

Stuff The Internet Says On Scalability For July 10th, 2015

High Scalability

High Scalability

— 14 min read

Hey, it's HighScalability time:


Spying on an ant holding its favorite microchip. (@SciencePorn)

  • 1,425%: malware attack ROI; 33333/sec: BP oil well datapoints generated; 8 million: mumified dogs; 5 billion: Apple map requests per week; 10 billion: parameter neural net in your basement; 1 trillion: records in Yahoo's Sherpa.
  • Quotable Quotes:
    • @warriors: "It's ironic but what the unexpected thing was that everything went exactly as we hoped. That never happens." 
    • @georgeblazer: At scale, architecture dominates material. Alan Kay #gophercon
    • Nassim Nicholas Taleb: How unpredictability rises faster and faster with dimensionality: add one variable to 100 and error doubles.
    • Elon Musk~ one of the biggest challenges in the CRS-7 event is matching events to the exact time. When you are talking a matter of milliseconds it's hard to match the ground track video to the sensors
    • @MeltingIce: NYSE halted, United grounds flights due to computer glitch, WSJ website is down. This is how it starts.
    • The Shut-In Economy: They end up asking each other which apps they work for: Postmates. Seamless. EAT24. GrubHub. Safeway.com
    • @aphyr: At a Devops Chicago meetup I asked how many people had experienced partitions in their datacenters, and over half the room raised hands.
    • @mjpt777: Simplifying code rocks. @toddlmontgomery and I are seeing new throughput highs on Aeron after simplifying and cutting indirection.
    • aphyr: Real software is fuzzier: our processes are usually not realtime, which means the network effectively extends within the node. Garbage collection, in particular, is a notorious cause of “network” partitions, because it delays messages.
    • @fgens: 2 key msgs at #AWSSummit : "Developers, we love you (IaaS is so yesterday)!" and "Go 'server-less' -- deploy code as Lambda microservices"
    • @BenedictEvans: @Jim_Edwards but the boom is caused by having 3bn people online and a generational change in the tech platform from PC to mobile.
    • @Obdurodon: "Our distributed file system doesn't work quite right and performs like crap." "OK, we'll just call it an object store then."
    • @viktorklang: Suspected main causes for slow programs: A) Doing unnecessary work B) Waiting for unnecessary work to be done
    • @jasongorman: There are only so many times one can re-learn the same patterns for remote procedure calls before one gets mightily sick of it
    • @BenedictEvans: Devices in use, end of 2014:  ~1.5bn PCs 7-800m consumer PCs 1.2-1.3bn closed Android 4-500m open Android 650-675m iOS 80m Macs, ~75m Linux.
    • vardump: When performance matters, don't chase pointers (references) and don't ever do random memory accesses if you can avoid it. Ensure variables needed about same time are near each other, so that a single cache line fill will get all. Except when other threads/cores frequently update a variable, try to keep those separately to reduce false sharing.
    • @JZdziarski: Quantum entanglement dictates if one programmer is writing productive good code, another somewhere is inexplicably porting to JavaScript.
    • buro9: Best way to defend against an L7 DDoS is to have the origin hidden, and to cache everything at a large number of geographically distributed PoPs. This helps in 99% of cases, and where it doesn't it is simply because there is a resource that cannot be cached and that the edge must revisit the origin for. 
    • markbnj: I feel like microservices as a thing is one of the least meaningful tech fads I have seen. Minimality and cohesiveness aren't surprising new challenges. They were desirable concepts of C++ class libraries two decades ago. An implementation of a service should always be both minimal and cohesive, and it should be complete. It should be as small as possible. Whether that is "micro" or not is entirely too subjective for me.
    • @tabdulla: programmers paradox: repeatedly taking the simplest option leads to greatest complexity
    • @SciencePorn: If the Sun was scaled down to the size of a white blood cell, the Milky way would be the size of the continental United States.
    • @guypod: Removing all images made pages 30% faster on 3G & 80% of images aren't visible w/out scrolling. Critical images FTW!
    • @BenedictEvans: Posit, one big change from 2000: spend big up front for 'first mover advantage' replaced by 'get it right first, then spend big to scale it'
    • @volothamp: "Async/Concurrent is not about performance (primarily)—it is about scalability." Clear explanation by @viktorklang
    • @rbranson: at Instagram we deploy our backend code ~40 times/day. In the last 2 months 71 site-breaking deploys were caught by test or canary failures.
    • @LusciousPear: Theory: a company's avg ping time is directly correlated with its importance on the internet. Google: 6.7ms, Twitter: 10.1ms, Yahoo: 28.2ms
    • @chrismunns: AWS Lambda is the next thing and like Pizza; cheap, easy to consume, accessed whenever you need it in almost any volume.
    • @bradfordcross: "Goldman Sachs had more engineers and programmers, at 9,000, than Facebook, Twitter, or LinkedIn"
    • @Chris_Skinner: "Airbnb have 500k rooms filled a night and will soon be the biggest hotel chain in the world" Salim Ismail #usi2015
    • @deanwampler: Very meaty talk on Spark Performance by @kayousterhout . CPU overhead tends to be more important than network overhead. @spark_summit
    • @blaine: I have some hats for you. Any engineering team who claims to know exactly what their algorithms do is lying*.
    • @mdudas: "~6% of investments representing 4.5% of dollars invested generated ~60% of the total [VC] returns" http://cdixon.org/2015/06/07/the-babe-ruth-effect-in-venture-capital/ … via @cdixon
    • jacques_chester: Creation is more fun than selection. And in the same position, I'd ditch Wordpress like a stolen sports car.

  • Love this! jamiesonbecker: AMAZON DEPRECATES EC2 November 3, 2017, SEATTLE At the AWS Re:Invent in Las Vegas today, Amazon Web Services today announced the deprecation of Elastic Compute Cloud as it shifts toward lighter-weight, more horizontally scalable services. Amazon announced that it was giving customers the opportunity to migrate toward what it claims are lower cost "containers" and "Lambda processes".

  • Horace Dediu with an interesting view of Humanism++. This time it's preferring humans over algorithms instead of humans over faith. Curate, don't automate...or sermonize. Also interesting was the discussion on functional vs divisional organizations. Apple is the largest functional org outside of the army. Functional orgs prevent cross divisional fights for resources and power. 

  • OK, this is funny: Using locks to solve your concurrency problems.

  • Do we have a precedence for the rise of walled gardens? 1492: The Year the World Began: shift of initiative—the upset in the normal state of the world—started in 1492, when the resources of the Americas began to be accessible to Westerners while remaining beyond the reach of other rival or potentially rival civilizations.

  • Here's how the new StackExchange blog was rebuilt. Pretty radical. They got rid of WordPress for a static blog built on Jekyll. While there was some contention with the move on HackerNews, jonhmchan set the record straight: Performance also wasn't the only plus here. Closing major security holes, making more of our content and technology more open, and moving to a platform that our devs liked working in are just some of the other wins. It's too early to say definitively now, but we think the change is probably a good one.

  • Should Uber be the new McDonalds? While I viscerally agree with the following statement, the idea that everyp place in the world must have the same laws is equally obviously wrong. Vive la difference! @paulg: Uber is so obviously a good thing that you can measure how corrupt cities are by how hard they try to suppress it.

  • Bigger is not always better. Scalability! But at what COST?: Rather than making your computation go faster, the systems introduce substantial overheads which can require large compute clusters just to bring under control. In many cases, you’d be better off running the same computation on your laptop.

  • Does the network matter for graph analytics? It depends on your stack: A recent NSDI paper argued that data analytics stacks don’t get much faster at tasks like PageRank when given better networking, but this is likely just a property of the stack they evaluated (Spark and GraphX) rather than generally true. A different framework (timely dataflow) goes 6x faster than GraphX on a 1G network, which improves by 3x to 15-17x faster than GraphX on a 10G network.

  • Interesting discussion on Hacker News on AWS Best Practices for DDoS Resiliency. Simply scaling up to out compute the attack doesn't seem to be cost effective. Most seem to think DDoS mitigation is the right approach and CloudFlare makes a good case for their service. 

  • How We [Parse] Moved Our API From Ruby to Go and Saved Our Sanity. When the “one process per request” model used by Rails started failing Parse turned to a rewrite in golang. The result: reliability improved by an order of magnitude, the code base became cleaner, the API is less fragile, support became easier, it was easier to add metrics to the API, integration tests ran much faster, and  reliability-impacting events are almost never tracked backed to the API layer.

  • Is Azure four times as expensive as Amazon Web Services?: And Azure…. what’s there to say: disappointing performance at a higher cost with less available memory. We like the IAAS functionality Azure almost as much as we like the AWS IAAS, but facts don’t lie: Azure has some catching up to do before it can compete with Amazon… The relevant VM’s Azure offers are 3.6 - 4.4 times as expensive as the t2.large Amazon instances.

  • Improving journalling latency in LMAX: With our latest set of changes, we are aiming for sub-100 microsecond mean latency and significantly reduced jitter...upgrading our operating system and hardware made a huge difference to the amount of jitter that we saw in our journaling performance...a modern kernel on commodity hardware seems to be capable of write latencies as low as 5 microseconds when asynchronous I/O is used. 

  • It would be surprising if this wasn't true. Discovery of Nanotubes Offers New Clues About Cell-to-Cell Communication: Certain types of stem cells use microscopic, threadlike nanotubes to communicate with neighboring cells, like a landline phone connection, rather than sending a broadcast signal.

  • The cloud can be cheaper, at least if you are an enterprise. When Yamaha moved to Amazon. Moving nearly all of the company's 200 computer servers to Amazon's cloud and getting rid of the company's data centers saved $500,000 a year. Sources of savings: not having hardware leases, eliminating our data center presence, no license for Windows Servers software.

  • Good to see the container industry bypassing the virtualization war phase of the cycle. Amazon, CoreOS, Docker, Google, Microsoft and others team up to create a common container standard for deploying Web apps. Notice, no standards bodies, which is why it can happen quickly. Perhaps industry has learned something? Instead of fighting the standards wars again and again, we've learned not to do that and converge early on common standards. It saves money and puts more work into more results, making everyone more productive.

  • Why it’s hard to meet SLAs with SSDs: GC is the likeliest explanation for the big performance hit when VMs share a partition. The GC process affects all the VMs sharing the partition, causing all of them to slow down...Another variable is the degree of over provisioning in the SSD...Yet another variable is the how the OPS is shared among partitions. 

  • Instagram with a great post on how they built their trending feature. Lots of detail. The trick is differentiating between popularity and novelty.

  • Use hash tables says Trie vs Hash Table Deathmatch!, tries are slower than hash tables, and consume a lot more memory. Sure, they're simple, but then so are hash tables.

  • What happened at the NYSE? Same old same old. There was a new software release that conflicted with some configuration bits. 

  • Assembly found a rabbit could defeat an elephant. It turns out the elephant was doing too many curls. Now that Rabbit is everywhere.

  • Don't think about what happens to your data when everything is going good. Think of what happens to it when the situation becomes desperate. When a Company Is Put Up for Sale, in Many Cases, Your Personal Data Is, Too

  • Something 'scary' went wrong with NASA's probe to Pluto. In an embedded system a huge fear is not being able to contact a device after it has been asleep for a while. So much can go wrong even when a system is doing "nothing." I can't imagine that feeling in the pit of the stomach that must occur when your "device" is flying near Pluto.

  • Scale-free gives humans a competitive edge: Humans may have evolved into scale-free structures to provide more rational responses to emergencies such as infection epidemics or natural disasters. When a society is structured as a scale-free network it responds more rationally to threats even when individuals in the society have not had time to gather information about a disease or disaster

  • Interesting idea. Tutorial: Serve a static website as a Unikernel: build a unikernel which hosts your very own static site served by Nginx...You'll be able to scale with server loads up and down in the blink of an eye. You also know the basics to build on more interactive things like running a PHP applications in an immutable image, or run an application without the operating system directly on hardware.

  • Storage is at a tipping point: much of the existing investment in the software stack will be obsolete within two years...In a nutshell, the performance optimization technologies of the last decade – log structured file systems, coalesced writes, out-of-place updates and, soon, byte-addressable NVRAM – are conflicting with similar-but-different techniques used in SSDs and arrays...In the next decade new storage technologies will enable a radical rethink and simplification of the I/O stack beyond what flash has already done.

  • Why We [iHeartRadio] Picked AKKA Cluster as our Microservice Framework: Out-of-the-box clustering infrastructure; Loose coupling without the cost of JSON parsing; Transparent programming model within and across microservices; Strong community and commercial support; High resiliency, performance and scalability.

  • How SignalFx uses Cassandra as a Time Series Database. Metric timeseries data is batched in a memory tier that sits in front of Cassandra. Using packed writes improved write performance 5-8x and CPU utilization decreased by roughly 25% and Disk utilization growth reduced to roughly half the rate.

  • While informative, it's not really that gentle. ECCHacks: a gentle introduction to elliptic-curve cryptography: This talk will explain how to work with elliptic curves constructively to obtain secure and efficient implementations, and will highlight pitfalls that must be avoided when implementing elliptic-curve crypto (ECC).

  • A great look Inside NGINX: How We Designed for Performance & Scale.

  • The Fight in the Clouds: I attribute our success to a couple of things. Number one was the fact that our group always performed liked a team. If you were ever in trouble, somebody would be there to help you out. When you’re in the middle of a dogfight and bullets and airplanes are flying every which way, it becomes a little hectic, to say the least. Most of us thought of ourselves as hot pilots. You had to have that attitude to survive; otherwise you had no business being in the middle of it. But help was always appreciated.

  • A good overview of some of the talks from Facebook's Inside Data@Scale 2015 conference.

  • Here's Knife Depot's server farm setup. In order to have multiple servers they moved assets to S3/CloudFront. Session data is stored in MongoDB. The public facing server runs on DigitalOcean 8GB Droplets ($80/month).  Varnish is used for caching and load balancing. Nginx is used for SSL termination. Their Application/Web Servers run Apache on DigitalOcean 16GB Droplets ($160/month). The Master Database is MySQL on use a single DigitalOcean 16GB Droplet. A slave database runs on its on droplet. RabbitMQ is used for the message Queue. Scout is used for monitoring.

  • Sounds familiar. Military Misfortunes: The Anatomy of Failure in War: To continue our metaphor, one might explain failure by suggesting that highly complicated machines, even when well-designed and competently operated, require universal habits or practices to make them run smoothly. Favored all-purpose lubricants for the machinery of military organizations include such measures as forcing decision making lower down the hierarchy rather than at its summit, ensuring that information is transmitted up as well as down the chain of command, and maintaining the best communications between different components.

  • How to achieve low latency with 10Gbps Ethernet: Userspace busy polling; Pin processes; RSS - Receive Side Scaling; Indirection Table; Flow steering; Interrupt coalescing; SO_TIMESTAMPNS;  OpenOnload kernel bypass; ethtool toggles. The result: 25us roundrip times. 

  • Sounds about right. Your Infrastructure is not a Unique Snowflake, and that's Ok. Everyone thinks that they need to build a bunch of unique software to run their business, but you really don't, and if everyone would share a little bit more, the world would be a better place.

  • A question worthy of some very deep thought. Redundancy vs dependencies: which is worse?: Redundancy sucks. Redundancy always means duplicated efforts, and sometimes interoperability problems. But dependencies are worse. The only reasonable thing to depend on is a full-fledged, real module, not an amorphous bunch of code. 

  • Real-time Resizing of Flickr Images Using GPUs. Flickr uploads over 27 million photos a day. To resize them Flickr created a dedicated tier f resize servers, each with an GPU co-processor. Each of these boards has two GPUs, each with 1500+ “cores”, running at just under 1GHz. They are now able to resize 2048px JPEGs to 1600px in under 16ms. This is more than 15x faster than GraphicsMagick and nearly 10x faster than Ymagine. 

  • Excellent explanation, but REST seems a very nuanced fit for something that's quite straightforward from a protocol POV. REST and long-running jobs.

  • First description of a selfie? Four Boots-One Journey: A Story of Survival, Awareness & Rejuvenation on the John Muir Trail: Before we moved on, I pulled her close for one of my famous one-handed camera shots, for which I hold the camera out in front of the both of us with one hand, aim it as well as I can without actually being able to see the background, and snap a picture.

  • Burnout amongst developers: an industry disease? There is nothing more satisfying than making software with great tools, great interactivity, and great libraries so that you don't have to reinvent the wheel. Most people just aren't dealing with that though. The biggest problem I see is that as a project progresses progress slows instead of accelerates. Ideally it would be the other way around as you abstract away small prickly details.

  • Why Information Grows: The Evolution of Order, from Atoms to Economies: one lesson to remember is that there is a relationship between the size of a productive network and the volume of knowledge and knowhow it can embody, with larger networks being able to embody larger volumes of knowledge and knowhow, all else being equal. Second, our ability to weave large networks depends on the costs of establishing links, with cheaper links favoring the creation of the large networks needed to amass large volumes of knowledge and knowhow.

  • Trash Day: Coordinating Garbage Collection in Distributed Systems: In this paper, we show that distributed applications suffer from each node’s language runtime system making GC-related decisions independently...We present initial results to demonstrate that this Holistic GC approach is effective both in reducing the impact of GC pauses on a batch workload, and in improving GC-related tail-latencies in an interactive setting.

  • TRANSFORMING+PERSONAL MOBILITY (2012): By combining five business model and technology enablers (the “Mobility Internet,” self driving/driverless vehicles, share! vehicle systems, specific purpose vehicle designs and advanced propulsion systems) better mobility experiences at radically lower cost can be realized in a variety of settings. This mobility system is compelling compared with other modes of public and!private transportation in Ann Arbor, Babcock!Ranch, and Manhattan, because of its potential to be more convenient, more energy and resource efficient, safer, and less costly.

  • Snow: A layer 3 virtual network that uses public keys instead of IP addresses.

  • Memory-Centric Data Storage for Mobile Systems: This design trades off data staleness for better app responsiveness and energy efficiency, in a quantitative manner. Evaluations show that MobiFS achieves 18.8× higher write throughput and 11.2× more database transactions per second than the default Ext4 filesystem in Android.

  • Don’t stack your Log on my Log: We show that multiple log layers affects sequentiality and increases write pressure to flash devices through randomization of workloads, unaligned segment sizes, and uncoordinated multi-log garbage collection. All of these effects can combine to negate the intended positive affects of using a log.

  • privacytools.io: one stop shopping for security knowledge.

  • Mega-processor: The Mega-processor is a micro-processor built large. Very large.  Like all modern processors the Mega-processor is built from transistors. It's just that instead of using teeny-weeny ones integrated on a silicon chip it uses discrete individual ones like those below. Thousands of them. And loads of LEDs.

Read more