hot links

Stuff The Internet Says On Scalability For August 28th, 2015x

High Scalability

High Scalability

— 10 min read

Hey, it's HighScalability time:


The oldest known fossil of a flowering plant. 130 million years old. What digital will last so long?

  • 32.6: Ashley Madison password cracks per hour; 1 million: cores in the Human Brain Project's silicon brain; 54,000: tennis balls used at Wimbledon; 4 kB: size of first web page; 1.2 million: million messages per second Apache Samza performance on a single node; 27%: higher conversion for sites loading one second faster; 

  • Quotable Quotes:
    • @adrianco: Apple first read about Mesos on http://highscalability.com  and for a year have run Siri on the worlds biggest cluster 
    • @Besvinick: Interesting recurring sentiment from recent grads: We lived most of our college lives on Snapchat—now we don't have any "tangible" memories.
    • Robin Hobb: For most moments of our lives, we have forgotten almost all of the world around us, except for what currently claims our interest.
    • @Carnage4Life: I'd like to thank all the Amazon employees who cried at their desks to make this possible 🙏👍 🚚🍷🍸🍹🍺 
    • Jim Handy: The single most interesting thing I learned at the 2015 Flash Memory Summit was that 3D NAND doesn’t have a natural limit, after which some other memory type will need to be adopted.
    • @mccv: them: is that written down? me: we communicate in the viking tradition. Let me tell you the saga of that system.
    • The Handmade Manifesto: that amazing speed we'd been granted was wasted, by us, in a death by a thousand abstraction layers
    • Peter Thiel: For us to really have a greater productivity gains as a society, we have to do things more in the world of atoms and not just the world of bits.
    • @lxpollitt: Verizon announced today as paying customer of @Mesosphere DCOS. Cool on stage demo with 22k cores: 50k containers in 100s - @flo #MesosCon
    • Matthew Brunwasser: Technology has transformed this 21st-century version of a refugee crisis, not least by making it easier for millions more people to move.
    • @rsingel: Stephen Hawking says to never give up hope if caught in a black hole. He has never evidently used a mobile browser.
    • @lxpollitt: Siri has been running on Mesos for exactly one year today. “Mesos scales” - Apple #MesosCon
    • @Jimminy: "The cheapest, fastest, and most reliable components are those that aren’t there. — Gordon Bell
    • @mathiasverraes: There are only two hard problems in distributed systems:  2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery
    • Horace Dediu: If I were Tim Cook I would not have the goal of tripling revenue over the next decade...The objective of the company is not to triple revenues, the objective of the company is to make great products...That's the goal. End of story. You don't talk about money. You talk about product. Money comes from product not the other way around...The purpose of the firm is to delight the customer. 
    • @t_blom: “The hardest thing about MVP — you decide what’s Minimum, the customer decides what's Viable” — @davidjbland 
    • @adrianco: #mesoscon @pbailis reading list. 
    • @kelseyhightower: Based on my twitter stream, it seems the theme coming out of #mesoscon is the major benefits of increasing resource utilization at scale.
    • lorenzhs: We need new algorithms that - require communication volume and latency significantly sublinear in the local input size (ideally polylogarithmic) - don't depend on randomly distributed input data (most older work does)
    • @clstokes: #MesosCon @pbailis on coordination-free systems - "Scalable systems can just shut up and comfortably share silence."
    • frankmcsherry: if you want to do any big data computation, please sort your records. Stop talking sass about how Hadoop sorts things it doesn't need to, read some papers, run some tests, and then sort your damned data. Or at least run faster than me when I sort your data for you.
    • @RFFlores: There's always lock-in. You have to choose where. My latest blog is about this.
    • Jared Diamond: People in the first world are terrified by the wrong things. The real danger isn’t terrorism, serial killers or sharks, which kill a very, very small percentage of people annually. The real risks are those things that we do daily that carry a low risk but that eventually catch up with you – driving, taking stairs, using step ladders.

  • Something tells me we can expect this list to get much larger as the future fumbles forward. T-Rex large. The 20 Most Infamous Cyberattacks of the 21st Century (Part I).

  • Getting to Datacenter Zero. Catchy buzzword from @swardley around Netflix sloughing off the last of its non AWS datacenter operations. Netflix shuts down its last data centre, but it still runs a big IT operation. Finally, all of Netflix IT will run in the public cloud. We'll likely hit Datacenter Zero long before we hit Inbox Zero.

  • She's so humble! Q: Alexa, what do you think of M, Facebook's new Human-Powered assistant? A: I don't have preferences or desires. 

  • Have you ever wanted to know how WiFi in a plane works? Have you ever wondered why it's so expensive? Have you ever wondered why it's just a tad slow? Then Why Gogo's Infuriatingly Expensive, Slow Internet Still Owns the Skies is your story. In my mind I thought the system would use a satellite. It doesn't! There's a vast air-to-ground system. The plane talks to 225 towers spread across the US. Newer systems do use a satellite. It's expensive because with a first mover advantage Gogo was able to lock in long term contracts and achieve a near monopoly. There are competitors, but switching costs are high. And with only 40,000 planes in the world making more money requires raising prices on relatively price insensitive business users. There's a sophisticated dynamic pricing scheme aimed at keeping traffic within capacity limits while maximizing profits. It's slow because the signal is shared by everyone on the plane and the hardware on 2/3rds of the planes tops out at 3Mbps. Yet it's still hard to deny: “Everything’s Amazing and Nobody’s Happy.” 

  • Isn't it wonderful how new technology uplifts humankind? Two men arrested after trying to drop drugs, porn into prison via drone.

  • This is a crackin' good cautionary tale. Practical Microservice Architecture and Implementation Considerations. Makes me think the book will be good. Imagine a microservices based project where they defined 25 REST services as part of their services layer...approximately 90% of transactions would require traversal of all 25 REST services to complete an end-to-end business transaction...they decided to have a one-to-one relationship between REST services and the JVM onto which it was deployed...Over time this pattern led to an environment made up of 400 JVMs, each with 1 GB heap, across 16 physical hosts.

  • How about a Beowulf cluster of these? Use Autonomous Car Computers When They Are Parked. Or is that just called a parking lot? 

  • NoSQL envy no more. It looks like Postgres will be getting autosharding. PostgreSQL moves toward NoSQL-style scaling: In the next year or two, we're actually doing a major push to add auto-sharding to Postgres so that you can have dozens of Postgres servers all working in a unified way, sharing transactions, sharing workloads

  • There's a lot of truth here. Fighting SDKs really kicks the enthusiasm out of you. Why App Developers Are Fed Up With SDKs: The software developer kit (SDK) is becoming a four-letter word to many developers. The same goes for application programming interfaces (API) or any other third-party software that developers need to plug into their apps to perform any and all types of functions.

  • The fascinating story behind The Secret of Airbnb’s Pricing Algorithm: We selected a machine-learning model called a classifier. It uses all of the attributes of a listing and prevailing market demand and then attempts to classify whether it will get booked or not. Our system calculates price tips based on hundreds of attributes, such as whether breakfast is included and whether the guest gets a private bath.  < And I was surprised that photos of cozy bedrooms decorated in warm colors work best.

  • Videos from MesosCon are available. And here's a good overview of MesosCon 2015 and how Apple, Bloomberg, Netflix, Two Sigma, PayPal, Orbitz, and TellApart make use of Mesos. And here's another present from Netflix: Fenzo: OSS Scheduler for Apache Mesos Frameworks.

  • I did not know that. Errata Security: A lesson in BitTorrent: But 9 women can't make a baby in 1 month. The same goes for BitTorrent. You can only download chunks from peers if they've got all the chunks. That's the current problem with the AshMad dump: everyone combined has only 85% of all possible chunks. The remaining 15% of the chunks haven't been uploaded to the swarm yet. Nobody has a complete copy. The original tracker is seeding at a rate of 37-kilobytes/second, handing off the next chunk to a random person in the swarm, who quickly exchanges it with everyone else in the swarm.

  • SSL termination may not need hardware accelerators. RPM: OpenSSL can do close to 900 RSA signatures per second per CPU core (which coincides nicely with ~3000 TPS quoted by F5 LTM-VE); A few thousand TPS might be more than enough for most web properties, particularly if you use persistent HTTP connection and TLS session resumption (so you do TLS negotiation only for truly new users).

  • For some reason I find comforting the idea we'll always be able to be unobserved. Quantum ‘spookiness’ passes toughest test yet: It’s a bad day both for Albert Einstein and for hackers. The most rigorous test of quantum theory ever carried out has confirmed that the ‘spooky action at a distance’ that the German physicist famously hated — in which manipulating one object instantaneously seems to affect another, far away one — is an inherent part of the quantum world.

  • Epic Graph Battles of History: Chaos vs Order: To the extent that the single-threaded code did well, it was because we took advantage of our ability to write programs as we see fit. This is a super-powerful ability, and not one that computer scientists should so quickly dispose of. Restrictive programming models make it easier to write apparently efficient systems, but if that efficiency comes by forcing you to write shit programs, we haven't obviously improved the world yet.

  • It's all about hacking networks. How the Brain Loses and Regains Consciousness: researchers believe that these alpha and low-frequency oscillations, which they also detected in last year’s study, produce unconsciousness by disrupting normal communication between different brain regions. The oscillations appear to constrain the amount of information that can pass between the frontal cortex and the thalamus, which normally communicate with each other across a very broad frequency band to relay sensory information and control attention.

  • Are you missing what is missing?  Missing what’s missing: How survivorship bias skews our perception. Don't look at where the bullet holes are in the bombers that make it back. Those survived. Put armor where the holes aren't because that must be where the holes are in the planes that didn't make it back. If you see a super cluster of successful businesses run away. It's not an indication that you should start a business just like it. These businesses survived in a super competitive environment. You don't see the failures. What you are seeing are the successes that skew your evaluation of the risks of opening a restaurant. Skill will allow you to put more bets on the table but it doesn't guarantee you success. So be aware of advice from the successful. All there decisions worked out. They can't tell you how not to fail.

  • Here's another reason you might want to split into different companies. Google Now’s Staff Exodus Reveals Hurdles for New CEO Pichai: A former Googler who worked on Now recalled Pichai’s response to their protests: “Look, I’ve got a lot on my plate. Chrome and Android are my top priorities. Google Now is not on that. I can’t fight that battle for you."

  • Another interesting commentary in your Morning Paper: MillWheel: Fault-Tolerant Stream Processing at Internet Scale.

  • What do AIs have to talk about? It turns out not much, not much at all. 

  • Kendra Little with The Top 5 Mistakes DBAs Regret Later: TURNING ON AUTO-SHRINK; ENABLING XP_CMDSHELL AND GIVING THE SQL SERVER ACCOUNT ADMINISTRATIVE RIGHTS IN WINDOWS; RUNNING TRANSACTION LOG BACKUPS EVERY 30 MINUTES, ONLY DURING THE DAY;  LOOKING FOR ONE MAGICAL FEATURE THAT WILL FIX ALL PERFORMANCE PROBLEMS; BEING RUDE TO ALL THE DEVELOPERS.

  • Good talk about Scaling Square Register (video). This is about the process they use, not their architecture.

  • Great explanation. Scaling Analytics at Amplitude: We’re excited to share what we learned while scaling to hundreds of billions of events and two of the key design choices of our system: pre-aggregation and lambda architecture...It turns out that the easiest way to make queries fast is perhaps the most obvious one: pre-aggregate partial results beforehand so that at query time it is faster to construct the final result...The main benefit [of the lambda architecture] is that the system supports real-time updates without needing to maintain infinitely growing mutable state...we built a high-performance database similar to Redis that performs set operations in memory... S3: the per-GB rate is 25x cheaper than using SSDs.

  • Papers are available from the SIGCOMM Preview Session 2015.

  • InterTubes: A Study of the US Long-haul Fiber-optic Infrastructure: We show how both risk and latency (i.e., propagation delay) can be reduced by deploying new links along previously unused transportation corridors and rights-of-way. In particular, focusing on a subset of high-risk links is sufficient to improve the overall robustness of the network to failures. Finally, we discuss the implications of our findings on issues related to performance, net neutrality, and policy decision-making.

  • Congestion Control for Large-Scale RDMA Deployments: Using a 3-tier Clos network test bed, we show that DCQCN dramatically improves throughput and fairness of RoCEv2 RDMA traffic. DCQCN is implemented in Mellanox NICs, and is being deployed in Microsoft’s datacenters.

  • Sophia: an advanced Append-Only MVCC storage designed for fast write and read, small to medium key-values. It supports ACID Transactions, Consistent Cursors, Compression, Snapshots and much more. Sophia has unique hybrid architecture that was created as a result of research and reconsideration primary algorithmic constraints of Log-file based data structures.

  • Aerosolve. A machine learning package built for humans from Airbnb. Aerosolve provides sophisticated machine learning features. Aerosolve was designed to make feature engineering fast and painless. Easily integrate with JVM projects. 

Read more