hot links

Stuff The Internet Says On Scalability For May 8th, 2015

High Scalability

High Scalability

— 9 min read

Hey, it's HighScalability time:


Not spooky at all. A 1,000 robot self-organizing flash mob.

  • 400 ppm: global CO2 concentration; 13.1 billion: distance in light-years of farthest galaxy
  • Quotable Quotes:
    • Pied Piper: It’s built on a universal compression engine that stacks on any file, data, video or image no matter what size.
    • Bokardo: 1 hour of research saves 10 hours of development time
    • @12Knocksinna: Microsoft uses Cassandra open source tech to help manage the 500+ million events generated by Office 365 hourly (along with SQL and Azure)
    • @antirez: Redis had a lot of client libs ASAP. By reusing the Redis protocol, Disque is getting clients even faster, and 2700 Github stars in 9 days!
    • @blueben: AWS Glacier seems like a great DR option until you realize it costs $180,000 to retrieve your 100TB archive in an emergency.
    • Peter Diamandis: The best way to become a billionaire is to solve a billion-person problem.
    • Cordkillers: YouTube visits up 40% from last year
    • @acroll: "It's about economics not innovation, otherwise we'd all be flying Concorde instead of Jumbo Jets." @JulieMarieMeyer #StrataHadoop
    • @DLoesch: Start time delayed because cable systems are overloaded due to PPV buys. Insane. Don't snooze, don't lose! #MayPac
    • grauenwolf: This is where unit test fanboys piss me off. They claim that they can't use integration tests because they are too slow. I claim that they need integration tests to find their slow queries.
    • nuclearqtip: The open source world needs a standardized trust model for binary artifacts. 
    • Greg Ferro: SDN and SNA are about as similar Model T Ford & any modern car. For the record, no drives a Model T Ford to work everyday. Stop comparing SDN to SNA. Its pointless.
    • Urs Hölzle: Now the decade of work we put into NoSQL is available to everyone using GCP.  One way it shows that we've been working on this longer than anyone else: 99% read latency is 6ms vs ~300ms for other systems.
    • Swardley: Cloud is not about saving money - never was. It's about doing more stuff with exactly the same amount of money. That can cause a real headache in competition. 
    • Johns Hopkins: scientists have discovered that neurons are risk takers: They use minor "DNA surgeries" to toggle their activity levels all day, every day. 

  • Tesla's Powerwall has already sold out. So will Tesla's next gigafactory be a terafactory or a petafactory?

  • Something to keep in mind when hiring: 21% of [NFL] Hall of Fame players were selected in the 4th round or later.

  • Move along, nothing to see here. Brett Slatkin: I wonder how long it will be before people realize that all of this server orchestration business is a waste of time? Ultimately, what you really want is to never think about systems like Borg that schedule processes to run on machines. That's the wrong level of abstraction. You want something like App Engine, vintage 2008 platform as a service, where you run a single command to deploy your system to production with zero configuration.

  • Can any product withstand Aphyr's Jepsen partition torture test? Aeropspike, Elasticsearch, MongoDB, RabbitMQ, Riak, Cassandra, Kafka, NuoDB, Postgres, Redis, all had problems when stress tested under network partitions. Not surprising really, as Aphyr says, "Distributed systems design is really hard." That we find problems in popular well regarded products indicates that "We need formal theory, written proofs, computer verification, and experimental demonstration that our systems make the tradeoffs we think they make. As systems engineers, we continually struggle to erase the assumption of safety before that assumption causes data loss or downtime. We need to clearly document system behaviors so that users can make the right choices. We must understand our systems in order to explain them–and distributed systems are hard to understand." gmagnusson has a good sense of things: "I admire the work that Aphyr does - though at the end of the day, I need to build systems that work for the problem I'm trying to solve (and I have to choose from real things that are available). These technologies in general are trying to address really hard problems and design and architecture is the art of balancing tradeoffs. Nothing is going to be perfect. Yet."  

  • Adrian Colyer, inspired by James Hamilton's "On Designing and Deploying Internet-Scale Services", created an excellent  Internet Scale Services Checklist for designing and developing internet scale services.

  • Is Amazon ECS SOA friendly? This question came up on We are the Amazon EC2 Container Service team – AUAA!: we have Service-Oriented Architecture + Docker, so we have lots of internal APIs behind the scene, we have 3 environments (prod, test, dev), ~40 Services per env (foo-api, bar-api, foobar-api ...) so in current shape (ECS), we need to spend at least 18 USD per service with ELB * 40 services * 3 envs = 2160 USD per month only for ELB (not including traffic). 

  • If you have peaky loads then buying Provisioned IOPS on RDS may be a mistake. Use general purpose SSD drives instead. For every GB of storage you buy, you get 3 free base IOPS, so for 100 GBs of space you get 300 "provisioned" IOPS for free. The result for TextIt: That gave us base IOPS of 750, only slightly less than our previous max of 1,000 IOPS, but now with the ability to burst to 3,000 IOPS for up to 40 minutes at a time. Not only that, but our bill for storage on our instance went from $220/mo to less than $60/mo, even though we now have two and half times the capacity.

  • So you are the boss of me. Google’s Next Calendar Could Manage Real Life for You. More proof that we are transitioning out of the collecting lots of data phase into the telling people what do phase.

  • James Hamilton with a great omnibus on 50 Years of Moore’s Law: IEEE Spectrum Report: What’s interesting to me is there is enough innovation going on in the cloud computing industry that, even though it’s provably not entirely Moore driven, the industry is delivery a near Moore’s law pace of price reductions.

  • This is different. Racking Mac Pros. Imgix does a lot of image processing and they really like the power of Mac Pros to do the job. So they built their own racks to host the oddly shaped Mac Pros. Is it creative or crazy? The comments on Hacker News are dubious. But they did their homework. They like OS X’s graphics frameworks and find the GPUs in the Mac Pro are actually an exceptionally good value per gflop. So why not?

  • The medium is the message now has a whole new meaning. How Facebook’s Algorithm Suppresses Content Diversity (Modestly) and How the Newsfeed Rules Your Clicks: Facebook’s algorithm is a modest suppressor of diversity of content people see on Facebook, and that newsfeed placement is a profoundly powerful gatekeeper for click-through rates. This, not all the roundabout conversation about people’s choices, is the news. 

  • Emoji's are becoming a near-universal method of expression in all languages. That's what Instagram is finding in Emojineering Part 1: Machine Learning for Emoji Trends and Part 2. Scary or a natural evolution?

  • Private or public cloud? That is a question. Zynga, as an early hybrid cloud innovator, has gone public. Zynga's Z Cloud - Scale Fast Or Fail Fast By Merging Private And Public Clouds. Now they are closing their datacenters: “There’s a lot of places that are not strategic for us to have scale and we think not appropriate, like running our own data centers,” Zynga CEO Mark Pincus told investors on a conference call. “We’re going to let Amazon do that. Running a data center is expensive. There are lots of mouths to feed when you have your own data center.”

  • Good story of Stackdive: the evolution of Wayfair’s stack: Wow! Stored procedures along with application code, colo instead of cloud.  We’re really checking all the boxes for Luddite internet companies, aren’t we!?  I can’t tell you how many times I’ve gotten a gobsmacked look and a load of snark from punks at meetups who basically ask me how we can still be in business with those choices.

  • Orleans, Distributed Virtual Actors for Programming and Scalability Comparison. Cool "side-by-side comparison of the Orleans work by Microsoft, a virtual actor system for large-scale distributed programming, with Erlang and Basho's distributed programming abstraction, Riak Core." They are hard to compare in a TLDR. Orleans is a higher level abstraction compared Erlang, providing load balancing, placement, and failure recovery. But when you need more control lower level capabilities can be a savior.

  • OpenDNS is using Elasticsearch as a realtime search engine for log data. They give the setup and lots of configuration details. Looks like it could work nicely. Also, 500 Billion Documents & Counting: Scaling Elasticsearch for Production.

  • At 351 pages this really makes for some light/good reading: Notes on Theory of Distributed Systems CS 465/565: Spring 2014.

  • Interesting talk by Christopher Berner on PrestoML, which is machine learning in SQL. Presto is a distributed SQL query engine. Christopher works on the Presto team at Facebook and previously worked on the news feed ranking team. He added machine learning capabilities to Presto and presto, we get PrestoML. The advantage is now everyone can have access to machine learning, even if they don't have the background, and it's easy because it's SQL!

  • How can you improve performance? Here are some things Neo4j has done in their new release: a new Cost-Based Optimizer has made queries 100x faster; a new In-Memory page cache improved read scaling by 10x; throughput has been increased by batching transaction writes to disk and "by buffering writes to a single transaction log, rather than blocking transactions one at a time where each transaction committed to two transaction logs"; the OS controlled memory mapped cache has been replaced with a purpose built cache was created.

  • Lots of good Course Lecture Videos and Materials from Carnegie Mellon. Includes videos on Computer Architecture, Parallel Computer Architecture, and Memory Systems.

  • Priority is too important to leave to your operating system. Starting in Chrome 41: we've created a task scheduler integrated into the Blink rendering engine. The scheduler is capable of evaluating pending tasks and reordering them so that the most urgent tasks, such as animating and responding to a user's action, are prioritized ahead of others. Lower priority tasks, such as clearing out unused memory, are delayed until the processor has available time. In practice, this results in up to a 40% higher responsiveness to user input when Chrome is working hard to draw frames.

  • Really surprised at Google's position in Just Say No to More End-to-End Tests. Unit tests are fast, but that's because they don't test the interactions that users experience using the real system. If your tests are slow then perform them in parallel all the time. If the test infrastructure is flakey then your system is flakey and should be fixed. If you tests are hard to write then make them easier to write. If you think end-to-end tests aren't written by developers then you are doing it wrong. The only advocate for the user in a testing cycle is the end-to-end test. Don't lose their perspective because of fixable problems.

  • Love this interpretation of the Tower of Babel story. Jeremy England Interview: A New Theory of Life. People try to build a tower and they have just one language, which means they only have one way of talking about the world. So god sends many languages to them because one language isn't enough. Many languages makes collaborating more difficult, but that is preferable to a complete unity of perspective and purpose that sharing a single language gives. You need multiple ways of describing the world if you are going to understand the world.

  • Murat with a bunch more great paper summaries: A Taxonomy of Partitioned Replicated Cloud-based Database Systems; On the use of Clocks to Enforce Consistency in the Cloud; The OS is the control plane; Large-scale cluster management at Google with Borg.

  • MaxCDN with a good explanation of ​How to Use HAProxy to Handle Traffic Spikes. 

  • Fast Approximate Logarithms, Part I: The Basics. A very deep dive David Goldberg at eBay.

  • The Injector: A new Executor for Java: The injector takes a different approach: as far as possible, the worker threads attempt to schedule themselves. The only time a producer schedules a thread is if the pool has been idle for a period, or the work queue is full (in which case it’s stuck waiting for room anyway). In all other situations the worker threads coordinate with each other to match the rate of work arrival.

  • This is disturbing. Globalization unraveling: Yet despite all this activity and enthusiasm, hardly any of the promised returns from globalization have materialized, and what was until recently a taboo topic inside multinationals — to wit, should we reconsider, even rein in, our global growth strategy? — has become an urgent, if still hushed, discussion.

  • Tardigrade: Leveraging Lightweight Virtual Machines to Easily and Efficiently Construct Fault-Tolerant Services: Tardigrade, a system that deploys an existing, unmodified binary as a fault-tolerant service. Tardigrade replicates the service on several machines so that it continues running even when some of them fail. Yet, it keeps the service states synchronized so clients see strongly consistent results. To achieve this efficiently, we use lightweight virtual machine replication.

  • amplab/ampcrowd: A RESTful web service that runs microtasks across multiple crowds, provides quality control techniques, and is easily extensible.

  • logcabin/logcabin: LogCabin is a distributed storage system built on Raft that provides a small amount of highly replicated, consistent storage. It is a reliable place for other distributed systems to store their core metadata and is helpful in solving cluster management issues.

  • Pony: Pony is an object-oriented, actor-model, capabilities-secure, high performance programming language.

  • Greg Linden with another tasty set of Quick links.

Read more