hot links

Stuff The Internet Says On Scalability For July 24th, 2015

High Scalability

High Scalability

— 8 min read

Hey, it's HighScalability time:


Walt Disney doesn't mouse around. Here's how he makes a goofy business plan.

  • 81%: AWS YOY growth; 400: hours of video uploaded to YouTube EVERY MINUTE; 9,000: # of mineable asteroids near earth; 1,400: light years to Earth's high latency backup node; 10K: in the future hard disks will be this many times faster 
  • Quotable Quotes:
    • @BenedictEvans: Chinese govt: At the end of 2014 China had 112.7 billion static webpages and 77.2 billion dynamic webpages. They used 9,310,312,446,467 KB
    • Michael Franklin (AMPLab): This is always a pendulum where you swing from highly distributed to more centralized and back in. My guess is there’s going to be another swing of the pendulum, where we really need to start thinking about how do you distribute processing throughout a wide area network.
    • Sherlock Holmes: Singularity is almost invariably a clue. 
    • @jpetazzo: OH: "In any team you need a tank, a healer, a damage dealer, someone with crowd control abilities, and another who knows iptables"
    • Jeff Sussna: Ultimately, the impact of containers will reach even beyond IT, and play a part in transforming the entire nature of the enterprise. 
    • @CarlosAlimurung: Impressive.  The number of #youtube channels making six figures grew by 50%. 
    • harlowja: Overall, no the community isn't perfect, yes there are issues, yes it burns some people out, but software isn't rainbows and butterflies after all.
    • werner: BTW nobody wants eventual consistency, it is a fact of live among many trade-offs. I would rather not expose it but it comes with other advantages ...
    • @VideoInkNews: We’re focused on our top three priorities – mobile, mobile and mobile, said @YouTube CEO @SusanWojcicki #VidCon2015 #keynote
    • Ivan Pepelnjak: Use a combination of MPLS/VPN and Internet VPN, or Internet VPN with 3G backup. Use multiple access methods, so the cable-seeking backhoe doesn’t bring down all uplinks.
    • @randybias: Repeat after me: containers do little to enable application portability.  If you want portability use a PaaS.  PaaS != Containers.
    • William Chan~ I see the tech industry working on helping rich people find taxis more easily, selling ads more effectively, or building sexting apps, solving abstract problems that enable them to make money off silly products for the 1%, rather than solving important problems for people that need help.
    • Web Design: The First 100 Years: A further symptom of our exponential hangover is bloat. As soon as a system shows signs of performance, developers will add enough abstraction to make it borderline unusable. Software forever remains at the limits of what people will put up with. Developers and designers together create overweight systems in hopes that the hardware will catch up in time and cover their mistakes.
    • Our Measures of the Economy Don’t Work: Right now, we are trying to measure a 21st-century economy with 20th-century tools. Technology makes life easier and more convenient in all sorts of ways, most of which simply aren’t captured by tools that were developed to measure things like pig iron output.
    • Deep Consequences: So it seems to me we should think of language, and cognitive sciences in general, as complex phenomena (like climate science or financial economics), rather than law-based systems (like those found in chemistry or physics).

  • Can't we all just get along? And by "we" I mean humans and robots. Maybe. Inside Amazon shows by example how one new utopian community is bridging the categorical divide. Forget all your skepticism and technopanic, humans and robots can really work together in a highly efficient system.

  • A Brief History of Scaling LinkedIn. Not so brief actually. Lots of really good details. They of course started off with a monolith and ended up with a service oriented architecture. One of the most interesting ideas is the super block: groupings of backend services with a single access API. This allows us to have a specific team optimize the block, while keeping our call graph in check for each client.

  • If you want to move at the speed of software doesn't your datacenter infrastructure have to move at the same speed? Network Break 45 from Packet Pushers talks about an open source virtual software router, CloudRouter, running the latest release of OpenDaylight's SDN controller and ONOS. The idea is to make a dead simple router you can just instantiate as needed. Greg Ferro makes the point that if you don't have to care if you are starting 100 or 1000 virtual routers it changes how you go about building infrastructure. Running a Cisco Router, and F5 load balancer, and a virtual firewall, how much will it cost to spin up virtual datacenters for 100s of developers? How long will it take? How much will it cost? How does it even work? 

  • Jeff Atwood is a rare combination of a kind, well spoken, intelligent, and wise human. Jeff Atwood on building Discourse, Stack Exchange, and Coding Horror. Well worth a listen. Discourse is RoR on the backend. A big ball of Javascript on the frontend. Ember.js. PostgreSQL. Redis. Discourse was open source so they needed an open source stack, which is why they chose Ruby. They get a lot of help from the open source community such that they can take their team relatively small and get a lot done. Philosophy of three things: what 3 things are you going to accomplish today?

  • In the future will we really play and work together in virtual worlds? Robert Scoble thinks so. That's how excited he is about Oculus Rift. I think those are third level virtual worlds, just another subcreation of the simulation we already live in. And of course our brain-in-a-vat minds are the second level. Do we need another level?

  • Uh-oh, a robot just passed the self-awareness test. Now that you know how easy it is to appear to be self-aware, do you now look at your fellow humans any differently?

  • Google Analytics-driven game development by example. Love the process here: A few months ago my friend and I performed something like a gamedev experiment. It was 'just create a trashy game, update it as frequently as possible and make it better based on users’ and Google Analytics’ feedback'. We didn't want to spend 6 months on developing another game that will get stuck among thousands of other games on Google Play. Results: By adding more content to the game we got more retention (from 46.2% to 70.5%); By adding more content to the game we got longer average session duration (from 3:03 to 8:20); We do not only gather the Google Analytics events but also we analyze them!

  • The container wars started they have. Under the Hood of Amazon EC2 Container Service. For an overview by jpetazzo on the original announcement take a look at My notes on Amazon's ECS (EC2 Container Service), aka Docker on AWS along with comments on Hacker News. To hard to handicap at the moment. This stuff is so complex and moving so fast...who can know?

  • The Computer Chip That Never Forgets: Melding spin-based logic and memory could lead to low-power, instant-on electronics. Wow, what an incredible tour through different memory technologies, their properties, their strength, and their weaknesses. MeRAM is a thin-film spintronic memories that can be densely packed and written to in a nanosecond using very little energy. It could potentially supplant all but the fastest SRAM on a chip. There's also the possibility of making logic that is nonvolatile and so acts as its own form of memory.

  • How do SSDs work? ExtremeTech with a good and relatively short explanation. I was however disappointed that the role of ritual magic was not explored more fully.

  • Transitioning to Flux Architecture. If you are interested in the declarative UI building approach using Facebook's React and Flux then this is a good post because it uses actual code examples that evolve with different approaches. Which isn't to say I understand it yet.

  • Time is an apriori category in our minds, but the concept is a bit mushy. How the brain perceives time. 

  • Using your database as a queue? Mikhail Panchenko with a good analysis of why that's not the best idea. Thoughts Evoked By CircleCI's July 2015 Outage. Inserting into table requires building indexes which require locking. Dequeing requires updating indexing and even more locking. So at a certain scale and update load the "queue" just stops performing. When this happened at Flickr says Mikhail they moved to a bunch of lists in Redis. Next is a great section on the dangers of unnecessary coupling and resource sharting between systems. The bright idea: think about the sources of traffic to your infrastructure and how they interact with each other. Over time, it pays even more to have parallel, as-decoupled-as-time-allows paths through your system, any of which can be shut off in isolation.

  • Microsoft Azure vs Amazon Web Services (AWS) Services & Feature Mapping. Very comprehensive. Too large event to take a whack at. Essential if you need to compare.

  • Feature flags can be the goto of production runtimes. Here are some thoughts how to them better from Yeller. Effective Feature Flags: . The Double Flag; Keep feature flags visible; Don’t let feature flags break your reliability. 

  • Some surprises here. 14 Common DDR DRAM Myths Debunked: All DDR3 Is The Same; Just Add More DRAM; There Are Only A Few DIMM Manufacturers; 3200 MT/s Support Means You Can Use Any DRAM; Mixed DRAM Runs At The Speed (Or Timings) Of The Slowest DIMM; It’s Cheaper To Buy Two Sets Of DIMMs Than Larger, More Expensive Sets; and more.

  • BGP and the System of Trust that Runs the Internet Part 1 and Part 2: What most Internet users don’t know, but many are beginning to discover, is the entire framework that holds the Internet together can be quite fragile. In fact the system in charge of correctly routing Internet traffic is largely built on a collective trust.

  • Nice story of how one person can do it all, with a little help from pretty much everyone. How I run a bootstrapped marketplace with eleven different properties and 5,000 vendors — by myself: I have free accounts, and I rely on my users to spread the word and drive more users onto my platform; Heavy use of drip marketing and Buffer so I only need to work one day a week; I launched a free wedding planning app to drive potential customers to my users so I didn't have to rely on SEO;  launched a blog so I have an excuse to repeatedly engage with my users; Heavy, heavy use of canned responses and Boomerang; I hire contractors/remote workers for any easily repeatable task; Take advantage of your users; 

  • Designed to Fail: Embracing failure means making a conscious decision to purposely fail...The key to being highly available is learning to be partially available...we made explicit decisions to drop messages on the floor if we detect the system is becoming overloaded...Other techniques include building kill switches into service calls and routers...Many times, it’s our own internal services which cause the biggest DoS attacks on ourselves.

  • Excellent introduction. Distributed Data Stores for Mere Mortals. Some good discussion on reddit.

  • A meditation on How we [SparkTG] store 400M phone number data with fast lookups: 1 million numbers in the same series can be searched in 3 seconds as compared to PostgreSQL taking 18 seconds after caching.

  • Workers: no longer needed?: the only time in recent history where IT appears to have led to an acceleration in productivity growth was the late 1990s, which was the result of increased investment in high-tech equipment. I went into writing After the New Economy thinking that the productivity acceleration was a mirage, but it wasn’t. But then it fell apart, because corporations prefer shoveling out cash to their shareholders to investing it.

  • The Power of Ten – Rules for Developing Safety Critical Code for NASA/JPL Laboratory for Reliable Software. More permissive than you might imagine. Um Ah has a great comment explaining how the On-board Shuttle Group at JSC had only one bug in 400,000 lines of code. And yes, it does take a bit of process.

  • Ibis: a new Python data analysis framework with the goal of enabling data scientists and data engineers to be as productive working with big data as they are working with small and medium data today.

Read more