Stuff The Internet Says On Scalability For September 19th, 2014

Hey, it's HighScalability time:

Galactic bolt-hole or supermassive black hole, weighing as much as 21 million suns?

• Quotable Quotes:
  • @debuggist: Chief takeaway from #velocityconf for me: failure happens so monitor for the ones that are important albeit in systems or culture & fix them
  • @Carnage4Life: The real tech bubble is valuations of Google, Facebook & Twitter are inflated by app install ads from unprofitable startups 
  • Jay Parikh: Android has 2x as many users as iOS. However, iOS average revenue per user is 4x higher than Android. 
  • Joe Armstrong: We’ve made a mess. Need to reverse entropy. Quantum mechanics sets limits to ultimate speed of computation. We need Math. Abolish names and places. Build the condenser. Make low-power computers - no net environmental damaged.
  • @reneritchie: iOS 8 is a beefy update. The internet is feeling the strain of millions of downloads. If it’s slow or stuttering, just give it some time.
  • @vishyp: Quip is big on disconnected clients and clients as write-thru caches. Like it! Uses C++ for x-platform. @btaylor #atscale2014
  • @csabacsoma: Instragram: we use Postgres for almost everything now, some Redis and Memcached #AtScale2014
  • @rfberry: "To maximize throughput, we needed an integrated approach to backpressure." #AtScale2014
  • @sharonw: I asked this Facebook / Instagram panel about image upload. Key changes: retry aggressively, and resize and encode on device. #AtScale2014
  • Igor Zaika: Most of the developers who work on Microsoft Office are younger than the codebase. 
  • Chris Marra: There are about 10k different device models accessing Facebook. Designing for high-end smartphones does not cut it.
  • @Carnage4Life: Apple spends $100m on a U2 album you don't want. Microsoft spends that on ads you don't like. Amazon spends it on free shipping #Marketing
  • @beerops: Laziness as a multiplier: train other people to do what you can do to remove yourself as a bottleneck #velocityconf
  • @xaprb: "Failure is a feature of complex systems." - #velocityconf
  • @dalmaer: "Our mobile app is a write through cache (SQLite) to the source of truth (MySQL on AWS)" -- @btaylor #AtScale2014
  • @herminghaus: @BenedictEvans Designed an 11.4TB patent retrieval system in 1993 with slow WORM robots. Cost $140m. Now <$1000.- at BestBuy.
  • @BenedictEvans: You can't ask people to decide on a trade-off when they have experience of one side but not the other.


• Caching at Scale. There's a need to better manage caching, especially under failure conditions. Solutions are generally in the form of a proxy layer above memcached. Along these lines Box created Tron, Twitter created TwemProxy, and Facebook created a value meal in McRouter. Database people have always countered, why have a separate cache, just build a cache into the database? This hasn't worked for various reasons, mostly because a database always cares more about being a good database rather than being a good cache. Vitess wants to fix that. Vitess is an open-source system written in Go used at Youtube that: challenges the paradigm of treating caching as a separate layer by directly addressing the issues of database scalability and by modifying the handling of SQL queries.  


• Talk about your Chaos Godzilla. Facebook Turned Off Entire Data Center to Test Resiliency. Before flipping that switch there must have been a little pause, perhaps thinking this wouldn't be prudent, but damn the torpedoes and full speed ahead. Apparently some issues were found, but it went fairly smoothly. Hazaa for the chutzpah.


• Best LAN party ever? Researchers twist four radio beams together to achieve high data transmission speeds. The researchers reached data transmission rates of 32 gigabits per second across 2.5 meters of free space in a basement lab.


• This is an understatement. iOS 8, thoroughly reviewed. An amazing job. A big take away for me is Apple is systematically removing reasons not to buy an iPhone. Bigger phone. Check. Configurable keyboard. Check. Extensions that display in the today view and allow app cooperation. Check. Another take away is Apple is abandoning simplicity for configurability, which is embracing complexity, which is a potential experience killer.


• Why Loggly Chose AWS Route 53 over Elastic Load Balancing: Route 53 turned out to be the best way for Loggly to take advantage of our high-performance collectors given our huge log volumes, unpredictable variations, and constant growth in our business. It aligns with the collectors’ core purposes: To collect data at network line speed with zero loss, and it allows us to benefit from the elasticity of all of the AWS services we use at Loggly.


• Who do I talk to at Alaska Airlines to tell them it doesn't matter if you open both entrances on to the plane if you have a single queue with head-of-line blocking?


• Mohammad Almalkawi is the most awesomeness note taking machine ever. Here are his notes on the Mobile Track at Scale 2014. Great stuff.


• Facebook's iOS Infrastructure. I understand enough about iOS programming now to understand most of what's going on here, but it's still amazingly complicated to make GUIs work. Some points...The way to make really slow ios apps. Use core data. Slow setup, slow runtime, hard to use. Use many small caches. Really great stuff on creating performant code from real time update streams. Very difficult to handle share state across multiple threads. The secret is a declarative immutable programming model in both the model and the view. They created a very slick system to make this a reality. It simplified code greatly and is definitely non-obvious.


• Dropbox and Facebook use C++ as a cross platform layer for mobile apps. For Dropbox the UI is native and the app logic is all in C++.


• It's not magic, but it's close. KDD Retro: Google Knowledge Vault and Topic Modeling. Great overview of data mining techniques, including Google's Knowledge Vault, "focusing on entity extraction, data fusion and resolving schema heterogeneity."


• Some good advice. 10 things I learned about rapidly scaling websites with Azure: Measure everything early; Configure alerts; Max out the instance count from the beginning; Scale up early;  Azure is amazingly resilient;  Get lean early; Steal other peoples’ bandwidth now!; Always, always be ready to release at the drop of a hat; Experiment with instance size and auto scale parameters to find your happy place; Relax, it’s only money!


• Understanding weak isolation is a serious problem: As we develop new systems, we can avoid making the same mistakes as our architectural ancestors by focusing on applications, not mechanisms. I’d personally welcome a moratorium on writing papers on new isolation, data consistency, and read/write transaction models unless there’s a clear and specific set of motivating and specific application-driven use cases.


• dougrohrer is channeling the Strangeloop conference through a live notebook.


• Here are all the slides from CppCon 2014. 


• Jakob Engblom with expansive commentary on an old but still relevant article by Jim Gray: Why DO Computers Fail? Keys to high availability: fail fast, modularity, paired components, transaction mechanisms.


• Many videos for the @scale conference are up and available.


• Looks cool and gives good explanation. Suture - Supervisor Trees for Go: It is a process that tells the runtime it is interested in whether a target process dies, and when the runtime tells it that it has, it takes the desired action, usually restarting it (though there are some other exotic options).


• Here's a wiki with a great list of stream related papers. 


• Nothing like a good language war to stir things up. Go Channels Are Not Enough. Good discussions on reddit and on Hacker News. Idealists vs pragmatists vs realists. Those are the real levels of abstraction.


• Rest.li: an open source REST framework for building robust, scalable RESTful architectures using type-safe bindings and asynchronous, non-blocking IO. Rest.li fills a niche for applying RESTful principals at scale with an end-to-end developer workflow for buildings REST APIs that promotes clean REST practices, uniform interface design and consistent data modeling.


• Chisel: an open-source hardware construction language developed at UC Berkeley that supports advanced hardware design using highly parameterized generators and layered domain-specific hardware languages.


• PeerMesh: GIS tiles and objects will be quickly and cost effectively delivered by peers and it will reduce the server load. In the mean time more peers means more hosted content. If peers are slow and Peermesh server is inaccessible, maps and site will load normally and peermesh will handle errors.