Stuff The Internet Says On Scalability For February 22, 2013

Hey, it's HighScalability time:

• Quotable Quotes:
  • @p337er: I have committed some truly horrendous crimes against scalability today.
  • @ErrataRob: doubling performance doesn't double scalability.
  • @rsingel: In 2008 when Yahoo.com  linked out, I had a Wired story get 1M visitors in an hour from their homepage.
  • @philiph: Lets solve this scalability problem with a queuing system
  • @jaykreps: Transferring data across data centers? Read this page and go tune your TCP buffer sizes...
  • @gwestr: In which the node community showers schadenfreude upon the rails community for "scalability is not my problem" architectures
  • @pbailis:  Makes sense, though I think there's a tradeoff re: coordination and scalability (always homogeneous vs dynamically heterogenous)
  • @pembleton: To summarize Yoav's philosophy: we started as quick as we can and then we accelerated #operationgrandma in #reversim
  • @surfichris: “We chose Heroku because we believed we could just `heroku scale web=X` when needed.” - Yeah, because scalability is magic, and unlimited...
  • kent langley: cloud computing had already been brewing for decades with its roots reaching far back in time. Grids, clusters and more were all precursors. However, it is striking how far things have come in just about five years.
  • @caspereeko: So Youtube views count is being generated from MapReduce from Google CDNs access logs and stored later in cache
  • @michaellperry: Three reasons for using async: offloading (do work on different thread), concurrency (start multiple), and scalability (use fewer resources)
  • @otomillo: one thing to remember about big O notation, it’s that it’s a measure of scalability; not a measure of performance as is commonly believed


• Late refund? The IRS may be having a few Congress Induced Scalability Problems...IRS Statement on "Where's My Refund?" Tool. 


• The eternal Law of Frustration predicts every competent person eventually gets so tired of working with the messiness of the real world that they just want to press reset and and start all over again. Greenfields are so...well...green. No, I'm not talking about another master criminal trying to take over the world, but Google creating a replacement web protocol on top of UDP. But it always ends up the same, you realize Layer 1 needs to be replaced and that it's the only layer that really exists. Good discussion on Hacker News.


• As we've seen major web properties transition back to black hole portal type sites, where attention enters and never escapes the event horizon, Pinterest thinks we are also trending back to human curated content, with humans doing the indexing. Or is that just what the machines want them to think?


• Complexity accretes. Rails, You Have Turned into Java. Congratulations! Which is whey every few years we need to all get naked, have a party, declare a Jubilee, and start over again, hopefully a little wiser.


• What is NoSQL and is it pornography? I think Emin's lesson is you can't turn something into pornography by adding more layers of clothing.


• Somebody recently mentioned I only seem to talk about Amazon on HighScalability. That's mostly true because as a hungry beast I forage off the land and what I usually find are little packages of Amazon fruits and nuts. If you have other consumables let me know. For example, Microsoft Azure overtakes Amazon's cloud in performance test: Azure was 56% faster than AWS S3 when it came to writing data into its cloud, and 39% faster when reading data. All the usual benchmark hues and cries apply. 


• Scalability: it's the question that drives us: The point I'm trying to make is that "performance" and "scalability" are orthogonal problems. When trying to teach engineers how to fix scalability, their most common question is "but won't that hurt performance". The answer is that it almost doesn't matter how much you hurt performance as long as it makes the application scale.


• I've never been a big fan of REST outside of the world of documents, multiplying verbs is the same as multiplying objects to me, but LinkedIn has some services handling several thousand queries per second per instance, and they have a great description of their approach in Rest.li: RESTful Service Architecture at Scale: we needed a standard way to describe the resources available in our service-oriented architecture...standardize common API operations...scalable and asynchronous...APIs would be fast and easy for our developers.


• Backblaze pioneered DIY massive and storage and they now have a new release: 180TB of Good Vibrations – Storage Pod 3.0, looking fantastic in red. Also an informative list of lessons learned: firmware revisions matter, let your vendors do your testing, don't make random design changes, power is more than just watts, KISS, things change. 


• A bunch of good ways to improve Erlang Scalability: NIFs are considered harmful, BIFs can also be harmful, Binary strings use less memory, Use binaries for buffering and streaming, Prefer list comprehensions over lists:map/2, It's a bad idea to use gen_server for everything, ets public or protected tables will give you the performance boost.


• The Stinger Initiative: Making Apache Hive 100 Times Faster. Not quite real-time, but it's good at answering:  human-time use cases (i.e. queries in the 5-30 second range) such as big data exploration, visualization, and parameterized reports. 


• For realtime processing look at Building a real-time data pipeline with Storm:  Storm can read the stream of logs from the adservers, aggregate them on the fly and store the aggregated data directly in the data warehouse as soon as they’re available.


• At certain point the database becomes the bottleneck and Scaling PostgreSQL Performance Using Table Partitioning describes how to uncork the bottle. Table partitioning: take one massive table and split it into many smaller tables – these smaller tables are called partitions or child tables. 


• First reports don't find Amazon's Redshift customers feeling blue. Amazon Redshift - Review and First Impressions: price is unbeatable...getting started is easy...yet to see it deliver the sub-second performance... the real hurdle for most companies is going to be database design and optimization.


• Open Letter: Web Scalability. A good punchy intro into the subject.


• Look straight into the face of complexity and know: there's an algo­rithm capable of iden­ti­fying the subset of components—or nodes—that are nec­es­sary to reveal a com­plex system’s overall nature.


• Markus Winand with an excellent trip report on the Free and Open source Software Developers’ European Meeting (FOSDEM).


• AppNeta explores how request queueing affects applications and what you can do about it. Cool graph of random load balancing vs dynos with explanation. Look for build up of queue depths and increased processing times. Random load balancing means one worker queue can have several entries while others have zero. Also, A randomized algorithm that fails with near certainty. 


• Lots of processing in parallel you need a lot of cores: Tilera unveils 72-core processor chip for data networks and infrastructure.


• Excellent getting started look at How-to: Use Apache ZooKeeper to Build Distributed Apps (and Why) from Cloudera. Tricky, but very useful for membership and name services, distributed mutexes, master election, event broadcasting, async message passing, centralized configuration management.