Stuff The Internet Says On Scalability For March 14th, 2014

Hey, it's HighScalability time:

  • Quotable Quotes:
    • The Master Switch: History shows a typical progression of information technologies: from somebody’s hobby to somebody’s industry; from jury-rigged contraption to slick production marvel; from a freely accessible channel to one strictly controlled by a single corporation or cartel—from open to closed system.
    • @adrianco: #qconlondon @russmiles on PaaS "As old as I am, a leaky abstraction would be awful..."
    • @Obdurodon: "Scaling is hard.  Let's make excuses."
    • @TomRoyce: @jeffjarvis the rot is deep... The New Jersey pols just used Tesla to shake down the car dealers.
    • @CompSciFact: "The cheapest, fastest and most reliable components of a computer system are those that aren't there." -- Gordon Bell
    • @glyph: “Eventually consistent” is just another way to say “not consistent right now”.
    • @nutshell: LinkedIn is shutting down access to their APIs for CRMs (unless you’re Salesforce or Microsoft). Support open APIs!
    • Tim Berners-Lee: I never expected all these cats.
    • @muratdemirbas: "Simple clear purpose&principles give rise to complex&intelligent behavior.
      Complex rules&regulations give rise to simple&stupid behavior."
    • @BonzoESC: “Duplication is far cheaper than the wrong abstraction.” @sandimetz @rbonales 
    • @BenedictEvans: Umeng: there are 700m active smartphones and tablets in China.

  • Scale matters object lesson number infinity: HBO Go Crashes During True Detective Finale. Perhaps make HBO Go available without a cable package and maybe you'll have money to scale the service? Think peak. But wait, Dan Rayburn says bandwidth was not the problem, it's other parts of the system, which is why Internet TV will never be as reliable as broadcast TV. Still, I'd like to cut the cord.

  • Turns out ecommerce over messaging works well...quite well. Retailers Are Striking Gold with Instagram: Fox and Fawn, items often sell out within minutes of the picture being posted on Instagram.

  • Even Facebook's infrastructure struggles when a new feature becomes an unexpected hit. That's the situation described in an engaging story: Looking back on “Look Back” videos. Look Back's are one minute videos generated from a user's pics and posts. For the release they planned on 187 Gbps more bandwidth and 25 petabytes of disk. To get the rendering done they highly parallelized the pipeline. CDNs were alerted. Internal tests on employees found a few bugs. Less storage was actually needed because the video could be regenerated so a high replication factor wasn't needed. Go time! The videos were an unexpected hit with a 40% reshare instead of the projected 10% reshare. It seems people like themselves...a lot. Overnight 30 teams cooperated to move tens of thousands of machines over to rendering. Good story. Though I'm disappointed it didn't have its own Look Back video. Stories are people too.

  • Why Google's services aren't really free. We all help train the beast. A Glimpse of Google, NASA & Peter Norvig + The Restaurant at the End of the Universe: Algorithms behave differently as they churn thru more data. For example in the figure, the Blue algorithm was better with a million training dataset. If one had stopped at that scale, one would think of optimizing that algorithm for better performance. But as the scale increased the purple algorithm started showing promise – in fact the blue one starts deteriorating at larger scale; In general, Google prefers algorithms that get better with data. Not all algorithms are like that, but Google likes to after the ones with this type of performance characteristic. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...


Paper: Scalable Eventually Consistent Counters over Unreliable Networks

Counting at scale in a distributed environment is surprisingly hard. And it's a subject we've covered before in various ways: Big Data Counting: How to count a billion distinct objects using only 1.5KB of Memory, How to update video views count effectively?, Numbers Everyone Should Know (sharded counters).

Kellabyte (which is an excellent blog) in Scalable Eventually Consistent Counters talks about how the Cassandra counter implementation scores well on the scalability and high availability front, but in so doing has "over and under counting problem in partitioned environments."

Which is often fine. But if you want more accuracy there's a PN-counter, which is a CRDT (convergent replicated data type) where "all the changes made to a counter on each node rather than storing and modifying a single value so that you can merge all the values into the proper final value. Of course the trade-off here is additional storage and processing but there are ways to optimize this."

And there's a paper you can count on that goes into more details: Scalable Eventually Consistent Counters over Unreliable Networks:

Click to read more ...


Douglas Adams - 3 Rules that Describe Our Reactions to Technologies

Chris Dixon unearthed a great quote from Douglas Adams on the nature of technological adoption that unsurprisingly hits the mark in our ever changing and evolving world:

  1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
  2. Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
  3. Anything invented after you’re thirty-five is against the natural order of things

Some that come to mind: horse to car, index card to online search, PC to mobile, web to app, portal to messaging, Newton to Einstein, oil to electric, rock to rap, Aquinas to Bacon, buying to renting, files to streaming, network TV to cordkilling, broadcast to social, programming CPUs to programming biology, server to cloud, vm to container, wired to wireless, long read to TLDR, privacy to public to ephemeral, paper based news aggregation to digital aggregation, checks to online banking, gold to fiat to bitcoin, linear to exponential growth, large to small teams, to a world that ignores you to a world the responds to you, nation states to who knows what, a military of people to a Military of Things, and weekly versus binge watching. Any others?


Building a Social Music Service Using AWS, Scala, Akka, Play, MongoDB, and Elasticsearch

This is a guest repost by Rotem Hermon, former Chief Architect for, on the architecture and scaling considerations behind making a startup music service. is a social music service that helps people discover great music shared by their friends, and also introduces them to their “music soulmates” - people outside their immediate social circle that shares a similar taste in music.

Serendip is running on AWS and is built on the following stack: scala (and some Java), akka (for handling concurrency), Play framework (for the web and API front-ends), MongoDB and Elasticsearch.

Choosing the stack

One of the challenges of building serendip was the need to handle a large amount of data from day one, since a main feature of serendip is that it collects every piece of music being shared on Twitter from public music services. So when we approached the question of choosing the language and technologies to use, an important consideration was the ability to scale.

The JVM seemed the right basis for our system as for its proven performance and tooling. It's also the language of choice for a lot of open source system (like Elasticsearch) which enables using their native clients - a big plus.

When we looked at the JVM ecosystem, scala stood out as an interesting language option that allowed a modern approach to writing code, while keeping full interoperability with Java. Another argument in favour of scala was the akka actor framework which seemed to be a good fit for a stream processing infrastructure (and indeed it was!). The Play web framework was just starting to get some adoption and looked promising. Back when we started, at the very beginning of 2011, these were still kind of bleeding edge technologies. So of course we were very pleased that by the end of 2011 scala and akka consolidated to become Typesafe, with Play joining in shortly after.

MongoDB was chosen for its combination of developer friendliness, ease of use, feature set and possible scalability (using auto-sharding). We learned very soon that the way we wanted to use and query our data will require creating a lot of big indexes on MongoDB, which will cause us to be hitting performance and memory issues pretty fast. So we kept using MongoDB mainly as a key-value document store, also relying on its atomic increments for several features that required counters.
With this type of usage MongoDB turned out to be pretty solid. It is also rather easy to operate, but mainly because we managed to avoid using sharding and went with a single replica-set (the sharding architecture of MongoDB is pretty complex).

For querying our data we needed a system with full blown search capabilities. Out of the possible open source search solutions, Elasticsearch came as the most scalable and cloud oriented system. Its dynamic indexing schema and the many search and faceting possibilities it provides allowed us to build many features on top of it, making it a central component in our architecture.

We chose to manage both MongoDB and Elasticsearch ourselves and not use a hosted solution for two main reasons. First, we wanted full control over both systems. We did not want to depend on another element for software upgrades/downgrades. And second, the amount of data we process meant that a hosted solution was more expensive than managing it directly on EC2 ourselves.

Some numbers

Click to read more ...


Let's Play a Game of Take It or Leave It - Game 1

The way this game is played is you read a few statements on some hot topics below. If you agree with a statement then you “take it”; if you disagree then you “leave it.” And if you are so moved please write a convincing comment as to why. Got it?

  1. Snowden vs. the State. Snowden represents true the spirit of freedom and is not a threat to all we hold dear.

  2. Walled Garden vs. Federated Freedom. The Walled Garden has won the last decade. The cycle of life will return the balance and federated services will once again win the day.

  3. Mobile + messaging vs. Le Web. Mobile + messaging is eating search and the web, changing the way things are found, discovered, and bought.

  4. Fiat vs. Cryptocurrency. BitCoin has had its 400 million dollars of fame, it’s on the way out, a tulip gone out of bloom.

  5. True Detective vs. The Field. True Detective is the best show on TV, ever. Wired and Breaking Bad need not apply.


Stuff The Internet Says On Scalability For March 7th, 2014

Hey, it's HighScalability time:

Twitter valiantly survived an Oscar DDoS attack by non-state actors.
  • Several Billion: Apple iMessages per Day along with 40 billion notifications and 15 to 20 million FaceTime calls. Take that WhatsApp. Their architecture? Hey, this is Apple, only the Shadow knows.
  • 200 bit quantum computer: more states than atoms in the universe; 10 million matches: Tinder's per day catch; $1 billion: Kickstarter's long tail pledge funding achievement
  • Quotable Quotes:
    • @cstross: Let me repeat that: 100,000 ARM processors will cost you a total of $75,000 and probably fit in your jacket pocket.
    • @openflow: "You can no longer separate compute, storage, and networking." -- @vkhosla #ONS2014
    • @HackerNewsOnion: New node.js co-working space has 1 table and everyone takes turns
    • @chrismunns: we're reaching the point where ease and low cost of doing DDOS attacks means you shouldn't serve anything directly out of your origin
    • @rilt: Mysql dead, Cassandra now in production using @DataStax python driver.
    • @CompSciFact: "No engineered structure is designed to be built and then neglected or ignored." -- Henry Petroski
    • Arundhati Roy: Revolutions can, and often have, begun with reading.
    • Brett Slatkin: 3D printing is to design what continuous deployment is to code.
  • Well Facebook got on that right quick: Facebook wants to use drones to blanket remote regions with Internet. We talked about a drone driven Internet back in January. This is good news IMHO. Facebook will have the resources to make this really happen. Hopefully. Maybe. Cross your fingers.

  • A vast hidden surveillance network runs across America, powered by the repo industry. This intelligence database was powered by individuals driving around and taking pictures of licence plates to track cars. Imagine how Google Glass will enable the tracking of people, without any three letter government agencies in the loop. Crowdsourcing is fun!

  • Francis Bacon way back in the 1700s was all over BigData with his ant, spider, and honey bee analogy:  Good scientists are not like ants (mindlessly gathering data) or spiders (spinning empty theories).  Instead, they are like bees, transforming nature into a nourishing product. This essay examines Bacon's "middle way" by elucidating the means he proposes to turn experience and insight into understanding.  The human intellect relies on "machines" to extend perceptual limits, check impulsive imaginations, and reveal nature's latent causal structure, or “forms.”

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...


10 Things You Should Know About Running MongoDB at Scale

Guest post by Asya Kamsky, Principal Solutions Architect at MongoDB.

This post outlines ten things you need to know for operating MongoDB at scale based on my experience working with MongoDB customers and open source users:

  1. MongoDB requires DevOps, too. MongoDB is a database. Like any other data store, it requires capacity planning, tuning, monitoring, and maintenance. Just because it's easy to install and get started and it fits the developer paradigm more naturally than a relational database, don't assume that MongoDB doesn't need proper care and feeding. And just because it performs super-fast on a small sample dataset in development doesn't mean you can get away without having a good schema and indexing strategy, as well as the right hardware resources in production! But if you prepare well and understand the best practices, operating large MongoDB clusters can be boring instead of nerve-wracking.
  2. Successful MongoDB users monitor everything and prepare for growth. Tracking current capacity and capacity planning are essential practices in any database system, and MongoDB is no different. You need to know how much work your cluster is currently capable of sustaining and what demands will be placed on it during times of highest use. If you don't notice growing load on your servers you'll eventually get caught without enough capacity. To monitor your MongoDB deployment, you can use MongoDB Management Service (MMS) to visualize your operations by viewing the opscounters (operation counters) chart:
  3. The obstacles to scaling performance as your usage grows may not be what you'd expect. Having seen hundreds of users' deployments, the performance bottlenecks usually are (in this order):

    Click to read more ...


Sponsored Post: Uber, ScaleOut Software, Couchbase, Tokutek, Logentries, Booking, Apple, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7  

Who's Hiring?

  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • C++ Senior Developer and Architect- Maps. The Maps Team is looking for a senior developer and architect to support and grow some of the core backend services that support Apple Map's Front End Services. Please apply here.  
    • Senior Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives that drive projects forward to completion. Please apply here.
    • Software Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives that drive projects forward to completion. Please apply here.
    • Quality Assurance Engineer. The iOS Systems team is looking for a Quality Assurance engineer. In this role you will be expected to work hand-in-hand with the software engineering team to find and diagnose software defects. Please apply here.

  • Join the team that scales Uber supply globally! Our supply engineering team is responsible for prototyping, building, and maintaining the partner-facing platform. We're looking for experienced back-end developers who care about developing highly scalable services. Apply at

  • We need awesome people @ - We want YOU! Come design next
    generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Apply:

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

Fun and Informative Events

  • How to Scale MySQL for Big Data Applications. A Guide to Evaluating TokuDB on March 20th at 1pm ET. You can do more than you think with the MySQL you already have. Learn how to use MySQL or MariaDB in Big Data applications by simply upgrading the storage engine with TokuDB. Register now.

  • Snapdeal Selects Aerospike over MongoDB, Couchbase and Redis to Improve Shopper Satisfaction. After experiencing 500% growth in 2013, Snapdeal, India’s largest online marketplace, switched from 10 MongoDB servers to just two Linux servers on Amazon EC2 with Aerospike, and reduced response times to less than a millisecond. Read the case study.

Cool Products and Services

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • As one of the fastest growing VoIP services in the world Viber has replaced MongoDB with Couchbase Server, supporting 100,000+ operations per second in the short term and 1,000,000+ operations per second in the long term for their third generation architecture.  See the full story on the Viber switch.

  • Log management made easy with Logentries Billions of log events analyzed every day to unlock insights from the log data the matters to you. Simply powerful search, tagging, alerts, live tail and more for all of your log data. Automated AWS log collection and analytics, including CloudWatch events. 

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • MongoDB Backup Free Usage Tier Announced. We're pleased to introduce the free usage tier to MongoDB Management Service (MMS). MMS Backup provides point-in-time recovery for replica sets and consistent snapshots for sharded systems with minimal performance impact. Start backing up today at

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...


The “Four Hamiltons” Framework for Mitigating Faults in the Cloud: Avoid it, Mask it, Bound it, Fix it Fast

This is a guest post by Patrick Eaton, Software Engineer and Distributed Systems Architect at Stackdriver.

Stackdriver provides intelligent monitoring-as-a-service for cloud hosted applications.  Behind this easy-to-use service is a large distributed system for collecting and storing metrics and events, monitoring and alerting on them, analyzing them, and serving up all the results in a web UI.  Because we ourselves run in the cloud (mostly on AWS), we spend a lot of time thinking about how to deal with faults in the cloud.  We have developed a framework for thinking about fault mitigation for large, cloud-hosted systems.  We endearingly call this framework the “Four Hamiltons” because it is inspired by an article from James Hamilton, the Vice President and Distinguished Engineer at Amazon Web Services.

The article that led to this framework is called “The Power Failure Seen Around the World.  Hamilton analyzes the causes of the power outage that affected Super Bowl XLVII in early 2013.  In the article, Hamilton writes:

As when looking at any system faults, the tools we have to mitigate the impact are: 1) avoid the fault entirely, 2) protect against the fault with redundancy, 3) minimize the impact of the fault through small fault zones, and 4) minimize the impact through fast recovery.

The mitigation options are roughly ordered by increasing impact to the customer.  In this article, we will refer to these strategies, in order, as “Avoid it”, “Mask it”, “Bound it”, and “Fix it fast”...

Click to read more ...


Stuff The Internet Says On Scalability For February 28th, 2014

Hey, it's HighScalability time:

  • Quotable Quotes:
    • @ML_Hipster: A machine learning researcher, a crypto-currency expert, and an Erlang programmer walk into a bar. Facebook buys the bar for $27 billion.
    • OH: Network effects don't happen on toll roads.
    • Benedict Evans: Google is a vast machine learning engine... and it spent 10-15 years building that learning engine and feeding it data.
  • Mining Experiment: Running 600 Servers for a Year Yields 0.4 Bitcoin. Yes, this is a far superior way of doing things. Chew up the commons for marginal gain. It's like old times.
  • Game designers, forget the sardines and go hunt some whale. Swrve found: half of free-to-play games’ in-app purchases came from 0.15 percent of players. Only 1.5 percent of players of games in the Swrve network spent any money at all.
  • Google has a beta version of their cloud pricing calculator. The interface is a little funky with separate "Add to Estimate" sections, but the prices look good. 5 servers, with 2 cores, 7.5GB RAM, 24x7, 3TB storage, 100 million IOPS, 1TB snapshot storage, 1TB light Cloud SQL operations, 4TB cloud storage, all for $1,559.24 a month.
  • So scalability doesn't matter? After the WhatsApp acquisition here's a tweet from Telegram Messenger: 4 million users joined Telegram within the last 18 hours. We're doing our best, but the service is getting unstable due to high'll take some time to transport and install the new equipment.
  • Maybe content can make money rather than being cheap commodity chum for aggregators. Financial Times’ CTO John O’Donovan: We make more money from our content than from advertising which is a really interesting shift – we are pushing boundaries in terms of how we are getting our content into these different services and platforms.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...