Stuff The Internet Says On Scalability For November 21st, 2014

Hey, it's HighScalability time:

Sweet dreams brave Philae. May you awaken to a bright-throned dawn for one last challenge.


  •  80 million: bacteria xferred in a juicy kiss;
  • Quotable Quotes:
    • James Hamilton: Every day, AWS adds enough new server capacity to support all of Amazon's global infrastrucrture when it was a $7B annual revenue enterprise.
    • @iglazer: What is the test that could most destroy your business model?  Test for that. @adrianco #defragcon
    • @zhilvis: Prefer decoupling over duplication. Coupling will kill you before duplication does by @ICooper #buildstufflt
    • @jmbroad: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful." ~George Box
    • @RichardWarburto: Optimisation maybe premature but measurement isn't.
    • @joeerl: Hell hath no version numbers - the great ones saw no need for version numbers - they used port numbers instead. See, for example, RFC 821,
    • JustCallMeBen: tldr: queues only help to flatten out burst load. Make sure your maintained throughput is high enough.
    • @rolandkuhn: «the event log is a database of the past, not just of the present» — @jboner at #reactconf
    • @ChiefScientist: CRUD is dead. -- @jboner #reactconf 
    • @fdmts: 30T of flash disks cabled up and online.  Thanks @scalableinfo!
    • monocasa: Immutable, statically linked, minimal system images running micro services on top of a hypervisor is a very old concept too. This is basically the direction IBM went in the 60's with their hypervisors and they haven't looked back.
    • Kiril Savino: Scaling is the process of decoupling load from latency.

  • Perhaps they were controlled by a master AI? Google and Stanford Built Similar Neural Networks Without Knowing It: Neural networks can be plugged into one another in a very natural way. So we simply take a convolutional neural network, which understands the content of images, and then we take a recurrent neural network, which is very good at processing language, and we plug one into the other. They speak to each other—they can take an image and describe it in a sentence.

  • You know how you never really believed the view in MVC was ever really separate? Now this is MVC. WatchKit apps run on the iPhone as an extension, only the UI component runs on the watch. XWindows would be so proud.

  • Shopify shows how they Build an Internal Cloud with Docker and CoreOS: Shopify is a large Ruby on Rails application that has undergone massive scaling in recent years. Our production servers are able to scale to over 8,000 requests per second by spreading the load across 1700 cores and 6 TB RAM.

  • Machine learning isn't just about creating humavoire AIs. It's a technology, like electricity, that will transform everything it affixes with its cyclops gaze. Here's a seemingly mundane example from Google, as discussed on the Green (Low Carbon) Data Center Blog. Google has turned inward, applying Machine Learning to its data center fleet. The result:  Google achieved from 8% to 25% reduction in its energy used to cool the data center with an average of 15%.  Who wouldn’t be excited to save an average of 15% on their cooling energy costs by providing new settings to run the mechanical plant? < And this is how the world will keep those productivity increases reaching skyward.

  • Does anyone say "I love my water service"? Or "I love my garbage service"? Then why would anyone say "I love Facebook"? That's when you've arrived. When you are so much a part of the way things are that people don't even think of loving them or not. They just are. The Fall of Facebook

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


We are leaving 3x-4x performance on the table just because of configuration.

Performance guru Martin Thompson gave a great talk at Strangeloop: Aeron: Open-source high-performance messaging, and one of the many interesting points he made was how much performance is being lost because were aren't configuring machines properly.

This point comes on the observation that "Loss, throughput, and buffer size are all strongly related."

Here's a gloss of Martin's reasoning. It's a problem that keeps happening and people aren't aware that it's happening because most people are not aware of how to tune network parameters in the OS.

The separation of programmers and system admins has become an anti-pattern. Developers don’t talk to the people who have root access on machines who don’t talk to the people that have network access. Which means machines are never configured right, which leads to a lot of loss. We are leaving 3x-4x performance on the table just because of configuration.

We need to workout how to bridge that gap, know what the parameters are, and how to fix them.

So know your OS network parameters and how to tune them.

Related Articles


Aeron: Do we really need another messaging system?

Do we really need another messaging system? We might if it promises to move millions of messages a second, at small microsecond latencies between machines, with consistent response times, to large numbers of clients, using an innovative design.  

And that’s the promise of Aeron (the Celtic god of battle, not the chair, though tell that to the search engines), a new high-performance open source message transport library from the team of Todd Montgomery, a multicast and reliable protocol expert, Richard Warburton, an expert on compiler optimizations, and Martin Thompson, the pasty faced performance gangster.

The claims are Aeron is already beating the best products out there on throughput and latency matches the best commercial products up to the 90th percentile. Aeron can push small 40 byte messages at 6 million messages a second, which is a very difficult case.

Here’s a talk Martin gave on Aeron at Strangeloop: Aeron: Open-source high-performance messaging. I’ll give a gloss of his talk as well as integrating in sources of information listed at the end of this article.

Martin and his team were in the enviable position of having a client that required a product like Aeron and was willing to both finance its development while also making it open source. So go git Aeron on GitHub. Note, it’s early days for Aeron and they are still in the heavy optimization phase.

The world has changed therefore endpoints need to scale as never before. This is why Martin says we need a new messaging system. It’s now a multi-everything world. We have multi-core, multi-socket, multi-cloud, multi-billion user computing, where communication is happening all the time. Huge numbers of consumers regularly pound a channel to read from same publisher, which causes lock contention, queueing effects, which causes throughput to drop and latency to spike. 

What’s needed is a new messaging library to make the most of this new world. The move to microservices only heightens the need:

As we move to a world of micro services then we need very low and predictable latency from our communications otherwise the coherence component of USL will come to rain fire and brimstone on our designs.

With Aeron the goal is to keep things pure and focused. The benchmarking we have done so far suggests a step forward in throughput and latency. What is quite unique is that you do not have to choose between throughput and latency. With other high-end messaging transports this is a distinct choice. The algorithms employed by Aeron give maximum throughput while minimising latency up until saturation.

“Many messaging products are a Swiss Army knife; Aeron is a scalpel,” says Martin, which is a good way to understand Aeron. It’s not a full featured messaging product in the way you may be used to, like Kafka. Aeron does not persist messages, it doesn’t support guaranteed delivery, nor clustering, nor does it support topics. Aeron won’t know if a client has crashed and be able to sync it back up from history or initialize a new client from history. 

The best way to place Aeron in your mental matrix might be as a message oriented replacement for TCP, with higher level services written on top. Todd Montgomery expands on this idea:

Aeron being an ISO layer 4 protocol provides a number of things that messaging systems can't and also doesn't provide several things that some messaging systems do.... if that makes any sense. Let me explain slightly more wrt all typical messaging systems (not just Kafka and 0MQ). 

One way to think more about where Aeron fits is TCP, but with the option of reliable multicast delivery. However, that is a little limited in that Aeron also, by design, has a number of possible uses that go well beyond what TCP can do. Here are a few things to consider: 

Todd continues on with more detail, so please keep reading the article to see more on the subject.

At its core Aeron is a replicated persistent log of messages. And through a very conscious design process messages are wait-free and zero-copy along the entire path from publication to reception. This means latency is very good and very predictable.

That sums up Aeron is nutshell. It was created by an experienced team, using solid design principles sharpened on many previous projects, backed by techniques not everyone has in their tool chest. Every aspect has been well thought out to be clean, simple, highly performant, and highly concurrent.

If simplicity is indistinguishable from cleverness, then there’s a lot of cleverness going on in Aeron. Let’s see how they did it...

Click to read more ...


Stuff The Internet Says On Scalability For November 14th, 2014

Hey, it's HighScalability time:

Spectacular rendering of the solar system to scale. (Roberto Ziche)


  • 700: number of low-orbit satellites in a sidecar cheap internet; 130 terabytes: AdRoll ad data processed daily; 15 billion: daily Weather Channel forecasts; 1 million: AWS customers
  • Quotable Quotes:
    • @benkepes: Each AWS data center has typically 50k to 80k physical servers. Up to 102Tbps provisioned networking capacity #reinvent
    • @scottvdp: AWS just got rid of infrastructure behind any application tier. Lambda for async distributed events, container engine for everything else.
    • @wif: AWS is handling 7 trillion DynamoDB requests per month in a single region. 4x over last year. same jitter. #reinvent
    • Philae: If my path was off by even half a degree the humans would have had to abort the mission.
    • Al Aho: Well, you can get a stack of stacks, basically. And the nested stack automaton has sort of an efficient way of implementing the stack of stacks, and you can think of it as sort of almost like a cactus. That's why some people are calling it cactus automata, at the time.
    • Gilt: Someone spent $30K on an Acura & LA travel package on their iPhone.
    • @cloudpundit: Gist of Jassy's #reinvent remarks: Are you an enterprise vendor? Do you have a high-margin product/service? AWS is probably coming for you.
    • @mappingbabel: Things coming out from the AWS #reinvent analyst summit - Amazon has minimum 3 million servers & lights up own globe-spanning fibre.
    • @cloudpundit: James Hamilton says mobile hardware design patterns are future of servers. Single-chip offerings, semiconductor-level innovation. #reinvent
    • @rightscale: RT @owenrog: AWS builds its own electricity substations simply because the power companies can't build fast enough to meet demand #reInvent
    • @timanderson: New C4 instances #reinvent up to 36 cores up to 16TB SSD
    • @holly_cummins: L1 cache is a beer in hand, L3 is fridge, main memory is walking to the store, disk access is flying to another country for beer. 
    • @ericlaw: Sample HTTP compression ratios observed on @Facebook: -1300%, -34.5%, -14.7%, -25.4%. ProTip: Don't compress two byte responses. #webperf
    • @JefClaes: It's not the concept that defines the invariants but the invariants that define the concept.

  • It's hard to imagine just a few short years ago AWS did not exist. Now it has 1 million customers, runs 70 million hours of software per month, and their AWS re:Invent conference has a robust 13,500 attendees. Re:Invent shows if Amazon is going to be disrupted, a lack of innovation will not be the cause. The key talking point is that AWS is not just IaaS anymore, AWS is a Platform. The underlying subtext is lock-in. Minecraft-like, Amazon is building out their platform brick by brick. Along with GCE, AWS announced a Docker based container service. Intel designed a special new cloud processor for AWS, which will be available in a new C4 instance type. There's Aurora, a bigger, badder MySQL. To the joy of many EBS is getting bigger and faster. The world is getting more reactive, S3 is emitting events. With less fan fare are an impressive suite of code deployment and management tools. There's also a key management service, a configuration manager, and a service catalog. Most provocative is Lambda, or PaaS++, which as the name suggests is the ability to call a function in response to events. Big deal? It could be, though quite limited currently, only supporting Node.js and a few event types. You can't, for example, terminate a REST request. What it could grow in to is promising, a complete abstraction layer over the cloud so any sense of machines and locations are removed. They aren't the first, but that hardly matters.

  • It's not a history of the Civil War. Or WWW I. Or the Dust Bowl. But An Oral History of Unix. Yes, that much time has passed. Interviewed are many names you'll recognize and some you've probably never heard of. A fascinating window into the deep deep past.

  • No surprise at all. Plants talk to each other using an internet of fungus: We suggest that tomato plants can 'eavesdrop' on defense responses and increase their disease resistance against potential pathogen...the phantom orchid, get the carbon they need from nearby trees, via the mycelia of fungi that both are connected to...Other orchids only steal when it suits them. These "mixotrophs" can carry out photosynthesis, but they also "steal" carbon from other plants...The fungal internet exemplifies one of the great lessons of ecology: seemingly separate organisms are often connected, and may depend on each other. 

  • How do you persist a 200 thousand messages/second data stream while guaranteeing data availability and redundancy? Tadas Vilkeliskis shows you how with Apache Kafka. It excels at high write rates, compession saves lots on network traffic, and a custom C++ http-to-kafka accommodates performance.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


Three Reasons You Probably Don’t Need Multi-Data Center Capabilities

This is a guest post by Nikhil Palekar, Systems Architect, FoundationDB

For many organizations that care a lot about strong consistency and low latency or haven’t already built a fault tolerant application tier on top of their database, adding a multiple data center (MDC) database implementation may create more complexity or unintended consequences than meaningful benefits. Why might that be?

Click to read more ...


Sponsored Post: Apple, Asana, Hypertable, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Sr. Software Developer. We are looking for a solid senior-level Java/C programmer who will be working on security software development. This software will provide the data protection, integrity, and service authentication services for iOS devices. Please apply here.
    • DevOps Software Engineer - Apple Pay, iOS Systems.  The iOS Systems team is looking for an outstanding DevOps software engineer to help make our rapidly growing platform manageable, scalable, and reliable using state of the art technologies and cutting edge system automation. Come join the team to strategize, architect, and build infrastructure to help our systems perform and scale. Please apply here
    • Sr. Software Engineer-iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of performance tuning Java applications make your heart leap? Would you like to work in a fast-paced environment where your technical abilities will be challenged on a day to day basis? Do you want your work to make a difference in the lives of millions of people? Please apply here.
    • Apple Pay - Site Reliability Engineer. You already know this… every issue counts. A single ticket can be the key to discovering an issue impacting thousands of people. And now that you’ve found it, you can’t wait to fix it. You also know that owning the quality of an application is about separating the signal from the noise. Finding that signal is what motives you. Now that you’ve found it, you’re next step is to role up the sleeves and start coding. As a member of the Apple Pay SRE team, you’re expected to not just find the issues, but to write code and fix them. Please apply here.
    • Senior Software Engineer -iOS Systems. This role demands the best and brightest engineers. The ideal candidate will be well rounded and offer a diverse skill set that aligns with key qualifications. Practical experience integrating with a diverse set of third-party APIs will also serve to distinguish you from other candidates. This is a highly cross functional role, and the typical team member's day to day responsibilities on the Carrier Services team. Please apply here

  • As a production-focused infrastructure engineer at Asana, you’ll be the 
    person who takes the lead on setting and achieving our stability and uptime 
    goals, architecting the production stack, defining the on-call experience, the 
    build process, cluster management, monitoring and alerting. Please apply 

  • Platform Software Engineer, Sprout Social, builds world-class social media management software designed and built for performance, scale, reliability and product agility. We pick the right tool for the job while being pragmatic and scrappy. Services are built in Python and Java using technologies like Cassandra and Hadoop, HBase and Redis, Storm and Finagle. At the moment we’re staring down a rapidly growing 20TB Hadoop cluster and about the same amount stored in MySQL and Cassandra. We have a lot of data and we want people hungry to work at scale. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

Fun and Informative Events

  • Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you.

Cool Products and Services

  • Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at Connect with Hypertable: @hypertable--Blog.

  • FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.

  • Diagnose server issues from a single tab. The Scalyr log management tool replaces all your monitoring and analysis services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. It's a universal tool for visibility into your production systems. Log aggregation, server metrics, monitoring, alerting, dashboards, and more. Not just “hosted grep” or “hosted graphs,” but enterprise-grade functionality with sane pricing and insane performance. Trusted by in-the-know companies like Codecademy – try it free!

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...


Nifty Architecture Tricks from Wix - Building a Publishing Platform at Scale

Wix operates websites in the long tale. As a HTML5 based WYSIWYG web publishing platform, they have created over 54 million websites, most of which receive under 100 page views per day. So traditional caching strategies don’t apply, yet it only takes four web servers to handle all the traffic. That takes some smart work.

Aviran Mordo, Head of Back-End Engineering at Wix, has described their solution in an excellent talk: Wix Architecture at Scale. What they’ve developed is in the best tradition of scaling is specialization. They’ve carefully analyzed their system and figured out how to meet their aggressive high availability and high performance goals in some most interesting ways.

Wix uses multiple datacenters and clouds. Something I haven’t seen before is that they replicate data to multiple datacenters, to Google Compute Engine, and to Amazon. And they have fallback strategies between them in case of failure.

Wix doesn’t use transactions. Instead, all data is immutable and they use a simple eventual consistency strategy that perfectly matches their use case.

Wix doesn’t cache (as in a big caching layer). Instead, they pay great attention to optimizing the rendering path so that every page displays in under 100ms.

Wix started small, with a monolithic architecture, and has consciously moved to a service architecture using a very deliberate process for identifying services that can help anyone thinking about the same move.

This is not your traditional LAMP stack or native cloud anything. Wix is a little different and there’s something here you can learn from. Let’s see how they do it...


Click to read more ...


Stuff The Internet Says On Scalability For November 7th, 2014

Hey, it's HighScalability time:

Google's new hyper efficient, hyper secure datacenters. (keynotelarge, m, o, r e)
  • 2 billion: containers started every week at Google; 86%: Apple's share of handset profits; 50: number of parallel processes in the brain; $12 million: cost for every additional second it takes customers to pay at a Walmart store; 800 terabytes: rendering for the movie Interstellar
  • Quotable Quotes:
    • @IFTTTRecipe:  to email @ Send new articles to my mailbox
    • @chadfowler: I've decided to break up tomorrow's monolithic talk about microservices into a series of microtalks #gotober
    • Emily Balcetis: Attention can “be thought of as what you allow your eyes to look at.”
    • Adam Alter: researchers have shown that our names take root deep within our mental worlds, drawing us magnetically toward the concepts they embody
    • @mjpt777: Monitoring needs to reflect an issue within 10 seconds of it happening. Poll at 1 second granularity. - @adrianco #gotober
    • @adyranov: "Trust only benchmarks you've made yourself. All others are benchmarketing" #highload2014
    • @adrianco: #GOTOber @mjpt777 Aeron performance on a laptop - 20 million 40byte messages/sec. 90%ile latency of 7us. 100% 37us.
    • @mjpt777: Have your infrastructure optimised for speed of delivery rather than cost, stuff then gets done so fast you reduce cost. @adrianco #gotober
    • Seth Newburg: This is a first in mobile! This is a device running over an internal network, rather than just everything being connected to a CPU
    • @swardley: OpenStack vs AWS - IMHO OpenStack should have dominated but pisspoor strategic play makes it unlikely for 10-15yrs.
    • Rudiger Moller: There is another strong value in going off heap (even when using advanced collectors like Zing's): Datastructures expressed in Java frequently have a redundancy of 80-90%.
    • @zooko: Novice engineers have not yet grokked this: the number of modes or options in your system is the *exponent* in how hard it is to maintain.

  • OK Google...give me more cool functionality and lower prices. And they did. Google Cloud Platform Live: Introducing Container Engine, Cloud Networking and much more. This may take a while to process and put in some context. What Google is continuing to do is consumerize their internal services. The biggest example of this is opening up their container scheduling, management, and auto scaler services, while making it Docker compatible. If you want to run code as close as possible to your users you can now direct peer to any of Google's over 70 points of presence in 33 countries around the world. That's impressive. AWS has 14 and Azure has 11. Firebase is still behind Parse in features, but with Google's support look for it to blossom. The distributed debugger is truly impressive work. Game changing really. There's also a new improved Pub/Sub system. Slowly but surely the worm turns. 

  • A great list. I'd like to see what a similar list for the message dispatch and handlers would look like. Martin Thompson design principles for the Aeron messaging system: garbage free in steady state running; smart batching in the message path; wait-free algos in the message path; non-blocking IO in the message path; no exceptional cases in message path; apply the single writer principle; prefer unshared state; avoid unnecessary data copies. 

  • Nicely done André Staltz. The introduction to Reactive Programming you've been missing. If insight meditation has not been enough to guide your conversion from the object obsessed to the functionally enlightened then this article might be just what you are looking for. Or not.

  • This is cool, using Cassandra as an embedded database. Russian Facebook Utilizing Cassandra for 1 Million Operations per Second, Over 80TB in Largest Cluster: The way we use Casssandra is somewhat unusual – we don’t use thrift or netty based native protocol to communicate with Cassandra nodes remotely. Instead, we co-locate Cassandra nodes in the same JVM with business service logic, exposing not generic data manipulation, but business level interface remotely. This way we avoid extra network roundtrips within a single business transaction and use internal calls to Cassandra classes to get information faster. Also this helps us to do many little hacks on its internals, making huge gains on efficiency and ease of distributed servers development.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


A Few Really Good Tech Podcasts for Your Listening Pleasure

Podcasts are back. Since I listen to a bunch of podcasts every week I was quite surprised to learn about their resurrection. But facial hair is back too, so I will revel in my short lived trendiness.

Why are podcasts back? One suggested reason is a little sad. People commute more than 25 hours a day and since cars are now faithful little bluetooth slaves, broadcasting podcasts over luxury car speakers is as easy as a smart technology and a fast cell network can make it. Podcasts are now a viable P2P radio replacement.

That’s the demand, means, motive, and opportunity side of things. What about supply?

Historically podcasts start in fire, the passion quickly burning to ash as podcasters learn making podcasts is a lot of hard work...and poorly unremunerated work at that. So podcasts have a high mortality rate.

What’s changed? Money. Strangely, people will exchange effort for money, so if podcasts can make money they will have the fuel they need to burn bright through the night.

Money from Subscriptions is New

Click to read more ...


Improve small job completion times by 47% by running full clones.

The idea is most jobs are small. Researchers found 82% of jobs on Facebook's cluster were less than 10 tasks. Clusters have a median utilization of under 20%. And since small jobs are particularly sensitive to stragglers the audacious solution is to proactively launch clones of a job as they are submitted and pick the result from the earliest clone. The result is an average completion time of all the small jobs improved by 47% using cloning, at the cost of just 3% extra resources.

For more details take a look at the very interesting Why Let Resources Idle? Aggressive Cloning of Jobs with Dolly.

Related Articles