Stuff The Internet Says On Scalability For April 4th, 2014

Hey, it's HighScalability time:

The world ends not with a bang, but with 1 exaFLOP of bitcoin whimpers.

• Quotable Quotes:
  • @EtienneRoy: Algorithm:  you must encode and leverage your ignorance, not only your knowledge #hadoopsummit - enthralling
  • Chris Brenny: A material is nothing without a process. While the constituent formulation imbues the final product with fundamental properties, the bridge between material and function has a dramatic effect on its perception and use.
  • @gallifreyan: Using AWS c1, m1, m2? @adrianco says don't. c3, m3, r3 are now better and cheaper. #cloudconnect #ccevent
  • @christianhern: Mobile banking in the UK: 1,800 transactions per MINUTE. A "seismic shift" that banks were unprepared for


• While we are waiting for that epic article deeply comparing Google's Cloud with AWS, we have Adrian Cockcroft's highly hopped slide comparing the two. Google: no enterprise customers, no reservation options, need more regions and zones, need lower inter-zone latency, no SSD options. AWS: no per minute billing, need simpler discount options, need more regions and zones, no real PaaS strategy, not instance migration.


• Technology has change us from a demo or die culture to a deploy or die culture. We see this in software, but it's happening in hardware too says, Joi Ito (MIT Media Lab), in this interview. Some notes: It's cheap enough to make things that they don't have to give designs over to big companies to deploy. Full blown Manufacturing is something students want to do at the lab. Push manufacturing to the edges. Samsung is selling manufacturing in a box. It's now possible to A/B test hardware in fast iterations. Need open hardware to route around IP issues. Designing for manufacturing is like coding for deployment. Can't make a $12 cell phone unless you understand manufacturing completely. Modularity will help concentrate expertise in a piece of hardware yet allow the modules to be used easily. Modules are where the test correction happens. Ensures they can be assembled easily. 


• The Curmudgeon of Truth declares I reckon your message broker might be a bad idea: Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour." < Good discussion on Hacker News.


• High phive this. Your application may already achieved a degree of consciousness. An information integration theory of consciousness: consciousness corresponds to the capacity of a system to integrate information. This claim is motivated by two key phenomenological properties of consciousness: differentiation - the availability of a very large number of conscious experiences; and integration - the unity of each such experience. The theory states that the quantity of consciousness available to a system can be measured as the Phi value of a complex of elements. 


• People really really like to talk. Line, a messaging app that's big in Japan, surpasses 400 Million Registered Users, 10 billion chat messages per day, 1.8 billion stickers per day, and over 12 million calls per day.


• MCS locks and qspinlocks. Jonathan Corbet nicely defines spinlocks and goes into details about different attempts and using them to reduce contention within the kernel. 


• Google isn't just bringing cheaper prices to the cloud wars, but something new, a Managed Virtual Machine, which combines IaaS and PaaS. You don't want to use Google App Engine because you want to run some of your own software so why bother with GAE? Or you want to run a lot of your app on IaaS but would love the convenience of GAE services like Task Queues. Now you can skip that choice as long as you go Google. 


• Joe Stump is an interesting guy so you might think his new Business of Coding Podcast would also be interesting. And you would be right. The idea is to talk not just tech, but to explore the wizardly aspects of running a successful tech business. Ryan King of Twitter is one of the guests and Justin captured part of the talk which gives you an idea of what to expect: "A lot of engineers fall into the 'build it and they will come' trap. Have you found that?" - Joe Stump "People sometimes mistake hard problems (in the computer science sense) for valuable problems (in the business sense). Sometimes that venn diagram overlaps, like Google's search algorithms. But there's a number of cases where there's a really valuable problem, but it's not particularly hard on the technology side (AirBnB is an example of this). The other problem I see is this idea of 'scratch your own itch.' That advice works in certain contexts, and sometimes you'll also outgrow it." - Ryan King


• The business of business is business. The REAL Reason Facebook Got Rid Of Its Beautiful Redesign: Its Users Have Old Computers With Tiny Screens: Zhuo says some testing showed the photo-rich redesign would have been better for Facebook revenues. But testing also showed users who had it were less likely to share content through Facebook, post comments, or get to Facebook groups they liked to use. 


• Good overview of current Web Hosting For App Developers. 


• When nothing but the best will do, MariaDB on Fusion-IO improves its out-of-the-box performance by 30-50%, using atomic writes, page compression, and multi-threaded flush.


• Storage Mojo asks Is software’s free ride over?: The entire infrastructure world’s acceleration is slowing down. CPUs aren’t much faster. Storage densities aren’t improving as they were 10 years ago. Network speeds and more importantly costs aren’t dropping as they were. Several things have obscured the trend and blunted its impact: multiprocessing; NAND flash SSDs; 10GigE uptake; massively parallel GPUs; scale-out architectures. But those too are less and less effective.


• Engineyard on Share Nothing, Scale Everything. Apps not designed for the cloud typically expect the file system to act like a database, which doesn't work you want to scale across multiple servers. Adopting a stateless model "is both easy to setup and works well with cloud architecture."


• Twitter created a new database after my favorite cocktail, Manhattan, our real-time, multi-tenant distributed database for Twitter scale. I don't know if it's true or not, but teodimoff has a great sounding comment on the inner workings: Twitter literally pulls data from thousands of servers but we are missing the point of Manhattan. As some of us know twitter services scale dynamic according the load they are serving and some engineers at Twitter decide to buy a hole truck of steroids to this idea. Here is a trick - We make container agents (storage services) "clients" of the Manhattan database(the core). They are Mesos processes which scale dynamic to the needs of the service (i.e 1 container = 10000 reads/writes per second, 2 containers = 20000 read/writes per second and so on) which allows the dynamic scaling of requests per second and writes per second. The core handles finding actual machines which have the data, replicating it and so on. There might be realtime storage service containers which need fast data access, batch importer and timeseries they mention and so on. This requires a lot of gymnastics but offer a lot of nice features.The Manhattan database acts as virtual layer over thousands of machines and storage services allows for customized data manipulation. Cool...huh? According importance and scale (multi dc) this operates on i think it almost impossible to open source this. But who knows. miracles are happening now and then. 


• A nice Distributed Systems Reading List, all in one place, how ironic.


• Nested brain hierarchy might also be a good model for software: By nested hierarchy I mean a way of looking at the brain, looking at its layers and how they developed over the course of evolution. Humans go back to the Pleistocene [about 2.5 million years ago], but the emotional part of the brain goes back much further, all the way to the time when ancestral mammals evolved away from reptiles. Primary processes, based in deep subcortical regions, manifest evolutionary memories that are the basic emotional operating systems of the brain. Secondary processes, based on a series of way stations known as basal ganglia, are enriched with the mechanisms for learning—for linking external perceptions with associated feelings. Then on top, the tertiary level is programmed by life experiences through the neocortex, engendering our higher cognitive processes such as thinking, ruminating, and planning. Our capacity to think is fueled by our storehouses of memory and knowledge acquired by living in complex physical and social worlds. But the ancient feeling states help forge our memories in the first place. New memories could not emerge without the underlying states that allow animals to experience the intrinsic values of life.