Stuff The Internet Says On Scalability For February 7th, 2014

Hey, it's HighScalability time:

Google "Corkboard" Server, 1999

• 5 billion requests per day: Heroku serves 60,000 requests per second; 500 Petabytes: Backblaze's New Data Center; 25000 simultaneous connections: on a Percona Server


• How algorithms help determine that shape of our world. First we encode normative rules of an idealized world in algorithms. Second those algorithms help enforce those expectations by nudging humans in to acting accordingly. A fun example is the story of Ed Bolian's Record-Breaking Drive. Ed raced from New York to L.A. at speeds of up to 158 mph, "breaking countless laws – and the previous record, by more than two hours." His approach is one that any nerd would love. He had three radar detectors, two laser jammers, two nav systems, a CB radio, a scanner, two iPhones and two iPads running applications like Waze, lookouts in the back seat scanning for cops, and someone scouting ahead. Awesome! For the moral of the story, they were going so fast Ed said that "AmEx froze my credit card. They didn't think I could've traveled from one station to another as fast as I did." A human didn't act as expected so the human was denied access to the System. And that System is what will mediate all human interactions going forward. When we think of our AI mediated future be aware that it will destroy as many degrees of freedom as it creates.


• Not what you want to have happen when you've spent $4.5 million on a Super Bowl ad. Maserati’s Ghibli SuperBowl Ad Crashes Maserati Website. They fell back to a YouTube video, which is a good strategy. Still, nice looking car. 


• AWS cost savings alert: new prices for the M3 instances are now more cost-effective than the M1 instances. Take a look at rebalancing your instance portfolio.


• Urs Hölzle with a little Google nostalgia: Back in 2000, the main ads database was on a single machine, f41. The ads group was five engineers back then, so we took turns carrying a pager. I had to abort a dinner date one night (it was Outback Steakhouse in Campbell) to come back to the Googleplex because f41 was wedged; Jeff Dean chips in with the story of his first visit to a datacenter. 


• Chris Davis: My experience in working on Graphite has reaffirmed a belief of mine that scalability has very little to do with low-level performance but instead is a product of overall design. I have run into many bottlenecks along the way but each time I look for improvements in design rather than speed-ups in performance. I have been asked many times why I wrote Graphite in Python rather than Java or C++, and my response is always that I have yet to come across a true need for the performance that another language could offer.


• The government wants our cars to 'talk' to each other. Interesting that it's not a centralized protocol. But I agree with the security risk, smart things always contain the possibility of betrayal.


• Bitcoins the hard way: Using the raw Bitcoin protocol. Or why people understand Bitcoin about as much they understood collateralized debt obligations, mortgage backed securities, and credit default swaps. Hopefully the results will not be the same.


• This Open Source thing really can work. Facebook Saved Over A Billion Dollars By Building Open Sourced Servers. Facebook thought they would be the only ones contributing and low and behold, others joined in and the savings are shared by all. Now Microsoft is also sharing its server designs and is open sourcing some code (be sure and read the comment section).


• In Black sheep Murat talks about the Nobel-Prize effect, that after winning a Nobel Prize, scientists often freeze up, never doing great work again because they can never live up to the ghost of their previous work. We see this with startups and even projects within a product. You hit a home run and every new thing pales by comparison. This is more than a metaphorical haunting. It's a rare person who can start over from the beginning. It takes a true fascination and interest in the world to banish the ghost of success.


• An idea that works for the rise and fall of companies as well as industries. After Tamerlane: we should reject the idea of a linear change in the course of modern world history...it is more productive to think in terms of ‘conjunctures’, periods of time when certain general conditions in different parts of the world coincided to encourage (or check) the enlargement of trade, the expansion of empires, the exchange of ideas or the movement of people.


• 30 Random Technical Thoughts by a WiFi Engineer. Wide ranging tips and tricks. I especially liked 25) Wifi is half duplex .. Wave 2 802.11ac will change that .. Lets call it Half Duplex+ ; 6) Walls are your friends. Design using walls as attenuation points. Letting RF run amuck and leak into areas cause unnecessary CCI. 


• Google's experience with running a large backup and restore system for exabytes of data shows us a confounding irony. It’s the scale of the world that drives automation. As there is more and more in the world, linear growth becomes too expensive at scale, which means automation must be pushed into all layers, which means fewer people are needed in the system. In a technology driven society as population pressures increase the need for human labor decreases.


• Is this what insanity looks like? Bitcoin Miners Building 10 Megawatt Data Center in Sweden.  Or is it perfect rationality?


• Lines at Costco and the thundering herd problem. At our Costco the checkout station has a viewing angle on line length at the food pickup window. So it is my custom to look at the line and if it's long I don't order anything, if it's short I do. Often when it's short when I order it's long when I actually finish paying and get there. In my imagination I think everyone made the same calculation and came to the same conclusion, thus we became a thundering herd on the pickup queue. I wish I could send my drone over to pick up the food while I'm in line.


• Mobile first. 75% of Pinterest's traffic comes from mobile. Interestingly they are pursuing a dual strategy of high quality mobile web and mobile apps.


• A peak into a possible future of storage. Frontiers of storage: magnetic holography. Densities of 1Tb/cm2 could be achieved using spin wave interference to store data. No, I don't really know what that means.


• Great Quora thread on What went wrong at Zynga? Lots of disagreement and lots of Zynga hate. Answers range from Zynga didn't know how to make good games ("We're an analytics company masquerading as a games company"), Zynga relied too much on Facebook, Zynga's management team sucked, Zynga missed the transition to mobile. Regardless of the reason(s), a lot of people at Zynga did a lot of good work and contributed to the knowledge base of how to program in this complex new social and cloud based world.


• Khan Academy with a lesson on improving front page performance: use TinyPNG to achieve 50%-70% compression, save an HTTP request by embedding data in a CSS file, save two HTTP requests with CSS sprites, save 5+ HTTP requests using delay-loading for images and YouTube, save one request using FontAwesome subscribe icon, saved one request by tracing dead images loaded by CSS. Result is 65% reduction in load times. 


• Algorithms and Me with a nice take on Locking : Mutex vs Spinlocks: Spin-locks are best used when a piece of code cannot go to sleep state. Best example of such code would be interrupts request handlers. Mutexes are best used in user space program where sleeping of process does not affect the system in catastrophic way; Here's Part 2.


• Apple is extending its philosophy of controlling the whole user experience out to the network. Apple Building Out Their Own CDN To Deliver Content To Consumers. Makes sense. Microsoft, Google, Netflix, Yahoo!, Twitter, and Facebook all do it. It's hard to fix something if it's not under your complete control. Hardware, software, services, and content make for multiple and integrated touch points.


• A 96-Antenna System Tests the Next Generation of Wireless: some experiments have been geared toward pushing existing 4G LTE technology further. The technology can, in theory, deliver 75 megabits per second, though it is lower in real-world situations. But some research suggests it can go faster by stitching together streams of data from several wireless channels.


• Rate-limiting State: Every reflection-friendly protocol mentioned in this article is going to have to learn rate limiting. This includes the initial TCP three-way handshake, ICMP, and every UDP-based protocol. In rare instances it's possible to limit one's participation in DDoS reflection and/or amplification with a firewall, but most firewalls are either stateless themselves, or their statefulness is so weak that it can be attacked separately.


• Hardware Transactional Memory in Java, or why synchronized will be cool again: Hardware Transaction Memory has the potential to allow multiple threads to speculatively access the same data structure at the same time and let the cache coherence protocol determine if a conflict occurred.  HTM aims to give you the scalability of fine grain locking, the simplicity of course grain locking, and performance close to no locking at all.  If supported by the JVM, your program or library was written with course grain lock it could mean your application can scale to many more cores with little change.


• What is Deep Mind all about? Playing Atari with Deep Reinforcement Learning: We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.