Stuff The Internet Says On Scalability For October 3rd, 2014

Hey, it's HighScalability time:

Is the database landscape evolving or devolving?

• 76 million: once more through the data breach; 2016: when a Zettabyte is transfered over the Internet in one year
• Quotable Quotes:
  • @wattersjames: Words missing from the Oracle PaaS keynote: agile, continuous delivery, microservices, scalability, polyglot, open source, community #oow14
  • @samcharrington: At last count, there were over 1,000,000 million containers running in the wild. http://stats.openvz.org  @jejb_ #ccevent
  • @mappingbabel: Oracle's cloud has 30,000 computers. Google has about two million computers. Amazon over a million. Rackspace over 100,000.
  • Andrew Auernheimer: The world should have given the GNU project some money to hire developers and security auditors. Hell, it should have given Stallman a place to sleep that isn't a couch at a university. There is no f*cking justice in this world.
  • John Nagle: The right answer is to track wins and losses on delayed and non-delayed ACKs. Don't turn on ACK delay unless you're sending a lot of non-delayed ACKs closely followed by packets on which the ACK could have been piggybacked. Turn it off when a delayed ACK has to be sent. I should have pushed for this in the 1980s.
  • @neil_conway: The number of < 15 node Hadoop clusters is >> the number of > 15 node Hadoop clusters. Unfortunately not reflected in SW architecture.


• In the meat world Google wants devices to talk to you. The Physical Web. This will be better than Apple's beacons because Apple is severely limiting the functionality of beacons by requiring IDs be baked into applications. It's a very static and controlled world. In other words, it's very Apple. By using URLs Google is supporting both the web and apps; and adding flexibility because a single app can dynamically and generically handle the interaction from any kind of device. In other words, it's very Google. Apple has the numbers though, with hundreds of millions of beacon enabled phones in customer hands. Since it's just another protocol over BLE it should work on Apple devices as well.


• Did Netflix survive the great AWS rebootathon? The Chaos Monkey says yes, yes they did: Out of our 2700+ production Cassandra nodes, 218 were rebooted. 22 Cassandra nodes were on hardware that did not reboot successfully. This led to those Cassandra nodes not coming back online. Our automation detected the failed nodes and replaced them all, with minimal human intervention. Netflix experienced 0 downtime that weekend. 


• Google Compute Engine is following Moore's Law by announcing a 10% discount. Bandwidth is still expensive because networks don't care about silly laws. And margin has to come from somewhere.


• Software is eating...well you've heard it before. Mesosphere cofounder envisions future data center as ‘one big computer’: The data center of the future will be fully virtualized, with everything from power supplies to storage devices consolidated into a single pool and managed by software, according to an executive whose company intends to lead the way.


• Companies, startups, hacker spaces, teams are all intentional communities. People choose to work together towards some end. A consistent group killer is that people can be really sh*tty to each other. There's a lot of work that has been done around how to make intentional communities work. Holacracy is just one option. Here's a really interesting interview with Diana Leafe Christian on what makes communities work. It requires creating Community Glue, Good Process and Communication Skill, Effective Project Management, and good Governance and Decision making. Which is why most communities fail. Did you know there's even something called Non-Defensive Communication? If followed the Internet would collapse.


• Universal SSL: How It Scales: Through a combination of modern hardware, modern algorithms, lazy loading, and session resumption techniques, we were able to reduce the CPU usage of Universal SSL to almost nothing.


• A good discussion of deployment options from How We Use AWS, Ansible, and Packer to Make Deployment Awesome. 


• You might enjoy the books David Brin, Bruce Sterling & Daniel Suarez think we would need to sustain or rebuild civilization. Brin liked De Nuptiis Philologiae Et Mercurii Et De Septem Artibus Liberalibus Libri Novem and the Feynman Lectures. Sterling liked Last and First Men and Star Maker : Two Science Fiction Novels and Endless Frontier. Suarez liked Demon-Haunted World: Science as a Candle in the Dark and The Medieval Machine: The Industrial Revolution Of The Middle Ages.


• Containers are in the money as C3DNA and StackEngine snag millions in funding. 


• AWS is getting faster. What are the Fastest EC2 and S3 Regions?: we see an average 32% improvement in individual upload speeds compared to our results from last year. 


• It looks like Postgresql as a Nosql Document Store could really work.


• So how does Facebook store all those cat pictures that nobody ever looks at again? More on Facebook's "Cold Storage". Both disks and Blu-ray drives are used, but the fascinating bit is adapting to meet power goals: It is this careful, organized scheduling of the system's activities at data center scale that enables the synergistic cost reductions of cheap power and space. 


• Talk about nerve-racking! NASA Reformatted an SSD... on Mars! Amazing. 


• Macromolecular storage: the next frontier: Unlike current storage, where data locations are fixed, molecular storage has floating data and fixed read/write locations. The molecules are transported by microfluidics plumbing.


• Why are those connections dropping? Network Detective Pepelnjak investigates. An inspection of the scene revealed strange packet drops and out-of-order packets. Eventually clues led directly to the perp, who turned out to be a short connection keep alive time out. An early suspect was bufferbloat. You just never know until you investigate and look for means, motive, and opportunity.


• Craig Venter in Boulder: Throughout Venter’s talk there was one theme in the way he views Nature’s code for life. He calls it software. He sees genetic code as the software which runs a system. Because computers have advanced so incredibly since his first human genome project was begun, it has become easy for geneticists today to study and use this living code.


• Jeremy Cole on Visualizing the impact of ordered vs. random index insertion in InnoDB: You can see here that the random-insertion index is 41% larger at 1043 pages compared to just 737 pages for the ordered-insertion index. Additionally, 206 more pages are left unused, making the actual disk space usage 57% larger.


• A big list of BigData resources. Lots to keep you mystified and amazed. 


• Murat with his usual great deep looks: Paper summary: Tango: Distributed Data Structures over a Shared Log and Paper Summary: High-availability distributed logging with BookKeeper.


• Jeremiah Peschka drills deep to answer one of life's most important questions: Is Azure Really 60% Faster? The answer is as always, it depends: Not all clouds are created equal and 60% more doesn’t mean that it’s any better than it was before. It’s up to you, dear reader, to determine what 60% faster means and how that applies to your environment. For companies dipping their toes in the cloud waters, be very wary with the new improved Azure performance. You may find that you’re deploying far more VMs than you thought, just to handle the same workload.


• OpenDNS is using Redis to reduce query latencies: Keys are set to expire 24 hours after their most recent update. HBase is not going anywhere and remains where historical data is being stored. However, the API our tools rely on in order to access the DNS database now transparently sends queries both to Redis and to HBase, and merges the results...They are also using HyperLogLog to compute reputation scores.


• Groupon on Optimizing Redis Storage: The end effect? The Redis CPU usage doubled, which was still very reasonable – about 33% of one core. The Redis storage dropped to 9 GB – less than 1/10th of the original storage. The latency in loading a complete experiment data set rose slightly – about 10% on average, based on the size and duration of the experiment. Everything we liked about Redis: fast, simple, robust, and persistent, we were able to keep.


• MySQL 5.7.5 has a ton of new features. Many of them performance related. Many of them scalability related. But they key takeaway seems to be the reports of MySQL's death have been greatly exaggerated.


• Escape From the Data Center: The Promise of Peer-to-Peer Cloud Computing: What would happen if you took this trend in geographically distributing cloud infrastructure to its logical conclusion? You’d end up with clouds made up of millions of individual computers distributed across the globe and connected through the Internet. We would call this a peer-to-peer (P2P) cloud because it shares many of the characteristics of various P2P systems developed for file sharing, content distribution, and the payment networks of virtual cryptocurrency schemes such as Bitcoin.


• Diverse Introspectives: a conversation with John Harte: What does turn me on is seeking out very general principles that must be true from which very general conclusions can be based, which can be tested, which are falsifiable, and which potentially, if the theory is right, can explain a huge amount of information.


• A good discussion of Why You Should Not Use an RDBMS for Messaging. You can do it, but once you've done it you've built a message system, so why not just start with a message system?


• When BMW needed a high performance database they turned to CortexDB: datasets are stored as independent entities (cf. objects). To achieve this, the system transforms all content into a new type of index structure. This ensures that every item of content and every field “knows” the context in which it is being used. As a result, the database isn’t searched. Instead, queries are run on information that is already known and the results are combined using simple procedures based on set theory.


• LinkedIn's Real-time Analytics at Massive Scale with Pinot. Pinot is a web-scale real-time analytics engine that slices, dices and scans through massively large quantities of data in real-time across a wide variety of products. Pinot is a distributed system that supports columnar indexes with the ability to add new types of indexes.  Apache Helix is used for cluster management. Kafka and Hadoop to support real-time needs. 


• Architecture of a Database System: This paper presents an architectural discussion of DBMS design principles, including process models, parallel architecture, storage system design, transaction system implementation, query processor and optimizer architectures, and typical shared components and utilities. Successful commercial and open-source systems are used as points of reference, particularly when multiple alter- native designs have been adopted by different groups.


• Greg Linden's Quick links always take a long time to savor.