hot links

Stuff The Internet Says On Scalability For January 27th, 2017

High Scalability

High Scalability

— 15 min read

Hey, it's HighScalability time:

Tired of noisy drones? Use the same dedrone tech used at Davos. It's the future.
If you like this sort of Stuff then please support me on Patreon.

  • 1+ trillion: messages Twitter handles per day; 695 million: Internet users in China; >350k: Twitter Star Wars bots; $90 million: value of LasVegas.com domain name; 45%: WiFi connection failure rate; 80: threads in Slack Mac OS app; 364: slides in Adrian Cockcroft's microservices deck; 5180%: increases at Etsy in daily visits to pages related to Donald Trump; 465,000: cars sold by Costco last year; 14 Million: one day of DuckDuckGo searches; 58 million: science papers online; ~3x: use of Kubernetes in production settings; 54: r3.2xlarge instances used for Reddit caching; $14 billion: Microsoft’s Azure's annual run rate; 

  • Quotable Quotes:
    • Carlo Rovelli: the basic ingredient is down there in the physical world: physical correlation between distinct variables. The physical world is not a set of self-absorbed entities that do their selfish things. It is a tightly knitted net of relative information, where everybody’s state reflects somebody else’s state. 
    • Charles Stross: There’s a saying that goes something like this: “Lieutenants study tactics, colonels study strategy, generals study logistics, and field marshals study economics.” But economists—the smart ones—study education.
    • Kirk Pepperdine: I would suggest that with 200 JVMs running on 80 core you should consider using the serial collector.
    • @alicegoldfuss: Things containers improve: - testing - deploying Things containers shit on: - security - troubleshooting - managing systems resources  Note: this is a long thread of comments, enjoy!
    • @pewinternet: In 2005, just 5% of Americans used at least one social media platform. Today, 69% do. 
    • Manu Saadia: He [Peter Thiel] was a bigger fan of “Star Wars” or “Star Trek,” Thiel replied that, as a capitalist, he preferred the former. “ ‘Star Trek’ is the communist one,” he said. “The whole plot of ‘Star Wars’ starts with Han Solo having this debt that he owes, and so the plot in ‘Star Wars’ is driven by money.
    • @asymco: Google's costs-per-click — essentially its pricing — fell 16% y/y
    • Anna MacLachlan: In order to follow best practices for performance when building PWAs [progressive web app] and otherwise, the Chrome team goes by the Rail performance model: Respond: 100ms / Animate: < 8ms / Idle work in 50ms chunks / Load: 1,000ms to interactive
    • Deepak Singh (AWS): There is a certain scale where specialized hardware and infrastructure make a lot of sense and for those who need special infrastructure, we think FPGAs are one clear way to go
    • @MarcWilczek: Containerization: 19% using it, 15% testing it, 13% considering it; 15% are curious, 38% have no plans or clue. #Cloud #CIO @interop #Docker
    • Clarke Illmatical: The death of net neutrality will severely impact IoT solutions which rely on an open internet concept.
    • @mipearson: OH "I'm the Technical Debt Fairy. If you leave technical debt under your pillowcase at night I hire away your best developers"
    • Reddit:  When you vote, your vote isn’t instantly processed—instead, it’s placed into a queue. Depending on the backlog of the queue, this can mean if you were to vote and quickly refresh the page, your vote may not have been processed yet, and it would appear that your vote had been reverted. 
    • Martin Kleppmann: in a 8,000-node cluster, the chance of permanently losing all three replicas of some piece of data (within the same time period) is about 0.2%. Yes, you read that correctly: the risk of losing all three copies of some data is twice as great as the risk of losing a single node!
    • Tammy Everts: Always remember that if you’re competing online, you’re competing with Amazon.
    • Marco Arment: I'm no spending more on [Apple] search ads than I am servers.
    • dijit: the big issue with databases I've worked with is not how many inserts you do per second, even spinning rust, if properly reasoned can do -serious- inserts per second in append only data structures like myisam, redis even lucene. However the issue comes when you want to read that data or, more horribly, update that data. Updates, by definition are a read and a write to commuted data, this can cause fragmentation and other huge headaches. I'd love to see someone do updates 1,000,000/s
    • @m0biusloop: things kubernetes can't do: ipv6, multiple host networks, prefix based policy, egress policy.
    • Dr Zhou: What is really surprising is our questioning on the whole effort of bot detection in the past years. Suddenly we feel vulnerable and don't know much: how many more are there? What do they want to do?
    • Marianne Bellotti: 15 years ago, everybody was telling us ‘Get off the mainframe, get on AT&T applications, build these thick clients. Mainframes are out.’ And now thick clients are out, and everybody’s moving to APIs and microservices, which basically are very similar to the thin client that a terminal uses to interact with a mainframe.
    • @garybernhardt: Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.
    • @jennschenker: #DLD17: BMW says it will evolve from being a car maker to a mobility services company.
    • Nick Craver (StackOverflow): We try to be boring. Boring is stable ...scalable. The simpler something is, the higher it scales...We are not against anything. We have loyalty to nothing. If there's a better option that comes along, move to it!
    • Romesberg: evolution works by starting with something close, and then changing what it can do in small steps
    • bitwiseand: The CAP theorem states that in the event of a network-partition you have to choose one of C or A. More intuitively, any delay between nodes can be modeled as a temporary network partition and in that event you have but two choices either wait to return the latest data at a peer node (C) or return the last available data at a peer node (A).
    • Gvaireth: We just had a discussion in the team, and we decided, that we need add-one microservice that would get a number and return the number increased by one. A nice separation of concerns in modern distributed web application :)
    • Russ Cox: When I first started thinking about generics for Go in 2008, the main examples to learn from were C#, Java, Haskell, and ML. None of the approaches in those languages seemed like a perfect fit for Go. Today, there are newer attempts to learn from as well, including Dart, Midori, Rust, and Swift.
    • RaptorXP: Do your virtual reality wearables usually connect to deep learning drones on the blockchain?
    • Twitter: Hadoop: We have multiple clusters storing over 500 PB divided in four groups (real time, processing, data warehouse and cold storage). Our biggest cluster is over 10k nodes. We run 150k applications and launch 130M containers per day.
    • arnon: GPUs tend to lend themselves well to analytics, contrary to transactions. Specifically, columnar databases. When the columns are all of the same data type, and the data locality is high, GPUs perform /very/ well.
    • Ed Sim: Despite the amazing productivity gains from open source, AWS, microservices and other new technologies, we have seen the time to launch extending and the cost of getting a minimally viable product (MVP) out the door increasing.
    • Daniel Miessler: It’s [AMP] poisonous to the underlying concept of an open internet. If this were to become widely adopted, you’d search for something, get results, consume the content, and you’d never leave Google.

  • Great detailed discussion on all things serverless. AWS Podcast #171: Serverless Special. Serverless is an implementation detail, not an architectural pattern. If you look at serverless as just a way to run existing code then it’s an implementation detail.  If you take it as an opportunity to think about how your application could be structured then it tends more towards the architectural pattern/microservices conversation; Serverless as a concept is a spectrum not binary. Serverless is an important concept but the boundaries are not clear. Example is compare Redshift and BigQuery. Redshift you have to specify a cluster size and with BigQuery you don’t, so BigQuery is more serverless; The more serverless you go the harder it is to inject errors in like with Chaos Monkey; There’s no best way to do end-to-end testing in serverless environments; There are two ways things can be serverless: Software as a Services and Functions as a Service. FaaS is often thought of being Serverless itself. FaaS only exists because there’s isn’t SaaS yet that does your business logic; Easy to get something something done quickly with Serverless, but people underestimate how hard it is to get a large team working on a project with a lot of functions. Serveless gets rid of heavy lifting on instracture side but there’s more complexity on the operational side when working with a team; What is the model where a team of developers working together on a logical unit uses a larger pipeline to deliver business functionality? Where do we have visibility as to what’s going on? How to go to from development to production? What is the pipeline where developers have their own serverless cluster then it goes to master then we have unit tests the integration tests, load tests, amd canary tests?; Serverless doesn’t mean noops. It is lessops. The tradeoff is observability; With Serverless it's easy to get something up and running, but the unit of deployment is not the unit of design. You get serverless sprawl. Pain is conserved, it just moves elsewhere. You are still writing a distributed system.

  • Information wants to be free. Sci-Hub the first pirate website in the world to provide mass and public access to tens of millions of research papers.

  • Nick Craver in an excellent interview on ScaleYourCode talks about many things, including how StackOverflow works: Primarily, it's a Lisp stack. It's Windows IES SQL .NET code, mostly C# for our programs...We're using HAProxy without exotic configs... We have Elasticsearch for our search on the background running on Linux 3 node cluster...With SQL, there is a primary server for Stack Overflow and a primary server for everything else on the network...There are nine web servers running production. We can run on one...behind that are three service boxes running...One of the cool things we might be doing on those is buying new ones and sticking GPUs in them...With a GPU, you can take a simple task and put it across thousands of cores, it's an excellent use case. That is the greatest use case...With the GTX 1080s which have 8GB of RAM on them now ( the old ones were 4), we can take a consumer graphics card for $700 and make a Tag engine 10 to 20 times faster than our CPU rewrite...The simplest thing is to grab something that pre exists and plug it in. A lot of people try to build it the first time. Don't do that. Take an existing tool and learn from it if nothing else for a day, or a week.

  • Around and around it goes. Where will it stop? Nobody knows. Making the move from Scala to Go, and why we’re not going back. Why was the bloom off the Scala rose?: slow compilation; slow deployments; poor IDE support; object oriented programming sucks; the future of the language is uncertain; it takes too long to learn. Why all the Go love?: simple to learn; simpler more readable code; channels and goroutines make programming easier; compiles fast; runs fast; code is small; Go microservices are a great fit for distributed systems; good tooling.

  • The Infrastructure Behind Twitter: Scale. Twitter is a complex beast. Too much to summarize. They cover Network Traffic, Data Center Traffic, Backbone Traffic, Edge Traffic, Storage, Cache, Running Puppet at Scale, Branching, and Monitoring. So there's something for everyone. Good discussion on HN on why Twitter is different than WhatsApp (hint: fanout).

  • You know Moore's Law is deadish. Here's interesting reasoning as to why cloud computing is more compatible with the future technologies that will help keep Moore's Law a law. Vanishing point: the rise of the invisible computer. Quantum computing and 3D chips, two projected next gen technologies,  require specialised care and feeding, so the cloud is the perfect zoo for keeping the future sources of compute cycles up and running.

  • Some free events that might be of interest: 

  • Tradeoffs are destiny. Upgrade #125. Concrete example of why Apple's privacy first approach of processing data on device makes the whole personal assistant thing a lot harder when compared to Google's cloud based processing approach. Example is populating calendar from events found in email and suggesting names, addressees and phone numbers. Apple only processes email when it's actually downloaded to the inbox of the mail app on your device. So it doesn't happen if your devices are off or you aren't reading email or you are using a third party interface. Apple does recognize events from some companies (doesn't understand Virgin Atlantic booking info, for example) but far fewer than Google supports. Also Apple doesn't have access to your calendar because it's encrypted. Also there's the problem of should Apple process the same data on all devices?

  • You can reduce your WiFi connection failures to under 4% and speed up connection setup times by picking the private network with the best signal. Why It Takes So Long to Connect to a WiFi Access Point.

  • To the chagrin of many, Facebook's open networking hardware revolution continues. Cumulus Networks supports Backpack and Wedge 100. Big Switch Networks announced a fully functional network operating system for Wedge 40 and Wedge 100. And so on. 

  • Ask HN: Why is the community so negative towards blockchain technology? patio11 captures it nicely: Blockchain is the world's worst database, created entirely to maintain the reputations of venture capital firms who injected hundreds of millions of dollars into a technology whose core defining insight was "You can improve on a Ponzi scam by making it self-organizing and distributed; that gets vastly more distribution, reduces the single point of failure, and makes it censorship-resistant."...No meaningful applications run on top of "blockchain", because it is a marketing term. You cannot install blockchain just like you cannot install database. (Database sounds much cooler without the definitive article, too.) If you pick a particular instantiation of a blockchain-style database, it is a horrible, horrible database. Can I pick on Bitcoin? Let me pick on Bitcoin. Bitcoin is claimed to be a global financial network and ready for production right now. Bitcoin cannot sustain 5 transactions per second, worldwide.

  • Excelling in-depth explanation of Caching at Reddit. Caches are split by workload type rather than having one big pool. Advantages: can scale workload independently based on utilization; make reasonable predictions about what will happen in case of a failure; prevent memory fragmentation because sizes are similar. Caching is also used for rate-limiting and locking. Caches are fronted by  mcrouter which is a connection pooler and proxy developed at Facebook. It can also route keys to specific pools based on their functionality; warm up caches slowly; manage failover pools. There's lots more too.

  • Store images in a database? The answer is usually heck no. But that's just what Walmart is doing. Building Object Store — Storing Images in Cassandra at Walmart Scale: "the read/write numbers (95th and 99th percentile) were simply outstanding for 90% of our use cases! Our performance tests involved storing millions of images of different sizes worth terabytes of data. The remaining 10% were all large images with 10’s of MB in size and the response times were still acceptable." A CDN is used to cache product images and different sized images will be produced on the fly. 

  • Like a spaceship finding the best path through a meteoroid belt here's another example of using AI in tuning, this time for NAND. NVMdurance – Making NAND Flash Last Longer: a machine learning algorithm that determines how best to tune the part.  This product has been named the “Pathfinder” since it works out the best few paths a chip can take to reach maximum endurance.  Another algorithm called the “Navigator” resides in the flash controller chip and manages the flash chip’s internal parameters during operation to follow one of the paths recommended by the Pathfinder. The exact set of paths chosen by the Navigator differs from chip to chip and depends on the kind of wear the particular application is causing for the flash. 

  • When Stack Overflow went down they did something cool, they went into read-only mode. It's not that easy to do. You have to be very careful when accessing the database. When Always On Isn’t: Handling Outages in Your Application. 

  • How Discord Stores Billions of Messages Using Cassandra. Their requirements: Linear scalability , Automatic failover, Low maintenance, Proven to Work, Predictable performance, Not a Blob Store, Open Source. Cassandra fit, but not perfectly. Messages had to bucketed by time so as to not createLarge partitions which put a lot of GC pressure on Cassandra during compaction, cluster expansion, and more. The eventual consistency model made for some problems as the last write wins policy caused some data overwrites. Only write non-null values to prevent generating tombstones during deletes. As expected writes were sub-millisecond and reads were under 5 milliseconds. They also had some surprising problems due to GC on deletes. Result: went from over 100 million total messages to more than 120 million messages a day, with performance and stability staying consistent.

  • The brain uses REM sleep to cut unneeded connections. Similar tactics in software: timeouts; garbage collection; caches; upgrading by starting a new software load in a new VM; in a primary-secondary pair periodically failing over to the secondary.

  • Apparently nature hasn't discovered ransomware yet. Nature is old school, malware is introduced for command and control. One Wasp Takes Control of Another.

  • Why Container-Based Deployment Is Preferred for Microservices: Ability to Deploy and Undeploy Independently of Other Microservices; Ability to Scale at Each Microservices Level; Ability to Build and Deploy Microservices Quickly.

  • Wizard level exploration of different coding approaches. “Same Fringe” Revisited: Both functional closures and actor behaviors capture algorithm fragments that can be passed around as values and invoked (or not) whenever we choose. These mechanisms are related to continuations [5], which have been discovered in various contexts by different people. If you squint, you could view all of these as kinds of coroutine implementations.

  • When to expect peak loads? Black Friday overtakes Cyber Monday for holiday traffic (and other lessons learned from studying 10B user experiences): more consumer activity to move online in the not-so-distant future; the week before; be ready for spikes throughout the entire holiday season, especially on Mondays. a 2-second improvement to median load times, they almost doubled mobile conversions. 

  • Looks good. The Internals of PostgreSQL: Chapters include: Database cluster, databases and tables; Process and memory architecture; Concurrency Control; VACUUM processing; Buffer Manager; Write Ahead Log (WAL); Base Backup and Point-In-Time Recovery (PITR); Streaming-Replication (SR).

  • Nice list of Advanced SQL Recipes to jump start your Analysis. Recipes like Calculating Running Total; Calculating Running/Moving Average; Calculating Z-Score.

  • 1.1 Billion Taxi Rides on kdb+/q & 4 Xeon Phi CPUs: Q is a programming language with a built-in, column-oriented, in-memory and on-disk database called kdb+. Q both includes and extends SQL. Q is native to the database engine so, unlike most databases, there is no shipping of data between the client and the server.

  • The Must Know Checklist For DevOps & Site Reliability Engineers. Seems reasonable. There's surely a lot to know.

  • It's hard to explain why your PaaS is worth the money. PaaS (false) economics. Good advice from NiteLite: Yeah, you should probably look into displaying your pricing information differently. Your pricing model right now is very similar to how AWS etc do their pricing. By breaking down the pricing in a different way you can probably make it a lot more clear that you are selling a service that lets the customer spend less time configuring servers and more time actually making money...Usually you want to make the customer feel like he is saving money if he uses your service. At the moment your pricing information might come across a little too much like "the cost of doing business" instead of a "simple way for your customer to be more efficient" :D

  • Best Practices For Using A Multi-CDN Strategy: How To Balance, Prioritize and Optimize Traffic: There are three types of strategies, those being balanced, prioritized, and optimized. Using a balanced strategy, one merely distributes traffic with different thresholds (like traffic served or concurrent streams) to spill over into secondary CDNs upon reaching a specified limit. A prioritized schema provides a criteria hierarchy, which may include the platform, ISP, device, or protocol utilized until a certain level.

  • randsinrepose on three rules that define a good game: Do I have a continual healthy sense of progression?; Am I learning and mastering the game via timely and effective feedback?; Do I have the impression that I can win?

  • This is different. MHRD is a hardware design game, in which you design various hardware circuits in a hardware description language. The hardware circuits you design get more complex as you go until you create a fully functional CPU design.

  • Need some Extreme Scale Computing training? Here's a series of 82 videos: 2016 Argonne Training Program on Extreme Scale Computing.

  • AMD Infinity Fabric underpins everything they will make: So that is the key to the new Infinity Fabric, the granularity, especially in mesh topologies it should allow bandwidth to scale with nodes. Topology is not protocol defined or restricted and the coherent links will work across sockets, CPUs, GPUs, and more. If the level of granularity is as fine as was intoned, it allows a CPU core to pass info to a shader ‘directly’ regardless of the two being on the same silicon or across a system. The separate control and data fabrics bring AMD up to modern SoC structures too, and in some ways beyond. Infinity Fabric is a really big deal

  • fission.io: Designed to be the open source alternative to Lambda. Also tries to make Kubernetes easy to consume. Good coverage on The ServerlessCast #1 - Project Fission. 

  • urweb/urweb: a domain-specific functional programming language for web applications. Lets web developers completely specify their sites’ functionality using just one programming language. The Ur/Web compiler then automatically generates all the different types of code required to power a website — HTML, JavaScript, SQL database queries, and cascading style sheets — while providing certain performance and security guarantees. More info: Faster websites with fewer bugs - System that automatically handles database caching in server farms increases speed and reliability.

  • Deep Probabilistic Programming: We propose Edward, a Turing-complete probabilistic programming language. Edward builds on two compositional representations---random variables and inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. or efficiency, Edward is integrated into TensorFlow, providing significant speedups over existing probabilistic systems. For example, on a benchmark logistic regression task, Edward is at least 35x faster than Stan and PyMC3.

Read more