hot links

Stuff The Internet Says On Scalability For February 10th, 2017

High Scalability

High Scalability

— 16 min read

Hey, it's HighScalability time:

It was a game of drones.
If you like this sort of Stuff then please support me on Patreon.

  • Half a trillion: Apple’s cash machine; 4,000-5,000: collected data points per adult in US; 10 million: gallons of gas UPS saves turning right; 2.27: Tesla 0-60 time; 40: complex steps to phone security; $2.3 billion: VR/AR investment in 2016; 18%: small players make up public cloud services market; 500°C: first chip to survive on Venus; 5 billion: ever notes; 375,000: images from The Metropolitan Museum of Art in public domain; 18 million: queries per minute against Facebook's Beringei database; 159: jobs per immigrant founder; 2.5 miles: whales breach for stronger signal; 10,000x: computers faster in 2035; 

  • Quotable Quotes: 
    • @martin_casado: Chinese factory replaces 90% of human workers with robots. Production rises by 250%, defects drop by 80%
    • Jure Leskovec: [Trolling is] a spiral of negativity. Just one person waking up cranky can create a spark and, because of discussion context and voting, these sparks can spiral out into cascades of bad behavior. Bad conversations lead to bad conversations. People who get down-voted come back more, comment more and comment even worse.
    • sudhirj: The first concrete thing I learnt is this - implement pull first, it works 100% of the time, but may be inefficient with regards to time. Then implement push, it works 99% of the time but is much faster. But always have both running.
    • Tom Randall: California’s goal is considerable, but it’s dwarfed by Tesla’s ambition to single-handedly deliver 15 gigawatt hours 1 of battery storage a year by the 2020s—enough to provide several nuclear power plants–worth of electricity to the grid during peak hours of demand
    • @aphyr: Like I can't show that it's 100% correct, but so far I haven't found a way to break 3.4.0. Opens up a bunch of new use cases for MongoDB.
    • Azethoth666: The coming fast non-volatile memory architectures will be interesting. Everything will be in memory, but it will not go away. The infection cycle will have to clean up after itself or remain in the super fast volatile memory parts.
    • StorageMojo: In five years the specter of AWS cloud dominance will be a distant memory. The potential cloud market is enormous and we are, in effect, where the computer industry was in 1965. AWS will be successful, just not dominant. No tears for AWS.
    • @johnrobb: ~ 'Bots make public conversation a synthetic conversation. This makes it very difficult to know what consensus looks like.
    • W. Daniel Hillis: One day when I was having lunch with Richard Feynman, I mentioned to him that I was planning to start a company to build a parallel computer with a million processors. His reaction was unequivocal, “That is positively the dopiest idea I ever heard.”
    • @supershabam: Every database is a message bus if you try hard enough
    • mlechha: Boltzmann machines are a stochastic version of the Hopfield network. The training algorithm simply tries to minimize the KL divergence between the network activity and real data. So it was quite surprising when it turned out that the algorithm needed a "dream phase" as they call it. Francis Crick was inspired by this and proposed a theory of sleep.
    • @benjammingh: OH "Docker is Latin for a fire consisting predominantly of tires
    • UweSchmidt: "Real" bitcoining doesn't use services like coinbase; the coins are on your computer which you have to secure yourself. At least this is what you get told in cryptocurrency forums when one of the exchanges get hacked.
    • @axleyjc: 'Think of your System as a "Set of annotated request trees"' to manage microservice complexity @adrianco @ExpediaEng
    • @happy_roman: VW CEO on Tesla: "We'll win in the end, because of our abilities to scale & spread production."
    • edejong: Many engineers I have worked with like to throw around terms like: "CQRS", "Event sourcing", "no schema's", "document-based storage", "denormalize everything" and more. However, when pushed, I often see that they lack a basic understanding of DBMSes, and fill up this gap by basically running away from it. For 95% of the jobs, a simple, non-replicated (but backed-up) DBMS will do just fine.
    • aaron bell: Whichever cloud provider you pick based on your needs and their specific offering, I beg of you — please don’t try hybrid
    • zebra9978: Kubernetes introduces a lot of upfront complexity with little benefit sometimes. For example, kargo is failing with Flannel, but works with Calico (and so on and so forth). Bare metal deployments with kubernetes are a big pain because the load balancer setups have not been built for it - most kubernetes configs depend on cloud based load balancers (like ELB). In fact, the code for bare metal load balancer integration has not been fully written for kubernetes.
    • a13n: This is huge. 87-99% shared code between iOS and Android. Someday companies as big as Instagram won't need to have entire separate product teams for separate platforms.
    • David Rosenthal: I've always said that the chief threat to digital preservation is economic; digital information being very vulnerable to interruptions in the money supply. 
    • YZF: There are no channels [in C++], there are no lightweight/green threads, there's no standard HTTP library, no standard crypto libraries, no standard test framework. For certain classes of applications this makes Go significantly more productive and significantly less bug/error prone. Not to mention compile times.
    • jdwyah: Kinesis firehose to S3 and then query with Athena is pretty great. I've been very happy with the combo.
    • mcherm: Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me.
    • Aras Pranckevičius: Moral of the story is: code that used to do something with five things years ago might turn out to be problematic when it has to deal with a hundred. And then a thousand. And a million. And a hundred billion. Kinda obvious, isn’t it?
    • kordless: I've come to a hypothesis that technology's purpose is to gently erode the concept of "self"
    • Microsoft: Close to a year ago we reset and focused on how we would actually get Git to scale to a single repo that could hold the entire Windows codebase (include estimates of growth and history) and support all the developers and build machines.
    • XorNot: I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster). Any clustered database setup should recover from failed nodes. Any regular relational database should be pretty close to automated failover with replicated backups and an alert email. Containerization doesn't make this better or worse, but it helps a lot with testing and deployment.
    • Dan Luu: When I was at Google, someone told me a story about a time that “they” completed a big optimization push only to find that measured page load times increased. When they dug into the data, they found that the reason load times had increased was that they got a lot more traffic from Africa after doing the optimizations. The team’s product went from being unusable for people with slow connections to usable, which caused so many users with slow connections to start using the product that load times actually increased.

  • There's a quintessential Silicon Valley moment in The Founder, a movie about the more interesting than expected McDonald's origin story. Brothers Mac and Dick McDonald kicked around from startup to startup. Nothing stuck. Drive-ins ruled the day, but were ripe for disruption. They were slow, used lots of servers, had too many options, attracted the wrong user base, and often delivered the wrong results. Metrics told them users mostly bought burgers, fries, and milkshakes. So the brothers decided to completely rethink how burgers were made and sold. What they came up with disrupted the food industry: a serverless drive-in based on a new low latency pipeline for making burgers called the Speedy System. An order was delivered within 30 seconds of being made; metrics helped control the latency distribution. Here's a short vignette showing how it was done. You'll love it. They traced the exact dimensions of the kitchen and conducted numerous simulations to figure out the optimal configuration. Users walk up to the window to order, so no servers. The API was narrow, only a few items could be ordered. Waste was reduced because utensils were done away with using innovative packaging design. Automation and a proprietary tool chain delivered a consistent product experience. And as often happens in Silicon Valley the founders were out maneuvered. While the McDonald brothers innovated the tech, Ray Kroc innovated the business model. Guess who ended up with everything? 

  • There's a disturbingly deep similarity between advertising and cyber-criminality. AI isn't just for the good guys anymore: What artificial intelligence does is it lets them automate the tailoring of content to the victim.

  • Snap has always liked the cloud for its agility. Agility has a price. A high price. Snap commits $2 billion over 5 years for Google Cloud infrastructure. But they note "relying on Google Cloud could also potentially limit Snap’s ability to expand into markets like China." So Snap will spend $1 billion on Amazon cloud services and they hint they may choose to build their own infrastructure in the future.

  • When is 2FA not 2FA? When it uses SMS. Hackers Have Stolen Millions Of Dollars In Bitcoin -- Using Only Phone Numbers: A hacker had faked his identity and transferred his phone number from T-Mobile to a carrier called Bandwidth that was linked to a Google Voice account in the hacker’s possession. Once all the calls and messages to Kenna’s number were being routed to them, the hacker(s) then reset the passwords for Kenna’s email addresses by having the SMS codes sent to them (or, technically, to Kenna’s number, newly in their possession). Within seven minutes of being locked out of his first account, Kenna was shut out of of up to 30 others, including two banks, PayPal, two bitcoin services — and, crucially, his Windows account, which was the key to his PC.

  • So much for static type checking. @necolas: Today we moved all of Twitter's mobile web traffic (that's like, a lot) to our new web stack – Node.js, Express, React PWA It took about a year. For the last 2 years the stack serving logged-out mobile web has been Scala, Google Closure Templates, and a sprinkling of JavaScript. Reducing the amount of json parsed on the server has been very effective. Twitter is looking at graphql down the road.

  • Ever considered Google Cloud Platform? That's where Evernote is migrating too and they've published a 5 part article  [1,2,3,4,5] on the process. Migration uses a 20 day Accelerated Phased Cutover model. They evaluated Google on Organizational controls, Infrastructure security, Product security, Physical security, Data use related to privacy. Went with a Primary location (serving production traffic): US-West1 and Secondary location (disaster recovery site): US-Central1. Significantly simplified their architecture by using a cloud-based, queuing mechanism and redesigning the application to rely on both the availability of jobs in the queue and speed of notification. Copying hundreds of terabytes per day across thousands of nodes was a major pain point in the migration. The migration took a lot of special code. Migrations were carefully scheduled. Zero downtime was not possible. 

  • If you are an iOS developer should you learn Swift or React Native? huangc10: I chose React-Native simply because I thought it will allow me to broaden my mobile development experience; RubenSandwich: I picked React Native for a small project over Swift. It was worth it. Not only because immutable render functions make view code extremely pleasurable to write, but also because it's mostly JS and JSX it allowed me to quickly spin up a web developer to work on this project with me; htormey: I'm pretty happy with React Native, its waaay better than any other cross platform solutions I have tried.

  • Ever considered Firebase as your BaaS? Firebase review — 9 months in. It's much loved. The good: Beautiful user authentication API and management; One of the best (and cheapest) hosted database solutions around; Fantastic documentation; Amazing UI; Decent integration libraries. The bad: does not fully replace a backend; SDK library is closed source. Good comments on reddit. Others like it too but have some issues: can't query on more than one field; reading from multiple locations in the database is a pain; Consistent transactions are expensive; not good at handling JSON arrays. 

  • @thetinot on the blessed union of Snapchat and Google AppEngine. Snapchat has been reported to run almost entirely on Google AppEngine. GAE manages deployment, scale, availability, versioning, etc.  This is not the same as buying a bunch of VMs and storage. This is Snapchat outsourcing much of their Ops and support to Google. Hence Snapchat is able to focus on what's important to them - velocity of innovation on product. Leave the boring tedious bits to Google. By contrast, listen to Netflix's story on @awscloud . Crazy amount of tooling, resiliency, scaling Netflix had to build bespoke.

  • Storage may no longer be free. Another article about the supply crisis hitting #SSD, #flash, #NVMe, #HPC #storage in general: 1) There is insufficient SSD/Flash supply in market 2) Hard disks which have been slowly dropping in manufacturing rates are not quite able to take up the demand slack at the moment, there are shortages there as well 3) 10k and 15k RPM drives are not long for this world 4) New fab capacity needed to increase SSD supply is coming online (I dispute that enough is coming online, or that it is coming online fast enough, but more is, at least organically coming online) 5) There is no new investment in HDD/SRD capacity … there is a drying up of HDD/SRD manufacturing lines

  • If your system has over 2000 microservices, as Uber's does, then distributed tracing is something you really need to nail. Evolving Distributed Tracing at Uber. 

  • Great story. How I built the IMDb message boards, in 2001. It's like reading the diary of a settler who went out West. Everything tried to kill him, but the homestead was raised and built into a glorious farm. After years of production the farm is abandoned, but the stories live on. 

  • Can you feel it? There's a force blowing through infrastructure, reorganizing everything in its wake. Container orchestration: Moving from fleet to Kubernetes. 

  • That was quick. We may have already reached peak “sharing” economy. mdb333: is this surprising news? at least living in SF its pretty clear we're at full saturation. Companies have been throwing all the money at fending off competition and expanding to new markets... As well, most all of these gigs are means to an end, so equally unsurprising about the employment trends. Go ask folks if they want to be a taxi driver or delivery person and you'll get pretty negative response. Ask, hey do you want to work for Uber or Caviar and it's sure, why not. I imagine the novelty wears off pretty quickly.

  • Changing schemas sucks, which is one reason NoSQL is popular. Here's Stripe with an excellent example of migrating their subscriptions so they can support more complex billing models. Online migrations at scale. Why is it hard? Scale, uptime, accuracy. Process: dual writing, changing all read paths, changing all write paths, removing old data. 

  • Good question. What do you mean by “Event-Driven”?: Event Notification - a system sends event messages to notify other systems of a change in its domain; Event-Carried State Transfer - This pattern shows up when you want to update clients of a system in such a way that they don't need to contact the source system in order to do further work; Event-Sourcing - whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future; CQRS - the notion of having separate data structures for reading and writing information.

  • Lots and lots of good detail. Monitoring and Tuning the Linux Networking Stack: Sending Data

  • This looks like a fun class. Lessons from Real-Time Programming class: Don’t trust anything. Don’t make any assumptions; You are as only good as your tools; It’s ok to take on technical debt. Keep it simple, stupid; Pair programming is effective team working.

  • Yes, the world is getting scarier, but it's also getting cooler. TensorFlow Image Recognition on a Raspberry Pi. What if you want to predict the arrival time of a Caltrain train based on real observations? Old: hire someone to sit down and record every arrival in a notebook like an animal. A hidden benefit of this approach is there are many kinds of trains running on a track and a human can tell which is which. New: train TensorFlow to recognize different kinds of trains, embed the recognition algorithm in a Raspberry Pi, let it watch and do its thing. Cost $130.

  • What are the negatives of React Native? tunaoftheland: React Native development, on a good day, feels like the best of web dev and native dev (reload to see the change, `yarn add <package> && react-native link` to add a new NPM package, run the app on devices while debugging with Chrome dev tool). On a bad day, it feels like the ultimate force unification from the worst of both worlds (nothing ever stays stable, ever; were you expecting the app reload to work every time? hah!; running the debugger makes the app crawl, even on recent devices; some packages require you to do lots of stuff for post-install configuration; upgrading RN version will fix some issues but you'll lose half day trying to resolve conflicts between RN and your code and RN and the dependent packages -- then you might get some issues that are unique to the new version of RN; CSS styling -- can cut both ways)

  • Build a NodeJS cinema microservice and deploying it with docker — part 1. Good example with lots of code.

  • I want to host WordPress multisite for 10-20 sites on AWS, what size instance should I use?: It's gonna depend on whether you're running the DB on the webhost or using an RDS instance for the database, but I've got >100 separate WP installs running on a t2.medium (with DB on RDS), and it's holding up just fine. You can probably get away with a t2.micro for your use case, but I'd try it on-demand for a little bit at first before buying a reserved instance in case you need a little more horsepower than the micro offers.

  • Why MyRocks?: Efficient performance is the reason for MyRocks. We want to provide performance similar to InnoDB but with much better storage efficiency. The technical message is that MyRocks has less space and write amplification than InnoDB without sacrificing too much read amplification. For a workload I care about it uses 1/2 the space and writes at 1/10 the rate of InnoDB. The less technical message is that with MyRocks a deployment needs less SSD and the SSD will last longer.

  • Good description of an active-passive dual autoscaling group approach to Simple canary releases in AWS: instead of having a single autoscaling group we run two... the “active” autoscaling group has a size of 3 and the “passive”(canary) group is in a disabled mode...to deploy a new version, the new one is added to the passive autoscaling group...Once the new instance becomes active it will take 25% of the traffic...After monitoring the new version in production for a period time, we gain the confidence to start sending it 100% of the traffic. The next step is to disable the canary autoscaling group by draining all active connections and then do a more standard Rolling upgrade on the “active” group.

  • Facebook's Beringei: A high-performance time series storage engine: Operating large-scale, globally distributed services requires accurate monitoring of the health and performance of our systems to identify and diagnose problems as they arise...Each system and service at Facebook writes hundreds to thousands of counters to our storage engine, and this data is available to query in real time by engineers looking at dashboards and doing ad-hoc performance analysis...Beringei is different from other in-memory systems, such as memcache, because it has been optimized for storing time series data used specifically for health and performance monitoring. We designed Beringei to have a very high write rate and a low read latency, while being as efficient as possible in using RAM to store the time series data. In the end, we created a system that can store all the performance and monitoring data generated at Facebook for the most recent 24 hours...uses a lossless streaming compression algorithm to compress points within a time series with no additional compression used across time series...we discovered that the value in most time series does not change significantly when compared to its neighboring data points...Ultimately, this algorithm resulted in compressing the entire data set by at least 90 percent...the delay between a counter being written to Beringei and available for consumption is approximately 300 microseconds, and our p95 server response time for a read request is approximately 65 microseconds. Compared with our old disk-based system, Beringei’s in-memory system is several orders of magnitude faster...Beringei currently stores up to 10 billion unique time series and serves 18 million queries per minute.

  • Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part I): One of Shopify’s secret weapons is our edge tier, which uses a combination of Nginx and OpenResty’s Lua module. This module integrates into Nginx’s event model allowing us to write Lua scripts which operate on requests and responses. This is especially powerful because it allows a small team of production engineers to have impact on the entire Shopify core Rails app without doing open heart surgery. We’ve found this approach useful in many cases, including to build out an HTTP router to enable us to go multi-DC, to fanout requests across shards, and as a DDoS mitigation layer. 

  • Warning, entering double black diamond territory. “The king is dead, long live the king”: Our Paxos-based consensus: MySQL Group Replication is a multi-master update everywhere solution and transaction’s changes may be originated at any member in the group. Having a leader-based protocol would clearly harm scalability as all updates would have to go through the leader which then would be responsible for disseminating them...So there is no leader election and each member is a leader of its own slots in the stream of messages. Members can propose messages for their slots without having to wait for other members...Batching and pipelining are two optimizations that can increase performance and we implemented them...We also transmit the application data only once.

  • awslabs/lambda-refarch-imagerecognition: The Image Recognition and Processing Backend demonstrates how to use AWS Step Functions to orchestrate a serverless processing workflow using AWS Lambda, Amazon S3, Amazon DynamoDB and Amazon Rekognition. This workflow processes photos uploaded to Amazon S3 and extracts metadata from the image such as geolocation, size/format, time, etc. It then uses image recognition to tag objects in the photo. In parallel, it also produces a thumbnail of the photo.

  • LIQUi|>: a software architecture and toolsuite for quantum computing. It includes a programming language, optimization and scheduling algorithms, and quantum simulators. 

  • IntegriDB: Verifiable SQL for Outsourced Databases: a system allowing a data owner to outsource storage of a database to an untrusted server, and then enable anyone to perform verifiable SQL queries over that database. Our system handles a rich subset of SQL queries, including multidimensional range queries, JOIN, SUM, MAX/MIN, COUNT, and AVG, as well as (limited) nestings of such queries. Even for tables with 105 entries, INTEGRIDB has small proofs (a few KB) that depend only logarithmically on the size of the database, low verification time (tens of milliseconds), and feasible server computation (under a minute). Efficient updates are also supported.

Read more