hot links

Stuff The Internet Says On Scalability For October 27th, 2017

High Scalability

High Scalability

— 25 min read

Hey, it's HighScalability time:

How a mutex works pic.twitter.com/TwFLAVs2ydOctober 21, 2017


Perfect! Now, imagine a little dog snuck under Big Dog's cone of shame and covered the food with its own cone of shame, and it won't leave. That's deadlock. Imagine a stream of little dogs sneaking under Big Dog's cone so Big Dog nevers gets a bite. That's livelock.

If you like this sort of Stuff then please support me on Patreon.

  • $100 billion: projected 2021 combined app store spend; 11 TB: SSD; 16: billion dollar disasters in the US this year; 8: meter long 3D printed bridge; 43%: employees who worry about losing their job due to their age; 125 TFLOPS: new AWS EC2 P3 instances; 7%: global Internet traffic flowing over QUIC; 50%: improvement in new in-package DRAM cache-management scheme; 43%: CockroachDB speed improvement executing parallel SQL statements; 325 billion: hours spent in Android apps in Q3; 4.5 million: C++ programmers; 3 trillion: ops per second in Pixel's Image Processing Unit; 80%: drop in Facebook referrals; 1,300 years: longest running business in the world; 40: age when tech workers start worrying about age discrimination; 400GE: first test by China Telecom Guangzhou and Huawei; 

  • Quotable Quotes:
    • @DynamicWebPaige: "One of our customers has 1 billion invocations daily; they told us their [Azure] bill was only $72 due to serverless computing." #RedShirtDevTour
    • Tim O'Reilly: “He realized then that history is a wave that moves through time slightly faster than we do.” If we are honest with ourselves, each of us has many such moments, when we realize that the world has moved on and we are stuck in the past.
    • @ehashdn: A primer: Site Reliability Engineers = sysadmins with Go / DevOps Engineers = sysadmins with Ruby / Systems Administrators = sysadmins with Perl
    • @ramez: Exponential gains in compute power produces only linear gains in AI accuracy. --> No runaway intelligence explosion 
    • @abnerg: $AMZN AWS reports YoY growth of 41.9% to $4.58B for the Q. $MSFT says Azure was up 90%. The ☁️ is on 🔥
    • @ryan_sb: Programming training doesn't need to start at birth. It's like plumbing: it's a skill, anyone can start anytime, it has hard and easy parts
    • Chuck Hollis: [Oracle] has re-implemented its entire business on a modern cloud platform – SaaS, PaaS and IaaS. Remember we’re talking a ~$200bn market cap company here – no easy trick.  The fun thing is that I’m part of a project to document the before and after around a whole raft of internal business metrics. The comparison is stunning, to say the least.
    • psyc: I can't be sure yet, but I'm starting to be concerned. I turned 40 recently. My current job search has lasted about 5x longer than any previous job search. I've been turned down for nonsensical reasons, such as not having enough experience in a specific language that I have a lot of experience in. (That was an assertion by the interviewer, not the result of technical questioning.) I've been interviewed several times by managers and directors 10 years younger than myself. I've noticed a distinct pattern where they'll ask very basic questions, I'll give a detailed answer than I know to be correct and insightful, and they'll say it's wrong - and I'm just dumbfounded, like what can I say? I'm not going to argue with them.
    • jquery: So even if there’s not explicit bias against older devs there is implicit bias by favoring quick whiteboard speed (memorization/practice) over the practiced thoughtfulness of older devs. And a one-off interview focused heavily on algos can sink anyone. It only takes one.
      This isn’t limited to FB. The $Elite companies I got offers from are the ones where I lucked through that one tricky interview by knowing it offhand. I worry as I get older, even as I become a stronger developer, I will become less and less able to marathon through these interviews. No wonder so many older devs switch to management.
    • Jonathan Solórzano-Hamilton: “You will never be able to understand any of what I’ve created. I am Albert F***ing Einstein and you are all monkeys scrabbling in the dirt.”
    • Franz Faerber: Hekaton achieves a roughly 15.7X performance improvement at 12 cores, while the scalability of the traditional engine is limited due to the overheads inherent in a disk-based architecture running a memory-bound workload.
    • @mhall119: I'm convinced that 90% of good software development in knowing what code not to write
    • Mohit Kumar: Coinhive has been hacked — a popular browser-based service that offers website owners to embed a JavaScript to utilise their site visitors' CPUs power to mine the Monero cryptocurrency for monetisation.
    • @timallenwagner: Couple updates: local execution is available via SAM Local. Lambda is also HIPAA eligible and PCI compliant.
    • @erictartanson: Amazon has 540K employees, nearly 7x that of Google and 4x that of Microsoft, 2nd largest US employer 
    • @NetflixUIE: Removing client-side React.js (but keeping it on the server) resulted in a 50% performance improvement on our landing page
    • Jonathan Lin Ern Sheong: Stack Overflow takes a hybrid approach, where links to questions are of the form https://stackoverflow.com/questions/42764046/ses-port-is-blocked-in-gcp with a perma ID and a vanity portion. It is friendly to both humans and computers. The URL remains valid even if the vanity portion changes when the Question is edited.
    • Vincent Lanaria: Alphabet says that deploying Project Loon in Puerto Rico is the first time it has used "machine learning powered algorithms" to ensure the balloons are over Puerto Rico. In other words, it hasn't found the optimal way of going at it just yet.
    • jhall1468: Create a problem, then offer a solution. In this case, invent a problem that doesn't exist, and your solution is one among a sea of shitty pre-mined Ethereum ICOs. It's pretty sad, really. 
    • dominotw: I was recently rejected by a big tech company for not knowing Java 8. I code in scala in my day job and haven't kept up with changes in java. My coding exercise language options were nodejs,java, python, I submitted solution java 7. Interviewer said to my face ( well..on the phone) that they won't be "moving forward" because I don't keep up with changes in technology and they are a 'fast moving' tech environment.
    • David Grinspoon: when faced with existential threat, we have risen to the challenge by completely reinventing ourselves through invention and collaboration
    • uncletammy: Nope. ~80% of miners, 90% of Bitcoin businesses, and at least half of the community are forking Bitcoin.
    • Kulyukin: If you can think about the beehive as an intelligent, immobile robot, essentially monitoring the bee colony inside and letting all interested parties know of deviations from the norm, then it [BeePi hive sensors will] cut down on hive inspections and transportation costs immensely
    • majke: There are three highlights in this blog post: - epoll() seem to have LIFO behavior which can result in uneven load balancing across workers (in case of accept()ing from a shared socket) - using REUSEPORT can worsen latency in a high-load case. - blocking accept() on Linux is pretty buggy, you basically can't close() underlying socket
    • Google: Using Pixel Visual Core, HDR+ can run 5x faster and at less than one-tenth the energy than running on the application processor (AP). A key ingredient to the IPU’s efficiency is the tight coupling of hardware and software—our software controls many more details of the hardware than in a typical processor. 
    • @DynamicWebPaige: "If [on Azure] you provision 4 million transactions/sec, latency <9ms, 1 trillion transactions/day, you get it. Guaranteed." @CosmosDB #RedShirtDevTour
    • @NelsonMRosario: 2/ there is no such thing as crypto economics, there's economics
    • Mike Elgan: Despite the headlines, the fact is that physical stores represent the overwhelming majority of all sales, and will continue to do so for decades. In fact, only around 10% of all sales are from online stores. The rest is good old-fashioned stores.
    • @suzenfylke: Azure: I heard you like running VMs so I put VMs inside your VMs so you could run VMs while you run VMs. #RedShirtDevTour
    • rothbardrand: Bitcoin has already gone from 1-4mb max blocks this August, going to 8mb is unproven.... and it does reduce security because it increases centralization. The great firewall of china is a real thing, and I've seen first hand that 1mb blocks have trouble crossing it. Absolute number of full nodes is not a relevant measure in this issue... miner centralization is.
    • @marcusmeurs: I hit 42.4 hashes/sec [mining in browser]. To compare: GPU mining Siacoin with a GTX 980 ti hits  1,650 MH/s
    • @rg9119: Gul Agha to John McCarthy: Why did LISP have dynamic scope? McCarthy to Agha:  To be honest, I didn’t understand lambda calculus. (AGERE17)
    • agamemnon: Study Finds Expressing Anger In Unhealthy Ways Incredibly Satisfying
    • @sheeshee: sometimes things are so obvious and yet so valuable... "move historic data ahead of time" before your actual migration.
    • Andrea Passwater: Object Oriented Programming is to Death Star as, as functional programming is to river delta in this distributed world, that's kind of the analogy. And, I believe the functional programming and the ideas and principles behind functional programming, does scale really well to a distributed case
    • @Carnage4Life: Facebook asks @Oatmeal to pay $2K for a post to reach 10% of his followers. FB has become the ultimate bait & switch platform.
    • @viktorklang: Mechanical Sympathy is important for Performance. Distributed Empathy is important for Scalability.
    • cryptonector: you assume things in Linux-land are "designed". It's more like natural evolution. It's very organic. As you'd expect, there's a lot of decomposition around. It kinda smells as you'd expect. The Linux community is known to be infected with a drug-resistant strain of NIHV -- the Not Invented Here Virus. This means that if there's a thing like kqueue, or NT I/O Completion Ports, or Solaris Event ports, and Linux could copy and/or improve one (or more) of those, then you can count on Linux to invent a new thing that isn't as good as any of the others.
    • Christopher Monroe: I believe in the next five to 10 years we will have 100-plus-qubit machines that will be available to anyone, and this will be when useful applications will be found
    • @unscriptable: I can't emphasize enough how serverless architecture helped me focus on microservices and forget about infrastructure (and devops)
    • @peterseibel: Infrastructure to make it trivial to migrate containers between your data center and different clouds: Dockers Without Borders.
    • Erik Nordström: PostgreSQL 10 with declarative partitioning is a big step forward, but it is a general implementation that seems more targeted towards medium-sized traditional workloads than for typical time-series data.
    • Joanne Itow: Microcontroller (MCU) sales are expected to grow over 9% this year while units increase over 16%. IoT and automotive are the major reason for the growth in general MCUs; however, the product group has been transitioning into several distinct segments 
    • codedokode: Google Page Speed is an awful tool. For example, it often advices to compress images and even suggests to download "optimally" compressed versions. Those images have awful quality, and have visible compression artifacts. The person who chose this level quality is probably blind. Also it often gives crazy advices, for example something like "embed CSS code necessary to display the top of the page, right into the page". 
    • @chrismunns: People pushing multi-cloud by talking about a container based abstraction layer ignoring the other 90% of the stack as a Service.
    • Dimitri: So far, we're expecting to see a significant progress on RW performance with MySQL 8.0 ! However, regardless positive overall benchmark results comparing to previous MySQL version, we're still far from scaling on Writes.. -- work in progress, stay tuned, yet more to come.. ;-))
    • Mark Callaghan: I just hope the devices I use right now will last long enough to finish my current round of testing. One day I will switch to EC2 and GCE and wear out their devices, but I like the predictability I get from my home servers.
    • @kelseyhightower: After reviewing a few Serverless platforms, I can't see the value in dealing with source code. Container images provide a better contract.
    • Wardley: We’re moving away from a product stack — things like LAMP and .NET — to a much more utility-based, code-execution environment, and that’s what we’re getting with [Amazon Web Services’] Lambda. We’re going to see an explosion of things built on top.
    • kev009: DCTCP is designed for.. data center networks. You have first party control and can ensure ECN works end to end on your equipment. BBR is designed for the "hostile internet" where you can't rely on ECN marking, and basically tons of people are willingly and unwillingly plotting against you.. middle boxes that do policing and shaping and just plain bizarre things, routers that clear options, other worse/unfair congestion controls, extreme variation in buffer sizes, etc
    • gaius: Cars are such an emotive topic because they are a powerful democratizing force. Any ordinary citizen can have at their disposal a machine that can transport them, their family/friends and their stuff, anywhere they want to go at any time, can be packed in advance and unpacked at leisure when you get back, can act as a base or a shelter at the destination, can extend a bubble of personal space on long or stressful journeys that no other means of transport can, and so on and so on. The self-driving car utopia breaks all of this
    • DIANA GITIG: Random, stochastic mutations allow species to diversify. But selective pressures push them toward sameness, by forcing them to thrive under limiting conditions. The 60,000 generations of E. coli already in Richard Lenski’s freezer have started to show how these opposing forces shape evolution;
    • Ashley Yeager: The finding suggests that the building blocks of life, including DNA, RNA, and materials used in cell walls and energy storage, may have been generated in mists and sea sprays on early Earth.
    • ChuckMcM: There has been a lot of work looking at this issue inside of Google. One of the natural outcomes of trying to schedule a bunch of different 'types' of services on CPU and maximize utilization, is to look at each subsystem (memory, network, disk i/o) and work on ways of partitioning them. Of course if they can block everything (like disk i/o can) then you need to take extra precautions. co-processes using the same spindle (physical disk drive, even if they aren't on the same partition) can easily become priority inverted if the low priority process is doing a lot of disk i/o.
    • adzicg: For me, charging for actual usage instead of reserved capacity actually aids in implementing those good design guidelines that were collected in books, such as low coupling and high cohesion of code, and this is what we wrote about in the paper as well. charging for reserved capacity creates financial incentives to bundle tasks and features into applications, creating runtime coupling (eg sharing the same /tmp dir, reusing security roles etc), even though they are designed to be isolated. charging for actual usage removes that incentive, so stuff that was designed to be isolated stays isolated when deployed.
    • zwischenzugs: Lean, Agile, Waterfall, whatever, the fact is we need some kind of common ideology to co-operate in large numbers. None of them are evil, so it’s not like you’re picking racism over socialism or something. Whichever one you pick is not going to reflect the reality, but if you expect perfection you will be disappointed. And watch yourself for unspoken or unarticulated collective fictions. Your life is full of them. Like that your opinion is important. 
    • David Gerard: The DAO was shut down soon after, and on 20 July the Ethereum Foundation – several of whose principals were curators of The DAO362 and/or heavily invested in it – changed how the actual code of Ethereum interpreted their blockchain (the “immutable” ledger) so as to wind back the hack and take back their money. The blockchain was “immutable,” so they changed how it was interpreted. The “impossible” bailout had happened.
    • Retr0spectrum: Note that enabling stack protection is often not enough to completely mitigate stack smashing attacks if there is a vulnerability in the code. For example if you can leak the value of the protector somehow, then you can just replace it with its original value when you smash the stack. Alternatively, it may be possible to brute-force the stack protector value, particularly on 32-bit systems where the protector often only has 24 bits of entropy.
    • dracodoc: Some disadvantage of OP's method: - hash code is difficult to read for human. Urls under some hierarchy share some common patterns, it also bear some meanings. All hash code will look same and have nothing to hint on the content. - You have to copy it, almost impossible to type it, or compare two visually similar string. - You may end up with some link shortening service for hash code, but you can use link shortening to solve the portable file host problem already. The Merkle Trees can solve some problems, but I don't think portable urls are the right one.

  • Matt Cutts (early in Google) is the new administrator of the US Digital Service. Could anyone be better for the job? Nope. He has the technical chops and the personality to make it work. And it's working, he's changing the government one line of code at a time. He explains how in Government is a System (long version). Medicare has 550 million beneficiaries, $620 billion in payouts, and is 3.5% of the US economy. How it works is important. Matt talks about using agilish practices to improve Medicare. They asked contractors to build prototypes, looked at the quality of their code, and selected the best. Very different than how the gov usually works. The move to AWS is projected to save $20 million over 5 years. They're implementing an API first architecture. Improving group communications. At the end he makes a powerful pitch to come join him, he asks you to ask yourself: Where can I do the most good for the most people?

  • Emit Conf 2017 videos will be available soon. Also, Key takeaways for the future of serverless - Emit 2017 recap, Bobby Calderwood - toward a functional programming analogy for microservices. Though I would caution, just because you invoke functions in FaaS, that's very different than functional programming.

  • rothbardrand: Bitcoin has layers. One of the layers people don't understand is the layer called "core". The creators of bitcoin were cypherpunks and so they had a decade before bitcoin existed where people worked out how such a system would work. Those cypherpunks became core. While people think of bitcoin as the software, or the network or the white paper, what it really is, is a collection of engineers (you have to trust the engineers to trust the software and network and ultimately money) that is very diverse, and a genuine meritocracy. Since bitcoin attracted a lot of libertarians and wanna-be-I-think-I-am-libertarians its relatively easy to say "core is the new evil overlords! a monopoly!" when the reality is that core is a large number of people who don't agree on a lot of stuff and have developed a process of reaching consensus. So, unfortunately, some uneducated non-engineer felon like Roger Ver can spread a lot of claims about core and those get believed despite core people explaining patiently in engineering terms why the claims are false (eg: big blocks are not free scaling, segwit is more efficient, etc.) Non-engineers simply cannot understand the technology arguments and so they are manipulatable via social engineering, which in this case is literally legions of paid shills (http://birds.bitcoin.com is the attackers site where shills get paid.) That's the layer that's under attack now- because the powers that be want to control bitcoin and they can't beat the software, they can't beat the hash power, but they think they can maybe beat the people. (And in Ver's case, he shills BCH to keep its price up while unloading it to get more BTC.)

  • What's the future of programming? The answer lies in functional languages. That's the future with a linear growth curve. It's not a very different future at all.

  • @clemensv: Let me highlight some key aspects of the Azure Event Grid Service for which we announced the preview yesterday. #AzEventGrid 1/18; The name "grid" reflects that we just added a cloud-wide, cloud-scale event distribution fabric to the Azure platform foundation 2/18; We made this accessible early so you can start playing with it and planning for it, even while many Azure teams still build publishers 3/18; Like power, water, and gas in a city, Event Grid is part of the infrastructure. You don't make one, it exists. 4/18; There's no deployment, updates, management work or scale and capacity monitoring to worry about. Capacity is there. 5/18; You just subscribe for events raised anywhere by your apps or Azure resources to be delivered to your handler functions. 6/18; Azure storage will raise an event when a blob was created. Subscribe for .jpg suffix and trigger a function that analyzes the picture. 7/18; The subscription gesture for this acts right on the storage container. Just three concepts: Storage container, subscription, function. 8/18  ...

  • LambdaConf 2017 videos are now available.

  • Is it a good thing when a lot of the money being made by apps don't depend on your platform? Global App Downloads & Consumer Spend Hit Record Levels: Q3 2017 Recap: In terms of categories, Entertainment had standout success in Q3 2017 on both iOS and Google Play. Worldwide consumer spend in this category grew 30% quarter over quarter on iOS and 45% on Google Play...Video streaming apps drove the category in both stores as consumers increasingly see value in consuming content on mobile apps. On iOS, Tencent Video experienced the highest quarter over quarter share change in the Entertainment category in Q3 2017, followed by Youku and STARZ. On Google Play, HBO Now had the highest share change followed by Netflix and STARZ.

  • Concurrency vs Parallelism and the Erlang Advantage: Suppose I work at a burrito shop. Two customers come in, and each orders a burrito. How do I get the work done? To do the work serially, I'd get out one tortilla, fill it with yummy stuff, and close it up. Then I'd get out the next tortilla and do the same. This is neither concurrent nor parallel. To do the work concurrently, I'd get out one tortilla and spread it out. Then I'd get out another tortilla and spread it out. Now I am making two burritos concurrently. I have two burritos in a partial state of completion at the same moment. I can switch between these burritos, doing a bit of work on the first, then the second, then the first, then the second. However, because I only have one set of hands, I can only be actively working on one burrito at any given moment. To do the work in parallel, I'd get a coworker to help me. We'd each spread out a tortilla. While I'm filling the first one, my coworker would be filling the second. If you took a photo at that moment, you'd notice that there are two burritos in a partial state of completion. So the work is concurrent. But there are also two burritos being actively worked on at that moment. This means the work is being done in parallel. Now we're actually getting more burrito-making done per second.

  • Just in case you thought the cloud was secure by default, it still takes work. Many-faced threats to Serverless security: AWS takes over the responsibility for maintaining the host OS as part of their core competency...12 of the top 50 data breaches were caused by applications using components with known vulnerabilities... 77% of the top 5000 URLs from Alexa include at least one vulnerable library...security updates to these 3rd party dependencies are usually bundled with feature and API changes that need to be integrated and tested... he managed to gain direct push rights to 14% of NPM packages...SQL injection and other forms of injection attacks are still possible in the serverless world, as are cross-site scripting attacks...we can accidentally expose all kinds of sensitive data by accidentally leaving directory listing options ON...you need to apply the Least Privilege Principle when configuring Lambda permissions... IAM policies are not versioned along with the Lambda function...account level isolation can help mitigate the problems of adding/removing permissions...Delete old Lambda functions that you are no longer using...scaling your serverless architecture aggressively to fight a DoS attack with brute force has a significant cost implication...ensure [data is] protected at rest...Use secure transport when transmitting data to and from services...You can use API keys but I think it’s better to use IAM roles. It gives you fine grained control over who can invoke which actions on which resources...A good approach to prevent AWS credential leaks is to use git pre-commit hooks.

  • DDoS is a form of asymmetric warfare. The Economics of DDoS Attacks: DDoS attacks are cheaper than ever for the perpetrator, extremely lucrative for the attack service provider, and financially devastating for the target. Low prices and the turnkey nature of attack services — which require nothing to build or configure — have “democratized” DDoS attacks and the threat actor population...individual DDoS attacks can now be launched for as little as $5. As such, attack service providers look to make their money on volume — explaining why Arbor’s ATLAS infrastructure has recorded 6.2 million attacks so far this year, roughly 22,000 per day. 

  • CppCon 2017 videos are now available. Also, C++17 is Here: Interview with Herb Sutter: Now that we have C++17, some people will be excited by the “big” features like Parallel STL, but I think the things you’ll see and appreciate every day are things like structured bindings (such as writing for (auto [key,value] : my_map) {…}) and class template argument deduction (such as writing pair p{1, 2.0}; instead of pair<int, double>{1, 2.0};) and being able to initialize variables in the scope of if and switch just as we can already do with for loops 

  • New hardware creates new industry categories. What can you build when GPS is accurate to within 30 centimeters? Superaccurate GPS Coming to Smartphones in 2018: Broadcom announced that it is providing customers samples of the first mass-market chip to take advantage of a new breed of global navigation satellite signals. This new chip will give the next generation of smartphones ­30-centimeter accuracy as opposed to today’s 5 meters. Even better, it works in a city’s concrete canyons, and it consumes half the power of today’s generation of chips...Why is this only happening now? “Up to now there haven’t been enough L5 satellites in orbit,” says Manuel del ­Castillo, associate director of GNSS product marketing at Broadcom. At this point, there are about 30 such satellites in orbit, counting a set that flies only over Japan and Australia. Even in a city’s “narrow window of sky you can see six or seven, which is pretty good,” del Castillo says. “So now is the right moment to launch.”

  • New hardware changes architecture. Introducing Amazon key. Get your Amazon packages securely delivered just inside your front door. Security is ensured using an Amazon Cloud Cam indoor security camera and a smart lock. Will we see houses built again with specialized mudrooms to accommodate human product delivery? Will these mudrooms include landing zones for drones? It's easy to imagine a drone on approach signaling a door to open, float on in, drop off the goods, and zoom away.

  • New hardware changes culture. Introducing Amazon key. Once the home was the castle. Enter without invitation and you can justly expect a load of buckshot as a reward. Will our homes become a semi-shared space? Convenience trumps privacy. It's not hard to imagine Amazon delivery drivers entering a house, taking stock of what needs replenishing, making the order, delivering the goods, and stocking the shelves, all without the home owner being present. Just like how jobbers restock store shelves. It's not hard to imagine the dog walker being let in, or the cable repair person let in, or the dishwasher repair person let in, or the house keeper let in, all without the home owner being present. All secured with the ever vigilant camera and a smart lock controlling access. It's a radical change to our conception what a home is. Own the home, own the world.

  • Short and sweet. A Survey of CPU Caches: CPU caches are very fast and small memories. They are part of the CPU and store a subset of the data present in main memory (RAM) that is expected to be used again soon. Their purpose is to reduce the frequency of main memory accesses...The hidden constant separating the time complexities of two reasonable algorithms under asymptotic analysis can get quite big because of cache effects. Understanding how CPU caches work helps make good choices for writing fast programs and I hope this article provided some insight.

  • Filesystem error handling: Filesystem error handling seems to have improved. Reporting an error on a pwrite if the block device reports an error is perhaps the most basic error propagation a robust filesystem should do; few filesystems reported that error correctly in 2005. Today, most filesystems will correctly report an error when the simplest possible error condition that doesn’t involve the entire drive being dead occurs if there are no complicating factors. Most filesystems don’t have checksums for data and leave error detection and correction up to userspace software. When I talk to server-side devs at big companies, their answer is usually something like “who cares? All of our file accesses go through a library that checksums things anyway and redundancy across machines and datacenters takes care of failures, so we only need error detection and not correction”. While that’s true for developers at certain big companies, there’s a lot of software out there that isn’t written robustly and just assumes that filesystems and disks don’t have errors.

  • Ontogeny recapitulates phylogeny. Anytime we get a new thing it seems we have to recreate all the old things. Serverless is about Automation, not Functions: Originally, the function was the unit of “Serverlessness” in the Serverless community...What we found is simply that when you have more than a few functions, and a small number of elements and you had to start managing it, that there were issues...Serverless became more about Automation than it did about Functions...The power to automate deployment. The power to automate scale. The power to automate maintenance. The power to automate repeating tasks. Serverless is all about the automation at all levels.

  • Turning on Project Loon in Puerto Rico: This is the first time we have used our new machine learning powered algorithms to keep balloons clustered over Puerto Rico, so we’re still learning how best to do this.   Want to track Google's Loon balloons? https://www.flightradar24.com/18.22,-66.59/8

  • You know how it's easier to learn a new thing by comparing it to something you already know? That's what Azure is doing with AWS. Cloud Service Map for AWS and Azure Available Now. They even showed parts of the map where there is no mapping. There be dragons one would assume. Azure has a lot of stuff AWS doesn't have and vice versa. Azure wins in areas of enterprise integration.

  • If you're around Palo Alto on Thursday, November 9, you might be interested in Gopherfest SV 2017 - Architectures, Design Patterns, and Lessons Learned.  

  • It's fun to watch programmers relearn the power of real message formats. Bridging Rails & Node.js with gRPC: Our first foray with gRPC has been a great alternative to hand rolling http clients that use JSON over the wire for internal service communication. gRPC also provides a number of new features that we’re eager to try in our stack: async clients, streaming messages, and a very high throughput service library. For internal service communication in a polyglot environment, gRPC provides excellent tooling and documentation and I suspect we’ll be continuing to experiment with it as we grow.

  • Great detailed example with code. Low-risk Monolith to Microservice Evolution. An act in three parts: I, II, III. Too much to gloss.

  • The Evolution of a Real-World Recommender System: Related Pins is the Web-scale recommender system that powers over 40% of user engagement on Pinterest...Our first version of pin ranking was pre-computed in offline map-reduce jobs and served from Terrapin [23], an immutable key-value lookup service. This required a massive map-reduce job to join raw data for every query and candidate, compute features and score the items. Due to limitations of the cluster we could only rank a few million queries at a time, yielding 50% coverage of user queries. We scaled up by running the reranking job on different segments at a time and combining the results, but this approach was inherently unable to give full coverage in a reasonable amount of time. Offline ranking also significantly slowed development velocity...we moved to an online ranking serving system. The pin raw data is stored on a sharded key-value store called RealPin [16], keyed by image signature. To perform ranking, we assemble a request with the list of candidate pins and other raw data that will be needed to compute the features and score...The RealPin root server replicates the request to the leaves, routing the appropriate subset of the candidates to each leaf server. The leaves locally retrieve pin raw data and invoke our custom feature extractor and scorer. The leaves send the top candidates and scores to the root node, which gathers and returns the overall top candidates. We chose this serving architecture to increase data locality...By pushing computation down to the nodes that store the candidates’ pin raw data, the bulk of the data transfer can be avoided

  • It's like watching a crash in slow motion knowing you are powerless to stop it. IoT_reaper: A Few Updates. So many interesting details about how it works, yet that doesn't help. Knowledge isn't power anymore, it's frustration, impotence, and rage.

  • How Complex Systems Fail: 1) Complex systems are intrinsically hazardous systems; 2) Complex systems are heavily and successfully defended against failure; 3) Catastrophe requires multiple failures – single point failures are not enough..; 4) Complex systems contain changing mixtures of failures latent within them.; 5) Complex systems run in degraded mode; 6) Catastrophe is always just around the corner; 7) Post-accident attribution accident to a ‘root cause’ is fundamentally wrong; 8) Hindsight biases post-accident assessments of human performance.; 9) Human operators have dual roles: as producers & as defenders against failure ...

  • Creating 3,000 columns is probably not the best data model to use. Best DB structure for survey data with 3000+ columns? Also, The day I reached the 1600 columns limit in PostgreSQL. 

  • Very clearly explained with helpful code examples. Dynamic Programming: First Principles: The aim of Dynamic Programming is the use of past values for the procurement of a solution to a problem. This is done to avoid unnecessary computation of the same calculation that problems may contain, optimizing, enabling computer algorithms to run as efficiently as possible. It aims to find the optimal substructure of a problem (if it exists), and to eliminate any occurrences of overlapping sub problems...A solution is said to have an optimal substructure if it can be defined based upon optimal solutions of its sub problems...A problem or function contains overlapping problems if it can be broken down into a series of smaller problems, and some are duplicates...Memoisation , also known as caching, is a technique used in Computer Science, which stores the results of functions and calculations, and uses them if the calculations are needed again.

  • danilop/ServerlessByDesign: A visual approach to serverless development. Think. Build. Repeat. This is a visual approach to serverless development: An application is a network of nodes (serverless resources, such as Lambda functions or S3 buckets) connected by edges (their relationships, for example a trigger or a data flow) Edit an application adding nodes and edges following an event-driven design

  • netifi/proteus-java: A fast, and easy RSocket-based RPC layer. 

  • Colaboratory: a data analysis tool that combines text, code, and code outputs into a single collaborative document.

  • Brigade: Event-driven scripting for Kubernetes. 

  • plaidml/plaidml: a framework for making deep learning work everywhere. PlaidML is a multi-language acceleration framework that: Enables practitioners to deploy high-performance neural nets on any device; Allows hardware developers to quickly integrate with high-level frameworks; Allows framework developers to easily add support for many kinds of hardware

  • FireSim: a cycle-accurate, FPGA-accelerated datacenter simulation platform developed in the Berkeley Architecture Research Group...FireSim lowers that entry bar and makes it easier for everyone to leverage the power of an FPGA-accelerated compute environment. Whether you are part of a small start-up development team or working at a large datacenter scale, hardware-software co-design enables faster time-to-deployment, lower costs, and more predictable performance. We are excited to feature FireSim in this post from Sagar Karandikar and his colleagues at UC-Berkele

  • Verifying Strong Eventual Consistency in Distributed Systems: Despite decades of research, algorithms for achieving consistency in replicated systems are still poorly understood. Indeed, many published algorithms have later been shown to be incorrect, even some that were accompanied by supposed mechanised proofs of correctness. In this work, we focus on the correctness of Conflict-free Replicated Data Types (CRDTs), a class of algorithm that provides strong eventual consistency guarantees for replicated data. We then obtain the first machine-checked correctness theorems for three concrete CRDTs: the Replicated Growable Array, the Observed-Remove Set, and an Increment-Decrement Counter. We find that our framework is highly reusable, developing proofs of correctness for the latter two CRDTs in a few hours and with relatively little CRDT-specific code.

Hey, just letting you know I've written a new book: Explain the Cloud Like I'm 10. It's pretty much exactly what the title says it is. If you've ever tried to explain the cloud to someone, but had no idea what to say, send them this book.

I've also written a novella: The Strange Trial of Ciri: The First Sentient AI. It explores the idea of how a sentient AI might arise as ripped from the headlines deep learning techniques are applied to large social networks. Anyway, I like the story. If you do too please consider giving it a review on Amazon.

Thanks for your support!

Read more