Stuff The Internet Says On Scalability For September 28th, 2018

Hey, it's HighScalability time:

@danielbryantuk: "A LAMP stack is a good thing. Never inflict a distributed system on yourself unless you have too..." @mipsytipsy #CloudNativeLondon

Do you like this sort of Stuff? Please support me on Patreon and you'll get 100 free cloud credits in heaven. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

• $2 billion: Pokémon GO revenue since launch; 10: say happy birthday to StackOverflow; $148 million: Uber data breach fine; 75%: streaming music industry revenue in the US;  5.2 TB: Fastly peak per second traffic; 10 billion: Ethereum requests per day; 01%: DNS resolution issues when the KSK rolls; 15B: projected gaming community views on Reddit; £4.1bn: saved by UK Government's Digital Transformation Journey; 10X: Core ML model runs  faster on the A12 processor; 4 million: cores managed by Open Stack at Yahoo; 1PB: Azure's data box; 21 million: US Apple music subscribers; .675: Curry's league leading true shooting percentage;  $3 trillion: taxes collected by IRS; 61,000: network of Mayan structures discovered using lidar; 90%: China's percentage of $4.2 billion increase in pure-play foundry market; 


• Quotable Quotes:
  • WhatsApp cofounder: I am a sellout. I acknowledge that.
  • MrTonyD: I was writing production code over 30 years ago (C, OS, database). It is much worse to be a software developer now. It used to be a very high autonomy job - where you were trusted to figure out your work process and usually given lots of freedom to dynamically define many of your deliverables (within reason). I remember when I first read about Agile - I looked at the practices and thought "I've done most of those." But when our nightly builds broke it was no big deal, we would just fix them when we got around to it (as opposed to managers now assigning blame and calling in people on weekends to "fix" it). And if things weren't going well then we might have daily brief meetings for a couple of weeks. But now there are managers who insist on daily standups irregardless of their actual business necessity. I could go on and on. There is a reason why I'm not a practicing programmer anymore - even though I love to code.
  • Tiger Woods: "This was different," Woods said, acknowledging that the scene on the final hole was unlike anything he experienced in his previous 79 career victories. "I guess it’s different now because the art of clapping is gone. You can’t clap with a cell phone in your hand, so people yell."
  • @tomgara: "Slate makes more money from a single article that gets 50,000 page views on its site than it does from the 6 million page views it receives on Apple News in an average month"
  • Steve Case: We are seeing the beginnings of a slowing [in Silicon Valley] of what has been a brain drain the last 20 years. It’s not just watching where the capital flows, it’s watching where the talent flows. And the sense that you have to be here or you can’t play is going to start diminishing.
  • Martin Sústrik: Philosophers, by and large, tend to be architecture astronauts. Programmers' insight is that architecture astronauts fail. Or, maybe, they can succeed as in getting comfy job at IBM, but their designs don't translate into useful products. What else? After decades of pain, we have finally concluded that hierarchies of concepts don't work. That's not an obvious statement at all. After all, nobody gets fired from creating a hierarchy. 
  • vl: I have a hilarious story about this from Google: I wanted second 30" monitor, so I filed a ticket. They sent me long email listing reasons why I shouldn't get a second monitor, including (numbers are approximate, employee count from 2013 or so) "If every googler gets an extra monitor, in a year it would be equivalent to driving Toyota Camry for 18000 miles."
  • @dhh: The iPhone XS is faster than an iMac Pro on the Speedometer 2.0 JavaScript benchmark. It's the fastest device I've ever tested. Insane 45% jump over the iPhone 8/X chip. How does Apple do it?! 
  • @davidcrespo: how is this different from making entries in a traditional database how is this different from making entries in a traditional database how is this different from making entries in a traditional database how is this different from making entries in a traditional database  re:@marshallk: Wow: Walmart will *require* food suppliers to upload data to IBM's blockchain in order to trace quality issues through the supply chain. They say issues that take 7 days to track today can be tracked in 2.2 seconds with the system. 
  • @DrQz: Somewhat more revealing USL curves that go BEYOND the performance measurements (also corrects previous legend): • XDP-1 saturates single NIC at ~4 cores • DPDK scales "infinitely" but reaches NIC satn at ~13 cores • XDP-2 peaks before satn at ~15 cores due to non-zero β term.
  • Mark Szulyovszky: Would I have thought 3 years ago that we end up building a mobile tech stack on Microsoft’s & Facebook’s open source code, wishing that Google’s and Apple’s clunky and slow middle layer would disappear? No way.
  • Want more? There more...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Stuff The Internet Says On Scalability For September 21st, 2018

Wake up! It's HighScalability time:

Like sparkle from a fairy godmother's wand a swarm of 300 autonomous burner drones take flight. (mnn) 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 60: Marchetti's constant—minutes humans allocate to commuting over history; 300 billion: eBay data queries processed each day;  $1: 1GHz ARM Cortex-A8 processor capable of running Linux; 20M:Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset; 1 billion: Lyft rides; 600+: engineers @ Pinterest (+175B Pins to +250M people); 50%: workplace tasks handled by machines by 2025; 28: years of Hubble Space Imagery on AWS; 500 petabytes: eBay data footprint; 10 million: Gmail spam messages prevented per minute; 5 trillion: Apple's A12 operations per second; $220 billion: investment in new fabs and lines; 43%: Americans who something on Netflix each day; 


• Quotable Quotes:
  • Linus Torvalds: This week people in our community confronted me about my lifetime of not understanding emotions.  My flippant attacks in emails have been both unprofessional and uncalled for.  Especially at times when I made it personal.  In my quest for a better patch, this made sense to me. I know now this was not OK and I am truly sorry.
  • @jessfraz: To those in my mentions. This "foundation" which has tax breaks... Is paying people who don't contribute to the kernel 400k a year. Would you like to rethink your "they build the kernel" comments.
  • Aria Bracci: Using a previously established, peer-reviewed technique, the team conducted more than half a million data traffic tests across 161 countries. From this data, the team found that internet service providers are “giving a fixed amount of bandwidth—typically something in the range of one and a half megabits per second to four megabits per second—to video traffic, but they don’t impose these limits on other network traffic.”
  • @mjpt777: "Patterson indicated that rewriting Python into C gets you a 50 times speedup in performance".
  • @JoeEmison: The idea that any kind of planning will necessarily lead to overengineering is part of the anti-planning developer culture that misunderstands the agile manifesto.
  • Marie Hicks: The dynamic continues to this day. Silicon Valley reaps enormous profits at the expense of the majority of users, and calls it progress. But technology’s alignment with actual progress has a long and uneven history, and its effects are rarely straightforward or fully foreseen. Real progress isn’t synonymous with building another app—it involves recognizing the problems in our society and confronting the uncomfortable fact that technology is a tool for wielding power over people. Too often, those who already hold power, those who are least able to recognize the flaws in our current systems, are the ones who decide our technological future.
  • More quotes...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

How do you explain the unreasonable effectiveness of cloud security?

With the enormous attack surface of cloud providers like AWS, Azure, and GCP, why aren't there more security problems? Data breaches and cyber attacks occur daily. How do you explain the unreasonable effectiveness of cloud security?

Google has an ebook on their security approach; Microsoft has some web pages. Both are the equivalent of that person who is disgustingly healthy and you ask them how they do it and they say "I don't know. I just eat right, exercise, and get plenty of sleep." Not all that useful. Most of us want a hack, a trick to good health. Who wants to eat right? 

I'm sure Amazon also eats right, exercises, and gets plenty of sleep (probably not the people who work there), but AWS also has a secret that when that disgustingly healthy person starts talking about at a party, you just can't help leaning in and listening. 

What's the trick to 6-pack security? Proving systems correct. Does your datacenter do that? I didn't think so. AWS does. 

Dr. Byron Cook gave an enthusiastic talk on Formal Reasoning about the Security of Amazon Web Service. He's clearly excited about finally applying his research in a real-world setting. This is the trojan horse the FloC (Federated Logic Conference) community has been waiting for. It's almost as if he's a FLoC guy working at AWS rather than an AWS guy giving a FLoC talk.

The main take-aways for me were:

Click to read more ...

Sponsored Post: Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr, MemSQL

Who's Hiring? 

• Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here. 


• Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

• For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

• Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

• Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.


• MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.


• Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

---

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.

---

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today. 

---

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Stuff The Internet Says On Scalability For September 14th, 2018

Hey, it's HighScalability time:

The Cloud Native Interactive Landscape is fecund. You are viewing 581 cards with a total of 1,237,157 stars, market cap of $6.86T and funding of $20.1B. (changelog)

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 72: signals sensed from a distant galaxy using AI; 12M: reddit posts per month; 10 trillion: per day Google generated test inputs with 100s of servers for several months using OSS-Fuzz; 200%: growth in Cloud Native technologies used in production; $13 trillion: potential economic impact of AI by 2030; 1.8 trillion: plastic pieces eaten by giant garbage pac-man; 100: min people needed to restart humanity; 


• Quotable Quotes:
  • Joel Hruska: Farmers in California have lost the fight to be allowed to repair their own tractors and equipment thanks to the capitulation of their own lobbying group. This issue has been gathering steam for the last few years, after aggressive moves by John Deere to lock down the equipment.
  • @RachFrieee: 3.6B events processed to date 300k+ users globally 50% of the Fortune 100 use @pagerduty 10,500+ customers of every size 300+ integrations 🔒 Enterprise grade security #PDSummit18 @jenntejada
  • @editingemily: "In peacetime, a ticketing system is great... American Eagle loses $200K per minute of downtime. You gonna make a ticket? Burn a cool million?" 😂🤣😂 @jenntejada #PDSummit18
  • NSF: When the HL-LHC reaches full capability in 2026, it is expected to produce more than 1 billion particle collisions every second, marking a 10-fold increase that will require a similar 10-fold increase in data processing and storage, including tools to collect, analyze, and record the most relevant events. Uniting multidisciplinary teams of researchers and educators from 17 universities, IRIS-HEP will receive $5 million a year for five years from the NSF with a focus on producing innovative software and training the next generation of users.
  • @aallan: “Moore’s Law is really dead, but the lesser know Dennard Scaling Law is also dead… we’re almost at the end of the line, last year performance only improved 3 percent,” says David Patterson. #TheAIConf 
  • Gary Bronner~ DRAM scaling is starting to run out of steam. The industry continues to scale from the 1x, to the 1y, to the 1z generation, but the benefit from each generation is getting smaller and smaller. Because of that big servers and other memory systems need to have another kind of memory in the hierarchy. The new memory hierarchy is SRAM, DRAM, SCM (storage class memory), SSD, HDD. SCM slots between DRAM and flash in terms of latency, cost, and density. It's a way of adding more memory when DRAM alone can't give you all the performance you want.
  • @sethearley: 7 areas of opportunity for #AI in teaching. 1. Feedback & Scoring, 2. College readiness,  3. Empowering students with challenges,   4. Behavior management,   5. Curriculum development, 6. Higher order learning,     7. Professional development. #TheAIConf
  • @The_McJones: “How do we work out who wins? Well we don’t go to the bar and talk it out, instead we’ve decided to all spend billions of dollars and let the market figure it out” - David Patterson #theAIConf
  • DSHR: As I understand it, Amazon's S3 is redundant across three data centers, whereas Backblaze's B2 is not. Thus, despite both quoting 11 nines of durability against hardware failures, S3 is durable against failures that B2 is not, and is thus better.
  • @dr_c0d3: 2000: Write 100s of lines of XML to "declaratively" configure your servlets and EJBs 2018: Write 100s of lines of YAML to "declaratively" configure your microservices At least XML had schemas...
  • Programming Rants: This time PHP7 became the best performing programming language implementation, also the least memory consumption (I'm amazed with what they did in version 7).
  • @BenedictEvans: With each cycle in tech, companies find ways to build a moat and make a monopoly. Then people look at the moat and think it's invulnerable. They're generally right. IBM still dominates mainframes and Microsoft still dominates PC operating systems and productivity software. But...
  • @The_McJones: “So we strapped a bunch of FitBits onto cows” - Peter Norvig #TheAIConf
  • @noahsussman: Complex systems are intrinsically hazardous systems. Complex systems are heavily and successfully defended against failure. Catastrophe requires multiple failures – single point failures are not enough. Complex systems contain changing mixtures of failures latent within them. Complex systems run in degraded mode. Catastrophe is always just around the corner. Post-accident attribution accident to a ‘root cause’ is fundamentally wrong.
  • So many more quotes. Keep reading for No mo FOMO...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Public Cloud Postcentralization is the Thin Edge of the Wedge into the Enterprise

Like an amoeba the public cloud is extending fingerlike projections to the edge in a new kind of architecture that creates a world spanning distributed infrastructure under one centralized management, billing, and security domain.

This issue—the deep nature of centralization—came up as a comment on my article What Do You Believe Now That You Didn't Five Years Ago? Centralized Wins. Decentralized Loses.

Centralization can refer to the locus of computation, but it also refers to a boundary, to a domain of control.

Facebook, Netflix, and Google are all distributed across much of the world, but they are still centralized services because control is centralized. You know this because in a browser, no matter where you are in the world, you navigate to facebook.com, netflix.com, or google.com, you never enter the URL for independent shards, yet all your data and services magically follow you around like a hyperactive puppy. 

That's the world we've come to expect. That's how services built on a cloud work.

In an unexpected development, the public cloud is expanding control out to the edge. As I wrote in Stuff The Internet Says On Scalability For July 27th, 2018:

Click to read more ...

The Anna Key-Value Store Now Has 355x the Performance of DynamoDB for the Dollar

New databases used to be announced seemingly every week. While database neogenesis has slowed down considerably, it has not gone necrotic.

RISELabs, those wonderfully innovative folks over at Berkeley, have uplifted their Anna datatabase—a shared-nothing, thread-per-core architecture to achieve lightning-fast speeds by avoiding all coordination mechanisms—to become cloud-aware.

What's changed?

Anna is not only incredibly fast, it’s incredibly efficient and elastic too: an autoscaling, multi-tier, selectively-replicating cloud service. All that adaptivity means that Anna ramps down resource consumption for cold things, and ramps up consumption for hot things. You get all the multicore Anna performance you want, but you don’t pay for what you don’t need.

Just to throw out some numbers, we measured Anna providing 355x the performance of DynamoDB for the dollar. No, I don’t think that is because AWS is earning a 355x margin on DynamoDB! The issue is that Anna is now orders of magnitude more efficient than competing systems, in addition to being orders of magnitude faster.

They've posted about Anna's new superpowers in Going Fast and Cheap: How We Made Anna Autoscale:

Using Anna v0 as an in-memory storage engine, we set out to address the cloud storage problems described above. We aimed to evolve the fastest KVS in the cloud into the most adaptive, cost-effective one as well. We did this by adding 3 key mechanisms to Anna: Vertical Tiering, Horizontal Elasticity, and Selective Replication.

The core component in Anna v11 is a monitoring system & policy engine that together enable workload-responsiveness and adaptability. To meet user-defined goals for performance (request latency) and cost, the monitoring service tracks and adjusts resources to workload changes. Each storage server collects statistics about the requests it serves, the data it stores, etc. The monitoring system periodically scrapes and munges this data, and the policy engine uses these statistics to take action via one of three mechanisms listed above. The trigger for each action is simple:

• Elasticity: In order for a system to adapt to changing workloads, the system must be able to autoscale up and down to match the request volume it is seeing. When a tier is saturating compute or storage capacity, we add nodes to the cluster, and when resources are underutilized, they are deallocated to save cost.
• Selective Replication: In real workloads, there is often a hot set of keys, which should be replicated beyond fault-tolerance requirements to improve performance. This increases the cores and network bandwidth available to serve common requests. Anna v0 enabled multi-master replication of keys, but had a fixed replication factor for all keys. As you can imagine, that was unreasonably expensive. In Anna v1, the monitoring engine picks the most accessed keys and increases the number of replicas of those keys specifically, without paying extra to replicate cold data.
• Promotion & Demotion: Just like traditional memory hierarchies, cloud storage systems should store hot data in a high-performance, memory-speed tier for efficient access, while cold data should reside in a slower tier to save cost. Our monitoring engine automatically moves data between tiers based on access patterns.

In order to implement these mechanisms, we had to make two significant changes to the design of Anna. First, we deployed the storage engine across multiple storage media — currently RAM and flash disk. Each of these resulting storage tiers represents a different cost-performance tradeoff, akin to a traditional memory hierarchy. We also implemented a routing service that sends user requests to the correct servers in the correct tiers. This gives users a single, uniform API regardless of where the data is stored. Each one of these tiers has the same rich consistency model inherited from the first version of Anna, so the developer can work off a single (widely parameterizable) consistency model.

Our experiments show an impressive level of both performance and cost efficiency. Anna provides 8x the throughput of AWS ElastiCache’s and 355x the throughput of DynamoDB for a fixed price point. Anna is also able to react to workload changes by adding nodes and replicating data appropriately.

RISELab has the goal of enabling computers to make intelligent real-time decisions. You might not remember last year RISELab arose from the ashes of AMPLab. AMPLab produced such open source successes as Apache Spark and Apache Mesos, so don't make the mistake of thinking anything they do is of mere academic interest. They are about creating the future.  

Related Articles

• Anna Paper: Eliminating Boundaries in Cloud Storage with Anna
• Anna Code: fluent-project/fluent

Stuff The Internet Says On Scalability For September 7th, 2018

Hey, it's HighScalability time:

Get antsy waiting 60 seconds for a shot? Imagine taking over 300,000 photos over 14 years, waiting for Mount Colima to erupt. Sergio Tapiro studied, waited, and snapped.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 3.5 Pflop/s: fully synchronous tensorflow data-parallel training; 3.3 million: new image/caption training set; 32,408,715: queries sent to Pwned Passwords; 53%: Memory ICs Total 2018 Semi Capex; 11: story Facebook datacenter prison in Singapore; $740,357: ave cost of network downtime; 


• Quotable Quotes:
  • @BenedictEvans: Recorded music: $18 billion. Cars: $1 trillion. Retail: $20 trillion.
  • @JoeEmison: Lies that developers tell (themselves): (1) This is the best stack/IaaS for us to use [reality: I know it and want to start now] (2) DevOps doesn’t matter until scaling [you’ll spend 30% of your time dealing with ops then] (3) We’ll just rebuild it if we get traction [hahahaha]
  • @sapessi: Lambda simplifies concurrency at the frontend, enforcing one event per function at a time. This makes it easy to reason about complex distributed system. Once inside the function, there's nothing wrong with multi-threading to do the work as efficiently as possible
  • @JonErlichman~ Comparing valuations: Amazon: $1 trillion; Combined $960 billion: Best Buy Macy’s Target Costco Nike  Sears  Home Depot Starbucks McDonald’s Barnes & Noble J.C. Penney Dollar Tree Office Depot Nordstrom Kroger Kohls
  • Kevin Kelly: The biggest invention in Silicon Valley was not the transistor but the start-up model, the culture of the entrepreneurial start-up.
  • Dare Obasanjo: Amazon made $2.2B from search ads last quarter. This is twice as much as Snapchat ($262M) and Twitter ($711M) combined. However still far from Google ($28B) and Facebook ($13.2B). Expect next step is for Amazon ads to start show scale  
  • Matthew Dillon: This is *very* impressive efficiency.  Whod a thought that one would be
    able to run an 8-core/16-thread CPU at full load at only 85W and still reap
    most of the benefit in a memory-heavy workload! This is *very* impressive efficiency.  Whod a thought that one would beable to run an 8-core/16-thread CPU at full load at only 85W and still reapmost of the benefit in a memory-heavy workload!
  • @laurencetratt: When we set out to look at how long VMs take to warm up, we didn’t expect to discover that they often don’t warm up. But, alas, the evidence that they frequently don’t warm up is hard to argue with. Of course, in some cases the difference to performance is small enough that one can live with it, but it’s often bad enough to be a problem. We fairly frequently see performance get 5% or more worse over time in a single process execution. 5% might not sound like much, but it’s a huge figure when you consider that many VM optimisations aim to speed things up by 1% at most. It means that many optimisations that VM developers have slaved away on may have been incorrectly judged to speed things up or slow things down, because the optimisation is well within the variance that VMs exhibit. 
  • Eli Bendersky: Just for fun, I rewrote the same benchmark in Go; two goroutines ping-ponging short message between themselves over a channel. The throughput this achieves is dramatically higher - around 2.8 million iterations per second, which leads to an estimate of ~170 ns switching between goroutines [3]. Since switching between goroutines doesn't require an actual kernel context switch (or even a system call), this isn't too surprising. For comparison, Google's fibers use a new Linux system call that can switch between two tasks in about the same time, including the kernel time.
  • @ben11kehoe: So instead of a processor [for @iRobot] that is maxed out by the features it supports at launch (which is the route to the lowest build cost), we've got headroom to keep pace even with software features we're planning for the generation of robots after this.
  • It was a quiet week this week. Just a few more quotes...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Sponsored Post: NationBuilder, Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr, MemSQL

Who's Hiring? 

• NationBuilder — if you’re a systems engineer, SRE or DevOps focused developer and have been looking for a place where you can help other people while still working in tech? We can give that opportunity. Please apply here. 


• Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here. 


• Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.


• Need excellent people? Advertise your job here! 

Fun and Informative Events

• Advertise your event here!

Cool Products and Services

• InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

• For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

• Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

• Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.


• MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.


• Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

---

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.

---

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today. 

---

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Stuff The Internet Says On Scalability For August 31st, 2018

Hey, it's HighScalability time:

This mind blowing creation is from John Williamson. It's the first million integers, represented as binary vectors indicating their prime factors, laid out with UMAP. No, I really have no idea what that means either, but it did make me consider that our universe could be created by an algorithm. What are the wiggly cycles on the periphery? Groups of numbers that share a minimum amount of prime factors, further out groups are numbers that have increasing amounts of shared prime factors. So the primes are at the core, ungrouped as they have no prime factors to use to join groups. Primorials should be furthest out.

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

• 800 million: internet users in China; 19,000: hours of audio from historic Apollo 11 mission; 2: mph Frodo walking pace to Mordor; 67%: daily trade volume faked; 60 TFLOPS: Data's theoretical peak, which is 8 NVIDIA Voltas, which you can rent on ec2 for $25 an hour; 7nm: an expense too far for GlobalFoundries; 98%: atoms replaced every year in human body; 150 billion: hours of Fitbit heart data; 4: times per second human brain oscillates in and out of focus; 27.9%: DNS requests over UDP from China to Google Public DNS are intercepted; 


• Quotable Quotes:
  • Chris Hadfield: the best antidote to fear is competence.
  • @kellabyte: I’m making fun of GraphQL but I like it so far. I’m just having fun with it because it’s literally re-learning a new way to do the same thing a group of other people solved some of the same problems almost 20 years ago. But like I said, not a hater. I like GraphQL so far.
  • Corey Quinn: Kubernetes is Named After the Greek God of Spending Money on Cloud Services
  • @QuinnyPig: #SREcon @aaronblohowiak "Once upon a time when the stars were young we had an ELB outage" ...because ELBs are incredibly durable right up until "Oh shit! Load!" at which point TCP terminates on the floor.
  • @GossiTheDog: This is wild - the White House are rolling back cybersecurity baseline standards for government (saying they will issue new ones in the future), and ditching deployment of DNSSEC and IPv6 (as mandated under Obama admin) saying they are already in place (they aren’t).
  • @nicolefv: Outsourcing is a bad idea. (shocker) Low performers are almost 4x as likely to use functional outsourcing. Read about it and some of the consequences (search for “Misguided performers”) in the 2018 Accelerate State of DevOps Report 
  • Bernard Golden: I see Snowball Edge with EC2 as an AWS initiative designed to reduce the friction of migrating data into AWS with the ultimate goal of enabling applications to migrate to AWS, not a way to run AWS applications on-prem. In other words, this new Snowball variant isn’t about processing data on-prem, it’s about making Snowball a better on-ramp to AWS.
  • @allspaw: In "managing workload there are only four coping strategies: (1) shed load, (2) do all components but do each less thoroughly, thereby, consuming fewer resources, (3) shift work in time to lower workload periods, (4) recruit more resources." (Woods and Hollnagel, 2006)
  • Stacy Horn: In those days journalists wrote that I started Echo to provide a safe place for women on the Net. Bite me. I wanted to get more women on Echo to make it better.
  • @michael_beh: there are so many price comparisons between #serverless and non-serverless deployments missing costs for multi-AZ, multi-region, operational OS mgmt, load balancer, pre-prod envs, etc. All of them are part of GBsec price in serverless. It's too often an apples/oranges comparison.
  • Andy Hertzfeld: Steve got mad. He goes, “The Apple II was going to be dead. Your OS is going to be obsolete before it’s finished!” And then he said something like, “The Macintosh is the future of Apple and you are going to start now.” I just wanted one day, really. It was Thursday afternoon. I said “Monday” because Monday seems to be a good time to make a new start. But he goes, “No! You’re going to start on it now!” And he went and he pulled the plug on my Apple II, which had my code not saved. I was right in the middle of working on it. He just yanks the plug out, and then without pausing he picks up the Apple II and starts walking away.
  • More more more...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...