advertise
Friday
Dec072018

Stuff The Internet Says On Scalability For December 7th, 2018

Wake up! It's HighScalability time:

Do you like this sort of Stuff? Please support me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (31 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever. And if you know someone with hearing problems they might find Live CC very useful.

 

  • $181.5M: top 10 YouTube star earnings; 153 million/1.2 billion/27 billion: Reddit posts/comments/votes;  12%: cut for Epic's new Steam competitor; 2,368: Baidu AI patents; 360: Walmart cleaning robots; 2:1: AMD CPUs outselling Intel; 11157: images in Disguised Faces in the Wild dataset; 7,518: FCC approved SpaceX LEO satellites; $40 million: saved Down Under by Tesla’s giant battery; intense beam of ultra-high energy heavy ions: testing AI chips for space;  $100 million: bot heist; 16TB: HAMR hard drives; 12.8Tbps: new Tofino ASIC; 85%: cloud-hosted TensorFlow workloads run on AWS; 39%: tech workers depressed; 16%: drop in GPU shipments; 4 x 10⁸⁴: photons ever emitted in the universe; 0.00%: blockchain success rate; 

  • Quotable Quotes:
    • @math_rachel: About half of > 200 people seeking visas for @black_in_ai #NeurIPS2018 were denied. This is a huge issue harming our field.
    • @heipei: It's Friday, I've been in a jumpsuit doing manual labor all day (crazy, I know) and weighing my options between passing out on the couch over some Youtube videos, reading the Friday @highscal blog post or writing code. Currently favouring the path of least resistance πŸ˜›
    • Paul Tassi: Again, it’s hard to overstate just what a big deal this is, and it’s a massive undertaking for Epic, but one they believe they’re able to handle given how massive Fortnite has scaled to be. Epic famously dodged the Apple store entirely when releasing Fortnite on iOS, and it’s clear those were the first warning shots for something like this.
    • IBM: Our device uses phase-change memory (PCM) for in-memory computing. PCM records synaptic weights in its physical state along a gradient between amorphous and crystalline. The conductance of the material changes along with its physical state and can be modified using electrical pulses. This is how PCM is able to perform calculations. Because the state can be anywhere along the continuum between 0 and 1, it is considered an analog value, as opposed to a digital value, which is either a 0 or a 1, nothing in between.
    • @ivanveram: World R&D leading companies 2017 in US$. No.1: Amazon; No.2: Alphabet (Google); No.3: Samsung. :) That´s the way it is!!!. :)
    • 40acres: Full disclosure, I work for Intel. We have a fabrication plant in Chengdu, it's public knowledge that this fab is helping to manufacture products built on the latest process technology. As I've come to learn more about China's tactics when dealing with foreign companies it's become of great concern to me what this plant means for our future. I don't think it would be far-fetched to assume that some very protected and valuable IP has leeched through our doors and into China's hands. In all honestly I really can't fathom how the American government let this deal occur.
    • So many more quotes...
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Dec052018

How ShiftLeft Uses PostgreSQL Extension TimescaleDB

 

This article is written by Preetam Jinka, Senior Infrastructure Engineer at ShiftLeft. Originially published as Time Series at ShiftLeft

 

Time series are a major component of the ShiftLeft runtime experience. This is true for many other products and organizations too, but each case involves different characteristics and requirements. This post describes the requirements that we have to work with, how we use TimescaleDB to store and retrieve time series data, and the tooling we’ve developed to manage our infrastructure.

We have two types of time series data: metrics and vulnerability events. Metrics represent application events, and a subset of those that involve security issues are vulnerability events. In both cases, these time series have some sort of ID, a timestamp, and a count. Vulnerability events can also have an event sample that contains detailed information about the request that exercised a security vulnerability. In addition to those attributes, time series are also keyed by an internal ID we call an SP ID, which essentially represents a customer project at a certain version...

Click to read more ...

Tuesday
Dec042018

Sponsored Post: InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services


  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net
  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorialStream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Nov302018

Stuff The Internet Says On Scalability For November 30th, 2018

Wake up! It's HighScalability time:

 

We all know the oliphant in the room this week (reinvent)

 

Do you like this sort of Stuff? Please support me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

 

  • 8: successful Mars landings; $250,000: proposed price for Facebook Graph API; 33: countries where mobile internet is faster than WiFi; 1000s: Facebook cache poisoning; 8.2 million: US Nintendo Switch sales; 40+%: Rust users feel productive; 15 terabytes: monthly measurements of third-party web transparency tracking data; $133.20: total music sales by Imogen Heap on blockchain; 8.3 million: concurrent Fortnite players; 6.2 Billion: fuel costs saved by smart car drivers; 80: salad bags assembled per minute by smart machines, 2x the output of a worker; 1/10th: power used by ebike compared to Nissan Leaf; 100,000: new micro industries; 40MW: solar plant floats on water; 20%: car crashes reduced using Waze-fed AI platform; 14%: decline in Sillycon Valley median wage over 20 years; 36.7%: smartphone e-commerce sales; many: Google Earth datasets; 6 zeta tons: earth's mass; 

  • Quotable Quotes:
    • @william_r_kerr: "The number of new international students enrolling at American institutions fell by 6.6% during the 2017-18 academic year, on top of a 3.3% decline the year before."
    • @codinghorror: “Apple’s silicon is now so well ahead that we’re not really expecting Android vendors to catch up any time soon.” 
    • @emileifrem: Adobe used 125 MongoDB servers to run their activity feed. It was replaced by 48 Cassandra servers. Now it runs on 3 (THREE!) servers of Neo4j. With more data (yet smaller disk footprint), higher load and more functionality. The scalability of a native graph database.
    • @swardley: First up at #AWSreInvent is McKinsey revisiting their 2009 report on Cloud Computing with their talk "We were so wrong, you'd have been a muppet to follow our advice" ... damn, daydreaming again. X : What did McKinsey do? Me : Clearing the Air on Cloud, 2009. Basically an argument that cloud was for startups but proper enterprises would be better off investing in their data centres and adopting a cloud model would be a money-losing mistake 
    • @AstroKatie: Hold your hand up 12 inches from your face: you’re seeing your hand as it was a nanosecond ago. Everything you look at is, to one degree or another, in the past. The farther away in space, the more ancient in time.
    • Ms. Hou: I used to think the machines are geniuses. Now I know we’re the reason for their genius.
    • @lightbend: How to scale? Here you go: With over 125 million players, and supporting over 8 million *concurrent gaming sessions*, we are really happy to learn that #Fortnite is running #Akka under the hood!
    • @mattklein123: The Firecracker announcement from @awscloud is super, super cool. I love seeing real innovation in the OS/VMM space, and a willingness to toss away legacy in order to vastly simplify the problem space. Kudos on making this OSS as well.
    • @hichaelmart: So my hot take is that Firecracker is gVisor done right: at the VM level, and more importantly, NOT using a garbage collected language πŸ”₯
    • Need more? There's lots more...
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Nov232018

Stuff The Internet Says On Scalability For November 23rd, 2018

Wake up! It's HighScalability time:

 

Curious how SpaceX's satellite constellation works? Here's some fancy FCC reverse engineering magic. (Delay is Not an Option: Low Latency Routing in Space, Murat)

 

Do you like this sort of Stuff? Please support me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

 

  • 5: ios 12 jailbreaks;  ~43 million: dataset of atomic wikipedia edits; 1.88 exaops: peak throughput Summit supercomputer; 2 million: ESPN subscribers lost to cord cutters; 302: neurons show simple behavior of learning and memory; 224: seconds to train a ResNet-50 on ImageNet to an accuracy of approximately 75%; 120 milliseconds: space shuttle tolerances; $1 billion: Airbnb revenue;  900+ mph: tip of a whip;

  • Quotable Quotes:
    • @JeffDean: This thread is important.  Bringing the best and brightest students from around the world to further their studies in the U.S. has been a key to success in many technological endeavors and to lead in many scientific domains.
    • @davidgerard: > crypto has a provably lower carbon footprint than fiat actually, this is trivially false bitcoin: 0.1% of electricity, for 7tps THE WHOLE REST OF CIVILISATION: 99.9% of electricity, for more than 7000tps
    • Paris Martineau: Lotti recalls the investor saying that if she wanted Lashify to succeed, quality didn’t matter, nor did customer satisfaction—only influencers. And they didn’t come cheap. She was told to expect to shell out $50,000 to $70,000 per influencer just to make her company’s name known, an insane amount for a new startup. There was no way around it; that’s just how things worked.
    • @KevinBankston: Ford's CEO just said on NPR that the future of profitability for the company is all the data from its 100 million vehicles (and the people in them) they'll be able to monetize. Capitalism & surveillance capitalism are becoming increasingly indistinguishable (and frightening).
    • Waqas Dhillon:  The goal of in-database machine learning is to bring popular machine learning algorithms and advanced analytical functions directly to the data, where it most commonly resides – either in a data warehouse or a data lake. While machine learning is a common mechanism used to develop insights across a variety use cases, the growing volume of data has increased the complexity of building predictive models, since few tools are capable of processing these massive datasets. As a result, most organizations are down-sampling, which can impact the accuracy of machine models and created unnecessary steps to the predictive analytics process.
    • Can you eat more after Thanksgiving? Lots of leftovers. Dig in...
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Nov162018

Stuff The Internet Says On Scalability For November 16th, 2018

Wake up! It's HighScalability time:

 

Beautiful. Unwelcome Gaze is a triptych visualizing the publicly reachable web server infrastructure of Google, Facebook, Amazon and the routing graph(s) leading to them. 

 

Do you like this sort of Stuff? Please support me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

 

  • $1 billion: Alibaba sales in 85 seconds; 1 million: Human brain processor supercomputer; 1 billion: Uber quarterly loss; 640,000: AMD cores in Hawk supercomputer; 40x: drone lifts its own weight; 26: pictures in history of computing; 10^11: WeChat requests per day;  

  • Quotable Quotes:
    • Larry Ellison: It’s kind of embarrassing when Amazon uses Oracle but they want you to use Aurora and Redshift. They’ve had 10 years to get off Oracle, and they’re still on Oracle.
    • @Werner: Amazon's Oracle data warehouse was one of the largest (if not THE largest) in the world. RIP. We have moved on to newer, faster, more reliable, more agile, more versatile technology at more lower cost and higher scale. #AWS Redshift FTW!
    • Sheera Frenkel: “We’re not going to traffic in your personal life,” Tim Cook, Apple’s chief executive, said in an MSNBC interview. “Privacy to us is a human right. It’s a civil liberty.” (Mr. Cook’s criticisms infuriated Mr. Zuckerberg, who later ordered his management team to use only Android phones — arguing that the operating system had far more users than Apple’s.)
    • @copyconstruct: This is a fantastic companion to the first Aurora paper from Amazon. 
      The main takeaway is the distributed consensus kills performance, that local state can actually be a good thing, use an immutable log as the source of truth (basically what the Kafka folks have been preaching)
    • @yishengdd: However from my own experience, AWS Amplify is 10x better than Google Firebase
    • Daniel Lemir: Another way to look at the problem is to measure the “speedup” due to the memory-level parallelism: we divide the time it takes to traverse the array using 1 lane by the time it takes to do so using X lane. We see that the Intel Skylake processor is limited to about a 10x or 11x speedup whereas the Apple processors go much higher.
    • Shaun Raviv: Friston’s free energy principle says that all life, at every scale of organization—from single cells to the human brain, with its billions of neurons—is driven by the same universal imperative, which can be reduced to a mathematical function. To be alive, he says, is to act in ways that reduce the gulf between your expectations and your sensory inputs. Or, in Fristonian terms, it is to minimize free energy.
    • Max J. Krause: with the exception of aluminium, cryptomining consumed more energy than mineral mining to produce an equivalent market value. While the market prices of the coins are quite volatile, the network hashrates for three of the four cryptocurrencies have trended consistently upward, suggesting that energy requirements will continue to increase. During this period, we estimate mining for all 4 cryptocurrencies was responsible for 3–15 million tonnes of CO2 emissions.
    • Doubleslash: Back in 2014 OpenStack may have been a thing but nowadays, 4 years later, people learned a lesson. There is no value in being a full-stack operator of infrastructure for systems. The idea of running a complex, distributed architecture like a private cloud at the same or higher stability and comparable or even lower cost than public clouds eventually proved futile. That's why in 2018, consensus is, infrastructure for systems has been replaced by infrastructure for applications (k8s). The infrastructure is typically consumed on public clouds.
    • How many quotes are there do you think?
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Nov142018

New: The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition

 

Five years ago when Google published The Datacenter as a Computer: Designing Warehouse-Scale Machines it was a manifesto declaring the world of computing had changed forever. Hop aboard or be left behind at the station. Since then the world has chosen to ride along with Google.

The world is still changing, so Google published a new edition: The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition.

What's different?

The third edition reflects four years of advancements since the previous edition and nearly doubles the number of pictures and figures. New topics range from additional workloads like video streaming, machine learning, and public cloud to specialized silicon accelerators, storage and network building blocks, and a revised discussion of data center power and cooling, and uptime. Further discussions of emerging trends and opportunities ensure that this revised edition will remain an essential resource for educators and professionals working on the next generation of WSCs.

The abstract:

This book describes warehouse-scale computers (WSCs), the computing platforms that power cloud computing and all the great web services we use every day. It discusses how these new systems treat the datacenter itself as one massive computer designed at warehouse scale, with hardware and software working in concert to deliver good levels of internet service performance. The book details the architecture of WSCs and covers the main factors influencing their design, operation, and cost structure, and the characteristics of their software base. Each chapter contains multiple real-world examples, including detailed case studies and previously unpublished details of the infrastructure used to power Google’s online services. Targeted at the architects and programmers of today’s WSCs, this book provides a great foundation for those
looking to innovate in this fascinating and important area, but the material will also be broadly interesting to those who just want to under- stand the infrastructure powering the internet.

 

At 189 pages it's not exactly light reading, but it's packed with information you can't find anywhere else.

 

If you like this kind of stuff, you might also like Google's New Book: The Site Reliability Workbook.

Tuesday
Nov132018

Sponsored Post: Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr

Who's Hiring? 


  • Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services


  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net
  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorialStream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.
  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Nov092018

Stuff The Internet Says On Scalability For November 9th, 2018

Wake up! It's HighScalability time:

 

@b0rk

 

Do you like this sort of Stuff? Please support me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

 

  • $3 billion: Tesla's yearly spend on gigafactories; 18,000: GDPR data breach notifications since May; 30: happy birthday Morris Worm!; 1/3: not opting for new Java; 10x: MySQL TPS improvement in 9 years; 1,300: childhood photos posted by parents by the time they're 13; 100TB: hard drives by 2025; 1000x: faster graphics performance than the original iPad released eight years ago; 13.28B: transistors in  world's first 7nm GPU; 15 million: daily Uber trips; $725 million: opening weekend for Redemption 2; 

  • Quotable Quotes:
    • @kylecyuan: 10/ From computational bio, cloud bio, and digital therapeutics, AI has put Bio on the Moore's Law curve, not Eroom's Law. From wet lab problems to dry lab ones. @vijaypande 's "When Software Eats Bio." This industrializes and 10x existing processes and creates new ones.
    • @davidgerard: It really won't, because it can't possibly scale. This is the key  problem with every musical blockchain initiative I've ever seen.  Proposals whose "blockchain" would need to add 1GB/hour, that sort of thing.
    • @matei_zaharia: Since we opened #DAWNBench deep learning benchmark rolling submissions, there have been some cool entires. You can now train CIFAR10 for just $0.06 and ImageNet for $12.60 (4x less than in April!): https://dawn.cs.stanford.edu/benchmark/ . MLPerf's deadline is also soon (http://mlperf.org ).
    • Paul Alcorn~ The 7nm Rome CPUs come with 64 physical Zen 2 cores, which equates to 128 threads per processor, double that of the first-gen Naples chips. In a two socket server, that equates to 128 physical cores and 256 threads in a single box. Rome is also the first PCIe 4.0 CPU, which offers double the bandwidth per channel. AMD has improved the Infinity Fabric. AMD is now using the second-gen Infinity Fabric to connect a multi-chip design with a 14nm I/O die serving as the linchpin of the design. That central chip ties together the 7nm CPU chiplets, creating a massively scalable architecture. Amazon Web Services, one of the world's largest cloud service providers, announced that, beginning today, it is offering new EPYC-powered cloud instances. The R5a, M5a and T3a instances purportedly will offer a 10% price-to-performance advantage over AWS's other cloud instances.
    • @tony_goodfellow: He's World of Warcraft's first and only conscientious objector—a neutral pandaren shaman who spends his time picking flowers and mining rocks instead of fighting and killing. ...after roughly 8000 .... Doubleagent has reached the maximum level of 110."
    • ...
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Nov072018

Fighting the Ecosystem Wars in the Proactive Cloud

This article is a chapter from my book Explain the Cloud Like I'm 10. It has 30 reviews on Amazon! If you like this chapter then you'll love the book.

 

 

The cloud is always busy proactively working for you in the background. That’s how cloud services compete with each other to keep you in their ecosystem.

 

Most of the cloud services we’ve talked about so far have been request driven. You initiate a request and the cloud does something for you. You read a book. You search for the nearest coffee shop. You navigate to a destination. You send a message. You play a movie.

Handling direct requests is not all a cloud is good for. In fact, the biggest potential of the cloud is how it can proactively perform jobs for you in the background, without you asking or even knowing that it can be done.

Let’s set this up:

Click to read more ...