advertise
Friday
Jul202018

Stuff The Internet Says On Scalability For July 20th, 2018

Hey, it's HighScalability time:

 

 

World History Timeline from 3000BC to 2000AD. Yet we still program with text—in files.

 

 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

 

  • $150 billion: Bezos Prime; 49%: Amazon's share of US e-commerce; 1,000 terabytes: image size to represent one cubic millimeter of brain tissue;  7x: 4 year reduction in cost of computing power; 25x: faster code using SIMD; 4TB: RAM in GCE “ultramem” instance type; 4 months: half-life of an ICO; 80%: cost savings moving from AWS to DO; 130,000: square feet in biggest vertical farm; 14x: price increase for Google Maps;  

  • Quotable Quotes:
    • Chappell: In [CPU] architectures, we believe that aggressive specialization is a part of the answer to what happens next. That’s mapping applications to the specific architectural choices. And you already see that in machine learning, where there’s a really hot field in terms of deep neural nets and other implementations. The third wing of the architecture piece is the “domain specific system-on-chip.”
    • Troy Hunt: I love the way these two services work in unison: Azure Functions to ensure you can scale immediately without being bound by logical infrastructure, deployment slots that make it easy to test and rollover new releases with zero downtime, then of course Cloudflare Workers to give you heaps of control over traffic flow for testing and rollover purposes and all protected using their rate limit service. 
    • Cliff Click: The JVM is very good at eliminating the cost of code abstraction, but not the cost of data abstraction. That means multiple data indirections mean multiple cache misses. They are very expensive. This is where your performance goes. 
    • jaybo_nomad: The Allen Institute for Brain Science is in the process of imaging 1 cubic mm of mouse visual cortex using TEM at a resolution of 4nm per pixel. The goal is to complete this in about 4 months running in parallel on 5 scopes. 
    • @lowrykoz: Stolen from a co-worker "Every company has a test environment. Some are lucky enough to also have a production environment."  Thanks Charlie for my laugh today!
    • DonHopkins: NeWS differs from the current technology stack in that it was all coherently designed at once by James Gosling and David Rosenthal, by taking several steps back and thinking deeply about all the different problems it was trying to solve together. So it's focused and expressed in one single language, instead of the incoherent fragmented Tower of Babel of many other ad-hoc languages that we're stuck with today.
    • tef: You can use a message broker to glue systems together, but never use one to cut systems apart.
    • @CompSciFact: 'Fancy algorithms are slow when n is small, and n is usually small.' -- Rob Pike
    • @Grady_Booch: I interviewed John Backus shortly before his death. He told me his work in functional programming languages failed, and would likely always fail, because it was easy to do hard things but incredibly difficult to do simple things.
    • Mark LaPedus: MRAM, a next-generation memory type, is being touted as a replacement for embedded flash and cache applications. MRAM works in consumer applications, but it’s still unclear if it will ever meet the temperature requirements for automotive. Some say MRAM will never work in automotive.
    • crabbone: This is the prism through which Java programmers view the world. They never question this belief. It also works well to justify an acquisition of more servers to investors. Investing tons of efforts into IT, building complicated deployment and clustering software etc. This is what happens in the company I work for, and couple of mid-size to big companies I worked for before. The truth about it is that Java only gets you a good bang for your buck just a wee bit before it hits OOM. It doesn't scale down to the point that is ridiculous and painful. A typical example of modern "microservices-inspired" Java application would function along these lines:
    • Netflix: We observed during experimentation that RAM random read latencies were rarely higher than 1 microsecond whereas typical SSD random read speeds are between 100–500 microseconds. For EVCache our typical SLA (Service Level Agreement) is around 1 millisecond with a default timeout of 20 milliseconds while serving around 100K RPS. During our testing using the storage optimized EC2 instances (I3.2xlarge) we noticed that we were able to perform over 200K IOPS of 1K byte items thus meeting our throughput goals with latency rarely exceeding 1 millisecond. This meant that by using SSD (NVMe) we were able to meet our SLA and throughput requirements at a significantly lower cost.
    • paulddraper: The biggest lesson HN teaches for designing large scale systems is "use a large scale system someone else has already designed".
    • There are a few more quotes. Not so many this week.
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Jul132018

Stuff The Internet Says On Scalability For July 13th, 2018

Hey, it's HighScalability time:

 

Steve Blank tells the Secret History of Silicon Valley. What a long, strange trip it is.

 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

 

  • $27 billion: CapEx invested by leading cloud vendors in first quarter of 2018; $40 billion: App store revenue in 10 years; $57.5 billion: venture investment first half of 2018; 1 billion: Utah voting system per day hack attempts; 67%: did not deploy a serverless app last year; $1.8 billion: made by Pokeman GO; $13 billion: Netflix's new content budget; 

  • Quotable Quotes:
    • @davidbrunelle: The best developers and engineering leaders I've personally worked with do *not* have a notable presence on GitHub or public bodies of speaking or writing work. I worry that a lot of folks confuse celebrity and visibility with talent and ability.
    • Bernard Golden: The tech industry has never seen this level of investment [in datacenters]. The investment we’re seeing in cloud capacity really has no precedent, save perhaps Henry Ford’s manic factory building for his Model T, the US government’s armaments efforts in WWII, and Foxconn’s manufacturing support for smartphones. As Ford’s efforts presaged the boom growth of the industrial economy, so too do (cloud) investments augur the explosion of the digital economy.
    • @kellabyte: The issue with microservices is it’s taught people to stop thinking about cohesiveness.  Cohesiveness is really important. When you fight against it you experience major pain and Service Autonomy works much better as a cohesive unit not just making “micro” things everywhere.
    • Leslie Lamport: Today, programming is generally equated with coding. It's hard to convince students who want to write code that they should learn to think mathematically, above the code level, about what they’re doing. Perhaps the following observation will give them pause. It's quite likely that during their lifetime, machine learning will completely change the nature of programming. The programming languages they are now using will seem as quaint as Cobol, and the coding skills they are learning will be of little use. But mathematics will remain the queen of science, and the ability to think mathematically will always be useful.
    • Nolan Bushnell, founder of Atari: What everybody wanted was a party and some beer and some pizza, and they ended up going home with each other.
    • @Grady_Booch: Ada Lovelace devised the first program. Grace Hopper wrote the first complier. Margaret Hamilton started the field of software engineering. Women have always been and will always be essential to the advance of computing.
    • Hossein Fateh: The largest deal of 2001 was 3.5 megawatts. That same company leased 35 megawatts from us in 2016. The decimal place moved by a column. The next decimal place will move in 2022. Deals will be 350 megawatts.
    • Matt Alderman: Based on our estimates, AWS Fargate deployments should save you 5 percent to 10 percent in your compute bill as compared to highly optimized AWS ECS or EKS deployments.
    • Memory Guy: Conventional wisdom holds that SSDs will someday displace all HDDs, but in reality SSDs are proving to be more of a challenge to the DRAM market than to the HDD market...So, if you have a fixed budget, SSDs can help you get the most out of your system and are a better alternative than additional DRAM.
    • @JoeEmison: Given this context, it is insane to say that something like AWS AppSync is bad lock-in. If I want AppSync functionality, I have two choices: use AppSync, or write AppSync myself. Given those, how is using AppSync until it makes sense for me to write my own a bad move?
    • More, more, more, more, more...
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Jul102018

Sponsored Post: Datadog, InMemory.Net, Triplebyte, Etleap, Scalyr, MemSQL

Who's Hiring? 

  • Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services

  • Datadog is a cloud-scale monitoring platform that combines infrastructure metrics, distributed traces, and logs all in one place. With out-of-the-box dashboards and seamless integrations with over 200 technologies, Datadog provides end-to-end visibility into the health and performance of modern applications at scale. Build your own rich dashboards, set alerts to identify anomalies, and collaborate with your team to troubleshoot and fix issues fast. Start a free trial and try it yourself.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net
  • For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Jul062018

Stuff The Internet Says On Scalability For July 6th, 2018

Hey, it's HighScalability time:

 

Could RAINB (Redundant Array of Independent Neanderthal ‘minibrains’replace TPUs as the future AI core? 

 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

 

  • $100m: Fortnite iOS revenue in 90 days; $2.5 Billion: SUSE Linux acquisition; 500,000: different orgs on Slack; 6.1%: chance of breaking change in each library; 10: years of the Apple app store; 2021: Japan goes exascale;

  • Quoteable Quotes:
    • jedberg: > Yes, that's how Amazon creates lock-in.
      That is the cynical way to look at it. It also creates value because it lets you do more with what you already have.
    • Paul Ingles: We didn’t change our organisation because we wanted to use Kubernetes, we used Kubernetes because we wanted to change our organisation.
    • ThousandEyes: The Internet is made up of thousands of autonomous networks that are interdependent on one another to deliver traffic from point to point across the globe. The lesson to be learned from this outage is that the Internet is made up of many hidden dependencies, any of which can impact your ability to connect to sites and services—even if you don’t have a direct relationship with an affected ISP. 
    • tyingq: This is why AWS and Azure continue to gain market share in cloud, while Google remains relativity stagnant, despite (in many cases) superior technology. Their sales staff is arrogant and has no idea how to sell into F500 type companies. Source: 10+ meetings, with different clients, I attended where the Google sales pitch was basically "we are smarter than you, and you will succumb". The Borg approach. Someone needs to revamp the G sales and support approach if they want to grow in the cloud space.
    • Google: We introduce adversarial attacks that instead reprogram the target model to perform a task chosen by the attacker—without the attacker needing to specify or compute the desired output for each test-time input. This attack is accomplished by optimizing for a single adversarial perturbation, of unrestricted magnitude, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary when processing these inputs—even if the model was not trained to do this task. These perturbations can be thus considered a program for the new task.
    • Where Wizards Stay Up Late: There is no small irony in the fact that the first program used over the network was one that made the distant computer masquerade as a terminal. All that work to get two computers talking to each other and they ended up in the very same master-slave situation the network was supposed to eliminate. Then again, technological advances often begin with attempts to do something familiar.
    • Yep, there's more.
Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Monday
Jul022018

In Defense of Humanity—How Complex Systems Failed in Westworld  **spoilers**

 

The Westworld season finale made an interesting claim: humans are so simple and predictable they can be encoded by a 10,247-line algorithm. Small enough to fit in the pages of a thin virtual book.

Perhaps my brain was already driven into a meta-fugal state by a torturous, Escher-like, time shifting plot line, but I did observe myself thinking—that could be trueOr is that a thought Ford programmed into my control unit?

To the armies of algorithms perpetually watching over us, the world is a Skinner box. Make the best box, make the most money. And Facebook, Netflix, Amazon, Google, etc. make a lot of money specifically on our predictability.

Even our offline behaviour is predictable. Look at patterns of human mobility. We stay in a groove. We follow regular routines. Our actions are not as spontaneous and unpredictable as we'd surmise.

Predictive policing is a thing. Our self-control is limited. We aren't good with choice. We're predictably irrational. We seldom learn from mistakes. We seldom change.

Not looking good for team human.

It's not hard to see how those annoyingly smug androids—with their perfect bodies and lives lived in a terrarium—could come to take such a dim view of humanity. They see us at our worst. Who wants entry into heaven decided on how we behave playing Grand Theft Auto?

They think they understand us because they've observed us playing a game. They don't. The reason is in How Complex Systems Fail. The androids saw humans through a Russian doll set of nested games. The most obvious game was Westworld. Ford had his game. Numerous corporate games played them selves out with immortality as the payoff.

Each game constrains and directs behavior. In real-life we humans have greater degrees of freedom. When Delores wanted to solve a problem she—ironically—chose genocide as the solution. That's what she was used to. Even horribly flawed humans wouldn't see genocide as a valid move.

Human lives and human society are emergent. You can't attach a debugger to our DNA and find our repertoire of behaviors explicitly described and enumerated. What we are arises out of our interactions. Just like how evolution emerges from biology, which emerges from chemistry, which emerges from physics, which emerges from quantum mechanics, which emerges from who knows what. Shape and control those interactions and you change the world.

Predicting what emerges at each layer is impossible, so it's futile to generate a 10,247-line algorithm by testing fidelity to a remembered baseline. You're creating a game character. That's what's missing in Westworld. And that's why humans can never be counted out—they can always surprise you.

How Complex Systems Fail:

Click to read more ...

Friday
Jun292018

Stuff The Internet Says On Scalability For June 29th, 2018

Hey, it's HighScalability time:

 

Rockets. They're big. You won't believe how really really big they are. (Corridor Crew)

 

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

 

  • 200TB: GitLab Git data; $100 Billion: Instagram; ~250k: Walmart peak events per second; 10x: data from upgraded Large Hadron Collider; .3mm: smallest computer; 9.9 million: spam or automated accounts identified by Twitter per week; 1 million: facial image training set; 1/3: industrial robots installed in China; 24%: never backup; 7 billion: BuzzFeed monthly page views; 

  • Quotable Quotes:
    • @jason: would love to do a https://www.founder.university/  for Immigrants -- but we might need to do it in Canada or Mexico, so that, umm.... potential immigrants can actually attend! #america 
    • @kellabyte: LOL at racking up an AWS bill of $140,000 in 4 hours of compute time.
    • @kellabyte: Recently I got to work on a project that really stressed Amazon AWS scalability. You want to talk scale? We spun up a cluster of 100,000 AWS instances multiple times. 2+ million CPU cores. I got to work on something so big a cloud provider learned about their scale bottlenecks
    • jedberg: Despite being a strong advocate for AWS, this is where I will say Google completely outshines Amazon. Google's approach to pricing is, "do it as efficiently and quickly as possible, and we'll make sure that's the cheapest option". AWS's approach is more, "help us do capacity planning and we'll let you get a price break for it.". Google applies bulk discounts after the fact, AWS makes you ask for them ahead of time.
    • Mark Lapedus: Costs of developing a complex chip could run as high as $1.5B, while power/performance benefits are likely to decrease.
    • Quirky: Thus there is a tradeoff: separateness enables inventors to create heterodox ideas, but strong cohesive networks are likely to be better for getting them implemented.
    • karmakaze: The key difference is that NoSQL was being developed and used to solve specific issues with using traditional databases. They were implementations of solutions to problems. Blockchain is a solution in search of problems.
    • slivym: FPGAs in industry are used for a very small number of specific applications: Smart NICs, Early stages of wireless networks (5G whilst the standards are being hammered out), military (where you need high performance with no consideration of cost), and embedded, Prof Video (where the custom I/O is essential). Generally, unless you're doing something that fits those applications well, the FPGA will not look good, and there are the same mistakes made in research time after time. For data centre these are twice as bad.
    • @troyhunt: We [Cloudflare] peaked at 44M requests to @reporturi in an hour yesterday. The busiest single minute I saw was 949k requests or an *average* of 16k requests per second for that minute. 
    • NSA: Right now, almost all NSA’s mission is being done in [IC GovCloud], and the productivity gains and the speed at which our analysts are able to put together insights and work higher-level problems has been really amazing.
    • Robert Graham: This is the most pressing lesson organizations need to learn, the one they are ignoring. They need to do more to prevent desktops from infecting each other, such as through port-isolation/microsegmentation. They need to control the spread of administrative credentials within the organization. A lot of organizations put the same local admin account on every workstation which makes the spread of NotPetya style worms trivial. They need to reevaluate trust relationships between domains, so that the admin of one can't infect the others.
    • smidgie82: This article totally ignores the devops side of Docker. Sure, you can cobble together Docker-like process isolation using namespaces and cgroups, and you can run the process using a custom set of libraries using chroot -- though I definitely don't agree that for the average developer that approach is anywhere near as easy as Docker.
    • Valeriy Kravchuk: Partitioning bugs do not get proper attention from Oracle engineers. We see bugs with wrong status and even a bug with a clear test case and a duplicate that is "Open" for 4 years. Some typical use cases are affected badly, and still no fixes (even though since 5.7 we have native partitioning in InnoDB and changing implementation gave good chance to review and either fix or re-check these bugs).
    • Don't stop here. Follow the clues and you shall...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Jun222018

Stuff The Internet Says On Scalability For June 22nd, 2018

Hey, it's HighScalability time:

 

4th of July may never be the same. China creates stunning non-polluting drone swarm firework displays. Each drone is rated with a game mechanic and gets special privileges based on performance (just kidding). (TicToc)

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

  • $40 million: Netflix monthly spend on cloud services; 5%: retention increase can increase profits 25%; 50+%: Facebook's IPv6 traffic from the U.S, for mobile it’s over 75 percent; 1 billion: monthly Facebook, err, Instagram users; 409 million: websites use NGINX; 847 Tbps: global average IP traffic in 2021; 200 million: Netflix subscribers by 2020; $30bn: market for artificial-intelligence chips by 2022;

  • Quotable Quotes:

    • @evacide: Just yelled “Encryption of data in transit is not the same as encryption of data at rest!” at a journalist on the car radio before slamming it off. I am a hit at parties.

    • Drako: I deal with the security industry, where more than 90% of the security cameras are manufactured in China. The chips in those cameras used to be made in a lots of different places. They’ve since migrated to China, and a lot of the government customers I engage with are unwilling in any way, shape, or form to deploy those cameras. They have a huge problem sourcing cameras that are not based on those chips. There is a lot of concern about the Trojans in chips and Trojans near the chips. It’s the first situation I’ve encountered where the customer is honestly concerned about this.

    • Memory Guy: But there are even more compelling reasons for certain applications to convert from today’s leading technologies (like NAND flash, DRAM, NOR flash, SRAM, and EEPROM) to one of these new technologies, and that is the fact that the newer technologies all provide considerable energy savings in computing environments...Something consistent about all of them is that they are nonvolatile, so they don’t need to be refreshed like DRAM, they use faster and lower-energy write mechanisms than either NAND or NOR flash, and their memory cells can be shrunk smaller than current memory technologies’ scaling limits, which means that they should eventually be priced lower than today’s memory chips.

    • Charlie Demerjian: what does Intel have planned for their server roadmap? Why is it causing such consternation among OEMs, ODMs, and major customers? For the same reason the 14/10nm messaging is causing consternation among investors, but the server side is in much worse shape. How bad is it? Three major roadmap updates in 29 days with serious spec changes, and it got worse from there.

    • Xtracerx: for me the biggest value to serverless functions is how nicely they tie in to the ecosystem of a cloud provider. using them to respond to storage events on s3 or database events or auth events is super easy and powerful.

    • Daniel Lemire: So Roaring bitmaps can be thousands of times faster than a native JavaScript Set. they can can be two orders of magnitude faster than FastBitSet.js. That Roaring bitmaps could beat FastBitSet.js is impressive: I wrote FastBitSet.js and it is fast!

    • Bryan William Jones: Cool. "But it turns out that the Orbiters' photos were actually super-high-rez, shot on 70mm film and robotically developed inside the orbiters, with the negs raster-scanned at 200 lines/mm and transmitted to ground stations using... lossless analog image-compression technology."

    • Bcrescimanno: I worked at Netflix when OpenConnect was introduced and I don't remember anyone internally thinking it was unnecessary (though, even at the time, the company was large enough that you didn't know everyone). Quite contrary, this was the era of the 250GB / month cap from Comcast and we could observe clearly that they were throttling Netflix traffic. OpenConnect, the ability to deploy the CDN directly into the internal network of these ISPs served multiple purposes--not the least of which was to expose the fact that they were holding Netflix for ransom. So, to say that was the executive foreseeing things is a bit of revisionist history. It doesn't lessen the impact or importance of OpenConnect; but, it grew out of a very real impasse with a very large ISP. Ultimately, Netflix did end up paying Comcast in 2014 and, surprise surprise, the throttling stopped.

    • Eitally: there are a few critical differences between GCP and AWS or Azure. Setting aside the network quality & performance, which is objectively superior with Google, outside of GCE almost every other GCP product is offered as a managed service. Beyond that, there are several -- like Spanner -- that don't exist anywhere else. I fully appreciate a desire to avoid vendor lock-in, but there are plenty of situations where allying with a vendor that offers a superior product/service for your specific business need is absolutely the correct decision.

    • There are more quotes, more everything. Get them while they're hot.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Jun192018

Sponsored Post: Datadog, InMemory.Net, Triplebyte, Etleap, Scalyr, MemSQL

Who's Hiring? 

  • Triplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Make your job search O(1), not O(n). Apply here.

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services

  • Datadog is a cloud-scale monitoring platform that combines infrastructure metrics, distributed traces, and logs all in one place. With out-of-the-box dashboards and seamless integrations with over 200 technologies, Datadog provides end-to-end visibility into the health and performance of modern applications at scale. Build your own rich dashboards, set alerts to identify anomalies, and collaborate with your team to troubleshoot and fix issues fast. Start a free trial and try it yourself.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net
  • For heads of IT/Engineering responsible for building an analytics infrastructure, Etleap is an ETL solution for creating perfect data pipelines from day one. Unlike older enterprise solutions, Etleap doesn’t require extensive engineering work to set up, maintain, and scale. It automates most ETL setup and maintenance work, and simplifies the rest into 10-minute tasks that analysts can own. Read stories from customers like Okta and PagerDuty, or try Etleap yourself.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to see if they can help you scale your career faster.


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Monday
Jun182018

How Ably Efficiently Implemented Consistent Hashing

This is a guest post by Srushtika Neelakantam, Developer Advovate for Ably Realtime, a realtime data delivery platform. You can view the original article—How to implement consistent hashing efficiently—on Ably's blog.

Ably’s realtime platform is distributed across more than 14 physical data centres and 100s of nodes. In order for us to ensure both load and data are distributed evenly and consistently across all our nodes, we use consistent hashing algorithms.

In this article, we’ll understand what consistent hashing is all about and why it is an essential tool in scalable distributed system architectures. Further, we’ll look at data structures that can be used to implement this algorithm efficiently at scale. At the end, we’ll also have a look at a working example for the same.

Hashing revisited

Remember the good old naïve Hashing approach that you learnt in college? Using a hash function, we ensured that resources required by computer programs could be stored in memory in an efficient manner, ensuring that in-memory data structures are loaded evenly. We also ensured that this resource storing strategy also made information retrieval more efficient and thus made programs run faster.

The classic hashing approach used a hash function to generate a pseudo-random number, which is then divided by the size of the memory space to transform the random identifier into a position within the available space. Something that looked like the following:

location = hash(key) mod size

So, why can’t we use the same method for handling requests over the network?

Click to read more ...

Friday
Jun152018

Stuff The Internet Says On Scalability For June 15th, 2018

Hey, it's HighScalability time:

 

Scaling fake ratings. A 5 star 10,000 phone Chinese click farm. (English Russia)

Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. They'll love you even more.

  • 1.6x: better deep learning cluster scheduling on k8s; 100,000: Large-scale Diverse Driving Video Database; 3rd: reddit popularity in the US; 50%: increase in Neural Information Processing System papers, AI bubble? 420 tons: leafy greens from robot farms; 75%: average unused storage on EBS volumes; 12TB: RAM on new Azure M-series VM; 10%: premium on Google's single-tenant nodes; $7.5B: Microsoft's cost of courting developers; 100th: flip-flop invention anniversary; 1 million: playlist dataset from Spotify; 38GB torrent: Stackoverflow public database; 85%: teens use YouTube; 20%-25%: costs savings using Aurora; 80%: machine learning Ph.D.s work at Google or Facebook; 18: years of NASA satellite data; >1TB: Ethereum blockchain; 200,000 trillion: IBM's super computer calculations per second; 

  • Quotable Quotes:
    • Michael Pollan: “I have no doubt that all that Hubbard LSD all of us had taken had a big effect on the birth of Silicon Valley.”
    • @hisham_hm: Strongly disagree. Most of use who started coding in the 80s started with BASIC. We turned out just fine. The first thing that your first language should teach you is the _joy of coding_.
    • @bryanmikaelian: OH: GraphQL is SOAP for millennials
    • @JoeEmison: Another thing that serverless architectures change: how do you software development. I find myself using a local dev environment to do infra config mgmt, but then often use the web consoles for writing functions and testing; so much faster that way.
    • Jürgen Schmidhuber: And now we can see that all of civilisation is just a flash in world history. In just a flash, the guy who had the first agriculture was almost the same guy who had the first spacecraft in 1957. And soon we are going to have the first AIs that really deserve the name, the first true AIs.
    • Dave Snowden~ A key principle of complex design is shift a system to an adjacent possible. Once there's enough stability more conventional approaches can be used. Architect for discovery before architecting for delivery. Starting with delivery misses the discovery phases which misses opportunities as well as threats. Fractal engagement is they way to achieve change. People don't make decisions about what other people do. People make decision about what they can do tomorrow within their own sphere of influence. The system as a whole orientates through multiple actions. You scale be decomposition and recombination, not through aggregation, and not by imitation.
    • Alex Lindsay: When possibility is greater than circumstance you get action.
    • Andrew Barron: These are all this family of traits that at one time were considered to be the thing that separated humans from all other animals, and then was slowly recognized to appear in primates and then large-brained mammals. And then suddenly we’re recognizing that something like a honey bee, with less than a million neurons, is able to do all of these things.
    • John Hennessy & David Patterson~ We're entering a new golden age [in processors]. The end of Dennard Scaling and Moore's Law means architecture is where we have to innovate to improve performance, cost, and energy. Rasing the level of abstraction using Domain Specific Languages makes it easier for programmers and architects to innovate. Domain Specific Architectures are getting 20x and 40x improvements, not just 5-10%.
    • Hungry for more? Get a plate, click through, and pile it high.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...