advertise
Tuesday
Mar132018

Sponsored Post: Clover, Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

  • Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!

  • Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.

  • Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • 5 Signs You’ve Outgrown DynamoDB. Companies often select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk. This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. In this white paper we’ll cover elements of costing as well as the results of benchmark-based testing. Read 5 Signs You’ve Outgrown DynamoDB to determine if your organization has outgrown this technology.

  • Advertise your event here!

Cool Products and Services

  • Educative provides interactive courses for software engineering interviews created by engineers from Facebook, Microsoft, eBay, and Lyft. Prepare in programming languages like Java, Python, JavaScript, C++, and Ruby. Design systems like Uber, Netflix, Instagram and more. More than 10K software engineers have used Coderust and Grokking the System Design Interview to get jobs at top tech companies like Facebook, Google, Amazon, Microsoft, etc. Ace your software engineering interviews today. Get started now

  • Gartner’s 2018 Magic Quadrant for Data Science and Machine Learning Platforms. Read Gartner’s most recent 2018 release of the Magic Quadrant for Data Science and Machine Learning Platforms. A complimentary copy of this important research report into the data science platforms market is offered by Domino. Download the report to learn: 
    • How Gartner defines the Data Science Platform category, and their perspective on the evolution of the data science platform market in 2018. 
    • Which data science platform is right for your organization. 
    • Why Domino was named a Visionary in 2018.

  • Exoscale GPU Cloud Servers. Powerful on-demand GPU. Perfect for your machine learning, artificial, and encoding workloads. GPU instances work exactly like other instances: they are billed by the minute and integrate seamlessly with your existing infrastructure. Tap the GPU's full power with direct passthrough access. Speed-up Tensorflow or any other Deep Learning, Big Data, AI, or Encoding workload. Start your GPU instances via our API or with your existing deployment management tools. Add parallel computational power to your stack with no effort. Get Started

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Scale your Job Search with Triplebyte

Triplebyte is unique because they're a team of engineers running their own centralized technical interview. The evaluation quality is so good that companies like Apple, Dropbox, Mixpanel, and Instacart now let every engineer Triplebyte recommends skip steps in the application process.

They give personal assistance to discover which roles you're most excited about, schedule your final interviews back-to-back, and help you negotiate with multiple companies at once.

Triplebyte now works with top tech companies and hundreds of the most exciting pre-screened startups.

It's free, confidential, and background-blind for engineers. Take Triplebyte's online coding quiz to see if they can help you scale your career faster. (Engineers with architecture and system design experience tend to do especially well.)


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Mar092018

Stuff The Internet Says On Scalability For March 9th, 2018

Hey, it's HighScalability time:

 

The largest simulation of the cosmos ever run finally produces a universe similar to our own. All it required was 24,000 processors, more than two months, and it produced 500 terabytes of data.

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

 

  • 72 bits: Google's new quantum computer; 50,000: sites infected with cryptocurrency mining malware; $40 billion: purchases via talking tubes by 2022; $12,000: value of 1 million YouTube views a month; $15 billion: Netflix 2018 revenue; 

  • Quotable Quotes:
    • @ValaAfshar: Jeff Bezos, CEO @amazon: I very frequently get the question: "what's going to change in he next 10 years?" I almost never get the question: "what's not going to change in the next 10 years?" I submit to you that the second question is actually the more important of the two.
    • @svscarpino: Sharing and consuming fake news is highly concentrated. 0.1% of Twitter accounts share 80% of the fake news and 1% share 100%!!!  Incredible results by @davidlazer and collaborators. #complenet18
    • Tim Wu: An unwelcome consequence of living in a world where everything is “easy” is that the only skill that matters is the ability to multitask. At the extreme, we don’t actually do anything; we only arrange what will be done, which is a flimsy basis for a life.
    • pauldjohnston: if the CGI execution model was horizontally scalable, fault tolerant, spread across availability zones, stable and with a managed infrastructure and a secure and certified API Gateway in front of it with DDoS protection built in... and someone else was looking after it for me... (Not to mention all the other stuff AWS provides) then yes it's exactly like cgi-bin was back in the day.
    • Abu Sebastian: Computational memory: A memory unit that performs certain computational tasks in place. 
    • @MIT_CSAIL: The world's first online transaction happened over 45 years ago between MIT & Stanford students - and it was for weed
    • @cloud_opinion: don't forget "good enough" often wins. Fargate will be seen by many as good enough. Will k8s continue to be adopted?. yes. But, Fargate will reduce TAM for k8s distro companies.
    • @kylewillett: After finally getting hands-on with #akka streams to implement a solution to what would normally be a tricky async problem, I can now agree with the praise I've heard - an awesome api and great tool to have in the toolbox.
    • @awscloud: Amazon Redshift uses machine learning to automatically hop short queries to an express queue for fast processing. 
    • @postwait: Look, I'm gonna be the last one to defend InfluxDB... but for f*ck's sake don't use units per day in computing.  Please, please, please.... use per second numbers so you don't mislead or attempt to look large.
    • @hypervisible: Researchers build AI to identify gang members. When asked about potential misuses, presenter (a computer scientist at Harvard) says "I'm just an engineer."
    • @Jason: 4/We should also allow folks to build apartments and homes with NO PARKING spaces BUT with carports capable of getting ridesharing cars and people off streets during drop off and pickup. Right now we force folks to build X spots per Y residents, which is dated.
    • antiviral: OK... so they [Facebook] are saying they don't use your microphone to target ads. But how about precisely enumerating how FB uses your microphone?
    • Yep, there are a lot more quotes. Go get 'em or forever live in darkness.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Mar022018

Stuff The Internet Says On Scalability For March 2nd, 2018

Hey, it's HighScalability time: 

 

Algorithms described like IKEA instructions. Can anyone assemble these? (Algorithms and data structures)

 

If you like this sort of Stuff then please support me on Patreon. And please consider recommending my new book—Explain the Cloud Like I'm 10—to whole entire world. 

 

  • $75 million: Dropbox saved moving out of S3; 159 million: Spotify monthly active users; 80 million: more records added to Have I Been Pwned; 9%: universe expanding faster than predicted; $2,222,279: Warren Buffett won his long bet against hedge fund mangers; 60,000: Mayan houses found in Guatemala using LiDAR; $14.2 billion: PaaS revenue; ~180 million: years until first sun after the big whatever it was; $1,599: cost of stolen Extended Validation (EV) certificate; 8,000X: query speedup using GPU database; 2.4 million: Google requests to be forgotten; 6 minutes: time to IoT device attack on the internet; 103 million: tweets sent about the Olympics; 320,000: increase in Chloe Kim's twitter followers; 150 kg: acorns stored by woodpeckers in a telecom antenna; 0.14ms: Fsync performance on Intel PC-3700; Q: earliest known article on Wikipedia; 800Gbps+: memcached reflection/amplification attacks; 2M+: Google-Landmarks image training set; 3 million: graphics cards purchased by cryptocurrency miners; 30%: Uber and Lyft drivers lose money; 

  • Quotable Quotes:
    • @mikko: Interesting point raised in a reddit thread: Satoshi's original bitcoins are now a quantum canary. Once we see them moving, we’ll know that someone has a functioning advanced quantum computer. It's just too big a prize not to be the first thing you’d do with a quantum computer.
    • @brettberson: I just learned from a former longtime Amazon employee, the idea for Prime came from an IC [individual contributor] engineer. He wrote up a 6 page memo. He was inspired by the Costco membership model. It was built as a test. It's now the key pillar of Amazon. The best ideas can come from anywhere.
    • Erica Klarreich: In a statistical analysis of nearly 1,000 networks drawn from biology, the social sciences, technology and other domains, researchers found that only about 4 percent of the networks (such as certain metabolic networks in cells) passed the paper’s strongest tests. And for 67 percent of the networks, including Facebook friendship networks, food webs and water distribution networks, the statistical tests rejected a power law as a plausible description of the network’s structure.
    • @kpkelleher: 531 ICOs that appeared in 2017 have already vanished. Together, they raised $233 million.
    • @darrenrovell: In November 2013, Jamie Siminoff came on Shark Tank valuing his WiFi enabled video doorbell at $7 million. Four sharks passed & @kevinolearytv offered his typical loan/royalty deal. Siminoff passed. That company became @ring & today sold for more than $1 billion to Amazon.
    • @tyleralove: As of today @bustle has fully adopted serverless. We’re down to 15 ec2 instances mostly comprised of self-managed HA Redis. We serve upwards of a billion requests to 80 million people using SSR preact and react a month. We are a thriving example of modern JavaScript at scale. We do all of this with a relatively tiny engineering team of 12 while simultaneously building compelling to use product that was never focused on social media audience gaming or egregious engagement metric hacking.
    • abetusk: I've heard, and agree with, that 95% of programming doesn't require any deep CS knowledge. The flip side of that is 5% of the time you will and for those 1/20 times you encounter a problem that requires theory, you're dead in the water unless you know how to identify it, how to solve it or where to look for solutions to it.
    • @geofft: At which point Trustico's CEO decided to EMAIL 23,000 CUSTOMER PRIVATE KEYS to Digicert, apparently in order to trigger that clause.
    • @swardley: The ONLY reason that Amazon is as big as it is today and continuing to grow rather than being constrained (as normally happens) is because competitor executives have utterly failed to adapt. This is not a market failure, it's a failure of executives ..
    • Andrei Barysevich: Contrary to a common belief that the security certificates circulating in the criminal underground are stolen from legitimate owners prior to being used in nefarious campaigns, we confirmed with a high degree of certainty that the certificates are created for a specific buyer per request only and are registered using stolen corporate identities, making traditional network security appliances less effective.
    • @KentonVarda: After I open sourced Protocol Buffers, the promo committee denied me for promotion (from Senior to Staff) because my packet contained no peer reviews from more-senior engineers who worked closely with me. (There were no such engineers.)
    • Jared Diamond: Why is there such widespread public opposition to science and scientific reasoning in the United States, the world leader in every major branch of science?
    • @sama: I wonder how much cryptocurrency is slowing the rate of AI progress by wildly driving up the price of GPUs...
    • So many more quotes. Don't miss out on all the smart stuff people have said. You will become so much smarter.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Feb272018

Sponsored Post: Clover, Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

  • Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!

  • Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.

  • Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • 5 Signs You’ve Outgrown DynamoDB. Companies often select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk. This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. In this white paper we’ll cover elements of costing as well as the results of benchmark-based testing. Read 5 Signs You’ve Outgrown DynamoDB to determine if your organization has outgrown this technology.

  • Advertise your event here!

Cool Products and Services

  • Gartner’s 2018 Magic Quadrant for Data Science and Machine Learning Platforms. Read Gartner’s most recent 2018 release of the Magic Quadrant for Data Science and Machine Learning Platforms. A complimentary copy of this important research report into the data science platforms market is offered by Domino. Download the report to learn: 
    • How Gartner defines the Data Science Platform category, and their perspective on the evolution of the data science platform market in 2018. 
    • Which data science platform is right for your organization. 
    • Why Domino was named a Visionary in 2018.

  • Exoscale GPU Cloud Servers. Powerful on-demand GPU. Perfect for your machine learning, artificial, and encoding workloads. GPU instances work exactly like other instances: they are billed by the minute and integrate seamlessly with your existing infrastructure. Tap the GPU's full power with direct passthrough access. Speed-up Tensorflow or any other Deep Learning, Big Data, AI, or Encoding workload. Start your GPU instances via our API or with your existing deployment management tools. Add parallel computational power to your stack with no effort. Get Started

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Scale your Job Search with Triplebyte

Triplebyte is unique because they're a team of engineers running their own centralized technical interview. The evaluation quality is so good that companies like Apple, Dropbox, Mixpanel, and Instacart now let every engineer Triplebyte recommends skip steps in the application process.

They give personal assistance to discover which roles you're most excited about, schedule your final interviews back-to-back, and help you negotiate with multiple companies at once.

Triplebyte now works with top tech companies and hundreds of the most exciting pre-screened startups.

It's free, confidential, and background-blind for engineers. Take Triplebyte's online coding quiz to see if they can help you scale your career faster. (Engineers with architecture and system design experience tend to do especially well.)


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Feb232018

Stuff The Internet Says On Scalability For February 23rd, 2018

Hey, it's HighScalability time: 

 

What does a bubble look like before it bursts? The image shows a brief period of stability before succumbing to molecular forces that pinch the film together and cause the bubble to burst (Mr Li Shen - Imperial College London)

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

 

  • 20 million: daily DuckDuckGo searches, 55% growth; 38.3 billion: WeChat messages sent per day; 500 million: database of pwned passwords to check against; 38%: China's consumption of world IC production; 92%: Fortune 500 traffic is from bots; $2: Blue Pill: A 72MHz 32-Bit Computer; 5,000: Martian days of operation for the Opportunity Rover; 30.72 terabytes: Samsung SSD; 185%: Golden State Warriors drive up ticket prices; $1.3 billion: loss that happens when Kylie Jenner tweets about your new UI; 

  • Quotable Quotes:
    • @slava_pestov: A senior programmer is merely a junior programmer who has given up
    • @gabrtv: This is why FaaS/PaaS on top of container orchestration is so important.  By running FaaS on Kubernetes, you hedge against the abstractions you've bet on today, while setting yourself up to benefit from the abstractions to come.
    • Karin Strauss~ Their storage system now offers random access across 400 megabytes of data encoded in DNA with no bit errors
    • @mipsytipsy: That reminds me ... about a year ago we posted our first open job rec.  Here are some things I have observed since then. 1) There is a huge, underserved pool of skilled senior engineers who are actively repelled by bro culture.  Of all genders. 2) there is an ocean of brilliant, overqualified, creative, entrepreneurial business folks who are sick of getting shit on by engineers, particularly technical founders. 
    • @KentLangley: “All the compute power of the top 500 supercomputer clusters today would represent less than 0.01% of all the Bitcoin mining operations worldwide.”
    • @allspaw: Problems in business-critical software are detected, identified, and handled via the coordinated *cognitive* work of people coping with uncertainty and ambiguity.
      No framework, tool, or process changes that. This is why I believe we need to take human performance seriously.
    • Vinod Khosla: There are, perhaps, a few hundred sensors in the typical car and none in the body. A single ad shown to you on Facebook has way more computing power applied to it than a $10,000 medical decision you have to make.
    • tryitnow: I've worked in finance for several SaaS companies and you're 100% right. Infrastructure is never the biggest cost, it's sales and marketing and most of that spend is a mystery in terms of attribution, so blended CAC is not terribly useful.
    • Ariel Bleicher: 5G Champion’s prototype radios, for instance, operate in a 1-Gigahertz-wide band around 28 Gigahertz—10 times the maximum spectrum available to today’s 4G networks.
    • @cl4es: I found a hack which reduce the one-off bootstrap overhead of using lambdas by ~75% (15ms or so)
    • underwater: I work for a publisher that zero ads. We have fast pages with minimal JS. We rolled out AMP purely for the SEO win and saw a huge uptick in traffic. If Google really cared about performance they’d reward publishers doing the right thing on their regular pages (which would benefit the web as a whole), not just those using AMP
    • Artur: Waze doesn’t get you to your destination any faster. So, is it better to spend 30-minutes following Waze through suburban neighborhoods and alleys than 27-minutes minutes following Google Maps into a highway traffic jam?
    • pradn: I'm in a team right now which I believe has a 10x programmer. He picked a simple threading model to prevent tons of wasted time on deadlocks and other threading issues. It's also easier to reason about and get new people on-boarded. When he reviews code, he finds bugs that prevent days of debugging down the road and suggests simpler architectures that make the code easier to understand and change. He has had this effect on ~20 people over ~5 years. I would not be surprised if he saved us ~1 year of dev time collectively.
    • Vince Cerf: It's the mind-stretching practice of trying to think what the implications of technology will be that makes me enjoy science fiction. It teaches me that when you're inventing something you should try to think about what the consequences might be.
    • Yes, there are a lot more quotes. Look away if you can't handle the truth.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Feb212018

How fast can a bird search a tree?

 

I was wondering if you could help me figure something out: what is the algorithmic complexity of a bird searching a tree for food?

Over the years I've had the pleasure of watching a lot of cute little birds feed in our oak trees. I've noticed they have a search pattern.

A bird will hop from branch to branch looking for insects. They don't hop on a branch and explore every square inch of it, so it's not an exhaustive search. They'll take a couple hops, peck at a branch a few times, and hop to a nearby branch. Birds also search the underside of branches, so the whole surface area of a tree is game. 

I've often marveled in wonder at how efficient this whole process is. They scour huge trees in no time. Then they'll move on to the next tree and repeat the process until they fly away to a completely different area.

My dog when searching for a ball seems to follow a similar Lévy flight sort of pattern. Search a local area by bouncing around for bit and then take a bee-line for a completely different area and repeat the process. 

Often a flock will search a single tree at the same time, so there's parallelism at work here. A bird being on a branch doesn't lock the branch, but a new bird will jump to a different part of a branch if a branch has another bird on it already.

I haven't been able to determine if there's some signal, like a certain threshold number of birds flying away that tells the flock this tree has played out, or if each bird makes their own decisions locally. I don't know how they select trees either. They, it seems to me, enter a tree at random, not always starting at the top or bottom, and not performing on obvious depth or breadth first search. They don't seem to want to visit every branch of a tree either. They spend a limited amount of time in each tree before moving on.

An analogy to computing I've noticed is that it's the shape of thee tree itself that contributes to the efficiency of the search. A tree efficiently fills a certain volume of space with leaves and branches, optimizing for sunlight exposure. It's because branches are located on a trunk, relatively close together, often almost touching, with a fractal branching pattern that means branches lead to more branches which lead to more branches, that birds can make quick work if searching a tree for food.

When I see a bird jump from branch to branch, tree to tree, I see some sort of search algorithm. What I don't see is good way to describe it. Any ideas?

Friday
Feb162018

Stuff The Internet Says On Scalability For February 16th, 2018

Hey, it's HighScalability time: 

 

Snow? Last march of the faeries? Nope. It's 1218 of Shooting Star drones forming the Olympic symbol. *chills*

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

  • 63.2%: Americans with one and only one cable provider; $1.5 billion: spend on chip startups last year; $7.5 billion: Uber sales; $4.5 billion: Uber loss; 180 TFLOPS: computation accessible via the TensorFlow programming model from a Google Cloud VM; 10x: computational capabilities of the human brain than previously thought; 10 million: went live on Facebook sharing 47% more Live videos than the previous year; 1.7 million: HQ players during Superbowl; 1/400: power to perform public-key encryption; 8 bit: custom CPU build from scratch; $8,500: daily take from mining Monero with your botnet; 10,000: datasets shared on Kaggle; 41%: NVIDIA revenue growth; 14x: real world 4G LTE vs. 5G bandwidth; 2: two SpaceX demonstration satellites ready to launch; 

  • Quotable Quotes:
    • Packet Pushers~ The only really good protocols are in people’s minds.
    • Georgia Dow~ I have an easier time getting people off of smoking and drinking that I do technology. 
    • Natalie Cheung: In order to create a real and lifelike version of the snowboarder with more than 1,200 drones, our animation team used a photo of a real snowboarder in action to get the perfect outline and shape in the sky.
    • Certhas: tl;dr: "With the Ryzen 5 2400G, AMD has completely shut down the sub-$100 graphics card market. As a choice for gamers on a budget, those building systems in the region of $500, it becomes the processor to pick."
    • AnalogOfDwarves: Better rule of thumb: Minimize the amount of covariant code. If you're repeating yourself, or thinking about doing it, ask yourself: "If I later change this in one place, will I want to change this in the other places?" And more importantly, "Are there changes I might make to this in one place that I'll want to avoid making in the other places?" If the answers to these questions are "yes" and "no", respectively, then you refactor to a common unit.
    • @Carnage4Life: Azure usage continues to explode with 35% year-over-year growth compared to AWS with 15%. Microsoft as the scrappy fast follower is a look no expected.
    • Scott Aaronson: Can we program a computer to find a 10,000-bit string that encodes more actionable wisdom than any human has ever expressed?
    • @tommorris: Bitcoin advocates: "We need cryptocurrencies for the farmers in sub-Saharan Africa with only a crappy Android phone and no bank account." Also Bitcoin advocates: "Oh, you lost your money? You should have stored your private key on an airgapped burner laptop."
    • J. M. Korhonen: Feel free to call me a luddite or whatever. It’s just that I’ve been studying the possibilities of blockchains for business for over a year now, and while it is certainly possible that I simply lack the imagination (or chutzpah) necessary for bold proclamations, I just don’t see the possibilities the marketers seem to see.’
    • George Church: All these things come together in a time of exponential change. It’s not necessarily some panacea that’s full of abundance and you don’t have to think and it’s easy, but there are some win-wins to be had if we think about it deeply and we talk about it as if science was a real thing rather than something that’s inconvenient. 
    • russellbeattie: For what it's worth, Windows Phone was actually an amazing platform for both users and developers, and shows a fundamental rule of technology: There Is No Third Ecosystsm. The most dominant hardware maker (at the time) and software/os maker teamed up with a really great product, but couldn't break the established smartphone duopoly, even though it was only a few years old by that point. I wasn't a Microsoft fan by any stretch (the opposite actually), but even I agreed with the decision at the time, especially after using Windows Phone. First mover advantage is huge, and developers only have so much bandwidth.
    • Quotes. Get your red hot quotes right here.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Feb132018

Sponsored Post: Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Scalyr, Domino Data Lab, MemSQL

Who's Hiring? 

  • Clover is looking for seasoned software engineers to help us solve the most complicated problem in the world: healthcare. We're using sophisticated data analytics, custom software, and machine learning to coordinate care and build a clearer model of our member's health and risk factors. We are on a mission to help seniors and low-income members live healthier while keeping costs down. This is an opportunity for those who want to be at the intersection of health and technology and thrive in a collaborative environment as well as the freedom of self-direction. If you're interested, please directly apply here!

  • Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.

  • Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • 5 Signs You’ve Outgrown DynamoDB. Companies often select a database that seems to be the best choice at first glance, as well as the path of least resistance, and then are subsequently surprised by cost overruns and technology limitations that quickly hinder productivity and put the business at risk. This seems to be the case with many enterprises that chose Amazon Web Service’s (AWS) DynamoDB. In this white paper we’ll cover elements of costing as well as the results of benchmark-based testing. Read 5 Signs You’ve Outgrown DynamoDB to determine if your organization has outgrown this technology.

  • Advertise your event here!

Cool Products and Services

  • Gartner’s 2018 Magic Quadrant for Data Science and Machine Learning Platforms. Read Gartner’s most recent 2018 release of the Magic Quadrant for Data Science and Machine Learning Platforms. A complimentary copy of this important research report into the data science platforms market is offered by Domino. Download the report to learn: 
    • How Gartner defines the Data Science Platform category, and their perspective on the evolution of the data science platform market in 2018. 
    • Which data science platform is right for your organization. 
    • Why Domino was named a Visionary in 2018.

  • Exoscale GPU Cloud Servers. Powerful on-demand GPU. Perfect for your machine learning, artificial, and encoding workloads. GPU instances work exactly like other instances: they are billed by the minute and integrate seamlessly with your existing infrastructure. Tap the GPU's full power with direct passthrough access. Speed-up Tensorflow or any other Deep Learning, Big Data, AI, or Encoding workload. Start your GPU instances via our API or with your existing deployment management tools. Add parallel computational power to your stack with no effort. Get Started

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


Scale your Job Search with Triplebyte

Triplebyte is unique because they're a team of engineers running their own centralized technical interview. The evaluation quality is so good that companies like Apple, Dropbox, Mixpanel, and Instacart now let every engineer Triplebyte recommends skip steps in the application process.

They give personal assistance to discover which roles you're most excited about, schedule your final interviews back-to-back, and help you negotiate with multiple companies at once.

Triplebyte now works with top tech companies and hundreds of the most exciting pre-screened startups.

It's free, confidential, and background-blind for engineers. Take Triplebyte's online coding quiz to see if they can help you scale your career faster. (Engineers with architecture and system design experience tend to do especially well.)


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Friday
Feb092018

Stuff The Internet Says On Scalability For February 9th, 2018

Hey, it's HighScalability time: 

 

To those living in the past: the future is coming. (launch, great sound).

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

  • 7.2 terabytes: data used during Super Bowl;  $220 million: projected podcast revenue this year; $127 billion: total addressable value of drone-powered solutions in all applicable industries; 100 billion billion billion: living microbial cells underlying all the worlds oceans, 200x biomass of humans; 110 billion: total market for memory; 1,000: drones North Korea may have, possibly with chemical or biological weapons, ready to attack South Korea; $123 billion: US apparel market; 100 million: iOS devices sold in Q4; $3: earnings in 24 hours from conscripting 5,000 Android devices into a mining botnet; 46%: cloud market growth in Q4; 104%: YoY Alibaba cloud growth; 

  • Quotable Quotes:
    • John Perry Barlow: The Internet is the most liberating tool for humanity ever invented, and also the best for surveillance. It's not one or the other. It's both. 
    • Fernando J. Corbato: Our use of the word daemon was inspired by the Maxwell's daemon of physics and thermodynamics. (My background is Physics.) Maxwell's daemon was an imaginary agent which helped sort molecules of different speeds and worked tirelessly in the background. We fancifully began to use the word daemon to describe background processes which worked tirelessly to perform system chores.
    • Monica Alleven: T-Mobile also released some additional tidbits: Tom Brady’s fumble and the Eagles’ field goal were the most shared moments of the game on social, with a 33% increase in posts. Social posts doubled during Timberlake’s halftime show versus the rest of the game, and nationwide, group and picture messaging went up by nearly 50%, with texting increasing nearly 10%.
    • @BretWeinstein: The most important patterns: 1. Prisoner's Dilemma 2. Race to the Bottom 3. Free Rider Problem / Tragedy of the Commons / Collective Action 4. Zero Sum vs. Non-Zero Sum 5. Externalities / Principal Agent 6. Diminishing Returns 7. Evolutionarily Stable Strategy / Nash Equilibrium
    • @Jason: US content companies should remove their content from @Facebook until facebook gives them 70% of revenue from the ads around their content — or a yearly license fee equal to 10-20% of their content budget. Facebook is the enemy of content companies — period.
    • @Carnage4Life: Slack officially states the only browser they support is Chrome. Striking to see history repeat itself and the dream of the Open Web die so ignominiously
    • @swardley: #noops, #nocode ... it's funny how some parts of the DevOps community are trying to create strawmen in the #serverless world. It's like EC2 in 2009 and the fight back from old practices against #DevOps. This time, the boot is on the other foot.
    • Troy Hunt: Every single minimum password length is an even number! How scientific do you think the process of determining the perfect minimum length is when all the big players just happened to land on 4, 6 or 8?
    • @seanjtaylor: TL;DR - There are about 100K-200K unique workers on MTurk. - On average, there are 2K-5K workers active on MTurk at any given time - 50% of the worker population changes within 12-18 months. - MTurk has a yearly transaction volume of a few hundreds of millions of dollars.
    • schappim: The TL;DR; of this article [How to Design a New Chip on a Budget] is: "...a simple ASIC (say one that is a few square millimeters in size, fabricated using the 250-nm technology node) might cost a few thousand bucks for a couple dozen samples." 
    • @j_s_n_d: Always store in UTC    @girlziplocked: What’s the best piece of dating advice you’ve ever encountered?
    • shadow31: Why? The entire point of doing a rewrite like that is to compare the implementations. No, you can't conclude that Rust is 100x more memory efficient than Java or something silly like that, but it's obvious that compiled, native code without a massive runtime VM layer is going to be more efficient. In this case, it's much more efficient.
    • Shar Darafsheh: Don’t start scaling until you have gathered all your requirements and understand the full scope of your SaaS scale processes. Give yourself enough time to scale; no software ever works well when development is rushed. Make sure there is proper architecture in place prior to development.
    • fackin_samsquamch: OK. I keep hearing this claim [There is no "skills gap." Employers are just cheapskates], and yet I see dozens of resumes a day and interview several candidates a week with unbelievably shit technical abilities and a barely passing ability to communicate with other humans.
    • @etiene_d: - How do french people send files? - Pierre-to-Pierre
    • Lots more quotes. Many many more. Don't miss out. Keep reading.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Friday
Feb022018

Stuff The Internet Says On Scalability For February 2nd, 2018

Hey, it's HighScalability time: 

 

Are silcon device designers also artists? Of course. (DAC Silicon/Technology Art Show)

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate if you would recommend my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll learn a lot, even if they're already familiar with the basics.

  • 2 billion: Siri requests per week; 1 trillion: semiconductor unit shipments, 9.1% compound annual growth rate over a 40 year span; 150 million: IPv4 addresses recirculated over 7 years; $100 Billion:  value lost in cryptocurrency markets in 24 hours; $1: microcontroller; 16 Gb/s: GDDR6 SGRAM; 1/3: IPv4 addresses registered to US entities; eight minutes and thirty-five seconds: breath-hold record; $32.32 billion: Google revenue, up 24%; 1.3 billion: active Apple devices, up 30% in 2 years; 500 petabytes: Backblaze; 6th or 7th: HTTP version of hypertex on the internet; $2.1 billion: Amazon's Q4 2017 operating income, up 69%; 45%: jump in Amazon cloud revenue; 9%: global smartphone market drop; 3 billion: photos uploaded to Google on New Year's eve; 1.5 billion: montly YouTube users; $530 million: stolen by hackers in the biggest cryptocurrency theft yet; 9,500: computers forced to be reinstalled by ransomware; 

  • Quotable Quotes:
    • ARM 2 106: Tell my lord: Your servant Yakim-Addu sends the following message: A short time ago I wrote to my lord as follows: "A lion was caught in the loft of a house in Akkaks. My lord should write me whether this lion should remain in that same loft until the arrival of my lord, or whether I should have borught to my lord." But letters from my lord were slow in coming and the lion has been in the loft for five days.
    • Brice Morrison: I'm predicting by 2020 there will be a billion dollar game where the primary way to play is with your voice. Right now Amazon Alexa and Google Home are simple utilities. The games on each of them are just toys - experiments to round out playing music and turning on smart lightbulbs. But each month signs are growing stronger that voice is becoming the next major growth platform. And as technology goes, games follow.
    • Geoff Huston: The days when the Internet was touted as a poster child of disruption in a deregulated space are long since over, and these days we appear to be increasingly looking further afield for a regulatory and governance framework that can continue to challenge the increasing complacency of the newly-established incumbents. 
    • @jckarter: Reminder: running 32 bit processes on a 64 bit CPU for prolonged periods can lead to burn-in of the unused high bits. Degauss your CPU regularly
    • Ted Nelson~ We had visions of democratization, citizen participation, great vistas of possibily participation for atistic expression in software. Software is an artform, though not generall recognized as such—exactly how you select the keys, exactly how you position things on the screen—has an impact. In the old days there was a greater shared citizen vision of the personal computing movement.
    • @n_srnck: Uber is buying 24,000 cars. Facebook is spending $1 billion on original TV shows. Alibaba is spending $2.6 billion for physical stores. Airbnb is opening branded apartment buildings.
    • @etherealmind: This is a big deal. Open sourcing the SDK changes whitebox market. 
      Link: Broadcom Expands Ethernet Switch Software Suite with Industry’s First Fully Open Source Software Development Kit. Enterprises can now easily develop their own Network Operating System. Open source projects are now free to flourish. Vendors based on open source can easily expand their feature sets. Obviously a reaction to SAI/Sonic. 
    • Geoff Huston: Time and time again we are lectured that NATs are not a good security device, but in practice NATs offer a reasonable front-line defence against network scanning malware, so there may be a larger story behind the use of NATs and device-based networks than just a simple conservative preference to continue to use an IPv4 protocol stack.
    • @Tanvim: This guy has been biking really slow outside the FCC to protest its decision to repeal net neutrality, and charging $5 to have vehicles pass him. Lol.
    • npz: The days of ASICs are long past. I guarantee you that NO ONE in the general community wants to repeat the same mistake bitcoin and subsequently litecoin made. Hence, all coins have been asic resistant since. And some modern coins / blockchain hashing algorithms are even complex enough to give GPUs a hard time, enough to allow CPUs to be competitive like Monero (XMR/cyryptonight). That's why when it comes to Monero, you'll often hear about XMR-"Stacks" because now even the CPU can be used!
    • Philippe Kahn: I met with all of them [Kodak, Polaroid]. Proposed our solution to no avail. They had an established business and thought that it would never go away and they could wait. They totally missed the paradigm shift. Paradigm shifts are challenges for any established player, look at the demise of Nokia for missing the smartphone.
    • Jakob: it would seem that reducing the precision and making timing sources more jittery won’t really help with the core problem. It is probably a good idea to do this in JavaScript to make it harder to do exploits, but it is not a panacea. It appears that in the end, it is the side channels themselves that have to be suppressed. Which is not a particularly appealing statement to make, since side channels by definition are not designed into a system. They are discovered as side-effects of otherwise reasonable decisions. In the end, there is no replacement for an adversarial mind-set, and putting resources into thinking about how things can be broken, not just made to work in the first place.
    • tw1010: There aught to be a name to the tendency that as tools get better and better, the more your time goes from having your mind in technical-space to social and news-space. It's like the authority to create goes from the individual first-principles (by necessity) maker, to the control over development being in the hands of an external group, and then all your time is spent keeping up with what they're doing. A similar thing happened with a lot of javascript frameworks. It also happened with the transition from building servers from the ground up, to it all being managed by AWS.
    • @daveixd: This notion that "it's all just a guess until we ship to production" flies in the face of decades of research in HCI, psych, etc. The point isn't knowing for sure, but increasing confidence as investment increases. Deciding to ignore those opportunities to learn is reckless.
    • @davidgerard: Dr Strangelove is actually a film about why immutable smart contracts that cannot be altered by human agency once they're in motion
    • So many more quotes. Get them while they're hot!

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...