Stuff The Internet Says On Scalability For July 28th, 2017s

Hey, it's HighScalability time:


Jackson Pollock painting? Cortical column? Nope, it's a 2 trillion particle cosmological simulation using 4000+ GPUs. (paper, Joachim Stadel, UZH)

If you like this sort of Stuff then please support me on Patreon.


  • 1.8x: faster code on iPad MacBook Pro; 1 billion: WhatsApp daily active users; 100 milliamps: heart stopping current; $25m: surprisingly low take from ransomware; 2,700x: improvement in throughput with TCP BBR; 620: Uber locations; $35.5 billion: Facebook's cash hoard; 2 billion: Facebook monthly active users; #1: Apple is the world's most profitable [legal] company; 500,000x: return on destroying an arms depot with a drone; 

  • Quotable Quotes:
    • Alasdair Allan: Jeff Bezos’ statement that “there’s not that much interesting about CubeSats” may well turn out to be the twenty first century’s “nobody needs more than 640kb.”
    • @hardmaru: Decoding the Enigma with RNNs. They trained a LSTM with 3000 hidden units to decode ciphertext with 96%+ accuracy. 
    • @tj_waldorf: Morningstar achieved 97% cost reduction by moving to AWS. #AWSSummit Chicago
    • Ed Sperling: Moore’s Law is alive and well, but it is no longer the only approach. And depending on the market or slice of a market, it may no longer be the best approach.
    • @asymco: With the end of Shuffle and Nano iPods Apple now sells only Unix-enabled products. Amazing how far that Bell Labs invention has come.
    • @peteskomoroch: 2017: RAM is the new Hadoop
    • Carlo Pescio: What if focusing on the problem domain, while still understanding the machine that will execute your code, could improve maintainability and collaterally speed up execution by a factor of over 100x compared to popular hipster code?
    • @stevesi: Something ppl forget: moving products to cloud, margins go down due to costs to operate scale services—costs move from Customer to vendor.
    • @brianalvey: The most popular software for writing fiction isn't Word. It's Excel.
    • @pczarkowski: How to make a monolithic app cloud native: 1) run it in a docker 2) change the url from .com to .io
    • @tj_waldorf: Morningstar achieved 97% cost reduction by moving to AWS. #AWSSummit Chicago
    • drinkzima: There is a huge general misunderstanding in the profitability of directing hotel bookings vs flight bookings or other types of travel consumables. Rate parity and high commission rates mean that directing hotel rooms is hugely profitable and Expedia (, trivago, expedia) and Priceline ( operate as a duopoly in most markets. They are both marketing machines that turn brand + paid traffic into highly profitable room nights.
    • Animats: This is a classic problem with AI researchers. Somebody gets a good result, and then they start thinking strong human-level AI is right around the corner. AI went through this with search, planning, the General Problem Solver, perceptrons, the first generation of neural networks, and expert systems. Then came the "AI winter", late 1980s to early 2000s, when almost all the AI startups went bust. We're seeing some of it again in the machine learning / deep neural net era.
    • Charity Majors: So no, ops isn't going anywhere. It just doesn't look like it used to. Soon it might even look like a software engineer.
    • @mthenw: As long as I need to pay for idle it’s not “serverless”. Pricing is different because in Lambda you pay for invocation not for the runtime.
    • Kelly Shortridge: The goal is to make the attacker uncertain of your defensive environment and profile. So you really want to mess with their ability to profile where their target is
    • @CompSciFact: 'About 1,000 instructions is a reasonable upper limit for the complexity of problems now envisioned.' -- John von Neumann, 1946
    • hn_throwaway_99: Few barriers to entry, really?? Sorry, but this sounds a bit like an inexperienced developer saying "Hey, I could build most of Facebook's functionality in 2 weeks." is THE largest spender of advertising on Google. They have giant teams that A/B test the living shite out of every pixel on their screens, and huge teams of data scientists squeezing out every last bit of optimization on their site. It's a huge barrier to entry. 
    • callahad: It's real [performance improvements]. We've [Firefox] landed enormous performance improvements this year, including migrating most Firefox users to a full multi-process architecture, as well as integrating parts of the Servo parallel browser engine project into Firefox. There are still many improvements yet-to-land, but in most cases we're on track for Firefox 57 in November.
    • Samer Buna: One important threat that GraphQL makes easier is resource exhaustion attacks (AKA Denial of Service attacks). A GraphQL server can be attacked with overly complex queries that will consume all the resources of the server.
    • wheaties: This is stupid. Really. Here we are in a world where the companies that own the assets (you know, the things that cost a lot of money) are worth less than the things that don't own anything. This doesn't seem "right" or "fair" in the sense that Priceline should be a middleman, unable to exercise any or all pricing power because it does not control the assets producing the revenue. I wonder how long this can last?
    • platz: Apparently deep-learning and algae are the same thing.
    • @CompSciFact: "If you don't run experiments before you start designing a new system, your entire system will be an experiment." -- Mike Williams
    • Scott Aaronson: our laws of physics are structured in such a way that even pure information often has “nowhere to hide”: if the bits are there at all in the abstract machinery of the world, then they’re forced to pipe up and have a measurable effect. 
    • The Internet said many more interesting things this week. To read them all please click through to the full article.

  • Cool interview with Margaret Hamilton--NASA's First Software Engineer--on Makers. Programmers, you'll love this. One of the stories she tells is how her daughter was playing around and selected the prelaunch program during flight. That crashed the simulator. So like a good programmer she wanted to prevent this from happening. She tried to get a protection put in because an astronaut could actually do this during flight. Management would certainly allow this, right? She was denied. They said astronauts are trained never to make a mistake so it could never happen. Eventually she won the argument and was able to add code to protect against human error. So little has changed :-)

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


7 Interesting Parallels Between the Invention of Tiny Satellites and Cloud Computing 


CubeSats are revolutionizing space exploration because they are small, modular, and inexpensive to build and launch. On an episode of, Professor Jordi Puig-Suari gives a fascinating interview on the invention of the CubeSat. 195: A BUNCH OF SPUTNIKS.

What struck me in the interview is how the process of how the CubeSat was invented parallels how the cloud developed. They followed a very similar path driven by many of the same forces and ideas. 

Just what is a CubeSat? It's a "type of miniaturized satellite for space research that is made up of multiples of 10×10×10 cm cubic units. CubeSats have a mass of no more than 1.33 kilograms per unit, and often use commercial off-the-shelf (COTS) components for their electronics and structure."

Parallel #1:  University as Startup Incubator

Click to read more ...


Stuff The Internet Says On Scalability For July 21st, 2017

Hey, it's HighScalability time:

Afraid of AI? Fire ants have sticky pads so they can form rafts, build towers, cross streams, & order takeout. We can CRISPR these guys to fight Skynet. (video, video, paper)

If you like this sort of Stuff then please support me on Patreon.


  • 222x: Bitcoin less efficient than a physical system of metal coins and paper/fabric/plastic; #1: Python use amongst Spectrum readers; 3x: time spent in apps that don't make us happy; 1 million: DigitalOcean users; 11.6 million: barrels of oil a day saved via tech and BigData; 200,000: cores on Cray super computer;$200B: games software/hardware revenue by 2021; $3K: for 50 Teraflops AMD Vega Deep Learning Box; 24.4 Gigawatts: China New Solar In First Half Of 2017; 

  • Quotable Quotes:
    • sidlls: I think instead there is a category error being made: that CS is an appropriate degree (on its own) to become a software engineer. It's like suggesting a BS in Physics qualifies somebody to work as an engineer building a satellite.
    • Elon Musk: AI is a fundamental existential risk for human civilization, and I don’t think people fully appreciate that
    • Mike Elgan: Thanks to machine learning, it's now possible to create a million different sensors in software using only one actual sensor -- the camera.
    • Amin Vahdat (Google): The Internet is no longer about just finding a path, any path, between a pair of servers, but actually taking advantage of the rich connectivity to deliver the highest levels of availability, the best performance, the lowest latency. Knowing this, how you would design protocols is now qualitatively shifted away from pairwise decisions to more global views.
    • naasking: You overestimate AI. Incompleteness is everywhere in CS. Overcoming these limitations is not trivial at all.
    • 451: Research believes serverless is poised to undergo a round of price cutting this year.
    • Nicholas Bloom: We found massive, massive improvement in performance—a 13% improvement in performance from people working at home
    • @CoolSWEng: "A Java new operation almost guarantees a cache miss. Get rid of them and you'll get C-like performance." - @cliff_click #jcrete
    • DarkNetMarkets: We're literally funding our own investigation. 
    • Tristan Harris: By shaping the menus we pick from, technology hijacks the way we perceive our choices and replaces them with new ones. But the closer we pay attention to the options we’re given, the more we’ll notice when they don’t actually align with our true needs.
    • xvaier: If I have one thing to tell anyone who is looking for business ideas to try out their new programming skills on, I strongly suggest taking the time to learn as much as possible about the people to whom you want to provide a solution, then recruiting one of them to help you build it, lest you become another project that solves a non-issue beautifully.
    • @sebgoa: Folks, there were schedulers before kubernetes. Let's get back down to earth quickly
    • Mark Shead: A finite state machine is a mathematical abstraction used to design algorithms. In simple terms, a state machine will read a series of inputs. When it reads an input it will switch to a different state. Each state specifies which state to switch for a given input. This sounds complicated but it is really quite simple.
    • xantrel: I started a small business that started to grow, I thought I had to migrate to AWS and increase my cost by 5xs eventually, but so far Digital Ocean with their hosted products and block storage has handled the load amazingly well.
    • danluu: when I’m asked to look at a cache related performance bug, it’s usually due to the kind of thing we just talked about: conflict misses that prevent us from using our full cache effectively6. This isn’t the only way for that to happen – bank conflicts and and false dependencies are also common problems
    • Charles Hoskinson: People say ICOs (Initial Coin Offering) are great for Ethereum because, look at the price, but it’s a ticking time-bomb. There’s an over-tokenization of things as companies are issuing tokens when the same tasks can be achieved with existing blockchains. People are blinded by fast and easy money.
    • Charles Schwab: There don't seem to be any classic bubbles near bursting at the moment—at least not among the ones most commonly referenced as potential candidates.
    • Sertac Karaman: We are finding that this new approach to programming robots, which involves thinking about hardware and algorithms jointly, is key to scaling them down.
    • Michael Elling: When do people wake up and say that we’ve moved full circle back to something that looks like the hierarchy of the old PSTN? Just like the circularity of processing, no?
    • Benedict Evans: Content and access to content was a strategic lever for technology. I’m not sure how much this is true anymore.  Music and books don’t matter much to tech anymore, and TV probably won’t matter much either. 
    • SeaChangeViaExascaleOnDown: Currently systems are still based around mostly separately packaged processor elements(CPUs, GPUs, and other) processors but there will be an evolution towards putting all these separate processors on MCMs or Silicon Interposers, with silicon interposers able to have the maximum amount of parallel traces(And added active circuitry) over any other technology.
    • BoiledCabbage: Call me naive, but am I the only one who looks at mining as one of the worst inventions for consuming energy possible?
    • Amin Vahdat (Google):  Putting it differently, a lot of software has been written to assume slow networks. That means if you make the network a lot faster, in many cases the software can’t take advantage of it because the software becomes the bottleneck.

  • Dropbox has 1.3 million lines of Go code, 500 million users, 500 petabytes of user data, 200,000 business customers, and a multi-exabyte Go storage system. Go Reliability and Durability at Dropbox. They use it for: RAT: rate limiting and throttling; HAT: memcached replacement; AFS: file system to replace global Zookeeper; Edgestore: distributed database; Bolt: for messaging; DBmanager: for automation and monitoring of Dropbox’s 6,000+ databases; “Jetstream”, “Telescope”, block routing, and many more. The good: Go is productive, easy to write and consume services, good standard library, good debugging tools. The less good: dealing with race conditions.

  • Professor Jordi Puig-Suari talks about the invention of CubeSat on 195: A BUNCH OF SPUTNIKS. Fascinating story of how thinking different created a new satellite industry. The project wasn't on anyone's technology roadmap, nobody knew they needed it, it just happened. A bunch of really bright students, in a highly constrained environment, didn't have enough resources to do anything interesting, so they couldn't build spacecraft conventionally. Not knowing what you're doing is an advantage in highly innovative environments. The students took more risk and eliminated redundancies. One battery. One radio. Taking a risk that things can go wrong. They looked for the highest performance components they could find, these were commercial off the shelf components that when launched into space actually worked. The mainline space industry couldn't take these sort of risks. Industry started paying attention because the higher performing, lower cost components, even with the higher risk, changed the value proposition completely. You can make it up with numbers. You can launch 50 satellites for the cost of one traditional satellite. Sound familiar? Cloud computing is based on this same insight. Modern datacenters have been created on commodity parts and how low cost miniaturized parts driven by smartphones have created whole new industries. CubeSats' had a standard size, so launch vehicles could standardize also, it didn't matter where the satellites came from, they could be launched. Sound familiar? This is the modularization of the satellite launching, the same force that drives all mass commercialization. Now the same ideas are being applied to bigger and bigger spacecraft. It's now a vibrant industry. Learning happens more quickly because they get to fly more. Sound familiar? Agile, iterative software development is the dominant methodology today. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


You'll Never Believe the Big Hairy Audacious Startup John Jacob Astor Created in 1808


Think your startup has a Big Hairy Audacious Goal? Along with President Thomas Jefferson, John Jacob Astor  conceived (in 1808), and implemented (in 1810) plan to funnel the entire tradable wealth of the westernmost sector of the North American continent north of Mexico through his own hands. Early accounts described it as “the largest commercial enterprise the world has ever known.”

Think your startup raised a lot of money? Astor put up $400,000 ($7,614,486 in today's dollars) of his own money, with more committed after the first prototype succeeded.

Think competition is new? John Jacob Astor dealt with rivals in one of three ways: he tried to buy them out; if that didn’t work, he tried to partner with them; if he failed to join them, he tried to crush them.

Think your startup requires commitment? Joining Astor required pledging five years of one’s life to a start-up venture bound for the unknownn.

Think your startup works hard? Voyageur's paddled twelve to fifteen hours per day, with short breaks while afloat for a pipe of tobacco. During that single day each voyageur would make more than thirty thousand paddle strokes. On the upper Great Lakes, the canoes traversed hundreds of miles of empty, forested shorelines and vast stretches of clear water without ports or settlements or sails, except for the scattered Indian encampment.

Think your product is complex? Astor planned, manned and outfitted one overseas and two overland expeditions to build the equivalent of a Jamestown settlement on the Pacific Coast.

Think your startup parties hard? Every nook and corner in the whole island swarmed, at all hours of the day and night, with motley groups of uproarious tipplers and whisky-hunters. It resembled a great bedlam, the frantic inmates running to and fro in wild forgetfulness. Many were eager for company and with a yen to cut loose—drinking, dancing, singing, whoring, fighting, buying knickknacks and finery from the beach’s shacks and stalls. 

Think your startup was an adventure you can never forget? I have been twenty-four years a canoe man, and forty-one years in service; no portage was ever too long for me. Fifty songs could I sing. I have saved the lives of ten voyageurs. Have had twelve wives and six running dogs. I spent all my money in pleasure. Were I young again, I should spend my life the same way over. There is no life so happy as a voyageur’s life!

Think people at your startup dress weird? Above the waist, the voyageurs wore a loose-fitting and colorful plaid shirt, perhaps a blue or red, and over it, depending on the weather, a long, hooded, capelike coat called a capote. In cold winds they cinched this closed with a waist sash—the gaudier the better, often red. From the striking sash dangled a beaded pouch that contained their fire-making materials and tobacco for their “inevitable pipe.”...The true “Man of the North” wore a brightly colored feather in his cap to distinguish himself from the rabble.

Think your startup takes risks? Half of them died.

And like most startups, they accomplished a lot, but ultimately failed to earn a payout.

Thomas Jefferson said to John Jacob Astor: Your name will be handed down with that of Columbus & Raleigh, as the father of the establishment and the founder of such an empire. Unfortunately, not so much Tom. How many have heard of Astor today? Not many, unless you've traveled to Astoria, Oregon. Astoria in the right weather is a gorgeous place with a hot beer scene.

It's trite to say the reward is in the journey, but in this case the saying is true, the journey was larger than digital life.

For the complete story read: Astoria: John Jacob Astor and Thomas Jefferson's Lost Pacific Empire: A Story of Wealth, Ambition, and Survival.


Stuff The Internet Says On Scalability For July 14th, 2017

Hey, it's HighScalability time:



We've seen algorithms expressed in seeds. Here's an algorithm for taking birth control pills expressed as packaging. Awesome history on 99% Invisible.

If you like this sort of Stuff then please support me on Patreon.


  • 2 trillion: web requests served daily by Akamai; 9 billion: farthest star ever seen in light-years; 10^31: bacteriophages on earth; 7: peers needed to repair ransomware damage; $30,000: threshold of when to leave AWS; $300K-$400K: beginning cost of running Azure Stack on HPE ProLiant; 3.5M: files in the Microsoft's git repository; 300M: Google's internal image data training set size; 7.2 Mbps: global average connection speed; 85 million: Amazon Prime members; 35%: Germany generated its electricity from renewables;

  • Quotable Quotes:
    • Jessica Flack: I believe that science sits at the intersection of these three things — the data, the discussions and the math. It is that triangulation — that’s what science is. And true understanding, if there is such a thing, comes only when we can do the translation between these three ways of representing the world.
    • gonchs: “If your whole business relies on us [Medium], you might want to pick a different one”
    • @AaronBBrown777: Hey @kelseyhightower, if you're surfing GitHub today, you might find it interesting that all your web bits come thru Kubernetes as of today.
    • Psyblog: The researchers were surprised to find that a more rebellious childhood nature was associated with a higher adult income.
    • Antoine de Saint-Exupéry: If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
    • Marek Kirejczyk: In general I would say: if you need to debug — you’ve already lost your way.
    • jasondc: To put it another way, RethinkDB did extremely well on Hacker News. Twitter didn't, if you remember all the negative posts (and still went public). There is little relation between success on Hacker News and company success.
    • Rory Sutherland: What intrigues me about human decision making is that there seems to be a path-dependence involved - to which we are completely blind.
    • joeblau: That experience taught me that you really need to understand what you're trying to solve before picking a database. Mongo is great for some things and terrible for others. Knowing what I know now, I would have probably chosen Kafka.
    • 0xbear: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator
    • Dissolving the Fermi Paradox: Doing a distribution model shows that even existing literature allows for a substantial probability of very little life, and a more cautious prior gives a significant probability for rare life
    • Peter Stark: Crews with clique structures report significantly more depression, anxiety, anger, fatigue and confusion than crews with core-periphery structures.
    • Patrick Marshall: Gu said that the team expects to have a prototype [S2OS’s software-defined hypervisor is being designed to centrally manage networking, storage and computing resources] ready in about three years that will be available as open-source software.
    • cobookman: I've been amazed that more people don't make use of googles preemtibles. Not only are they great for background batch compute. You can also use them for cutting your stateless webserver compute costs down. I've seen some people use k8s with a cluster of preemtibles and non preemtibles. 
    • @jeffsussna: Complex systems can’t be fully modeled. Failure becomes the only way to fully discover requirements. Thus the need to embrace it.
    • Jennifer Doudna: a genome’s size is not an accurate predictor of an organism’s complexity; the human genome is roughly the same length as a mouse or frog genome, about ten times smaller than the salamander genome, and more than one hundred times smaller than some plant genomes.
    • Daniel C. Dennett: In Darwin’s Dangerous Idea (1995), I argued that natural selection is an algorithmic process, a collection of sorting algorithms that are themselves composed of generate-and-test algorithms that exploit randomness (pseudo-randomness, chaos) in the generation phase, and some sort of mindless quality-control testing phase, with the winners advancing in the tournament by having more offspring.
    • Almir Mustafic: My team learned the DynamoDB limitations before we went to production and we spent time calculating things to properly provision RCUs and WCUs. We are running fine in production now and I hear that there will be automatic DynamoDB scaling soon. In the meantime, we have a custom Python script that scales our DynamoDB.

  • I've written a novella: The Strange Trial of Ciri: The First Sentient AI. It explores the idea of how a sentient AI might arise as ripped from the headlines deep learning techniques are applied to large social networks. I try to be realistic with the technology. There's some hand waving, but I stay true to the programmers perspective on things. One of the big philosophical questions is how do you even know when an AI is sentient? What does sentience mean? So there's a trial to settle the matter. Maybe. The big question: would an AI accept the verdict of a human trial? Or would it fight for its life? When an AI becomes sentient what would it want to do with its life? Those are the tensions in the story. I consider it hard scifi, but if you like LitRPG there's a dash of that thrown in as well. Anyway, I like the story. If you do too please consider giving it a review on Amazon. Thanks for your support!

  • Serving 39 Million Requests for $370/Month, or: How We Reduced Our Hosting Costs by Two Orders of Magnitude. Step 1: Just Go Serverless: Simply moving to a serverless environment had the single greatest impact on reducing hosting costs. Our extremely expensive operating costs immediately shrunk by two orders of magnitude. Step 2: Lower Your Memory Allocation: Remember, each time you halve your function’s memory allocation, you’re roughly halving your Lambda costs. Step 3: Cache Your API Gateway Responses: We pay around $14 a month for a 0.5GB API Gateway cache with a 1 hour TTL. In the last month, 52% (20.3MM out of 39MM) of our API requests were served from the cache, meaning less than half (18.7MM requests) required invoking our Lambda function. That $14 saves us around $240 a month in Lambda costs.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

  • Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.

  • Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While there are similarities between data science and software engineering, well intentioned engineering leaders may make assumptions about data science that lead to avoidable conflict and unproductive workflows. Read the full guide to data science for Engineering and IT leaders.

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Etleap provides a SaaS ETL tool that makes it easy to create and operate a Redshift data warehouse at a small fraction of the typical time and cost. It combines the ability to do deep transformations on large data sets with self-service usability, and no coding is required. Sign up for a 30-day free trial.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • : Monitor End User Experience from a global monitoring network. 

  • Working on a software product? Clubhouse is a project management tool that helps software teams plan, build, and deploy their products with ease. Try it free today or learn why thousands of teams use Clubhouse as a Trello alternative or JIRA alternative.

  • Build, scale and personalize your news feeds and activity streams with Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here:

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today

VividCortex Gives You Database Superpowers 

Database monitoring is hard, but VividCortex makes it easy. Modern apps run complex queries at large scales across distributed, diverse types of databases (e.g. document, relational, key-value). The “data tier” that these become is tremendously difficult to observe and measure as a whole. The result? Nobody knows exactly what’s happening across all those servers.

VividCortex lets you handle this complexity like a superhero. With VividCortex, you're able to inspect all databases and their workloads together from a bird's eye view, spotting problems and zooming down to individual queries and 1-second-resolution metrics in just two mouse clicks. With VividCortex, you gain a superset of system-monitoring tools that use only global metrics (such as status counters), offering deep, multi-dimensional slice-and-dice visibility into queries, I/O, and CPU, plus other key measurements of your system's work. VividCortex is smart, opinionated, and understands databases deeply, too: it knows about queries, EXPLAIN plans, SQL bugs, typical query performance, and more. It empowers you to find and solve problems that you can't even see with other types of monitoring systems, because they don’t measure what matters to the database.

Best of all, VividCortex isn’t just a DBA tool. Unlike traditional monitoring tools, VividCortex eliminates the walls between development and production -- anybody can benefit from VividCortex and be a superhero. With powerful tools like time-over-time comparison, developers gain immediate production visibility into databases with no need for access to the production servers. Our customers vouch for it: Problems are solved faster and they're solved before they ever reach production. As a bonus, DBAs no longer become the bottleneck for things like code reviews and troubleshooting in the database. 

Make your entire team into database superheros: deploy VividCortex today. It only takes a moment, and you’re guaranteed to immediately learn something you didn’t already know about your databases and their workload. 

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...


Stuff The Internet Says On Scalability For July 7th, 2017

Hey, it's HighScalability time:



What's real these days? I was at Lascaux II, an exact replica of Lascaux. I was deeply, deeply moved. Was this an authentic experience? A question we'll ask often in VR I think.

If you like this sort of Stuff then please support me on Patreon.
  • $400k: cost of yearly fake news campaign; $50,000: cost to discredit a journalist; 100 Gbps: SSDP DDoS amplification attack; $5.97BN: wild guess on cost of running Facebook on AWS; 2 billion: Facebook users; 80%: Spotify backend services in production run as containers; $60B: AR market by 2021; 10.4%: AMD market share taken from Intel; 5 days: MIT drone flight time; $1 trillion: Apple iOS revenues; 35%-144%: reduction in image sizes; 10 petabytes: data stored; 1 trillion: photos taken on iPhone each year; $70B: Apple App Store payout to developers; 355: pages in Internet Trends 2017 report; 14: people needed to make 500,000 tons of steel; 25%: reduced server-rendering time with Node 8; 50-70%: of messages Gmail receives are spam; 8,000: bugs found in pacemaker code; 

  • Quotable Quotes:
    • Vladimir Putin: We must take into account the plans and directions of development of the armed forces of other countries… Our responses must be based on intellectual superiority, they will be asymmetric, and less expensive.
    • @swardley: What most fail to realise is that the Chinese corporate corpus has devoured western business thinking and gone beyond it.
    • @discostu105: I am a 10X developer. Everything I do takes ten times as long as I thought.
    • DINKDINK: You grossly underestimate the hashing capacity of the bitcoin network. The hashing capacity, at time of posting, is approximately 5,000,000,000 Gigahashes/second[1]. Spot measurement of the hashing capacity of an EC2 instance is 0.4 Gigahashes/second[2]. You would need 12 BILLION EC2 instances to 51% attack the bitcoin network.[3] Using EC2 to attack the network is impractical and inefficient.
    • danielsamuels && 19eightyfour~ Machiavelli's Guide to PaaS: Keep your friends close, and your competitors hosted.
    • Paul Buchheit:  I wrote the the first version of Gmail in a day!
    • @herminghaus: If you don’t care about latency, ship a 20ft intermodal container full of 32GB micro-SD cards across the globe. It’s a terabyte per second.
    • @cstross: Okay, so now the Russian defense industry is advertising war-in-a-can (multimodal freight containerized missiles):
    • Dennett~ you don't need comprehension to achieve competence.
    • @michellebrush~ Schema are APIs. @gwenshap #qconnyc
    • Stacy Mitchell: Amazon sells more clothing, electronics, toys, and books than any other company. Last year, Amazon captured nearly $1 of every $2 Americans spent online. As recently as 2015, most people looking to buy something online started at a search engine. Today, a majority go straight to Amazon.
    • Xcelerate: I have noticed that Azure does have a few powerful features that AWS and GCP lack, most notably InfiniBand (fast interconnects), which I have needed on more than one occasion for HPC tasks. In fact, 4x16 core instances on Azure are currently faster at performing molecular dynamics simulations than 1x"64 core" instance on GCP. But the cost is extremely high, and I still haven't found a good cloud platform for short, high intensity HPC tasks.
    • jjeaff: I took about 5 sites from a $50 a month shared cPanel plan that included a few WordPress blogs and some custom sites and put them on a $3 a month scaleway instance and haven't had a bit of trouble.
    • @discordianfish: GCP's Pub/Sub is really priced by GB? And 10GB/free/month? What's the catch?
    • Amazon: This moves beyond the current paradigm of typing search keywords in a box and navigating a website. Instead, discovery should be like talking with a friend who knows you, knows what you like, works with you at every step, and anticipates your needs. This is a vision where intelligence is everywhere. Every interaction should reflect who you are and what you like, and help you find what other people like you have already discovered. 
    • @CloudifySource: Lambda is always 100% busy - @adrianco #awasummit #telaviv #serverless
    • @codinghorror: Funny how Android sites have internalized this "only multi core scores now matter" narrative with 1/2 the CPU speed of iOS hardware
    • @sheeshee: deleted all home directories because no separation of "dev" & "production". almost ran a billion euro site into the ground with a bad loop.
    • We have quotes the likes of which even God has never seen! Please click through to ride all of them.

  • The Not Hotdog app on Silicon Valley may be a bit silly, but the story of how they built the real app is one of the best how-tos on building a machine learning app you'll ever read. How HBO’s Silicon Valley built “Not Hotdog” with mobile TensorFlow, Keras & React Native. The initial app was built in a weekend using Google Cloud Platform’s Vision API, and React Native. The final version took months of refinement.  Google Cloud’s Vision API was dropped because its accuracy in recognizing hotdogs was only so-so; it was slow because of the network hit; it cost too much. They ended up using Keras, a deep learning library that provides nicer, easier-to-use abstractions on top of TensorFlow. They used on SqueezeNet due to its explicit positioning as a solution for embedded deep learning. SqueezeNet used only 1.25 million parameters which made training much faster and reduced resource usage on the device. What would they change? timanglade: Honestly I think the biggest gains would be to go back to a beefier, pre-trained architecture like Inception, and see if I can quantize it to a size that’s manageable, especially if paired with CoreML on device. You’d get the accuracy that comes from big models, but in a package that runs well on mobile. And this is really cool: The last production trick we used was to leverage CodePush and Apple’s relatively permissive terms of service, to live-inject new versions of our neural networks after submission to the app store. 

  • And the winner is: all of us. Serverless Hosting Comparison: Lambda: Unicorn: $20,830.83. Heavy: $120.16. Medium: $4.55. Light: $0.00; Azure Functions: Unicorn: $19,993.60. Heavy: $115.40. Moderate: $3.60. Light: $0.00; Cloud Functions: Unicorn: $23,321.20. Heavy: $138.95. Moderate: $9.76. Light: $0.00; OpenWhisk: Unicorn: $21,243.20. Heavy: $120.70. Medium: $3.83. Light: $0.00; depends on the cost of running your managed Kubernetes cloud. 

  • Minds are algorithms made physical. Seeds May Use Tiny “Brains” to Decide When to Germinate: The seed has two hormones: abscisic acid (ABA), which sends the signal to stay dormant, and gibberellin (GA), which initiates germination. The push and pull between those two hormones helps the seed determine just the right time to start growing...According to Ghose, some 3,000 to 4,000 cells make up the Arabidopsis seeds...It turned out that the hormones clustered in two sections of cells near the tip of the seed—a region the researchers propose make up the “brain.” The two clumps of cells produce the hormones which they send as signals between each other. When ABA, produced by one clump, is the dominate hormone in this decision center, the seed stays dormant. But as GA increases, the “brain” begins telling the seed it’s time to sprout...This splitting of the command center helps the seed make more accurate decisions.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...


What is NASA Doing with Big Data? Check this Out


Within the time you read the above sentence, NASA could have collected 1.73 gigabytes of data from around 100 missions which are active currently. NASA doesn’t stop doing this and the rate of collection is growing in an exponential manner. So, managing this kind of data is an uphill task for them. But the data which NASA collects is highly precious and its significance is immense in NASA’s science and research. NASA is trying extremely hard to make this data as approachable and accessible as possible for their daily tasks, various predictions in the universe, and for the human well-being through its innovations and creativity.

In version 2.0 of their “Open Government Plan” in the year 2012, NASA discussed, but did not go deeply into the work they have been doing regarding “Big Data” and they believed that they have much more to explore in this field.

We all know what big data is and what its uses are. So, I don’t think there is any need to mention what really big data is and let’s move on with other topic.

NASA’s Big Data Challenge

Click to read more ...


Gone Fishin'

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. Au revoir!


Stuff The Internet Says On Scalability For May 26th, 2017

Hey, it's HighScalability time:



Sport imitating tech. Cloud Computing chases down Classic Empire to win...the Preakness. (Daily News)

If you like this sort of Stuff then please support me on Patreon.
  • 42%: increase US wireless traffic since 2015; 44: age of Ethernet; $18.5m: low cost of Target data breach; 25 million: record set from Library of Congress; 98%: WannaCry infections on Windows 7; 100 terabytes: daily Pinterest logging; 2020: when Microsoft will have DNA storage in the cloud; 220 μm: size of microbots; 2 billion: lines of code in Google repository; 40%+: esports industry growth; 

  • Quotable Quotes:
    • @Werner: There is no compression algorithm for experience.
    • @colinmckerrache: We just crossed over 2m EVs on the road. So yeah, second million took just under 18 months. Next million in about 10 months.
    • @swardley: When discussing China, stop thinking cheap labour, communism & copying ... to understand changes, start thinking World's largest VC.
    • @JOTB17: "Cars generate more than 4Tb of data a day, humans are becoming irrelevant in data collection" 😳 @saleiva #JOTB17
    • Wojciech Kudla: that's why blacklisting workqueues from critical cpus should be on the jitter elimination check list. They can be affinitized just like irqs
    • @ryanhuber: Any sufficiently advanced attacker is indistinguishable from one of your developers.
    • @spolsky: "During peak traffic hours on weekdays, there are about 80 people per hour that need help getting out of Vim."
    • SrslyJosh: Basing anything on proof-of-work puts you in a perpetual race to control more compute than your adversaries.
    • gkoberger: So, in my mind, Mozilla won. It's a non-profit, and it forced us into an open web. We got the world they wanted. Maybe the world is a bit Chrome-heavy currently, but at least it's a standards compliment world.
    • NoGravitas: The basic argument of this article seems to be that the real benefit of cryptocurrencies, other than their speculative value, is that they provide a way of enforcing artificial scarcity in the digital realm, where scarcity does not come naturally.
    • Renee DiResta: The trouble is that “high-frequency trading” is about as precise as “fake news.”
    • Silicon Valley: I mean, that Ken doll probably thinks traversing a binary search tree runs in the order of "n," instead of "log n." Idiot.
    • @__apf__: If you look at another engineer's work and think, "That's dumb. Why don't you just..." Take a breath. Find out why the problem is hard.
    • Too many quotes. Please click through to read the full article.

  • Failing Kubernetes pods by playing whack-a-mole is an awesome idea. Funner than a barrel of chaos monkeys. You just have to see the video

  • There are times when specialized hardware absolutely destroys commodity hardware. TensorFlow Frontiers. The need for Google to create the TPU became urgent in 2013 when it was realized if all Android users spoke to their phone for just three minutes a day it might force Google to double its number of datacenters. That drove a crash program to develop the first TPU. The first-gen TPU was 15-30x faster than contemporary CPUs & GPUs, 30-80x more power-efficient, but it only worked for inference, not training. The second-gen TPU has up to 180 teraflops of floating point performance, 64 GB of ultra-high-bandwidth memory, works for both training and inference (simpler to use), and can be connected together using a 2-D toroidal mesh network (tackle largest problems). On one problem training time was reduced from 24 hours to 6 hours.

  • Another victim of Stacked ranking. T.J. Miller Is Leaving Silicon Valley.

  • The biggest every day risk from the massive data surveillance panopticon carried out by private corporations is not storm troopers busting down your door, it's this: everything will start costing you more. Whenever an algorithm calculates it has leverage over you it will exploit that advantage to charge you more. A computer mediated personalized world will anticipate your needs, but it will also invisibly shape them. What is being created is the ultimate Skinner Box. Uber Is Using AI to Charge People as Much as Possible for a Ride

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...