advertise
Tuesday
Sep132016

If Traffic is an Iterated Prisoner's Dilemma Game Can Smart Cars Evolve Co-operative Behavior?

 

Can small tribes of cooperating smart cars improve overall traffic even if they are not in the majority? Sure, if every car was a self-driving car maybe traffic jams could dissolve like blood clots on anticoagulants, but what about that messy in-between period? It will be some time before smart cars rule the road. Until then can smart cars make traffic better?

Adoption is hard. This is a general problem in tech. You want people to join your social network yet people won't join until enough people have already joined. What you really want is that virtuous circle to develop, where as more people adopt a technology it causes even more people to adopt it. So startups spend their VC money fast and furiously in hopes of acquiring new customers betting the lifetime value of a customer will be worth the investment. VC money is the dead corpse that feeds the rest of the ecosystem.

Traffic is already an example of a vicious cycle. Horrendous traffic jams are now the norm and "good" traffic windows are just tall tales texted to children. And it keeps on getting worse and not in a worse is better sort of way. Yet the incentives are still not enough for people to self-organize and batch themselves into cars. Cars are more of a synchronous streaming model. Traffic problems will need to be solved at a different level of abstraction. Human drivers are just so hopelessly human.

In some ways traffic is like an iterated game of Prisoner's Dilemma. So in an Evolution of Cooperation sense can overall flows improve if groups of self-driving cars cooperate together within a stream of muggle cars? If smart cars on the road choose to gang up together will that improve commute times in such a way that it will encourage more and more cars to join the gang, becoming part of the solution instead of the problem?

But we have the social network problem. Cars currently are individual, kept in silos organized by manufacturer. Tesla, Uber, Google, etc. don't cooperate at a global traffic planning level. Even cars within a manufacturer don't yet have the ability to slave themselves together in a self-driving conga line of traffic goodness.

Historically we know after individual point solutions are created the next step is to add a scheduling layer. After running a program on an entire CPU we create an OS (Linux, Windows, etc) to run multiple programs on the same CPU. After the container we create an OS (Swarm, Kubernetes, Mesos, etc) to run multiple programs on the same boxes.

We'll need a TrafficOS so all the cars that want to can cooperate together, you know like XMPP before the walls went up. Plus we'll need ecosystem incentives to help drive adoption. 

So many questions. Will drivers volunteer to be part of a smart car peloton even if it means their commute suffers in the short term? What's the tipping point? Will free riders ruin the whole thing? Like the fast lane, should incentives be created to encourage cooperating tribes of smart cars? Should traffic lights favor smart car trains? Should traffic laws allow bullet trains of smart cars to speed down the highway? Should insurance premiums be reduced for time spent protected in smart car convoys? Maybe smart car software should be seeded with altruism "genes" so they cooperate naturally? How can defectors be punished? Maybe we need a reputation system scoring for traffic reciprocity?

Unlike the weather traffic is something we can do something about. Let's just try to do a better job than we did with social networks and IM systems. Traffic is actually important.

Related Articles

Tuesday
Sep132016

Sponsored Post: ScaleArc, Spotify, Aerospike, Scalyr, Gusto, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

  • Spotify is looking for individuals passionate in infrastructure to join our Site Reliability Engineering organization. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. We are looking for an SRE Engineering Manager in NYC and SREs in Boston and NYC.

  • IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.

Fun and Informative Events

  • Learn how Nielsen Marketing Cloud (NMC) leverages online machine learning and predictive personalization to drive its success in a live webinar on Tuesday, September 20 at 11 am PT / 2 pm ET. Hear from Nielsen’s Kevin Lyons, Senior VP of Data Science and Digital Technology, and Brent Keator, VP of Infrastructure, as well as from Brian Bulkowski, CTO and Co-Founder at Aerospike, as they describe the front-edge architecture and technical choices – including the Aerospike NoSQL database – that have led to NMC’s success. RSVP: https://goo.gl/xDQcu4

Cool Products and Services

  • ScaleArc's database load balancing software empowers you to “upgrade your apps” to consumer grade – the never down, always fast experience you get on Google or Amazon. Plus you need the ability to scale easily and anywhere. Find out how ScaleArc has helped companies like yours save thousands, even millions of dollars and valuable resources by eliminating downtime and avoiding app changes to scale. 

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Click to read more ...

Tuesday
Sep132016

The Dollar Shave Club Architecture Unilever Bought for $1 Billion

This is a guest post by Jason Bosco, the Dollar Shave Club’s Director of Engineering, Core Platform & Infrastructure, on the infrastructure of its ecommerce technology.

With more than 3 million members, Dollar Shave Club will do over $200 million in revenue this year. Although most are familiar with the company’s marketing, this immense growth in just a few years since launch is largely due to its team of 45 engineers.

Dollar Shave Club engineering by the numbers:

Core Stats

Click to read more ...

Thursday
Sep082016

Stuff The Internet Says On Scalability For September 9th, 2016 

Hey, it's HighScalability time:

 

An alternate universe where Zeppelins rule the sky. 1929. (@AeroDork)

 

If you like this sort of Stuff then please support me on Patreon.
  • 15%: Facebook's reduction in latency using HTTP2's server push; 1.9x: nanotube transistors outperform silicon; 200: projectors used to film a "hologram"; 50%: of people fall for phishing attacks (it's OK to click); 5x: increased engagement using Google's Progressive Web Apps; 115,000+: Cassandra nodes at Apple; $500 million: Pokémon Go; $150M: Delta's cost for datacenter outage; 

  • Quotable Quotes: 
    • Dan Lyons: I wanted to write a book about what it’s like to be 50 and trying to reinvent yourself – that struggle. There are all these books and inspirational speakers talking about being a lifelong learner and it’s so great to reinvent yourself, the brand of you. And I wanted to say, you know, it’s not like that. It’s actually really painful.
    • Engineers & Coffee~ In modern application development everything is a stream now versus historically everything was a transaction. Make a request and the you're done. It's easier to write analytics on top of streams versus using Hive. It's cool that Kinesis is all real-time and has the power of SQL.
    • David Smith: The [iOS] market has been pulling me along towards advertising based apps, and I’ve found that the less I fight back with anachronistic ideas about how software “should” be sold, the more sustainable a business I have.
    • @tef_ebooks: (how do you keep a lisp user in suspense
    • @bodil: Use tests to verify your assumptions. Use a type checker to verify your implementations. Always.
    • tostitos1979: Here is a factoid for the youngins ... the Internet/Arpanet was created BEFORE the first microprocessor! In fact, Intel was originally founded to make RAM ICs. They only later created the first microprocessor (the 4004)!
    • gsubes:  Our tests showed than even with larger messages (100k price ticks per request) pipes were still a magnitude slower [than Memory Mapping].
    • Quincy Larson: Did you know the average developer only get two hours of uninterrupted work done a day? They spend the other 6 hours in varying states of distraction.
    • StorageMojo: Achieving lower-than-DRAM pricing requires volume, and that’s where NRAM has a competitive advantage over, say, 3D XPoint. Processing can be done on today’s flash, DRAM or logic lines. NRAM processing only needs spin coating and patterning – as well as carbon nanotubes – which modern fabs all support.
    • Xiao Mina: We’ve seen this story before: as cost of production and distribution go down, the range of creativity goes up.
    • @clarkkaren: Give humans a system and they'll game it. The End.
    • Jim Starkey: AmorphousDB is my modest effort to question everything database.
      The best way to think about Amorphous is to envision a relational database and mentally erase the boxes around the tables so all records free float in the same space – including data and metadata.
    • @jdub: On Reddit: “What is the use of Elastic IPs, if I can use ELB or an Auto Scaling Group instead?” STUDENT, YOU HAVE ACHIEVED ZEN OF CLOUD.
    • @BenedictEvans: A key premise for the next decade: it's easier for software to enter other industries than for other industries to hire software people
    • @jasongorman: To clarify, "dependency injection" literally just means passing an object's collaborators as constructor/method params. That's all it is.
    • jackpeterfletch: Grand solution to world hunger, available on Kindle!
    • @swardley: Optimise flow.  Often when you examine flows then you’ll find bottlenecks, inefficiencies and profitless flows.  There will be things that you’re doing that you just don’t need to. Be very careful here to consider not only efficiency but effectiveness. 
    • @PatrickMcFadin: #uber is fully replicated and active-active to make sure you never get stranded. #cassandrasummit
    • @FSVO: A monk named Chaitin found an algorithm for expressing the complexity of sutras. His master commented, “This monk could be shorter.”
    • Dotzler: We [Firefox] can learn from the competition [Chrome]. The way they implemented multi-process is RAM-intensive, it can get out of hand. We are learning from them and building an architecture that doesn’t eat all your RAM. 
    • @hichaelmart: Although CPU bound calculations [on OpenWhisk] seem about 4x slower than Lambda, so not too bad. Lambda still the winner so far though.
    • Shel Kaphan: Okay, I’m going to be building this website to run a bookstore [Amazon] and I haven’t done that before but it doesn’t sound so hard. When I’m done with that I’m not sure what I’ll do.
    • sixhobbits: "Our logger failed silently" "Shouldn't that have been recorded somewhere?" "I guess it's turtles all the way down"
    • @xmal: Trying to explain that CRDT causal contexts are a natural evolution of TCP sequence numbering and vector clocks in reliable causal broadcast
    • Joi Ito: Just like it is impossible to make another Silicon Valley somewhere else, although everyone tries—after spending four days in Shenzhen, I’m convinced that it’s impossible to reproduce this ecosystem anywhere else.
    • @adriancolyer: "My claim is that it is possible to write grand programs, noble programs, truly magnificent ones..." Knuth 1974
    • @Excellion: According to legend, if you say Blockchain three times fast, your databases will magically become immutable & your company a fintech leader.
    • bec0: The world has changed. Dennard scaling has mostly been replaced. The economic Moore's Law has morphed. It had too...we have all gotten used to its benefits.
    • @cloud_opinion: 5 stages of Cloud Grief: It's not secure / It's someone's computer / We do private cloud / Hybrid cloud  / Lambda is full of servers anyway
    • @DDD_Borat: "Why you not like framework annotations in your code?" - "Would you put bumper sticker on a Ferrari?" Rofl
    • @robert_winslow: Slow software is your fault. These are the real speed limits: billions of CPU instructions, GBs of RAM access, 100k+ SSD I/Os... per second.
    • Walter Bentley: I am proud to say, OpenStack held up to the torment. Did not experience not one single API request failure throughout my numerous load tests — yet another proof point that OpenStack is ready for enterprise/production use.
    • @xaprb: Let's fork it, say the people who have never put their heart and 5 years of their life into a product only to watch someone else fork it.
    • @adrianco: People asking Docker to slow down is like OpenStack folks asking AWS to standardize and slow down.
    • @amcafee: "In 1974, it was illegal for an airline to charge < $1,442 for a flight between New York City and Los Angeles."
    • Fairly Nerdy: For most real world scenarios, where you are betting against the house which has a house edge, f* becomes negative, which means that you shouldn’t be playing that game.  Truthfully it means that you should take the other side of the wager, become the house, and make them bet against you!
    • Judd Kaiser: Experience shows that good scalability can be achieved on 10 GigE networking provided that you stay above about 50,000 cells per core. That means, for example, that a 20 M cell problem shows good scaling up to about 400 cores; beyond that, interprocess communication latency begins to dominate and scaling degrades.

  • Maybe the real reason Uber wants driverless cars is hiring, er...onboarding drivers from across the globe is a really tough problem to solve. Each location has their own processes and that kills scalability. Screening processes and regulations vary, some countries have a very long list of required documents, and onboarding flows vary. Here's the story: How Uber Engineering Massively Scaled Global Driver Onboarding. So you can't use the same app everywhere. The solution was, as it often is, is to go meta and dynamic: the onboarding state machine (OSM)  easily configure a set of steps for each onboarding process in each country, state, city, or any level of granularity we need, coupled with an event system that allows us to easily switch users from one step to another depending on their actions or input. The onboarding API can then easily query the OSM to know at which step in the process a user is.  Clients are now stateless,  responsible only for their UI, 100% of the business logic in the shared back end. They went from Flask to Tornado and a lighter version of their initial JSON schema architecture, where only data is passed to the client, not UI definitions.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Sep072016

Code Generation: The Inner Sanctum of Database Performance

This is guest post by Drew Paroski, architect and engineering manager at MemSQL. Previously he worked at Facebook and developed HHVM, the popular real-time PHP compiler used across the company’s web scale application.

Achieving maximum software efficiency through native code generation can bring superior scaling and performance to any database. And making code generation a first-class citizen of the database, from the beginning, enables a rich set of speed improvements that provide benefits throughout the software architecture and end-user experience.

If you decide to build a code generation system you need to clearly understand the costs and benefits, which we detail in this article. If you are willing to go all the way in the name of performance, we also detail an approach to save you time leveraging existing compiler tools and frameworks such as LLVM in a proven and robust way.

Code Generation Basics

Click to read more ...

Friday
Sep022016

Stuff The Internet Says On Scalability For September 2nd, 2016 

Hey, it's HighScalability time:

 

Spectacular iconic drawing of Aurora Borealis as observed in 1872. (Drawings vs. NASA Images)
  • 4,000 GB: projected bandwidth used per autonomous vehicle per day; 100K: photos of US national parks; 14 terabytes: code on Github in 1 billion files held in 400K repositories; 25: age of Linux; $5 billion: cost of labor for building Linux; $3800: total maintenance + repairs after 100K miles and 5 years of Tesla ownership; 2%: reduction in Arizona's economy by deporting all illegal immigrants; 15.49TB: available research data; 6%: book readers who are "digital only";

  • Quotable Quotes
    • @jennyschuessler: "Destroy the printing press, I beg you, or these evil men will triumph": Venice, 1473
    • @Carnage4Life: Biggest surprise in this "Uber for laundry" app shutting down is that there are still 3 funded startups in the space
    • @tlipcon: "backpressure" is right up there with "naming things" on the top 10 list of hardest parts of programming
    • cmcluck: Please consider K8s [kubernetes] a legitimate attempt to find a better way to build both internal Google systems and the next wave of cloud products in the open with the community. We are aware that we don't know everything and learned a lot by working with people like Clayton Coleman from Red Hat (and hundreds of other engineers) by building something in the open. I think k8s is far better than any system we could have built by ourselves. And in the end we only wrote a little over 50% of the system. Google has contributed, but I just don't see it as a Google system at this point.
    • looncraz: AMD is not seeking the low end, they are trying to redefine AMD as the top-tier CPU company they once were. They are aiming for the top and the bulk of the market.
    • lobster_johnson: Swarm is simple to the point of naivety.
    • @BenedictEvans: That is, vehicle crashes, >90% caused by human error & 30-40% by alcohol, cost $240bn & kill 30k each year just in the USA. Software please
    • @joshsimmons: "Documentation is like serializing your mental state." - @ericholscher, just one of many choice moments in here.
    • @ArseneEdgar: "better receive old data fast rather than new data slow"
    • @aphyr: hey if you're looking for a real cool trip through distributed database research, https://github.com/cockroachdb/cockroach/blob/develop/docs/design.md … is worth several reads
    • @pwnallthethings: It's a fact 0day policy-wonks consistently get wrong. 0day are merely lego bricks. Exploits are 0day chains. Mitigations make chains longer.
    • andrewguenther: Speaking of [Docker] 1.12, my heart sank when I saw the announcement. Native swarm adds a huge level of complexity to an already unstable piece of software. Dockercon this year was just a spectacle to shove these new tools down everyone's throats and really made it feel like they saw the container parts of Docker as "complete." 
    • @johnrobb: Foxconn just replaced 60,000 workers with robots at its Kushan facility in China.  600 companies follow suit.
    • @epaley: Well publicized - Uber has raised ~$15B. Yet the press is shocked @Uber is investing billions. Huh? What was the money for? Uber kittens?
    • Ivan Pepelnjak: One of the obsessions of our industry is to try to find a one-size-fits-everything solutions. It's like trying to design something that could be a mountain bike today and an M1 Abrams tomorrow. Reality doesn't work that way
    • There were so many good quotes this week that they wouldn't all fit here. Please see the full post to read all the wonderfulness.

  • This should concern every iPhone user. Total ownage.
    • Steve Gibson, Security Now 575, with a great explanation of Apple's previously unknown professional grade zero-day iPhone exploits, Pegasus & Trident, that use a chain of flaws to remotely jail break an iPhone. It's completely stealthy, surviving both reboots and upgrades. The exploits have been around for years and were only identified by accident. It's a beautiful hack.
    • Your phone is totally open and it happens just like in the movies: A user infected with this spyware is under complete surveillance by the attacker because, in addition to the apps listed above, it also spies on: Phone calls, Call logs,  SMS messages the victim sends or receives, Audio and video communications that (in the words a founder of NSO Group) turns the phone into a 'walkie­talkie'
    • Bugs happen in complicated software. Absolutely. But these exploits were for sale...for years. The companies that sell these exploits do not have to disclose them. Apple should be going to the open market and buying these exploits so they can learn about them and fix them. Apple should be outbidding everyone in their bug bounty system so they can find hacks and fix them.
    • Paying for exploits is not an ethical issue, it's smart business in a realpolitik world. If you can figure out the Double Irish With a Dutch Sandwich you can figure out how to go to the open market and find out all the ways you are being hacked. Apple needs to think about security strategically, not only as a tactical technical issue

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Aug312016

My Test Tube Filled with DNA is Better than Your Mesos Cluster

 

We’ve seen computation using slime mold, soap film, water droplets, there’s even a 10,000 Domino Computer. Now DNA can do math In a test tube. Using addition, subtraction, multiplication, and division.

It’s not fast. Calculations can take hours. The upside: they are tiny and can work in wet environments. Think of running calculations in your bloodstream or in cells, like a programmable firewall, to monitor and alert on targeted health metrics and then trigger a localized response. Or if you are writing  science fiction perhaps the ocean could become one giant computer?

The applications already sound like science fiction:

Prior devices for control of chemical reaction networks and DNA doctor applications have been limited to finite-state control, and analog DNA circuits will allow much more sophisticated analog signal processing and control. DNA robotics have allowed devices to operate autonomously (e.g., to walk on a nanostructure) but also have been limited to finite-state control.
Analog DNA circuits can allow molecular robots to include real-time analog control circuits to provide much more sophisticated control than offered by purely digital control. Many artificial intelligence systems (e.g., neural networks and probabilistic inference) that dynamically learn from environments require analog computation, and analog DNA circuits can be used for back-propagation computation of neural networks and Bayesian probabilistic inference systems.

How does it work?

Click to read more ...

Tuesday
Aug302016

The cat-and-mouse story of implementing anti-spam for Mail.Ru Group’s email service and what Tarantool has to do with this

Hey guys!

In this article, I’d like to tell you a story of implementing the anti-spam system for Mail.Ru Group’s email service and share our experience of using the Tarantool database within this project: what tasks Tarantool serves, what limitations and integration issues we faced, what pitfalls we fell into and how we finally arrived to a revelation.

Let me start with a short backtrace. We started introducing anti-spam for the email service roughly ten years ago. Our first filtering solution was Kaspersky Anti-Spam together with RBL (Real-time blackhole list — a realtime list of IP addresses that have something to do with spam mailouts). This allowed us to decrease the flow of spam messages, but due to the system’s inertia, we couldn’t suppress spam mailouts quickly enough (i.e. in the real time). The other requirement that wasn’t met was speed: users should have received verified email messages with a minimal delay, but the integrated solution was not fast enough to catch up with the spammers. Spam senders are very fast at changing their behavior model and the outlook of their spam content when they find out that spam messages are not delivered. So, we couldn’t put up with the system’s inertia and started developing our own spam filter...

Click to read more ...

Tuesday
Aug302016

Sponsored Post: Spotify, Aerospike, Exoscale, Host Color, Scalyr, Gusto, LaunchDarkly, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?

  • Spotify is looking for individuals passionate in infrastructure to join our Site Reliability Engineering organization. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. We are looking for an SRE Engineering Manager in NYC and SREs in Boston and NYC.

  • IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.

Fun and Informative Events

  • High-Scalability Database Beer Bash. Come join Aerospike and like-minded peers on Wednesday, September 7 from 6:30-8:30 PM in San Jose, CA for an informal meet-up of great food and libations. You'll have the chance to learn about Aerospike's high-performance NoSQL database for mission-critical applications, and about the use cases of the companies switching to Aerospike from first-generation NoSQL databases such as Cassandra and Redis. Feel free to invite colleagues and peers! RSVP: bit.ly/DBbeer

Cool Products and Services

  • Do you want a simpler public cloud provider but you still want to put real workloads into production? Exoscale gives you VMs with proper firewalling, DNS, S3-compatible storage, plus a simple UI and straightforward API. With datacenters in Switzerland, you also benefit from strict Swiss privacy laws. From just €5/$6 per month, try us free now.

  • High Availability Cloud Servers in Europe: High Availability (HA) is very important on the Cloud. It ensures business continuity and reduces application downtime. High Availability is a standard service on the European Cloud infrastructure of Host Color, active by default for all cloud servers, at no additional cost. It provides uniform, cost-effective failover protection against any outage caused by a hardware or an Operating System (OS) failure. The company uses VMware Cloud computing technology to create Public, Private & Hybrid Cloud servers. See Cloud service at Host Color Europe.

  • Dev teams are using LaunchDarkly’s Feature Flags as a Service to get unprecedented control over feature launches. LaunchDarkly allows you to cleanly separate code deployment from rollout. We make it super easy to enable functionality for whoever you want, whenever you want. See how it works.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Click to read more ...

Friday
Aug262016

Stuff The Internet Says On Scalability For August 26th, 2016

Hey, it's HighScalability time:

 

 

The Pixar render farm in 1995 is half of an iPhone (@BenedictEvans)

 

If you like this sort of Stuff then please support me on Patreon.
  • 33.0%: of all retail goods sold online in the US are sold on Amazon;  110.9 million: monthly Amazon unique visitors; 21 cents: cost of 30K batch derived page views on Lambda; 4th: grade level of Buzzfeed articles; $1 trillion: home value threatened by rising sea levels; $1.2B: Uber lost $1.2B on $2.1B in revenue in H1 2016; 1.58 trillion: miles Americans drove through June; 

  • Quotable Quotes:
    • @bendystraw: My best technical skill isn't coding, it's a willingness to ask questions, in front of everyone, about what I don't understand
    • @vmg: "ls is the IDE of producing lists of filenames"
    • @nicklockwood: The hardest problem in computer science is fighting the urge to solve a different, more interesting problem than the one at hand.
    • @RexRizzo: Wired: "Machine learning will TAKE OVER THE WORLD!" Amazon: "We see you bought a wallet. Would you like to buy ANOTHER WALLET?"
    • @viktorklang: "The very existence of Ethernet flow control may come as a shock" - http://jeffq.com/blog/the-ethernet-pause-frame/ 
    • @JoeEmison: 4/ (c) if you need stuff on prem, keep it on prem. No need to make your life harder by hooking it up to some bullshit that doesn't work well
    • @grayj_: Also people envision more than you think. Wright Brothers to cargo flights: 7 yrs. Steam engine to car: 7 yrs.
    • David Wentzlaff: With Piton, we really sat down and rethought computer architecture in order to build a chip specifically for data centres and the cloud
    • @thenewstack: In 2015, there was 1 talk about #microservcies at OSCON; in 2016, there were 30: @dberkholz #CloudNativeDay
    • The Memory Guy: Now for the bad news: This new technology [3D XPoint] will not be a factor in the market if Intel and Micron can’t make it, and last week’s IDF certainly gave little reason for optimism.
    • @Carnage4Life: $19 billion just to link WhatsApp graph with Facebook's is mundane. Expect deeper, more insidious connections coming
    • Seth Lloyd~ The universe is a quantum computer. Biological life is all about extracting meaningful information from a sea of bits.
    • Facebookk: To automate such design changes, the team introduced new models to FBNet in which IPs and circuits were allocated using design tools based on predefined rules, and relevant config snippets were generated for deployment.
    • Robert Graham: Despite the fact that everybody and their mother is buying iPhone 0days to hack phones, it's still the most secure phone. Androids are open to any old hacker -- iPhone are open only to nation state hackers.
    • oppositelock: I'm a former Google engineer working at another company now, and we use http/json rpc here. This RPC is the single highest consumer of cpu in our clusters, and our scale isn't all that large. I'm moving over to gRPC asap, for performance reasons.
    • Gary Sims: The purposes and goals of Fuchsia are still a mystery, however it is a serious undertaking. Dart is certainly key, as is Flutter.
    • @mjpt777: "We haven't made all that much progress on parallel computing in all those years." - Barbara Liskov
    • @AnupGhosh_: Just another sleepy August: 1. NSA crown jewels hacked. 2. Apple triple 0-day weaponized. 3. Short selling vulnerabilities for fun & profit.
    • @JoeEmison: Hypothesis: enterprises adopted CloudFoundry because at least it gets up and running (cf OpenStack), but now finding it so inferior to AWS.
    • Robert Metcalfe: I predict the Internet will soon go spectacularly supernova and in 1996 catastrophically collapse.
    • Alan Cooper~ Form follows function to Hell. If you are building something out of bits what does form follows function mean? Function follows the user. If you are focussing on functions you are missing the point. 
    • @etherealmind: I've _never_ seen a successful outsourcing arrangement. And I've work on both sides in more than 10 companies.
    • @musalbas: Schools need to stop spending years teaching kids garbage Microsoft PowerPoint skills and teach them Unix sysadmin skills.
    • Dan Woods: With data lakes there’s no inherent way to prioritize what data is going into the supply chain and how it will eventually be used. The result is like a museum with a huge collection of art, but no curator with the eye to tell what is worth displaying and what’s not.
    • Jay Kreps: Unlike scalability, multi-tenancy is something of a latent variable in the success of systems. You see hundreds of blog posts on benchmarking infrastructure systems—showing millions of requests per second on vast clusters—but far fewer about the work of scaling a system to hundreds or thousands of engineers and use cases. It’s just a lot harder to quantify multi-tenancy than it is to quantify scalability.
    • Jay Kreps: the advantage of Kafka is not just that it can handle that large application but that you can continue to deploy more and more apps to the same cluster as your adoption grows, without needing a siloed cluster for each use. 
    • @vambenepe: My secret superpower is using “reply” in situations where most others would use “reply all”.
    • @tvanfosson: Developer progression: instead of junior to senior 1. Simple and wrong 2. Complicated and wrong 3. Complicated and right 4. Simple and right
    • Maria Konnikova: The real confidence game feeds on the desire for magic, exploiting our endless taste for an existence that is more extraordinary and somehow more meaningful.
    • gpderetta: Apple A9 is a quite sophisticate CPU, there is no reason to believe is not using a state of the art predictor. The Samsung CPU might not have any advantage at all on this area.
    • Chetan Sharma: For 4G, we went from 0% to 25% penetration in 60 months, 25-50% in 21 months, 50-75% in 24 months and by the end of 2020, we will have 95%+ penetration. By 2020, US is likely to be 4 years ahead of Europe and 3 years ahead of China in LTE penetration. In fact, the industry vastly underestimated the growth of 4G in the US market. Will 5G growth curves be any different?

  • You know what's cool? A rubberband powered refrigerator. Or trillions of dollars...in space mining. Space Mining Company Plans to Launch Asteroid-Surveying Spacecraft by 2020. Billionaires get your rockets ready. It's a start: Weighing about 110 pounds, Prospector-1 will be powered by water, expelling superheated vapor to generate thrust. Since water will be the first resource mined from asteroids, this water propulsion system will allow future spacecraft–the ones that do the actual mining–to refuel on the go.

  • False positives in the new fully automated algorithmic driven world are red in tooth and claw. We may need a law. You know that feeling when you use your credit and you are told it is no longer valid? You are cutoff. Some algorithm has decided to isolate you from the world. At least you can call a credit card company. Have you ever tried to call a Cloud Company? Fred Trotter tells a scary story of not being able to face his accuser in Google Intrusion Detection Problem: So today our Google Cloud Account was suspended...Google threatened to shut our cloud account down in 3 days unless we did something…but made it impossible to complete that action...Google Cloud services shutdown the entire project...It is not safe to use any part of Google Cloud Services because their threat detection system has a fully automated allergic reaction to anything that has not seen before, and it is capable of taking down all of your cloud services, without limitation. 

  • In the "every car should come with a buggy whip" department we have The Absurd Fight Over Fund Documents You Probably Don't Read. $200 million would be saved if investors got their mutual fund reports online instead of on paper. You guessed it, there's a paper lobby against it. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...