advertise
Friday
Nov292013

One Story of Life as Told Through Queues

Love this little example of the human condition from John Kellden via Ilya Grigorik. This happens so often to me shopping at Costco or making lane changes on the highway or picking stocks. Sometimes it's just never the right line and trying to make it better only makes it worse. Stick and stay. Buy and hold. Live to queue another day.
Wednesday
Nov272013

Hidden History: Driving the Last Spike of the Transcontinental Railroad was an Early Version of the Internet of Things

The story of driving the golden spike to symbolize the completion of the transcontinental railroad is famous in the US. What is not so well known is the story of how it also foreshadowed changes to come as an early version of both the Internet and the Internet of Things. But that was 1869, how can that possibly be?

Telegraph as Internet

Click to read more ...

Tuesday
Nov262013

Sponsored Post: Spokeo, Klout, Apple, NuoDB, ScaleOut, MongoDB, BlueStripe, AiScaler, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • Quality Assurance Engineer. The iOS Systems team is looking for a Quality Assurance engineer. In this role you will be expected to work hand-in-hand with the software engineering team to find and diagnose software defects. Please apply here.
    • Sr Software Engineer iPhone. Do you love building highly scalable, distributed web applications? Does the idea of a fast-paced environment make your heart leap? Do you want your technical abilities to be challenged every day, and for your work to make a difference in the lives of millions of people? Please apply here.
    • Sr. Software Engineer. You will primarily work with the domain team including project managers and engineers, as well as a large team of consultants in California and India. You will also work with many cross-functional and infrastructural teams during software delivery life cycle. Assignments can include design, delivery and oversight of incremental functionality, as well as a multi-year re-architecture of a complex in-flight application. Please apply here.
    • Enterprise Software Engineer.  Ability to develop detailed design and deliver a scalable implementation. Hands-on development of new code and/or managing existing code as part of a group and/or alone. Evaluating products including open-source modules and if need be incorporating them into projects. Should be able to lead a small group of developers to develop and maintain systems. Please apply here
  • Spokeo is hiring a Senior Backend Developer. We've spent years agonizing over the best way to construct an elegant, simplistic, yet highly powerful people search engine. Spokeo deals with problems of ginormous scale, so a strong understanding and appreciation for algorithms and efficiency is desired. Please apply here.

  • Spokeo is hiring a Senior Software Developer - Web Applications. Build features that involve any of our products, from the universal people search interface and functionality to the construction of family trees to a portal that connects customers to employees and employer data. Please apply here.

  • Apply to be a Senior Software Engineer  - Platform API at Klout, a fast-growing startup in SF! Implement solutions to service-oriented architecture problems and help engineer systems that serve over 3 billion API calls a day. Apply here: http://grnh.se/o1ynh8 

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

  • New Relic is looking for a Java Instrumentation Engineer, Java Scalability Engineer,  Distributed Systems Engineer and Android app engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter? 

Fun and Informative Events

  • Your amazing event here.

Cool Products and Services

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting

  • NuoDB Blackbirds Release 2.0 Birthday. They grow up so fast these days! What people love about NuoDB is that it’s stable, always there for you and its flexible. Which is why it’s winning all kinds of popularity competitions, from “Most Likely to Succeed” through “Least Likely To Fall Over Sharding” to “Most Likely to Be ACID Compliant”. 

  • Rapidly Develop Hadoop MapReduce Code. With ScaleOut hServer™ you can use a subset of your Hadoop data and run your MapReduce code in seconds for fast code development and you don’t need to load and manage the Hadoop software  stack, it's a self-contained Hadoop MapReduce execution environment. To learn more check out www.scaleoutsoftware.com/prototypehadoop/

  • MongoDB Backup Free Usage Tier Announced. We're pleased to introduce the free usage tier to MongoDB Management Service (MMS). MMS Backup provides point-in-time recovery for replica sets and consistent snapshots for sharded systems with minimal performance impact. Start backing up today at mms.mongodb.com.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • Gartner Magic Quadrant for Operational Database Management Systems 2013. Gartner has published the 2013 Magic Quadrant for Operational Database Management Systems and recognized Aerospike in the Visionaries Quadrant. The report examines the key capabilities of information management for transaction processing.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Monday
Nov252013

How To Make an Infinitely Scalable Relational Database Management System (RDBMS)

This is a guest post by Mark Travis, Founder of InfiniSQL.

InfiniSQL is the specific "Infinitely Scalable RDBMS" to which the title refers. It is free software, and instructions for getting, building, running and testing it are available in the guide. Benchmarking shows that an InfiniSQL cluster can handle over 500,000 complex transactions per second with over 100,000 simultaneous connections, all on twelve small servers. The methods used to test are documented, and the code is all available so that any practitioner can achieve similar results. There are two main characteristics which make InfiniSQL extraordinary:

Click to read more ...

Friday
Nov222013

Stuff The Internet Says On Scalability For November 22th, 2013

Hey, it's HighScalability time:

Test your sense of scale. Is this image of something microscopic or macroscopic? Find out.

  • 26,496: number of cores in Amazon's supercomputer; 128 billion: WDC's huge web graph publicly available; 400 million: Snapchat more photos than Facebook per day; 300,000: Microsoft's servers for Xbox; 1 million: MessageMe user growth in one week
  • Quotable Quotes:
    • Jony Ive: I feel that ideas are very fragile, so you have to be tender when they are in development. I realized that if he pissed on this, it would be so sad because I know it was so important.
    • @BenedictEvans: 100m users is the new 1m users. $4bn is the new $400m. 30 staff is the new 300 staff.
    • @postwait: Knowing the latency of every I/O on every spindle in every machine in every rack... over all of time. Done & eye-opening. Thanks @circonus.
    • @mrb_bk: Things that hurt: unbounded memory growth. Things that help: concurrency control.
    • @sogrady: the latest version of chaos monkey described by @adrianco at re:invent is positively sadistic. most weaponizable software ever?
    • Joe Landman: More efficient, faster units, pay for themselves very quickly when compared to inspecific designs that make fine mail servers, but terrible storage controllers, or network hubs, or … The argument makes sense at scale and at small scale.
    • Sync: In other words, a dumb rule (majority rule) running on a smart architecture (a small world) achieved performances that broke the world.

  • Jamestown was a startup established in ~1617. Lethal conditions soon required hiring more employees. The hiring bonus used to induce people to sign on was greater independence and self rule. The first American pivot.

  • Everyone hates cubicles, yet they are still the best packing algorithm for humans in rectangular spaces. And if we lose productivity to noise and other cube farm diseases, does that really matter? Research: Cubicles Are the Absolute Worst

  • My how the world has changed. In 1996 The Oracle of Bacon, a web site for automatically computing the shortest possible chain of costars between Kevin Bacon and any other film actor, was selected by Time magazine as a top 10 web site. At its height the site was bludgeoned with as many as 20,000 hits a day.

  • You can't compete with a entrenched technology like the cloud by doing more or better cloud. You need to take a leap. Here's an example of what a leap looks like. +PeerServer:  a peer-to-peer client server using WebRTC, where your browser acts as a server for other browsers across WebRTC peer-to-peer data channels. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Wednesday
Nov202013

How Twitter Improved JVM Performance by Reducing GC and Faster Memory Allocation

Netty is a high-performance NIO (New IO) client server framework for Java that Twitter uses internally as a protocol agonostic RPC system. Twitter found some problems with Netty 3's memory management for buffer allocations beacause it generated a lot of garbage during operation. When you send as many messages as Twitter it creates a lot of GC pressure and the simple act of zero filling newly allocated buffers consumed 50% of memory bandwidth. 

Netty 4 fixes this situation with:

  • Short-lived event objects, methods on long-lived channel objects are used to handle I/O events.
  • Secialized buffer allocator that uses pool which implements buddy memory allocation and slab allocation.

The result:

  • 5 times less frequent GC pauses: 45.5 vs. 9.2 times/min
  • 5 times less garbage production: 207.11 vs 41.81 MiB/s
  • The buffer pool is much faster than JVM as the size of the buffer increases. Some problems with smaller buffers.

Given how many services use the JVM in their messaging infrastructure and how many services have GC related performance problems, this is in impressive result others may want to consider.

Tuesday
Nov192013

We Finally Cracked the 10K Problem - This Time for Managing Servers with 2000x Servers Managed Per Sysadmin

In 1999 Dan Kegel issued a big hairy audacious challenge to web servers:

It's time for web servers to handle ten thousand clients simultaneously, don't you think? After all, the web is a big place now.

This became known as the C10K problem. Engineers solved the C10K scalability problems by fixing OS kernels and moving away from threaded servers like Apache to event-driven servers like Nginx and Node.

Today we are considering an even bigger goal, how to support 10 Million Concurrent Connections, which requires even more radical techniques.

No similar challenge was issued for managing servers in a datacenter, but according to Dave Neary from Red Hat, in a recent FLOSS Weekly episode, we have passed the 10K barrier for server management with 10,000 or more servers managed per sysadmin.

Should we let this milestone pass without mention?

Click to read more ...

Friday
Nov152013

Stuff The Internet Says On Scalability For November 15th, 2013

Hey, it's HighScalability time:


Test your sense of scale. Is this image of something microscopic or macroscopic? Find out.
  • Quotable Quotes:
    • fidotron: It feels like we've gone in one big circle, where first we move the DB on to a separate machine for performance, yet now more computation will go back to being done nearer the data (like Hadoop) and we'll try to pretend it's all just one giant computer again.
    • @pbailis: Building systems from the ground up with distribution, scale, and availability in mind is much easier than retrofitting single-node systems.
    • @merv: #awsreinvent Jassy: Netflix has 10,000s of EC2 instances. They are the final deployment scenario: All In. And others are coming.
    • Edward Capriolo: YARN... Either it is really complicated or I have brain damage
    • @djspiewak: Eventually, Node.js will reinvent the “IO promise” and realize that flattening your callback effects is actually quite nice.
    • @jimblomo: A Note on Distributed Computing advised against treating remote calls as local functions, I advise against treating the cloud as datacenter
  • Instagram: A Brief, Rapid History of Scaling Instagram (with a tiny team). Have a bigger team. Do simple first. Reduce moving parts. Test and monitor everything. Use Chef earlier. Match the team with the tools. Simple isn't always hacky. Rocketship scaling has been somewhat democratized. 
  • Most of your hard drives will live longer than a mouse (4 years) says Backblaze, the mad scientists of mass storage.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Wednesday
Nov132013

Google: Multiplex Multiple Works Loads on Computers to Increase Machine Utilization and Save Money

Jeff Dean gave a talk at SFBay ACM and at about 3 minutes in he goes over how Google runs jobs on computers, which is different than how most shops distribute workloads.

It’s common for machines to be dedicated to one service, say run a database, run a cache, run this, or run that. The logic is:

Click to read more ...

Tuesday
Nov122013

Sponsored Post: Klout, Apple, NuoDB, ScaleOut, FreeAgent, CloudStats.me, Intechnica, MongoDB, Stackdriver, BlueStripe, Booking, AiCache, Aerospike, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple is hiring for multiple positions. Imagine what you could do here. At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly.
    • Sr. Software Engineer. You will primarily work with the domain team including project managers and engineers, as well as a large team of consultants in California and India. You will also work with many cross-functional and infrastructural teams during software delivery life cycle. Assignments can include design, delivery and oversight of incremental functionality, as well as a multi-year re-architecture of a complex in-flight application. Please apply here.
    • Enterprise Software Engineer.  Ability to develop detailed design and deliver a scalable implementation. Hands-on development of new code and/or managing existing code as part of a group and/or alone. Evaluating products including open-source modules and if need be incorporating them into projects. Should be able to lead a small group of developers to develop and maintain systems. Please apply here
    • Senior Cocoa Engineer. Apple is seeking a senior Cocoa engineer to join the IS&T Client Frameworks team. Are you someone looking to solve technically challenging problems involving a wide range of technologies (client, server, web)? Please apply here.
    • Web Application Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives. Please apply here.
    • Web Application Engineer. We are looking for a team player with focus on designing and developing WWDR’s web-based applications. The successful candidate must have the ability to take minimal business requirements and work pro-actively with cross functional teams to obtain clear objectives. Please apply here.
    • Senior Web Developer: Worldwide Developer Relations. Responsible for the architecture, design and development of the user interface of WWDR’s web applications. Work with UI designer and marketing to analyze business requirements and contribute to functional requirements. Collaborate with server-side software engineers on design, document technical specifications, and implement proper solutions. Please apply here
    • Sr Software Engineer. The iOS Systems Team is looking for a Software Engineer to work on operations, tools development and support of worldwide iOS Device sales and activations. Please apply here
    • Sr. Software Engineer. The Identity Management Services team at Apple is in search of a motivated Senior Software Engineer who is self-driven and has a proven track record in design and development of complex, highly available and scalable systems. Please apply here
    • SQE and Operations Manager, iOS Systems. The iOS Systems team is looking for an experienced hands-on manager to lead the Quality Engineering, Build and Release Engineering team. Please apply here
    • Senior Engineer: Emerging Technology. Apple’s Emerging Technology group is looking for a senior engineer passionate about exploring emerging technologies to create paradigm shifting cloud based solutions. Please apply here. 

  • Apply to be a Senior Software Engineer  - Platform API at Klout, a fast-growing startup in SF! Implement solutions to service-oriented architecture problems and help engineer systems that serve over 3 billion API calls a day. Apply here: http://grnh.se/o1ynh8 

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

  • FreeAgent are looking for a talented Operations Engineer to come and work on the FreeAgent app, internal services and supporting infrastructure. You'll be working alongside our Ops team squashing single points of failure, fixing bottlenecks, profiling load and solving interesting scaling and automation problems. Please apply here

  • Intechnica is looking for Performance Architects, Performance Engineers, a Lead Automation Engineer, and a Solution Assurance Analyst. If making super-fast systems is your forte, send your CV with covering letter to careers@intechnica.co.uk.

  • Stackdriver is looking for systems + cloud + dev + ops guru to serve as our liaison within the DevOps community. If you are passionate about monitoring and automation, enjoy working on open source, and are excited by the prospect of sharing your expertise with your peers, get in touch with us today! http://bit.ly/143ARmy

  • We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.

  • New Relic is looking for a Java Instrumentation Engineer, Java Scalability Engineer,  Distributed Systems Engineer and Android app engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter? 

Fun and Informative Events

  • Your amazing event here.

Cool Products and Services

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting

  • NuoDB Blackbirds Release 2.0 Birthday. They grow up so fast these days! What people love about NuoDB is that it’s stable, always there for you and its flexible. Which is why it’s winning all kinds of popularity competitions, from “Most Likely to Succeed” through “Least Likely To Fall Over Sharding” to “Most Likely to Be ACID Compliant”. 

  • Rapidly Develop Hadoop MapReduce Code. With ScaleOut hServer™ you can use a subset of your Hadoop data and run your MapReduce code in seconds for fast code development and you don’t need to load and manage the Hadoop software  stack, it's a self-contained Hadoop MapReduce execution environment. To learn more check out www.scaleoutsoftware.com/prototypehadoop/

  • CloudStats.me - Monitor all your VPS, Dedicated and Cloud servers from one place. Whether you have only one server or hundreds of them, you will be able to check their status in seconds from the dashboard. Try server monitoring now for free.

  • MongoDB Backup Free Usage Tier Announced. We're pleased to introduce the free usage tier to MongoDB Management Service (MMS). MMS Backup provides point-in-time recovery for replica sets and consistent snapshots for sharded systems with minimal performance impact. Start backing up today at mms.mongodb.com.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • Gartner Magic Quadrant for Operational Database Management Systems 2013. Gartner has published the 2013 Magic Quadrant for Operational Database Management Systems and recognized Aerospike in the Visionaries Quadrant. The report examines the key capabilities of information management for transaction processing.

  • aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required. http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...