advertise
Monday
Dec282009

Zynga Needs a Server-side Systems Engineer

Ashleigh Anderson from Zynga let me know that they have an opening for a Systems Engineer working on some new games they are developing. Given the state of the job market I thought it worth posting. Here are more details...

Click to read more ...

Tuesday
Dec222009

Incremental deployment

Incremental deployment. Manual/automated hybrid deployment strategy we use at Forward for our critical, high availability distributed systems.

Monday
Dec212009

Hot Holiday Scalability Links for 2009

Thursday
Dec172009

Oracle and IBM databases: Disk-based vs In-memory databases 

Current disk based RDBMS can run out of steam when processing large data. Can these problems be solved by migrating from a disk based RDBMS to an IMDB? Any limitations? To find out, I tested one of each from the two leading vendors who together hold 70% of the market share - Oracle's 11g and TimesTen 11g, and IBM's DB2 v9.5 and solidDB 6.3.

read more at BigDataMatters.com

Wednesday
Dec162009

The most common flaw in software performance testing

How many times have we all run across a situation where the performance tests on a piece of software pass with flying colors on the test systems only to see the software exhibit poor performance characteristics when the software is deployed in production? Read More Here...

Wednesday
Dec162009

Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud

"But it is not complicated. [There's] just a lot of it." 
-- Richard Feynman on how the immense variety of the world arises from simple rules.

Contents:

  1. Have We Reached the End of Scaling?
  2. Applications Become Black Boxes Using Markets to Scale and Control Costs
  3. Let's Welcome our Neo-Feudal Overlords
  4. The Economic Argument for the Ambient Cloud
  5. What Will Kill the Cloud?
  6. The Amazing Collective Compute Power of the Ambient Cloud
  7. Using the Ambient Cloud as an Application Runtime
  8. Applications as Virtual States
  9. Conclusion

We have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.

Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.

Tomorrow the numbers foreshadow a new Cambrian explosion of connectivity that will look as different as the image of a bare lifeless earth looks to us today. We will have 10 billion people, we will have trillions of things, and we will have a great multitude of social networks densely interconnecting all these people to people, things to things, and people to things.

How can we possibly build planet scalable systems to handle this massive growth if building much smaller applications currently stresses architectural best practices past breaking? We can't. We aren't anywhere close to building applications at this scale, except for perhaps Google and a few others, and there's no way you and I can reproduce what they are doing. Companies are scrambling to raise hundreds of millions of dollars in order to build even more datacenters. As the world becomes more and more global and more and more connected, handling the load may require building applications 4 or 5 orders of magnitude larger than any current system. The cost for an infrastructure capable of supporting planet-scale applications could be in the 10 trillion dollar range (very roughly estimated at $100 million a data center times 10K).

If you aren't Google, or a very few other companies, how can you possibly compete? For a glimmer of a possible direction that may not require a kingdom's worth of resources, please take a look at this short video:

Click to read more ...

Tuesday
Dec152009

The Common Principles Behind the NOSQL Alternatives

This post draws some of the common patterns behind the various NOSQL alternatives, and how they address the database scalability challenge.

Read the full story here

Monday
Nov302009

Why Existing Databases (RAC) are So Breakable!

One of the core assumption behind many of today’s databases is that disks are reliable. In other words, your data is “safe” if it is stored on a disk, and indeed most database solutions rely heavily on that assumption. Is it a valid assumption?

Read the full story here

Thursday
Nov262009

Kngine Snippet Search New Indexing Technology

While Kngine just announce some improvement and new features, I would like you take you in small trip in Snippet Search research project at Kngine.

 

Click to read more ...

Thursday
Nov262009

What I'm Thankful For on Thanksgiving

I try to keep this blog targeted and on topic. So even though I may be thankful for the song of the tinniest sparrow at sunrise, I'll save you from all that. It's hard to tie scalability and the giving of thanks together, especially as it sometimes occurs to me that this blog may be a self-indulgent waste of time. But I think I found a sentiment in A New THEORY of AWESOMENESS and MIRACLES by James Bridle that manages to marry the topic of this blog and giving thanks meaningfully together:

I distrust commercial definitions of innovation, and particularly of awesomeness. It’s an overused term. When I think of awesomeness, I want something awe-inspiring, vast and mind-expanding.

So I started thinking about things that I think are awesome, or miraculous, and for me, it kept coming back to scale and complexity.

We’re not actually very good about thinking about scale and complexity in real terms, so we have to use metaphors and examples. Douglas Adams writes somewhere about how big the Hitchhiker’s Guide to the Galaxy actually is—imagine a sheet of paper, then a filing cabinet full of sheets of paper, then a room full of filing cabinets, then a skyscraper full of rooms, then a city full of skyscrapers, a country, a planet, a solar system and so on. I couldn’t find the exact quote, so his thoughts on space will have to do:

Just wonderful. I especially love the quote So I started thinking about things that I think are awesome, or miraculous, and for me, it kept coming back to scale and complexity. This perfectly sums up why the topic of scalability is so endlessly diverting. It can take you anywhere you want to go and everything eventually ends up back again.

Thanks for reading and...

Happy Thanksgiving!