Building a Scalable Key-Value Database: Project Hydracus

The world of NoSQL and alternative database implementations (i.e.  non-relational) is deeply fascinating to me. I can’t help but be swept  up in the whirl of planet-scale web development scalability techniques  and the evolution of how developers think about building their  applications knowing that with success comes the inevitable need to  scale to levels almost unimaginable just five or ten years ago.

I’m going to make a prediction: Developers will be expected to  understand the fundamentals of how different database systems can be  applied within a singular application; their strengths and weaknesses,  and when it is appropriate to leverage them.

There’s a lot out there about scaling relational databases.  Partitioning your data over application-managed shards is a topic that  has seen its fair share of attention. But I think there’s a whole slew  over NoSQL databases that have been built and designed by some very  smart people that are new enough that their internals aren’t well  understood by most developers to the same level as the typical  relational database. My hope is that I’m remedying that need with this  new article series and, in the process, building a functional  distributed key-value database with the community. I’ll be documenting  each stage of the process that I go through during my exploration into  designing and coding each piece of this database and, with some helpful  guidance from the community, evolving and sharing it for all to use – it  will of course be open sourced.

You can read the full story here.