Their setup is marvelously scalable but there's a catch ... they didn't actually need it. Those 1 billion reasons were mostly imaginary.

They tried providing a common infrastructure for various internal projects (storing common stuff). But the projects are different enough that data sharing doesn't really happen, a project's space being practically an island which have been better off served by its own individual database.

There are other advantages to that, like having a reusable storage, but you need your own database anyway because in HBase only certain kinds of data ends-up being stored (happened for one of the clients Cosmin's talking about). And then you end-up with consistency/local performance issues (to workaround that a lot of metadata is replicated in the local database, and there are synchronization issues anyway, only without guarantees ... like in the case of a master-master that dies loudly in case of a conflict).

This is not to downplay their effort (they are really top-notch engineers) ... but my impression is that they had way too much time on their hands, and their particular project is just a solution looking for a problem.

Of course, when you have the resources of Adobe, this may actually be a good idea. But in a smaller company that has to deliver before its competition does this is yak shaving.

I'm curious who manages their 'Usher' system? who's allowed to turn all the knobs that it supports?
Would be interesting to know more details regarding your web based source deployment system. Is it custom or some existing product?
How many sysadmins do they have?
What kind of Linux(?) distros do they use?
Is nginex used as a reverse proxy or regular direct http web server?
What are their thoughts on flash/h264/html5 video tag/ogg theora?