advertise
« Product: The Spread Toolkit | Main | WAN Accelerate Your Way to Lightening Fast Transfers Between Data Centers »
Thursday
Oct112007

How Flickr Handles Moving You to Another Shard

Colin Charles has cool picture showing Flickr's message telling him they'll need about 15 minutes to move his 11,500 images to another shard. One, that's a lot of pictures! Two, it just goes to show you don't have to make this stuff complicated. Sure, it might be nice if their infrastructure could auto-balance shards with no down time and no loss of performance, but do you really need to go to all the extra complexity? The manual system works and though Colin would probably like his service to have been up, I am sure his day will still be a pleasant one.

Reader Comments (3)

Colin snuck up upon the man behind the curtain! :)

The process isn't manual, actually, it is automatic. But even when auto-balancing, you need to make sure that the data isn't changed while it's being moved. Migrating accounts are for a very small time locked while they are moved, unlocked when they are done. What happened to Colin can happen when a user just happens to hit their shard at the time they are being migrated, which doesn't happen very often. I suspect that his account was done in a lot less time than 15 minutes, probably seconds at absolute most. Note that actual photos aren't being moved, just metadata. :)

That 15 minute possibility comes from the part where sometimes the migration could take longer depending on what's happening on the source or target shard at the time. Our auto-balancing shard migrations happen with feedback from the actual load on the site, so more migrations happen at low traffic periods to avoid this exact thing. :)

Colin should win some sort of prize for catching it in the act.
-john

November 29, 1990 | Unregistered Commenterjohn allspaw

I have a feeling that by "manual" he probably meant not-migrated-on-the-fly. In other words, that it isn't a completely transparent process at Flickr. :)

Todd, good point about transparency of a move not always being needed. I think far too often people try to make too much invisible, especially when it's completely unnecessary, as with Flickr.

--
Dustin Puryear
Author, "Best Practices for Managing Linux and UNIX Servers"
http://www.puryear-it.com/pubs/linux-unix-best-practices

November 29, 1990 | Unregistered CommenterDustin Puryear

Here's a http://www.bytebot.net/blog/archives/2007/10/11/moving-to-another-shard">direct link to the post.

http://www.callum-macdonald.com/" title="Callum" target="_blank">Callum

November 29, 1990 | Unregistered Commenterchmac

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>