Secrets to Fotolog's Scaling Success
Fotolog, a social blogging site centered around photos, grew from about 300 thousand users in 2004 to over 11 million users in 2007. Though they initially experienced the inevitable pains of rapid growth, they overcame their problems and now manage over 300 million photos and 800,000 new photos are added each day. Generating all that fabulous content are 20 million unique monthly visitors and a volunteer army of 30,000 new users each day. They did so well a very impressed suitor bought them out for a cool $90 million. That's scale meets success by anyone standards. How did they do it?
Site: http://www.fotolog.com
Information Sources
The Platform
The Stats
The Architecture
- Their new "Fotolog memberpage" feature is written in Java with significant performance improvement. Page is cleaner with an improved response time.
- They are now serving the site on less than half the boxes they were using.
- Daily registrations are up over 35% given the improved performance and a requirement to register to post a guest book message.
- The new code base allows them to innovate much more on the member experience.
- There are no tags, no APIs, no JavaScript widgets, no Ajax.
- They have a Spanish language option which extends the site to a broad user base.
- They use very little text. It's mostly visual so it usable by a broad range of users.
- Their interface is customizable and many people like to express their individual identities.
- Their unique visitors are 1MM less than Yahoo's, yet the total minutes on the site are twice that of Yahoo and pages are 3x.
- Gold camera member for about $5/month means you can upload 6 photos a day instead of 1, have 200 comments per photo instead of 20, a custom title image for your profile, a mini-thumbnail of your most recent photo displayed next to your name in guest books, plus the possibility of having your photo featured on the front page..
- Adsense. Revenue lift from Google is trending up approximately 15% given additional contextual data from guest books.
- Will move to a peer-to-peer advertising among their members.
- Members will have the ability to buy and sell real and virtual items using a micro-payment service.
- They have resisted the temptation to add feature after feature. Instead their vision is to offer a handful of features, similar to Craig's list, the focus being on content and the conversations.
- Pages need to be social.
- Pages need to include not only your images, but also images from across the network, providing a visual navigation that today drives much of the time their members spend on the site, a self formed, organic distribution system, letting members see and be seen.
- Complementing this social network of images are comments and guest book entries — making the experience one where media intersects with communications, day in day out, millions of images collide with billions of conversations.
- Photobucket stores image-based media, then distributes it to your page on social networking sites such as Myspace, Bebo, Piczo, Friendster, etc.
- Fotolog is a destination.
- The first generation of social-networking sites stressed self-publishing over connections (from Geocities, to Tripod to Blogger). The next generation focused mostly on connections (sixdegrees, and friendster are the classic examples here — tools to gather friends and connections, as social capital accrues in theory to the people with the most connections). The third and current generation of sites blends media with connections — each with a different emphasis.
- user, GB (guest book), PH (photos), FF (friends and favorites lists)
- Uses non-persistent connections.
- Connection pooling on the Java side.
- InnoDB
- Partitioning is handled by the application layer.
- Is fronted by a set of application servers.
- Divided into a set of shards.
- Each shard has MySQL write-only master-master configuration feeding a few read-only slaves.
- Application servers send their read requests to the slaves and their write requests to the masters.
- Data are assigned to shared based on some sort of cluster specific partioning key. Naive partitioning algorithms can lead to very uneven shard loads, you want a more balanced load on each shard.
- Created a distributed cluster of 50 memcached servers with a total cache size of approximately 150 gigabytes, supporting around 4 billion page views/month. Peak load times dropped from 10 seconds to 2 seconds.
- Quote from CTO:
I have a new memcached user to add to your list: we here at Fotolog, the world's largest photo blogging community, now use it and we love it. I just rolled our first code to use it into production today and it has been a lifesaver. I can't wait to start using it in places where we had been relying on Berkeley databases to offload some database work. We are not some wimpy million page a day site, either. Fotolog is a billion+ pages/month site (35 to 40 million views/day is pretty typical for us). We had recently overcome some significant DB-related performance issues which allowed our site traffic to explode, and it started to bog down again under the heavy traffic load (getting back up towards 10 seconds for a page to load sometimes during the peak periods). The servers were churning away each recreating a list every time when it could easily be shared in the same form for at least 5 or 10 minutes. So we introduced memcache, creating a distributed 30-server cluster with 4 gigs available in total and made a very minor code mod to use memcache, and our peak period load times dropped back down to the 2 second or so range. It has allowed for continued growth and incredible efficiency. I can't say when I've ever been so pleased with something that worked so simply."
Lessons Learned
- Find the source of the problem.
- Mature systems are mostly disk bound.
- The query cache may be hurting you.
- Add RAM to help dodge the bullet.
- Stripe your disks.
- Restructure tables for optimal performance.
- Use libumem.so to find memory leaks.
- Know the problem
- Know your application
- Know your storage engine
- Know your requirements
- Know your budget
- Use all this information to decide what parts of your system really require the investment of time, money, and testing to be highly available.