advertise
« Moving Beyond End-to-End Path Information to Optimize CDN Performance | Main | PaxosLease: Diskless Paxos for Leases »
Tuesday
Sep222009

How Ravelry Scales to 10 Million Requests Using Rails

Tim Bray has a wonderful interview with Casey Forbes, creator of Ravelry, a Ruby on Rails site supporting a 400,000+ strong community of dedicated knitters and crocheters.

Casey and his small team have done great things with Ravelry. It is a very focused site that provides a lot of value for users. And users absolutely adore the site. That's obvious from their enthusiastic comments and rocket fast adoption of Ravelry.

Ten years ago a site like Ravelry would have been a multi-million dollar operation. Today Casey is the sole engineer for Ravelry and to run it takes only a few people. He was able to code it in 4 months working nights and weekends. Take a look down below of all the technologies used to make Ravelry and you'll see how it is constructed almost completely from free of the shelf software that Casey has stitched together into a complete system. There's an amazing amount of leverage in today's ecosystem when you combine all the quality tools, languages, storage, bandwidth and hosting options.

Now Casey and several employees makes a living from Ravelry. Isn't that the dream of any small business? How might you go about doing the same thing?

Site: http://www.ravelry.com

Statistics

  • 10 million requests a day hit Rails (AJAX + RSS + API)
  • 3.6 million pageviews per day
  • 430,000 registered users. 70,000 active each day. 900 new sign ups per day.
  • 2.3 million knitting/crochet projects, 50,000 new forum posts each day, 19 million forum posts, 13 million private messages, 8 million photos (the majority are hosted by Flickr).
  • Started on a small VPS and demand exploded from the start.
  • Monetization: advertisers + merchandise store + pattern sales

    Platform

  • Ruby on Rails (1.8.6, Ruby GC patches)
  • Percona build of MySQL
  • Gentoo Linux
  • Servers: Silicon Mechanics (owned, not leased)
  • Hosting: Colocation with Hosted Solutions
  • Bandwidth: Cogent (very cheap)
  • Capistrano for deployment.
  • Nginx is much faster and less memory hungry than Apache.
  • Xen for virtualization
  • HAproxy for load balancing.
  • Munin for monitoring.
  • Tokyo Cabinet/Tyrant for large object caching
  • Nagios for alerts
  • HopToad for exception notifications.
  • NewRelic for tuning
  • Syslog-ng for log aggregation
  • S3 for storage
  • Cloudfront as a CDN
  • Sphinx for the search engine
  • Memcached for small object caching

    Architecture

  • 7 Servers (Gentoo Linux). Virtualization (Xen) creates 13 virtual servers.
  •  Front end uses Nginx and HAproxy. The request flow: nginx -> haproxy -> (load balanced) -> apache + mod_passenger. Nginx is first so it can provide functions like serving static files and redirects before passing a request to HAproxy for load balancing. Apache is probably used because it is more configurable than Nginx.
  •  One small backup server.
  • One small utility server for non-critical processes and staging.
  •  2 32 GB of RAM servers for the master database, slave database, Sphinx search engine.
  •  3 application servers running 6 Apache Passenger and Ruby instances, each capped at a pool size of 20. 6 quad core processors and 40 GB of RAM total. There's RAM to spare.
  • 5 terabytes of storage on Amazon S3. Cloudfront is used as a CDN.
  • Tokyo Cabinet/Tyrant is used instead of memcached in some places for caching larger objects. Specifically markdown text that has been converted to HTML.
  • HAproxy and Capistrano are used for rolling deploys of new versions of the site without affecting performance/traffic.

    Lessons Learned

  • Let your users create the site for you. Iterate and evolve. Start with something that works, get people in it, and build it together. Have a slow beta. Invite new people on slowly. Talk to the users about what they want every single day. Let your users help build your site. The result will be more reassuring, comforting, intuitive, and effective.
  • Let your users fund you. Ravelry was funded in part from users who donated $71K. That's a gift. Not stock. Don't give up equity in your company. It took 6 months of working full time and bandwidth/server costs before they started making a profit and this money helped bridge that gap. They key is having a product users feel passionate about and being the kind of people users feel good about supporting. That requires love and authenticity.
  • Become the farmer's market of your niche. Find an under serviced niche. Be anti-mass market. You don't always have to create something for the millions. The millions will likely yawn. Create something and do a good job for a smaller passionate group and that passion will transfer over to you.
  • Success is not about scale, it’s about sustainable execution. This lovely quote is from Jeff Putz.
  • The database is always the problem. Nearly all of the scaling/tuning/performance related work is database related. For example, MySQL schema changes on large tables are painful if you don’t want any downtime. One of the arguments for schemaless databases.
  • Keep it fun. Casey switched to Ruby on Rails because he was looking to make programming fun again. That reenchantment helped make the site possible.
  • Invent new things that delight your users. Go for magic. Users like that. This is one of Costco's principles too. This link, for example, describes some very innovative approaches to forum management.
  • Ruby rocks. It's a fun language and allowed them to develop quickly and release the site twice a day during beta.
  • Capture more profit using low margin services. Ravelry has their own merchandise store, wholesale accounts, printers, and fulfillment company. This allows them to keep all their costs lower so their profits aren't going third party services like CafePress.
  • Going from one server to many servers is the hardest transition to make. Everything changes and becomes more difficult. Have this transition in mind when you are planning your architecture.
  • You can do a lot with a little in today's ecosystem. It doesn't take many people or much money anymore to build a complex site like Ravelry. Take a look at all the different programs Ravelry uses to build there site and how few people are needed to run the site.

    Some people complain that there aren't a lot of nitty gritty details about how Raverly works. I think it should be illuminating that a site of this size doesn't need to have a lavish description of arcane scaling strategies. It can now be built from off the shelf parts smartly put together. And that's pretty cool.

    Related Articles

  • Ravelry gets funding from its own community.
  • Appache/Passenger vs Nginx/Mongrel by Matt Darby
  • The Ravelry Blog (note the number of comments on posts).
  • Podcast - Episode 4: Y Ravelry (featuring Jess & Casey)
  • Beta testing and beyond
  • Hacker News Thread - I included the reasoning from a user named Brett for why the HTTP request path is "Nginx out front passing requests to HAProxy and THEN to Apache + mod_rails."
  • Reader Comments (11)

    the website http://www.ravelry.com/ is down. Interesting.

    November 29, 1990 | Unregistered CommenterAnonymous

    Thanks for this story. gives all of us the hope we need to continue trying to make it on our own.

    November 29, 1990 | Unregistered CommenterJarrold

    10.000.000 req/day doesn't really sound like that much. It would be about 115 req/s if the load is distributed evenly. If there are heavy peak times or some write operations for each request, it's probably more of a challenge to handle.

    November 29, 1990 | Unregistered CommenterKenneth Falck

    It would be interesting to hear the decision process they used to determine it would be better to purchase their own hardware and use xen than using something like ec2. My initial guess is that it has something to do with access to funds early own to purchase hardware but maybe not?

    November 29, 1990 | Unregistered CommenterRyan Crumley

    The site is not down.

    Even though I'm not a knitter, or crocheter, I think Ravelry is great. It's one of the best sites I know for demonstrating how Ajax can really spice up a user interface without getting in the way. It feels really slick in action, and the simple, clean lines of the UI are very easy to like.

    November 29, 1990 | Unregistered CommenterBen

    The site is not down.

    Even though I'm not a knitter, or crocheter, I think Ravelry is great. It's one of the best sites I know for demonstrating how Ajax can really spice up a user interface without getting in the way. It feels really slick in action, and the simple, clean lines of the UI are very easy to like.

    November 29, 1990 | Unregistered CommenterBen

    Way to write a whole post about a site and not throw them a single link. At least the site you linkjacked (tbray) threw them a link.

    October 9, 2009 | Unregistered Commenterchris

    Chris, there are two links in the first sentence and multiple links in the related references section and there's a link with Site.

    October 9, 2009 | Registered CommenterTodd Hoff

    Almost 12 requests per second. Oh my dear, its great work!

    October 20, 2009 | Unregistered CommenterHotentot

    12 * 10 rps, maybe. What's the peak load like?

    October 21, 2009 | Unregistered CommenterReginald

    I see this post tweeted and linked occasionally so I can't help but comment.

    Yeah - peak load isn't crazy huge but it's respectable: Monday afternoons Eastern time are about 350 requests/second
    (also - we've grown since October)

    July 15, 2010 | Unregistered CommenterCasey

    PostPost a New Comment

    Enter your information below to add a new comment.
    Author Email (optional):
    Author URL (optional):
    Post:
     
    Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>