A Super Short on the Youporn Stack - 300K QPS and 100 Million Page Views Per Day

Eric Pickup from Youporn.com posted on a news group that Youporn is now 100% Redis based and will soon be revealing more about their architecture at the ConFoo conference. Some stunning, but not surprising numbers were revealed:

  • 100 million page views per day
  • A cluster of Redis slaves are handling over 300k queries per second.

Some additional nuggets:

  • Additional Redis nodes were added because the network cards couldn't keep up with Redis.
  • Impressed with Redis' performance.
  • All reads come from Redis;  we are maintaining MySQL just to allow us to build new sorted sets as our requirement change
  • Most data is found in hashes with ordered sets used to know what data to show.   
    • A typical lookup would be an zInterStore on: videos:filters:released, Videos:filters:orientation:straight,Videos:filters:categories:{category_id}, Videos:ordering:rating
    • Then perform a zRange to get the pages we want and get the list of video_ids back. 
    • Then start a pipeline and get all the videos from hashes.
    • Do use some key/value lookups and some lists, but the majority of our operations are using the above pattern.

Not much to see yet, but hopefully we'll learn more after their talk.