Example

Pinboard.in Architecture - Pay to Play to Keep a System Small

High Scalability

High Scalability

— 5 min read

How do you keep a system small enough, while still being successful, that a simple scale-up strategy becomes the preferred architecture? StackOverflow, for example, could stick with a tool chain they were comfortable with because they had a natural brake on how fast they could grow: there are only so many programmers in the world. If this doesn't work for you, here's another natural braking strategy to consider: charge for your service. Paul Houle summarized this nicely as: avoid scaling problems by building a service that's profitable at a small scale.

This interesting point, one I hadn't properly considered before, was brought up by Maciej Ceglowski, co-founder of Pinboard.in, in an interview with Leo Laporte and Amber MacArthur on their their net@night show.

Pinboard is a lean, mean, pay for bookmarking machine, a timely replacement for the nearly departed Delicious. And as a self professed anti-social bookmarking site, it emphasizes speed over socializing. Maciej considers Pinboard a personal archive, where you can keep a history of what you are reading: forever. When the demise of Delicious was announced, if Pinboard had been a free site they'd have been down immediately, but being a paid site helped flatten out their growth curve.

Bookmarking sites used to about sharing links with your friends, but Twitter has largely taken over that role. Twitter, however, is infamous for presenting only a small slice of your tweet history. What you really want is a big server sucking down your bookmarks from wherever you might bookmark them, and that's just what Pinboard does.

A few points struck me as particularly cool about Pinboard:

  • It uses a traditional style simple architecture based on MySQL, scale-up, PHP, Perl, and no frameworks.
  • It requires payment for services to cover costs and keep the user base manageable.
  • It charges based on the number of users to cover costs.
  • They built the site slowly and want to keep it small, simple fast, and reliable because that's a better way to run things.

Site: Pinboard.in

Sources

Stats

  • 16.3 million bookmarks
  • 52 million tags
  • 9.4 million urls
  • 989 GB archived content
  • A little under 1 hour cumulative downtime since July 8th, 2010.

Platform

  • MySQL
  • PHP
  • Perl
  • Ubuntu
  • APC
  • Sphinx
  • Cron jobs
  • S3

Hardware

  • Machine 1: 64 GB, runs a database master, stores user archives and runs search
  • Machine 2: 32 GB, runs the failover master, crawls various outside feeds, does background tasks
  • Machine 3: 16 GB, web server and database slave

Architecture

  • A copy of the database is kept on all three machines.
  • The website runs on the 16GB machine. The database fits entirely in RAM and page load times have improved by a factor of 10 or more.
  • Master-master architecture with an additional read slave.  All writes are pointed at one DB, these include bookmark, user, and tag tables.
  • The second master runs:
    •  Aggregate calculations like global link counts and per-user statistics.
    •  Nightly DB backups using mysqldump. The backup is stored to S3 in a compressed format.
  • Perl scripts run background tasks:
    • Downloading outside feeds, caching pages for users with the archive feature enabled, handling incoming email, generating tag clouds, and running backups.
    • Perl was chosen because of an existing skill set and the large support library available in CPAN.
    • Features like most popular bookmarks are generated by a cron job that is generally run each night, but are turned off when the load becomes too high.
  • PHP is used to generate HTML pages:
    • No templating engine is used. No frameworks are used.
    • APC is used to cache PHP files.
    • No other caching is used.
  • Sphinx is used for the search engine and for global tag pages.

Lessons Learned

  • Have a mantra. Pinboard has the goals of: being fast, reliable, and terse. They think these are the qualities that will earn and keep customers. When a problem comes up, like massive growth, they always prioritize so that these system qualities are maintained. For example, their first priority is preventing data loss, which dictated changing their server architecture. So the site seems conceptually confusing during a period of growth that's OK...if the site is quickly and reliably saving links.
  • Start as a paid site from the beginning. The advantage of being a paid site is you don't get the rush of new users, so you can stay small. When the demise of Delicious was announced, if they would have been a free site they would have been down immediately, but being a paid site helps smooth out the growth.
  • Charge based on the number of users. Pinboard has a unique pricing scheme that is designed to scale better than services with free accounts. The price is based on the current number of users. As the number of users goes up the price goes up. People are paying for the resources are using. This is similar, but enticingly different than the Amazon or Google App Engine payment model. This is a one time fee. For an extra $25/year all your bookmarks can be cached and searched.
  • Use boring and faded technologies. These help ensure the site will never lose data and be very fast.
  • A rule of thumb: if you are excited to play around with something, it probably doesn't belong in production.
  • Make switching as simple as possible. Pinboard removes  adoption objections by automatically importing and exporting to Delicious and by supporting the Delicious API.
  • Staying small is much more fun.  When you can offer personal customer support and interact directly with users you'll have a much better time.
  • Compare machine costs based on dollars per GB of RAM or storage. Pinboard originally ran on Slicehost and Linode, but they moved to a different service when the cost expressed in dollars per GB of RAM or storage was far higher, without any offsetting benefits.
  • Turn off features under load. Turn off search, for example, if you need performance elsewhere.
  • A medium to large site is the most expensive. Small sites are relatively cheap to run, but at some point during the growth curve the marginal cost of each new user increases. It costs more because data has to be split across multiple machines and those machines must be bought and managed. There's a scaling cost. Once you get into millions of users it gets cheaper again. Those first steps from where you go to a tiny site to a medium or medium large site are painful and expensive.
  • Call the shots on your own product. Depending on who you believe, Delicious was harmed by the continual layoffs at Yahoo, but the real problem was the Delicious team were not the decisions makers. New features were prioritized over reliability, stability, and innovation. It doesn't matter how hard or long you work when your fate is in the hands of others.
  • Small doesn't always work. A storm of new users added over seven million bookmarks, more than were collected over the entire lifetime of the service, and traffic to the site was over a hundred times normal. As a result normal background tasks like search, archiving, and polling outside feeds were suspended. An elastic strategy to handle spike loads like these isn't all bad.
  • Look at outlier page load times, not median page load times to judge the quality of your service. It's not acceptable if a page can take multiple seconds to load even is most of the page load times are acceptable.
  • Punt on features to build quickly. Pinboard was built quickly partly because  social and discovery features were deferred by saying "go somewhere else for that." Other sites will let you share links with friends and discover new and interesting content, but no other site acts like a personal archive and that's Pinboard's niche.
  • Segregate services by machine. When a web server shares a machine with other services the web server can take a hit. Another example is once each day the search indexer would wrestle with MySQL over memory while it did a full index rebuild.

Read more