Scaling Django Web Apps by Mike Malone
Mike Malone, who worked on Pownce, a blogging tool now owned by Six Apart, tells in this very informative EuroDjangoCon presentation how Pownce scaled using Django in the real world.
I was surprised to learn how large Pounce was: hundreds of requests/sec, thousands of DB operations/sec, millions of user relationships, millions of notes, and terabytes of static data. Django has a lot of functionality in the box to help you scale, but if you want to scale large it turns out Django has some limitations and Mike tells you what these are and also provides some code to get around them.
Mike's talk-although Django specific--will really help anyone creating applications on the web. There's a lot of useful Django specific advice and a lot of general good design ideas as well.
The topics covered in the talk are:
* The database is responsible for scaling state.
* Application servers are horizontally scalable because they are stateless.
* Vertical - buy bigger hardware
* Horizontal - the ability to increase a system’s capacity by adding
more processing units (servers)
* Invalidate when a model is saved or deleted.
* Invalidate post_save, not pre_save.
* This leaves a small race condition so:
** Instead of deleting, set the cache key to None for a short period of time
** Instead of using set to cache objects, use add, which fails if there’s already something stored for the key
* Their servers were not CPU bound, they were IO and memory bound so they compressed objects before caching.
* Hardware load balancers are expensive ($35K) and you need two for redunancy.
* Software load balancers are cheap and easy.
* Some options: Perlbal, Pound, HAProxy, Varnish, Nginx
* Chose a single Perlbal server. This was a Single Point of Failure but they didn't have the money for hardware. Liked Perlbal's reproxying feature.
* The RDBMS’s consistency requirements get in our way
* Most sharding / federation schemes are kludges that trade consistency
* There are many non relational databases (CouchDB, Cassandra, Tokyo Cabinet) but they aren't easy to use with Django.
* Start with a normalized database
* Selectively denormalize things as they become bottlenecks
* Denormalized counts, copied fields, etc. can be updated in signal handlers
* Since your typical web app is 80% to 80% reads adding MySQL master-slave replication can solve a lot of problems.
* Django doesn't support multiple database connections, but there's a library, linked to at the end of this document to help.
* A big problem is slave lag. When you write to the primary it takes time for the state to be transferred to the read slaves so readers may see an old value on the read.
* Federate. Split tables across different servers. Not well supported by Django.
* Vertical Partitioning: split tables that aren’t joined across database servers.
* Horizontal Partitioning: split a single table across databases (e.g., user table). Problem is autoincrement now doesn't work and Django uses autoincrement for primary keys.
* Products: Ganglia and Munin
* Server load, CPU usage, I/O
* Database QPS
* Memcache QPS, hit rate, evictions
* Queue lengths
* Anything else interesting
