This is a guest post by Doug Barth, a software generalist who has currently found himself doing operations work at PagerDuty. Prior to joining PagerDuty, he worked for Signal in Chicago and Orbitz, an online travel company.
A little over six months ago, PagerDuty switched its production MySQL database to XtraDB Cluster running inside EC2. Here's the story of why and how we made the change.
How the Database Looked Before
PagerDuty's MySQL database was a fairly typical deployment. We had:
A pair of Percona Servers writing data to a DRBD-backed volume.
EBS disks for both the primary and secondary servers backing the DRBD volume.
Two synchronous copies of the production database. (In the event of a failure of the primary, we wanted to be able to quickly switch to the secondary server without losing any data.)
A number of asynchronous replication slaves for disaster recovery, backups and accidental modification recovery.