Scaling Hotjar's Architecture: 9 Lessons Learned

Hotjar offers free website analytics so they have a challenging mission: handle hundreds of millions of requests per day from mostly free users. Marc von Brockdorff, Co-Founder & Director of Engineering at Hotjar, summarized the lessons they've learned in: 9 Lessons Learned Scaling Hotjar's Tech Architecture To Handle 21,875,000 Requests Per Hour.

In response to the criticism their architecture looks like a hot mess, Erik Näslund, Chief Architect at Hotjar, gives the highlights of their architecture:

• We use nginx + lua for the really hot code paths where python doesn't quite cut it. No language is perfect and you might have to break out of your comfort zone and use something different every now and then.
• Redis, Memcached, Postgres, Elasticsearch and S3 are all suitable for different kinds of data storage and we eventually needed them all to be able to query and store data in a cost effective way. We didn't start out using 5 different data-stores though...it's something that we "grew into".
• Each application server is a (majestic) monolith. Micro-services are one way of architecting things, monoliths are another - I'm still waiting to be convinced that one way is superior to the other when it comes to a smaller team of developers.

What have they learned?

• Don’t underestimate how soon you will need to grow. Build an infrastructure which enables you to quickly scale up by adding more instances under a load balancer.
• Whenever possible, serve frequently accessed static content from a CDN. Your site performance will increase dramatically.
• Sometimes you need to look beyond your core technologies for performance critical code paths.
• Store data which does not require low latency and that is accessed using a primary key on cloud storage instead of your database to benefit from big cost savings.
• Your primary database may not be the ideal solution for everything - be ready to move some data to other, more suitable databases as you scale.
• Think about your users - what setup do they have? Multiple users? Multiple accounts? Multiple projects? It’s important to understand how your users work early on to ensure your database schema reflects it.
• Sometimes, even a tiny structure change can deliver big savings, both in terms of cost and performance. If you want to cut down on costs, don't assume you will need weeks of work.
• Though you shouldn’t over think your DB scheme too early on, make sure you have the proper monitoring set up and investigate ways of doing schema changes without downtime.
• Monitoring is crucial - the more monitoring you have, the quicker you can identify issues that arise from changes you’ve deployed.