High Scalability Building bigger, faster, more reliable websites.

With more users interacting, working, purchasing, and communicating over the network than ever before, Web 2.0 infrastructure is taking center stage in many organizations. Demand is rising, and companies are looking for ways to tackle the performance and scalability needs placed on Web infrastructure without raising IT operational expenses. Today companies are turning to efficient, high-performance, open source solutions as a way to decrease acquisition, licensing, and other ongoing costs and stay within budget constraints.

Update: YouTube: The Platform. YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway.

YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?

Automattic recently purchase Gravatar and have switched the server onto their hosting platform. WordPress.com host over 1.7 million blogs with well over 60'000 new posts submitted each day generating 10 - 12 million page views per day.

Barry on WordPress.com has a great post on the changes they've introduced to help Gravatar scale.

ThemBid provides a market where people needing work done broadcast their request and accept bids from people competing for the job. Unlike many of the sites profiled at HighScalability, ThemBid is not in the popular press as often as Paris Hilton. It's not a media darling or a giant of the industry. But what I like is they have a strategy, a point-of-view for building websites and were gracious enough to share very detailed instructions on how to go about building a website. They even delve into actual installation details of the various software packages they use. Anyone can benefit by taking a look at their work.

lighttpd (pronounced "lighty") is a web server which is designed to be secure, fast, standards-compliant, and flexible while being optimized for speed-critical environments. Its low memory footprint (compared to other web servers), light CPU load and its speed goals make lighttpd suitable for servers that are suffering load problems, or for serving static media separately from dynamic content. lighttpd is free software / open source, and is distributed under the BSD license. lighttpd runs on GNU/Linux and other Unix-like operating systems and Microsoft Windows.

* Load-balancing FastCGI, SCGI and HTTP-proxy support
* chroot support
* select()-/poll()-based web server
* Support for more efficient event notification schemes like kqueue and epoll
* Conditional rewrites (mod_rewrite)
* SSL and TLS support, via openSSL.
* Authentication against an LDAP server
* rrdtool statistics
* Rule-based downloading with possibility of a script handling only authentication
* Server-side includes support
* Flexible virtual hosting
* Modules support
* Cache Meta Language (currently being replaced by mod_magnet)
* Minimal WebDAV support
* Servlet (AJP) support (in versions 1.5.x and up)
* HTTP compression using mod_compress and the newer mod_deflate ( 1.5.x )

Information Sources

* http://en.wikipedia.org/wiki/Lighttpd
* http://highscalability.com/paper-lightweight-web-servers

This paper is a great overview of different lightweight web servers. A lot of websites use lightweight web servers to serve images and static content. YouTube is one example: http://highscalability.com/youtube-architecture.

So if you need to improve performance consider changing over a different web server for some types of content.