Product: The Spread Toolkit

Complex applications coordinating work across a lot of machines often need a highly performing fault tolerant message layer. Though a blast to write, it's probably a better use of your time to use an off the shelf solution. And that's where Spread comes in. Flickr, for example, uses Spread to create real-time event feeds from their web server logs. What exactly is Spread?

From the Spread website:

Spread is an open source toolkit that provides a high performance messaging service that is resilient to faults across local and wide area networks. Spread functions as a unified message bus for distributed applications, and provides highly tuned application-level multicast, group communication, and point to point support. Spread services range from reliable messaging to fully ordered messages with delivery guarantees.

Spread can be used in many distributed applications that require high reliability, high performance, and robust communication among various subsets of members. The toolkit is designed to encapsulate the challenging aspects of asynchronous networks and enable the construction of reliable and scalable distributed applications.

Some of the services and benefits provided by Spread:

Reliable and scalable messaging and group communication.

A very powerful but simple API simplifies the construction of distributed architectures.

Easy to use, deploy and maintain.

Highly scalable from one local area network to complex wide area networks.

Supports thousands of groups with different sets of members.

Enables message reliability in the presence of machine failures, process crashes and recoveries, and network partitions and merges.

Provides a range of reliability, ordering and stability guarantees for messages.

Emphasis on robustness and high performance.

Completely distributed algorithms with no central point of failure.

In Building Scalable Web Sites Cal Henderson describes how Flickr uses Spread to create a log of real-time events, like photos uploaded and discussions started, as they happen. Spread is connected to their web servers. As photos are uploaded these web server events are messaged in real-time to agents consuming the feed.

The advantage of this architecture is it sheds load away from the database. Otherwise the database would have to be continuously polled for new events by each agent.

Related Articles