Update 2: Nice introductory New York Time's article Cloud Computing: So You Don’t Have to Stand Still. Good example of how Animoto used RightScale and Amazon to meet a Facebook driven demand of 25,000 test drives an hour. Update: Peter Laird in Understanding the Cloud Computing/SaaS/PaaS markets: a Map of the Players in the Industry paints a very cool visual map of all the cloud service players. It's a larger industry than you might think. Once upon a time I worked at an Asynchronous Transfer Mode (ATM) switch startup. Over a delicious Christmas punch my grandma asked me what I did for a living that I could afford such extravagantly inexpensive gifts. Always so subtle. I explained I worked on an ATM switch. Mistake. She sniffed, said that's nice, and asked me why the Automated Teller Machine ate her bank card that morning. No matter how hard I tried I couldn't convince her I didn't work on bank ATMs. To all future job interrogations I waxed off, protesting I do boring software stuff that nobody cares about. Not put off in the least, grandma asked me last night to explain this cloud computing thing she keeps hearing about at her church club. Afraid of being another victim of the distortion field surrounding cloud computing, I instead referred her to Kent Langley's excellent overview of the subject in Cloud Computing: Get Your Head in the Clouds. It does a good job demystifying the very confusing concept of cloud computing. It has nice diagrams, definitions, examples and is a great place to start. She agreed that she had learned a lot, but one thing still troubled her: what's the difference between cloud computing and utility computing? They seem to be the same to her. Always so perceptive. She felt sure if she could drive this point home she would score big points with her church group. Oh the pressure. I steadied myself and explained 3Tera’s take is that cloud computing is for service users and utility computing is for service builders. Cloud computing is essentially about the surrender of control. Users of a service like Salesforce.com don’t care how the site is implemented. They don’t care about how it scales, deals with failure, or any of the other 1000s of little details you have to care about when running a complicated operation. Users just want their service to work when they need it. Utility computing customers on the other hand require fine control over their resources because they are the builders of services like Salesforce.com. Cloud computing is built on utility computing. You couldn’t build a Salesforce.com on Google whereas you could build it on top of 3Tera or Amazon. StorageMojo thinks all this cloud/utility nonsense is just foggy thinking. Real computing will stay local because the cost of network access is too high. Memory and CPU are plentiful and cheap while bandwidth is neither. Distributed computing 1990s style will still rule the day. Mike Nygard thinks there’s A Cloud for Everyone in the future. Latency matters and “Keeping your endpoints on your own network at least lets you control your own latency.” Security matters and pushing your precious data into the hands of strangers isn’t secure. Yet we see SalesForce, Google Docs, Basecamp, SugarCRM, and hosted email all flourishing so is privacy really a concern for newer generations trying to get stuff done? HP’s Patrick Eitenbichler thinks “utility computing refers to a business model, while cloud computing describes the underlying IT architecture” with the real decision point being “utility/cloud computing vs. purchasing your own IT assets.” Geva Perry writing for GigaOM essentially agrees with Mr. Eitenbichler saying: Utility computing relates to the business model in which application infrastructure resources — hardware and/or software — are delivered. While cloud computing relates to the way we design, build, deploy and run applications that operate in a virtualized environment, sharing resources and boasting the ability to dynamically grow, shrink and self-heal. Krish tries to condense that down to: cloud computing is software as a service (where companies run their own software) and utility computing is hardware as a service (where you can run your own software). Margaret Rouse makes a good case for cloud computing being just a better marketing concept for utility/grid/cluster/distributed/parallel computing. Bits or Pieces smartly ignores saying the word cloud but my impression is they think providing Software as a Service on a utility computing basis is the game changing innovation. James Urquhart defines the cloud to include: SaaS, PaaS (e.g. force.com) and HaaS (e.g. Amazon, Mosso, etc.). SaaS is in clearly in play today, HaaS is being experimented with, but PaaS may be the most interesting facet of the cloud in the long term. Keystones and Rivets finds that “The Cloud” is grid computing wrapped up in a service offered by data centers. Confident I must have answered her original question, I asked “Now, doesn’t that clear things up grandma?” Grandma sniffed, said that's all very nice, but she still wanted to know why the ATM ate her bank card! I groaned and said “Goodnight grandma. I’ll call again next week.” “Excellent,“ she Cheshire smiled, “next week my church group is going to tackle if social networks are really monitizeable.”
Hi, I was wondering if I could borrow the collective minds of you all to draw up a list to the CDN's that you'd use/do use in the UK. If they're outside the UK but have decent support then also include. The service must be cheap and not require a huge setup fee, it's really only for a small time business; it shares video & high-res pics so mass cheap storage is a must and wondered whether you guys had any ideas, also costs? Mass storage isn't cheap in the UK compared to the states, for example, unless I go colo but as I say, it's a small setup but happens to require a fair bit of space. Would S3 be a good starting point? What is the service like? I hear mixed reviews about it. Many thanks, Jim
The Third International Conference on Scalable Information Systems will focus on a wide array of scalability issues and investigate new approaches to tackle problems arising from the ever-growing size and complexity of information of all kinds. Looking at their technical program a lot of interesting topics will be covered. I see sensor networks, a subject I'm really interested in, has a number of sessions. That's unusual. And it's in Italy!
A lot has been said already about Twitter's scalability issues. Many have given Twitter as an anti-pattern of how not to deal with scalability and have suggested different solutions for scaling it. As Twitter is famously a Ruby-on-Rails deployment, this case has also been used as a weapon in the language/platform wars between the RoR and Java camps, and to a lesser degree, also with the LAMP (PHP) camp
Searching around the HS website I noticed that there are no articles regarding db2, which has an express edition, free of charge and from what I know there aren't any restrictions. Being a powerful database system I thought it could make be an alternative to MySQL, PostgreSQL databases. Here is the IBM statement: "DB2 Express Edition for Community (DB2 Express-C) is a no charge data server for use in development and deployment. DB2 Express-C supports a full range of APIs, drivers, and interfaces for application development including PHP, C/C++, and .NET. In addition, DB2 Express-C V9 contains advanced XML features. DB2 Express-C provides ISVs an ideal starting database server for Web, enterprise, and eBusiness applications. This IBM Redbook provides fundamentals of DB2 application development with DB2 Express-C. It covers the DB2 Express-C installation and configuration for application development and skills and techniques for building DB2 applications with XML, PHP, C/C++, Java, and .NET. Code examples are used to demonstrate how to develop a DB2 application in a different language. By following the examples provided, you will be able to learn DB2 application development with XML, PHP, C/C++, Java, and .NET in a short time." Download the redbook about db2 express-c.
Nobody came up with an example of a website powered by a Websphere product (which has a community edition) and backed up by a DB2 database. I guess you all know about usopen.org so here's the story: While the re-emergence of 35-year-old Andre Agassi and the continued dominance of wunderkind Maria Sharapova have highlighted the on-court headlines at this year's U.S. Open Tennis Championships in Flushing Meadows, N.Y., IBM is hoping its new Power5 chip-based IT support for USOpen.org can make news among those more interested in .NET than tennis nets. Big Blue has partnered with the U.S. Tennis Association and the U.S. Open -- the most prestigious tennis tournament in the U.S. -- since 1992. Together, they launched USOpen.org in 1995 so racket heads could follow the matches online. The iSeries' role this year is in powering a Web-based end-user application called "Point Tracker," a graphics tool using autonomic technology that recreates the trajectory of every shot. On-court cameras capture and record ball position data for every forehand, ace and volley. Once that data is integrated with the scoring data, the shot data is pushed to the Web site to enable visitors to follow the action online. IBM is running the Web site on an eServer pSeries system, a Power5-based server. Two pSeries systems, models p550 -- released two weeks ago -- and p570, replaced Web and application servers to help automate the infrastructure that supports the Web site. The 2005 U.S. Open Web site traffic will be managed by Big Blue from a "virtualized" server environment at one of the three hosting locations. According to IBM, the pSeries systems allow IBM to consolidate several servers onto two larger boxes. The pSeries p5 systems handle USOpen.org workloads from Web serving to fan polling, feedback and player search applications, which are managed from each pSeries p5 server as a virtualized environment using Power-based virtualization technologies such as Micro-Partitioning, Virtual I/O Server and Partition Load Manager, which consolidate AIX 5L and multiple Linux operating environments onto a single system. Approximately 2.8 million fans visited USOpen.org during the two-week tournament in 2004. More information on this technologies can be found here: Quote from IBM redbooks: Building a high performance and high availability commerce site is not a trivial task -- from having right capacity hardware to handle the workload to properly testing the code change before deploying in production site. This redbook covers several major areas that need to be considered when using WebSphere Commerce Server and provide solution on how to address them. Here are some of the topics: 1. How to build a Commerce site to deal with various kind of unplanned outage? Topic including utilizing WebSphere Application Server Network Deployment 6.0 and IBM DB2 High Availability disaster Recovery (HADR) in Commerce environment. 2. How to build a Commerce site to deal with planned outages such as software fix and operation update? Topic including uses of WebSphere Application Server's Rolling update feature and uses of Commerce's Staging Server and Content Management. 3. How to proactively monitoring the commerce site prevent potential problem happening? Various Tools should be discussed including various WebSphere Application Server build-in tools and Tivoli's Composite Application Management. 4. How to utilized dynacache to future enhance your Commerce Site's performance? Topics includes additional Commerce command caching introduce in Commerce Fix pack and e-spot caching. 5. What's the methodology of doing performance and scalability testing on Commerce site? Tools that may be covered included Tivoli Performance Tester 6. Techniques on migrate a high volume Commerce site to newer Commerce release. " end Quote Maybe some of us can find this useful, Websphere Community Edition is a free Java™ EE 5 server for building and managing Java™ applications. Download this Redbook
Update: Erlang at Facebook by Eugene Letuchy. How Facebook uses Erlang to implement Chat, AIM Presence, and Chat Jabber support.
I've done some XMPP development so when I read Facebook was making a Jabber chat client I was really curious how they would make it work. While core XMPP is straightforward, a number of protocol extensions like discovery, forms, chat states, pubsub, multi user chat, and privacy lists really up the implementation complexity. Some real engineering challenges were involved to make this puppy scale and perform. It's not clear what extensions they've implemented, but a blog entry by Facebook's Eugene Letuchy hits some of the architectural challenges they faced and how they overcame them.
A web based Jabber client poses a few problems because XMPP, like most IM protocols, is an asynchronous event driven system that pretty much assumes you have a full time open connection. After logging in the server sends a client roster information and presence information. Your client has to be present to receive the information. If your client wants to discover the capabilities of another client then a request is sent over the wire and some time later the response comes back. An ID is used to map the reply to the request. All responses are intermingled. IM messages can come in at any time. Subscription requests can come in at any time.
Facebook has the client open a persistent connection to the IM server and uses long polling to send requests and continually get data from the server. Long polling is a mixture of client pull and server push. It works by having the client make a request to the server. The client connection blocks until the server has data to return. When it does data is returned, the client processes it, and then is in position to make another request of the server and get any more data that has queued up in the mean time. Obviously there are all sorts of latency, overhead, and resource issues with this approach. The previous link discusses them in more detail and for performance information take a look at Performance Testing of Data Delivery Techniques for AJAX Applications by Engin Bozdag, Ali Mesbah and Arie van Deursen.
From a client perspective I think this approach is workable, but obviously not ideal. Your client's IMs, presence changes, subscription requests, and chat states etc are all blocked on the polling loop, which wouldn't have a predictable latency. Predictable latency can be as important as raw performance.
The real scaling challenge is on the server side. With 70 million people how do you keep all those persistent connections open? Well, when you read another $100 million was invested in Facebook for hardware you know why. That's one hella lot of connections. And consider all the data those IM servers must store up in between polling intervals. Looking at the memory consumption for their servers would be like watching someone breath. Breath in- streams of data come in and must be stored waiting for the polling loop. Breath out- the polling loops hit and all the data is written to the client and released from the server. A ceaseless cycle. In a stream based system data comes in and is pushed immediately out the connection. Only socket queue is used and that's usually quite sufficient. Now add network bandwidth for all the XMPP and TCP protocol overhead and CPU to process it all and you are talking some serious scalability issues.
So, how do you handle all those concurrent connections? They chose Erlang. When you first hear Erlang and Jabber you think ejabberd, an open source Erlang based XMPP server. But since the blog doesn't mention ejabberd it seems they haven't used it .
Why Erlang? First, the famous Yaws vs Apache shootout where "Apache dies at about 4,000 parallel sessions. Yaws is still functioning at over 80,000 parallel connections." Erlang is naturally good at solving high concurrency problems. Yet following the rule that no benchmark can go unchallenged, Erik Onnen calls this the Worst Measurement Ever and has some good reasoning behind it.
In any case, Erlang does nicely match the problem space. Erlang's approach to a concurrency problem is to throw a very light weight Erlang process at each state machine you want to be concurrent. Code-wise that's more natural than thread pools, async IO, or thread per connection systems. Until Linux 2.6 it wasn't even possible to schedule large numbers of threads on a single machine. And you are still devoting a lot of unnecessary stack space to each thread. Erlang will make excellent use of machine resources to handle all those connections. Something anyone with a VPS knows is hard to do with Apache. Apache sucks up memory with joyous VPS killing abandon.
The blog says C++ is used to log IM messages. Erlang is famously excellent for its concurrency prowess and equally famous for being poor at IO, so I imagine C++ was needed for efficiency.
One of the downsides of multi-language development is reusing code across languages. Facebook created Thrift to tie together the Babeling Tower of all their different implementation languages. Thrift is a software framework for scalable cross-language services development. It combines a powerful software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, and Ruby. Another approach might be to cross language barriers using REST based services.
A problem Facebook probably doesn't have to worry about scaling is the XMPP roster (contact list). Handling that many user accounts would challenge most XMPP server vendors, but Facebook has that part already solved. They could concentrate on scaling the protocol across a bunch of shiny new servers without getting bogged down in database issues. Wouldn't that be nice :-) They can just load balance users across servers and scalability is solved horizontally, simply by adding more servers. Nice work.
Hi, First of all I want to to say that this is an extremely interesting and informative website. i have enjoyed reading the various posts on how the big sites scale to meet the needs of their customers. The service we are developing is a webcam service. The client application sends images to the server via HTTP POST and they are saved in folder specified by the users id. When a new image is sent to the server it will overwrite the current image. Users can then view the images via our web server. Ideally we want the images to upload as quickly as possible and allow users to view them as quickly as possible. Would I be correct to assume that when the number of uploading clients exceeds the capability of the server the only way to scale is to add more hardware. Also I assume that to use HTTP accelerator caches will not speed up viewing the images as the new images will invalidate the cache. I appreciate any input on the subject.
High Performance Multithreaded Access to Amazon SimpleDB is a great follow up to the idea in How SimpleDB Differs from a RDBMS that more programming is the price paid for performance in SimpleDB. It shows how much work and infrastructure is required to batter better performance out of SimpleDB. Remember, in SimpleDB you get keys to records from queries so if you want to get all the fields for records you need to make separate requests. Since SimpleDB isn't exactly a speed daemon the obvious strategy is to parallelize. Even if a job takes a 100 msecs you can get a lot done in a little time if you can execute enough jobs in parallel. Parallelization is the approach taken by Haakon@AWS in his Java code example of how to get the most out of SimpleDB. You can find the code at Indexing and Querying Amazon S3 Metadata with Amazon SimpleDB. We'll also consider how a back-end service architecture built on Erlang may be a better fit with cloud computing. Two general mechanisms of parallelism are available: threads and boxes. To get the most bang out of a single machine you need threads (events, etc). To scale beyond the load handled by a single machine you need multiple boxes. The example code uses the Executor Thread Pool for parallelism within a program. Thread pools are a pretty common idiom by now. Amazon's queue service SQS was used to distribute work amongst boxes. Work was queued to SQS in batches of 1000 work items. The items were pulled by the thread pool and processed. Why 1000? The idea is to balance processing overhead with work overhead. You don't want popping items off SQS to dominate your processing time so you have to do enough work in each pass to make it worth the investment. The architecture uses two thread pools: one to run queries and one to get record values. Applications must carefully tune the number of threads in each pool so the queries to overwhelm the gets. Using a query thread pool with 2 threads and a get thread pool with 32 threads it was possible to perform 300 TPS on a small EC2 instances. Theoretically the advantage of this architecture is that it will scale to any size you need. SQS is your work distribution backbone and you just spin up the number of thread pool instances you need. The disadvantage is that this is a lot of programmer effort. But let's consider that you had to do some serious processing on each record, you would need something like this approach anyway to scale out the processing. But to perform simple aggregation operations it's total overkill which is why more time needs to be spent on the write site of the equation in SimpleDB/BigTable than the read side as we are used to with a RDBMS. What's the best way to go parallel? On the front-end life is simple. Go shared nothing and compose your pages from scalable back-end services. This is how Amazon does it and it's how Google AppEngine does it. GAE completely punts on the back-end service layer architecture. Unfortunately we still need to create a back-end architecture for more complex applications. Thread pools and SQS is one parallelization approach. Instead of thread pools something like Java's fork/join framework could be used. Initially I thought piling on more low level primitive threading facilities into Java was the wrong way to go. Yes, it is a "'multicore-friendly lightweight parallel framework' that supports a style of parallel programming where problems are recursively split into smaller fragments, solved in parallel and recombined," but it's also a style of programming that is very difficult to program correctly. If cloud architectures will rely on these primitives for efficiency then I think we have regressed. Erlang style architectures described by Luke Hoersten in Scalable Web Apps: Erlang + Python is a simpler more reliable to programming model. An event driven actor based approach is much harder to screw up than closely cooperating threads in a shared memory space. Erlang originally ran in embedded systems where the requirement was to reliably squeeze the most work possible out of limited CPU and other compute resources. Oddly enough the embedded node of old closely parallels your basic cloud VM. Start your work horse Erlang (or other similar system) instances and let them efficiently chew up your work loads. Erlang's scheduling model fits perfectly with a service centric job engine cloud instance. It will get more work done then your typical thread based system ever would.
HSCALE - Handling 200 Million Transactions Per Month Using Transparent Partitioning With MySQL Proxy
Update 2: A HSCALE benchmark finds HSCALE "adds a maximum overhead of about 0.24 ms per query (against a partitioned table)." Future releases promise much improved results. Update: A new presentation at An Introduction to HSCALE. After writing Skype Plans for PostgreSQL to Scale to 1 Billion Users, which shows how Skype smartly uses a proxy architecture for scaling, I'm now seeing MySQL Proxy articles all over the place. It's like those "get rich quick" books that say all you have to do is visualize a giraffe with a big yellow dot superimposed over it and by sympathetic magic giraffes will suddenly stampede into your life. Without realizing it I must have visualized transparent proxies smothered in yellow dots. One of the brightest images is a wonderful series of articles by Peter Romianowski describing the evolution of their proxy architecture. Their application is an OLTP system executing 200 million transaction per month, tables with more than 1.5 billion rows, and a 600 GB total database size. They ran into a wall buying bigger boxes and wanted to move to a sharded architecture. The question for them was: how do you implement sharding? In the first article four approaches to sharding were identified: