advertise
Thursday
Feb212008

Product: Capistrano - Automate Remote Tasks Via SSH

Update: Deployment with Capistrano  by Charles Max Wood.  Nice simple step-by-step for using Capistrano for deployment.

From their website:
Simply put, Capistrano is a tool for automating tasks on one or more remote servers. It executes commands in parallel on all targeted machines, and provides a mechanism for rolling back changes across multiple machines. It is ideal for anyone doing any kind of system administration, either professionally or incidentally.

* Great for automating tasks via SSH on remote servers, like software installation, application deployment, configuration management, ad hoc server monitoring, and more.
* Ideal for system administrators, whether professional or incidental.
* Easy to customize. Its configuration files use the Ruby programming language syntax, but you don't need to know Ruby to do most things with Capistrano.
* Easy to extend. Capistrano is written in the Ruby programming language, and may be extended easily by writing additional Ruby modules.


One of the original use cases for Capistrano was for deploying web applications. (This is still by far its most popular use case.) In order to make deploying these applications reliable, Capistrano needed to ensure that if something went wrong during the deployment, changes made to that point on the other servers could be rolled back, leaving each server in its original state.

If you ever need similar functionality in your own recipes, you can introduce a transaction:

task :deploy do
transaction do
update_code
symlink
end
end

task :update_code do
on_rollback { run "rm -rf #{release_path}" }
source.checkout(release_path)
end

task :symlink do
on_rollback { run "rm #{current_path}; ln -s #{previous_release} #{current_path}" }
run "rm #{current_path}; ln -s #{release_path} #{current_path}"
end

The first task, “deploy” wraps a transaction around its invocations of “update_code” and “symlink”. If an error happens within that transaction, the “on_rollback” handlers that have been declared are all executed, in reverse order.

This does mean that transactions aren’t magical. They don’t really automatically track and revert your changes. You need to do that yourself, and register on_rollback handlers appropriately, that take the necessary steps to undo the changes that the task has made. Still, even as lo-fi as Capistrano transactions are, they can be quite powerful when used properly.


From the Ruby on Rail manual:

Ultimately, Capistrano is a utility that can execute commands in parallel on multiple servers. It allows you to define tasks, which can include commands that are executed on the servers. You can also define roles for your servers, and then specify that certain tasks apply only to certain roles.

Capistrano is very configurable. The default configuration includes a set of basic tasks applicable to web deployment. (More on these tasks will be said later.)

Capistrano can do just about anything you can write shell script for. You just run those snippets of shell script on remote servers, possibly interacting with them based on their output. You can also upload files, and Capistrano includes some basic templating to allow you to dynamically create and deploy things like maintenance screens, configuration files, shell scripts, and more.

Related Articles

 

  • Friends for Sale uses Capistrano for deployment.

  • Thursday
    Feb212008

    Tracking usage of public resources - throttling accesses per hour

    Hi, We have an application that allows the user to define a publicly available resource with an ID. The ID can then be accessed via an HTTP call, passing the ID. While we're not a picture site, thinking of a resource like a picture may help understand what is going on. We need to be able to stop access to the resource if it is accessed 'x' times in an hour, regardless of who is requesting it. We see two options - go to the database for each request to see if the # of returned in the last hour is within the limit. - keep a counter in each of the application servers and sync the counters every few minutes or # of requests to determine if we've passed the limit. The sync point would be the database. Going to the database (and updating it!) each time we get a request isn't very attractive. We also have a load balanced farm of servers, so we know 'x' is going to have to be a soft limit if we count in the app serevrs. (We know there will be a period of time between syncing the counts in the app servers where we'll overshoot the limit. That is okay since we'll catch the limit violation and stop the requests.) Other thoughts on how do to this? Thanks, Chris

    Click to read more ...

    Tuesday
    Feb192008

    Building a email communication system

    hi, the website i work for is looking to build a email system that can handle a fair few emails (up to a hundred thousand a day). These comprise emails like registration emails, newsletters, lots of user triggered emails and overnight emails. At present we queue them in SQL and feed them into an smtp server on one of our web servers when the queue drops below a certain level. this has caused our mail system to crash as well as hammer our DB server (shared!!!). We have got an architecture of what we want to build but thought there might be something we could buy off the shelf that allowed us to keep templated emails, lists of recipients, schedule sends etc and report on it. We can't find anything What do big websites like amazon etc use or people a little smaller but who still send loads of mail (flickr, ebuyer, or other ecommerce sites) Cheers tarqs

    Click to read more ...

    Tuesday
    Feb192008

    Hadoop Getting Closer to 1.0 Release

    Update: Yahoo! Launches World's Largest Hadoop Production Application. A 10,000 core Hadoop cluster produces data used in every Yahoo! Web search query. Raw disk is at 5 Petabytes. Their previous 1 petabyte database couldn't handle the load and couldn't grow larger. Greg Linden thinks the Google cluster has way over 133,000 machines. From an InfoQ interview with project lead Doug Cutting, it appears Hadoop, an open source distributed computing platform, is making good progress towards their 1.0 release. They've successfully reached a 1000 node cluster size, improved file system integrity, and jacked performance by 20x in the last year. How they are making progress could be a good model for anyone:

    The speedup has been an aggregation of our work in the past few years, and has been accomplished mostly by trial-and-error. We get things running smoothly on a cluster of a given size, then double the size of the cluster and see what breaks. We aim for performance to scale linearly as you increase the cluster size. We learn from this process and then increase the cluster size again. Each time you increase the cluster size reliability becomes a bigger challenge since the number and kind of failures increase.
    It 's tempting to say just jump to the end game, don't bother with all those errors and trials, but there's a lot of learning and experience that must be earned on the way to scaling anything.

    Click to read more ...

    Monday
    Feb182008

    How to deal with an I/O bottleneck to disk?

    A site I'm working with has an I/O bottleneck. They're using a static server to deliver all of the pictures/video content/zip downloads ecetera but now that the bandwith out of that server is approaching 50Mbit/second the latency on serving small files has increased to become unacceptable. I'm curious how other people have dealt with this situation. Seperating into two different servers would require a significant change to the sites architecutre (because the premise is that all uploads go into one server, all subdirectorie are created in one directory, etc.) and may not really solve the problem.

    Click to read more ...

    Monday
    Feb182008

    limit on the number of databases open

    Have a few doubts.. here are the qs 1) is there any limit on the number of databases that can be accessed simultaneously? (MySQL) 2) will it be a problem to scale in the future if there are large number of small databases(2-5 MB) each?

    Click to read more ...

    Sunday
    Feb172008

    Web Accelerators - snake oil or miracle remedy?

    Perhaps this question is borderline off-topic but since high scalability solutions often have a global aspect I will give it a try... Have anybody had any experience with different techniques for speeding up their application to places that have a problem with poor ping response time? Ideally I would love to be running only one data center world-wide but one day I know that our sales department will sign up a customer with an unacceptable response time... Could installing a web-accelerator in front of our application extend the reach of our current data center or will we just add complexity and another source of potential errors?

    Click to read more ...

    Saturday
    Feb162008

    S3 Failed Because of Authentication Overload

    Being an authentic human being is difficult and apparently authenticating all those S3 requests can be a bit overwhelming as well. Amazon fingered a lot of processor heavy authentication requests as the reason for their downtime: Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types. Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue. Interesting problem. Same thing happens with sites using a lot of SSL. They need to purchase specialized SSL concentrators to handle the load which makes capacity planning a lot trickier and more expensive. In the comments Allen conjectured What caused the problem however was a sudden unexpected surge in a particular type of usage (PUT's and GET's of private files which require cryptographic credentials, rather than GET's of public files that require no credentials). As I understand what Kathrin said, the surge was caused by several large customers suddenly and unexpectedly increasing their usage. Perhaps they all decided to go live with a new service at around the same time, although this is not clear. We see these kinds of bring up problems all the time. The Skype failure was blamed on software updates which caused all nodes to relogin at the same time. Bring up a new disk storage filer and if you aren't load balancing requests all new storage requests will go to that new filer and you'll be down lickity split. Booting is one of the most stressful times on large networks. Bandwidth and CPU all become restricted which causes a cascade of failures. ARP packets can get dropped or lost and machines never get their IP addresses. Packets drop which causes retransmissions which chews up bandwidth which uses CPU and causes more drops. CPUs spike which causes timeouts and reconnects which again spirals everything out of control. When I worked at a settop company we had the scenario of a neighborhood rebooting after a power outage. Lots of houses needing to boot large boot images over asymmetric low bandwidth cable connections. As a fix we broadcasted boot image blocks to all settops. No settops performed your typical boot image download. Worked like a charm. Amazon's problem was a subtle one in a very obscure corner of their system. It's not surprising they found a weakness. But I'm sure Amazon will be back even bigger and better once they get their improvements on line.

    Click to read more ...

    Wednesday
    Feb132008

    What's your scalability plan?

    How do you plan to scale your system as you reach predictable milestones? This topic came up in another venue and it reminded me about a great comment an Anonymous wrote a while ago and I wanted to make sure that comment didn't get lost. The Anonymous scaling plan was relatively simple and direct: My two cents on what I'm using to start a website from scratch using a single server for now. Later, I'll scale out horizontally when the need arises. Phase 1

  • Single Server, Dual Quad-Core 2.66, 8gb RAM, 500gb Disk Raid 10
  • OS: Fedora 8. You could go with pretty much any Linux though. I like Fedora 8 best for servers.
  • Proxy Cache: Varnish - it is way faster than Squid per my own benchmarks. Squid chokes bigtime.
  • Web Server: Lighttpd - faster than Apache 2 and easier to configure for me.
  • Object Cache: Memcached. Very scalable.
  • PHP Cache: APC. Easy to configure and seems to work fine.
  • Language: PHP 5 - no bloated frameworks, waste of time for me. You spend too much time trying to figure out the framework instead of getting work done.
  • Database - MySQL 5. I didn't consider Postgres because I've never used it. There are just a lot more tools available for MySQL. Phase 2
  • Max Ram out to 64 GB, cache everything Phase 3
  • Buy load balancer + 2 more servers for front end Varnish/Memcached/Lighttpd.
  • Use original server as MySQL database server. Phase 4 Depending on my load & usage patterns, scale out the database horizontally with an additional server. I don't expect the db to be a bottleneck for my website as only metadata info is stored there. I'll mostly be serving images stored on the file system. Possibly separate Varnish / Memcached / Lighttpd tier into separate tiers if necessary. But I'll carefully evaluate the situation at this point and scale out appropriately and use CDN for static content if necessary. Phase 5
  • Max all servers to 64gb of RAM, cache, cache, cache. Phase 6
  • If I get this far then I'm a multi-millionaire already so I'll replace all of the above machines with whatever the latest and greatest is at that time and keep scaling out. The important point is that I know how to scale each layer when/if the need arises. I'll scale the individual machines when necessary and scale horizontally too. In previous post we also read where ThemBid has a nice simple scalability plan too :
  • Use Munin to tell when to think about upgrading. When your growth trend will soon cross your resources trend, it's time to do something.
  • Move MySQL to a separate server. This frees up resources (CPU, disk, memory). What you want to run on this server depend on its capabilities. Maybe run a memcached server on it.
  • Move to a distributed memory cache using memcached.
  • Add a MySQL master/slave configuration.
  • If more webservers are needed us LVS on the front end as a load balancer. The great insight ThemBid had was to use monitoring to predict when performance is hitting a limit so you can take action before the world ends. Take a look at Monitoring for some options. Some examples of how monitoring is used: Feedburner Architecture, Scaling Early Stage Startups, Thembid Architecture, Flickr Architecture, Ebay Architecture. Most problems are pretty predictable, especially if you read this site. Have a plan in mind for what you want to do when you grow. You don't have to do all now, but make the path easier by starting in the right direction now. You'll also be much less stressed when all hell breaks loose.

    Click to read more ...

  • Tuesday
    Feb122008

    Search the tags across all post

    Let suppose i have table which stored tags .Now user can enter keywords and i have to search through all the records in table and find post which contain tags entered by user .user can enter more than 1 keywords. What strategy ,technique i use to search fast .There maybe more than millions records and many users are firing same query. Thanks

    Click to read more ...