Great Open Source Solution for Boring HA and Scalability Problems
This is a guest post about how boring and repetitive HA and scalability problems can be solved via Open Source so you can focus on the interesting tasks. The post was written by Maarten Ectors, responsible for Cloud Strategy and Frank Mueller, a Juju Core developer, at Ubuntu/Canonical.
High-availability and scalability are exciting in general but there are certain problems that experts see over and over again. The list is long but examples are setting up MySQL clustering, sharding Mongo, adding data nodes to a Hadoop cluster, monitoring with Ganglia, building continuous deployment solutions, integrating Memcached / Varnish / Nginx,… Why are we reinventing the wheel?
At Ubuntu we made it our goal to have the community solve these repetitive and often boring tasks. How often have you had to set-up MySQL replication and scale it? What if the next time you just simply do:
- juju deploy mysql
- juju deploy mysql mysql-slave
- juju add-relation mysql:master mysql-slave:slave
- juju add-unit -n 10 mysql-slave
It’s easy to see how these four commands work. After deploying a master and a slave MySQL both are related as master and slave. After this you simply can add more slaves like it is done here with 10 more instances.
Responsible for this easy approach is one of our latest open source solutions, Juju. Juju allows any server software to be packaged inside what is called a Charm. A Charm describes how the software is deployed, integrated and scaled. Once an expert creates the Charm and uploads it to the Charm Store, anybody can use it instantly. Execution environments for Charms are abstracted via providers. The list of supported providers is growing and includes Amazon AWS, HP Cloud, Microsoft Azure, any Openstack private cloud, local LXC containers, bare metal deployments with MaaS, etc. So Juju allows any software to be instantly deployed, integrated and scaled on any cloud.
There are over a 100 Charms already and many of the most common server solutions already can be deployed, integrated and scaled instantly, e.g. Hadoop/HBase/Hive, Cassandra, MongoDB, MySQL/Postgres clusters, Memcached, Redis, Varnish, Nginx, HAProxy, Logstash, ElasticSearch, OpenTSDB, Nagios, Ganglia, Storm, Munin, Rails / Rack / Passenger, Node.JS, Tomcat clusters, PHP Lamp, Django, etc.
While the command line is the best friend for many administrators it is often very helpful to get a graphical overview and simple way of configuration for larger environments. Here Juju provides a GUI which can be installed via the Juju GUI Charm. Here new services can be deployed and integrated via drag-and-drop.
So by having an ever growing eco-system of charms, you do not have to reinvent deployment and more importantly integration and scaling any more. Additionally if you have your stack “charmed up” then you have the choice of any public/private cloud or bare metal so no more lock-in.
The first question we always get is how is Juju different from Puppet or Chef? Juju is focusing on services, Puppet and Chef focus on server provisioning. This means that you can use Puppet Manifests or Chef Recipes inside Charms to do the software install. A good example is the Rack Charm that uses Chef Recipes. But unlike Puppet or Chef when you start scaling up your architecture you scale up services not servers:
As you can see from the above example Ganglia is monitoring 100 Cassandra nodes. If you would scale to 200 you would not have to update Ganglia because Ganglia is monitoring the “Cassandra Service” and it will automatically add the 100 extra nodes.
Another question we often get is how is Juju different from PaaS? Cloud Foundry is a Charm and runs on top of Juju so if you need “traditional PaaS” you can. However often there are software solutions that can not work inside a PaaS and need another framework, e.g. Django, Rails, etc. Juju allows you to combine the power of IaaS and PaaS by creating your own PaaS where you assemble all needed solutions, sort of a DIY PaaS.
Ubuntu is eating its own dog food. Juju and MaaS are the default deployers for Openstack. So all large Openstack deployments that are currently happening on Ubuntu use Juju and MaaS. Additionally Ubuntu One and other Ubuntu cloud services are all running on top of Juju.
Except for boring scalability problems, Ubuntu is also working with startups around designing an instant continuous deployment solution hence every startup in the world can launch new features every hour. Additionally we launched a contest around building new solutions on top of Juju. There are many categories but high availability, monitoring and continuous deployment are probably of most interest to readers of High Scalability. Each category has a $10,000 price attached and there are judges like Adrian Cockcroft, Netflix’s Chief Cloud Architect.
However we don’t want this blog post to be an advertisement. Every new open source product has its limitations. We are aware that several charms are still rough around the edges. Certain charms are missing key relations. We are missing Charms and Cloud providers. Certain Cloud providers have still limitations. We should even further simplify complete stack deployments. Make different Juju environments talk to one another. Although OSX and Windows are supported on the client side, we only support Ubuntu on the server for now. This is where we would hope, the community can assist us in filing bugs, feature requests and even solving bugs and adding features. That is the beauty about open source, each addition provides synergies for the whole community. We hope you enjoy Juju and if you want to just play around with the GUI, you can go to JujuCharms.com.
