Stop Running Twenty Services Locally

James Carr / June 21, 2013

Tell me if you're familiar with this scenario. You develop a handful of web applications and backend services at your company. As a result, for local development you need the following installed.

• mysql
• redis
• rabbitmq
• nginx
• elasticsearch
• graylog2

But that's not all! The work you do with a team located at the european branch of your company runs zookeeper, postgresql, mongodb, solr and Jetty. So as a result you have all these things installed and configured on your machine. And lord forbid you also do some freelance work on the side which adds several other services installed on your box!

Just Stop

It's my firm belief that installing all of these services on your laptop or workstation is a bad thing. You're mixing environments required for different applications into your personal workspace. You also lose repeatability as changes to the development environment are difficult to communicate aside from detailed emails or wiki entries. Finally, it's way too easy to make a trivial change that allows a new feature to work on your machine but completely fail for everyone else. What we need is a way to ensure repeatability of an environment in an isolated fashion that is easy to share with our peers.

Enter Vagrant

Here at zapier we use vagrant not just to emulate our infrastucture for local testing but to also allow developers to just clone our main project and run it locally with all the services it requires, no strings attached. But before we get into all of that, I thought it'd be a good introductory post to illustrate using vagrant to launch a virtual machine with mysql running with some data seeded into it.

The Vagrantfile

First we'll need a Vagrantfile that contains the plumbing so to speak that describes how we want to launch our virtualbox image. This allows you to tweak such things like the box name, the url to download the box from if it isn't present, a host-only IP address for the guest machine and the amount of RAM to give the guest.

For this example we're going to use puppet, but you could also use the configuration management tool of your choice whether it's chef, ansiable or salt. :)

Required Puppet Modules

Since we're using puppet, let's take advantage of the existing puppetlabs/mysql module! This means we're going to require the mysql and stdlib module. Although there are better ways to manage your puppet modules, for this tutorial we're going to keep it simple to lower the barrier to entry and just clone the modules locally. So we'll run the following commands in our project after creating a module directory:

git clone https://github.com/puppetlabs/puppetlabs-mysql.git modules/mysql

git clone https://github.com/puppetlabs/puppetlabs-stdlib.git modules/stdlib

With these modules in place, the next step we'll use the mysql module to install and configure mysql to our liking.

The Manifest

Finally, we'll tie this all together by defining a default node definition in manifests/site.pp that does the heavy lifting.

Some key points to take note of here is that we bind to 0.0.0.0 and configure the user to use a wildcard for its host. This allows all remote connections to connect to the guest, including the host machine. One last detail is that we set the sql attribute to /vagrant/loaddata.sql. First let's include that here and place it in the root directory of the project (alongside Vagrantfile).

This will basically get ran against the database when it is created.

Running It

Well that's enough details. Let's go ahead and run this thing! From the project directory type

vagrant up

And wait. The first time can always take a little while, especially since you're going to download a 200 to 300mb linux image from the internet. The good thing is that once you have this running you can run vagrant halt or vagrant suspend so that it starts in seconds next time you type vagrant up. :)

You should see a some console output fly by. Once it is finished, let's ssh into the box and poke around! Just run vagrant ssh to log in (this will be more difficult on windows, see these instructions for getting in on windows). Once on the box, take a look in the /vagrant directory and you'll notice the project directory is located here (which is where we specified the loaddata.sql file would be). Feel free to look around and type exit to return to the host machine when you are done.

Finally, let's connect to the box and run a couple queries! If you have an existing application in your favorite language, go ahead and pug the authentication details into it as an exercise. Otherwise if you have mysql-client installed on your host machine use that to connect to the mysql instance running on the virtual machine.

mysql -u bob -p -h 33.33.33.33 testdb

> SELECT * FROM album;

We now have a repeatable and shareable instance of mysql. It's possible to share this with our co-workers and allow them to just check the project out, type vagrant up and start working in the code base without having to figure out how to install mysql themselves on their machine.

The Road Forward

You don't have to stop at just a database. You can use vagrant and your favorite configuration management tool to completely define the infrastructure your application requires from the web server to the databases to even other fancy things like a message broker. And as a final touch, you can share the project directory on the machine, meaning developers can just type vagrant up and start working against an environment similar to production regardless of the OS they run. Epic win!:)

A complete working copy of the code described here can be found on github.