Developer Blog

Move fast and don’t break things! Testing with Jenkins, Ansible and Docker

One of our highest priorities at Mist.io is to never break production. Our users depend on it to manage and monitor their servers and we depend on them. At the same time, we need to move fast with development and deliver updates as soon as possible. We want to be able to easily deploy several times per day.

A big part of Mist.io is the web interface so we use tools like Selenium, Splinterand Behave for headless web interface testing. However testing the UI is time-consuming. Having to wait 40 minutes to get a green light before merging each pull request is not very agile.

In this post we'll describe the setup we've used to reduce testing time. It is based on Jenkins, Ansible and Docker and its main mission is to automate build steps and run tests in parallel and as quickly as possible.

Since execution time is essential for us, we opted to run our CI suite on one of Rackspace's High-Performance Cloud Servers due to their fast performance and short provisioning times. In general, make sure you give your test server enough RAM and a fast disk.

Our first stop for our testing suite, is Jenkins. We've used Jenkins in other projects before and we feel comfortable with it since it is mature, widely used and provides great flexibility through its plugin system. Jenkins has very good Github integration which is another plus for us. It is quite simple to set it up so that every commit in a branch will trigger the tests.

When our needs started growing, our first thought was to add more Jenkins nodes and have multiple tests running in parallel. But there are many different environments that we want to test every commit against:

  • Test the deployment of the app in a clean environment, a fresh build.
  • Test the deployment of the app in a staging environment where you want to ensure backwards compatibility
  • Test the web interface against all supported browsers.

All these meant that testing requirements would grow over time and a testing infrastructure based solely on Jenkins would not do the job fast enough.

Enter Docker

Docker helps you easily create lightweight, portable, self-sufficient containers from any application. It is fast, reliable and a perfect fit for our needs. The idea, is to set up Jenkins so that every pull request to a specific branch (e.g. staging) triggers a job. Jenkins then commands our testing infrastructure to spawn different docker applications simultaneously and runs the tests in parallel.

This way, we can test any environment. We just have to describe each application the same way we would've done manually. On the plus side, we can use pre-made images from the docker repository to jumpstart our tests.

For example we could do this:

docker pull ubuntu

And we will end up with an ubuntu image/application. We can then run the docker container by typing:

docker run -i -t ubuntu /bin/bash

And we will be in a fresh, newly created ubuntu environment.

But we don't want to manually run docker commands after every pull request. What we need, are Dockerfiles. Dockerfiles are sets of steps that describe an image/container. We'll use them to build our custom docker images.

Dockerfile commands are fairly simple. For example:

FROM ubuntu:latest

MAINTAINER mist.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list

RUN apt-get update

RUN apt-get upgrade -y

RUN apt-get install -y build-essential git python-dev python-virtualenv

RUN apt-get install -y xterm

RUN apt-get install -y -q x11vnc xvfb

RUN apt-get install -y xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic

RUN add-apt-repository -y ppa:mozillateam/firefox-next

RUN apt-get update

RUN apt-get install -y firefox

RUN mkdir MIST

RUN cd MIST && git clone https://github.com/mistio/mist.io

WORKDIR MIST/mist.io

RUN git pull

RUN cp settings.py.dist settings.py

RUN echo JS_BUILD = True >> settings.py

RUN echo CSS_BUILD = True >> settings.py

RUN echo SSL_VERIFY = True >> settings.py

RUN virtualenv . && ./bin/pip install --upgrade setuptools

RUN ./bin/python bootstrap.py && ./bin/buildout -N

ADD ./test_config.py src/mist/io/tests/features/

ADD ./init.sh /

ENTRYPOINT ./init.sh

FROM Chooses the base image (ubuntu/latest)

RUN Runs the following commands. First, we update the system ant then we install Xvfb, the latest firefox etc in order to run headless browser steps. Finally, we build the Mist.io app

ADD Adds files from our host machine to the docker image. In this example, we added the configuration for tests and an init.sh script. The init.sh script could be as simple as this:

#!/bin/bash

cd MIST/mist.io

git checkout $BRANCH

./bin/run_test

ENTRYPOINT This tells docker to start with ./init.sh script every time we run the image.

To build the docker image for future reuse:

docker build -t mist/iotest /path/to/Dockerfile

Now we have a test environment. If there are more tests in other branches that need to be spawned we can use it for every one of them:

docker run -e BRANCH=your_branch mist/iotest

If we had multiple test servers, we would have to build every custom image in every test server. Fortunately, docker lets you have your own private repository of docker images. You can build your custom image once, say mist/iotest, push it to the repository and run:

docker pull mist:iotest

This is a simple test scenario, but the possibilities are endless. For example, in another of our test scenarios we want to spawn a docker application with our monitor service and one with the mist web app.

The problem, is that we need every test server configured with docker and every docker image available. And we need to automate the procedure to be able to scale our test server infrastructure whenever needed.

Ansible to the rescue

Ansible automates deployment. It is written in Python and installation is relatively simple. It is available through most linux distro repositories, you can clone it from github or install it via pip.

Configuring ansible is also easy. All you have to do is group your servers in an ansible_hosts file and use ansible's playbooks and roles to configure them.

For example, this is a simple ansible_hosts file:

[testservers]

testserver1 ansible_ssh_host=178.127.33.109

testserver2 ansible_ssh_host=178.253.121.93

testserver3 ansible_ssh_host=114.252.27.128

[testservers:vars]

ansible_ssh_user=mister

ansible_ssh_private_key_file=~/.ssh/testkey

We just told Ansible that we have three test servers, grouped as testservers. For each one the user is mister and the ssh key is testkey, as defined in the [testservers:vars] section.

Each test server should have docker installed and a specified docker image built and ready for use. To do that, we have to define some playbooks and roles:

- name: Install new kernel
  sudo: True
  apt:
    pkg: ""
    state: latest
    update-cache: yes
  with_items:
    - linux-image-generic-lts-raring
    - linux-headers-generic-lts-raring
  register: kernel_result

- name: Reboot instance if kernel has changed
  sudo: True
  command: reboot
  register: reboot_result
  when: "kernel_result|changed"

- name: Wait for instance to come online
  sudo: False
  local_action: wait_for host= port=22 state=started
  when: "reboot_result|success"

- name: Add Docker repository key
  sudo: True
  apt_key: url="https://get.docker.io/gpg"

- name: Add Docker repository
  sudo: True
  apt_repository:
    repo: 'deb http://get.docker.io/ubuntu docker main'
    update_cache: yes

- name: Install Docker
  sudo: True
  apt: pkg=lxc-docker state=present
  notify: "Start Docker"

- name: Make dir for io docker files
  command: mkdir -p docker/iotest

- name: Copy io Dockerfiles
  template:
    src=templates/iotest/Dockerfile.j2 dest=docker/iotest/Dockerfile

- name: Copy io init scripts
  copy:
    src=templates/iotest/init.sh dest=docker/iotest/init.sh

- name: Build docker images for io
  sudo: True
  command: docker build -t mist/iotest docker/iotest

However, building the docker application on each one of our servers is time consuming, especially if we want to have a lot of test servers. Fortunately we can build our own docker registry, build the docker application once and then push it. On each test server, instead of building the docker image we can then just pull it from the registry:

docker pull :/

We can automate this through ansible by updating our Ansible hosts file:

[docker-grid]

docker-registry ansible_ssh_host=178.127.33.119

docker-node1 ansible_ssh_host=178.253.121.93

docker-node2 ansible_ssh_host=114.252.27.128

[docker-grid:vars]

ansible_ssh_user=mister

ansible_ssh_private_key_file=~/.ssh/testkey

Next, we'll add the following ansible tasks:

- name: Install dependencies for docker registry
  sudo: True
  apt:
    pkg: ""
    state: latest
    update-cache: yes
  with_items:
    - git
    - python-dev
    - liblzma-dev
    - python-gevent
    - libevent1-dev
    - build-essential
  register: kernel_result
  when: "inventory_hostname == 'docker-registry'"

- name: Intall docker-registry
  git:
    repo: https://github.com/dotcloud/docker-registry
    dest: docker-registry
    accept_hostkey: True
  when: "inventory_hostname == 'docker-registry'"

- name: Copy config.yml for docker registry
  shell: chdir=docker-registry/config cp config_sample.yml config.yml
  when: "inventory_hostname == 'docker-registry'"

- name: Install pip requirements for docker-registry
  sudo: True
  pip:
    requirements: /home/mist/docker-registry/requirements.txt
  when: "inventory_hostname == 'docker-registry'"

- name: Copy docker-registry init script
  sudo: True
  template:
    src: docker-registry.conf.j2
    dest: /etc/init/docker-registry.conf
  when: "inventory_hostname == 'docker-registry'"

- name: Start docker-registry service
  sudo: True
  service:
    name: docker-registry
    state: started
  when: "inventory_hostname == 'docker-registry'"

The docker-registry.conf is an upstart script (it could be an init script as well) that looks like this:

description "Docker Registry start"

start on runlebel [3]
stop on shutdown

expect fork

script
        cd /home/mist/docker-registry
        GUNICORN_WORKERS=${GUNICORN_WORKERS:-4}
        REGISTRY_PORT=${REGISTRY_PORT:-5000}
        GUNICORN_GRACEFUL_TIMEOUT=${GUNICORN_GRACEFUL_TIMEOUT:-3600}
        GUNICORN_SILENT_TIMEOUT=${GUNICORN_SILENT_TIMEOUT:-3600}

        gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout $GUNICORN_GRACEFUL_TIMEOUT -t $GUNICORN_SILENT_TIMEOUT -k gevent -b 0.0.0.0:$REGISTRY_PORT -w $GUNICORN_WORKERS wsgi:application &

        emit docker-registry_running
end script

We will build the docker image once and then push it to our registry:

docker build -t localhost:5000/mist.io

docker push localhost:5000/mist.io

Now on each docker-node server we can just pull the image:

docker pull 178.127.33.119:5000/mist.io

To automate things we will add these tasks to our ansible playbook:

- name: Build mist images
  sudo: True
  shell: chdir="/home/mist/dockers/" docker build -t localhost:5000/ .
  with_items:
    - mist.io
  when: "inventory_hostname == 'docker-registry'"

- name: Push images to docker registry
  sudo: True
  shell: "docker push localhost:5000/"
  with_items:
    - mist.io
  when: "inventory_hostname == 'docker-registry'"

- name: Pull images on the docker nodes
  sudo: True
  shell: "docker pull :5000/"
  with_items:
    - mist.io
  when: "inventory_hostname != 'docker-registry'"

After that, we just have to set ansible to trigger the tests and spawn these docker applications. All Jenkins has to do is catch the webhook of a new pull request and issue one command:

ansible-playbook runtests.yml

Thats it.

We have set up Jenkins to respond to Github commits and call Ansible to automatically spawn and configure our test servers, and we have optimized our testing speed by using pre-made Docker images sitting on our registry.

Thanks to Phil Kates from Rackspace and Nick Stinemates from Docker for proof reading early versions of this.

Comments


Racker Powered