Faisal

not for public consumption

View My GitHub Profile

Jenkins Job Builder by OpenStack

Jenkins is a flexible and easy to use continous integration server. Coming from a Linux background however, the GUI seems awkward and cumbersome to use. Enter Jenkins Job Builder (JJB). This is an excellent project by the good people at OpenStack that allows you to define your jenkins jobs via yaml files. JJB then parses these yaml files and converts them into the XML format that Jenkins expects.

JJB solves a few problems that I have encounted when using Jenkins:

  1. Configuration drift - this is the familiar problem usually associated with hand-configured servers. Over time it becomes impossible to say with any degree of certainty exactly what is on a server if it is hand-configured. This is true of GUI configured jobs within Jenkins as well. The root cause is the high labour overhead of clicking through 50 configuration screens for 50 jobs.

  2. Changing a common step across 50 build jobs means that you will need to step through 50 web pages. Very demoralising and from a developers perspective - a sin.

  3. Configuration done via the GUI screen cannot be versioned unless you start versioning the xml files that are generated by Jenkins. This gets ugly very fast and isnt a very elegant solution.

  4. Most importantly though - your Jenkins server becomes special i.e. indispensible and difficult to replicate.

Jobs

This is an example of a job definition written using JJB.

- job:
    name: 'Run PHP unit tests'
    description: >
                  This job runs the php unit tests in this project.
    project-type: freestyle
    logrotate:
      numToKeep: 20
    scm:
      - git:
          url: '[email protected]:atom/atom.git'
          branches:
            - 'origin/PreRelease'
          credentials-id: 'some-uuid'
    triggers:
      - pollscm: "* * * * *"
    builders:
      - shell: "make test"

The job definition above is for the most part quite self-explanatory. It creates a job in Jenkins that polls a github repo every minute for changes and runs a set of commands when it finds that the repo has been updated. The credentials-id is the api key that needs to be provided to JJB so that it can talk to Jenkins via the REST api.

You will need a minimal configuration file in the .ini format that specifies how to connect to your Jenkins server. This needs to be in the same directory as the job template.

[jenkins]
user=exampleuser
password=apikey_of_the_exampleuser
url=http://127.0.0.1:8080

You can now run the following command to have JJB build the job for you.

jenkins-jobs -i jenkins_config.ini update .

Job Templates

The example above is overly simplified. The power of JJB comes is when you parameterise these jobs like below. Note that for job templates, we use a different top level element.

- job-template:
    name: '{job_name}_unittests'
    description: >
                  {job_description}
    project-type: freestyle
    logrotate:
      numToKeep: 20
    scm:
      - git:
          url: '{git_repo_url}'
          branches:
            - '{branch_to_monitor}'
          credentials-id: '{api_key}'
    triggers:
      - pollscm: "{polling_interval}"
    builders:
      - shell: "make test"

Now we have a job template that can be fed with values for the variables. This paves the way for re-using the job template. So how do we supply these job templates with variables. Enter projects.

Projects

A project is again a yaml file that ties together job templates and variables.

- project:
    name: 'Demo jjb project'
    api_key: 'bzzxx234-41asdf-4780-asdf9-f1poasdfsf'
    branch_to_monitor:
      - 'pre-master':
           job_name: 'PRE-MASTER'
      - 'master':
           display_branch_name: 'MASTER'
    project_name: 'codename_alpha'
    git_repo_url: '[email protected]:company/project1.git'
    jobs:
      - '{job_name}_unittests'
      - '{job_name}_styletests'

Everything in the projects.yml file apart from the jobs list is a series of variable definitions that get substituted into the job templates listed under the jobs list.

A powerful feature here is that you can define variables as lists. That causes JJB to create a job for every combination in the Cartesian product of the variables that are defined. In the projects.yml file above, it will cause the following jobs to be created:

Thats four jobs in Jenkins written across 2 concise yaml files! We can now store these yaml files in a git repository. Our concerns with respect to Jenkins jobs via GUI are now addressed:

  1. Configuration drift - this is no longer possbile as the single source of truth is now the yaml files. Any changes to the jobs must be done via the yaml files.

  2. If you need to change a common step across 50 jobs and you have constructed your jobs in a modular, re-usable manner - this now becomes trivial.

  3. yaml files can be stored in a version control system alongside your code.

  4. Your Jenkins server is no longer special - you can now rebuild all your jobs with a single command. If you use configuration management systems like Ansible - you can recreate your Jenkins box and jobs in no time at all.