Setup Guide for Jenkins Continuous Integration¶

Personal Use, Local Machine (Ubuntu 16.04)¶

Ensure java is installed, then add jenkins apt-key and install:

wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -

sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'

sudo apt-get update

sudo apt-get install jenkins

What this does (From Jenkins Website):

Jenkins will be launched as a daemon up on start. See /etc/init.d/jenkins for more details.
The 'jenkins' user is created to run this service.
Log file will be placed in /var/log/jenkins/jenkins.log. Check this file if you are troubleshooting Jenkins.
/etc/default/jenkins will capture configuration parameters for the launch like e.g JENKINS_HOME
By default, Jenkins listen on port 8080.

Usage¶

If the process is running, the 'top' UNIX command should show that a user named jenkins is now consuming some %MEM. More information on the process can be found via:

ps -aux | grep jenkins

To access the server (which should be running already), go to "localhost:8080" in a browser

However...

To manually stop/start/restart the daemon process:

sudo service <stop/start/restart>

If desired, the auto-start on startup can be disabled (I haven't tried this yet, answer from here):

update-rc.d jenkins disable

Configuration¶

Set up security - Important as default settings allows any user on the network or VPN to run any bash command or executable through jenkins given the IP and port number.
- Manage Jenkins > Global Security Settings
- Good settings might be (with guidance from Jenkins best practices and Jenkins security reccomendations):
  - Enable Security: Yes
  - TCP port for JNLP slave agents: Random
  - Access Control: Unix user/group database
  - Use 'Test' under Unix user/group database access control to troubleshoot: I had to add jenkins to the group 'shadow' with sudo usermod -aG shadow jenkins
  - Authorization: Matrix Authorization Strategy - need to add user here as it doesn't add UNIX users to the table automatically (managed to lock myself out by not doing this). Note the tick all box on the right
  - Markup Formatter: Plain Text
  - Prevent Cross Site Request Forgery exploits: yes, default crumb issuer
  - Enable Slave -> Master access control: yes
Set up plugins (Github, Bitbucket...)
- Manage Jenkins > Manage Plugins > Available > search for 'git plugin' > download and install on restart
- Do the same for "Bitbucket plugin" (I haven't tested performance of this yet)
- This will now show 'Git' under source code management when creating a new project
- the Pipeline plugin is handy for advanced useage, but not required
Set Environment variables
- I want to use Anaconda python, but the path to anaconda is not set as jenkins does not use my .bashrc
- Go to Manage Jenkins > Configure System
- Tick the Environment variables box under "Global Properties" and add your anaconda path to PATH:
  - Name: PATH, Value: /path/to/anaconda/bin:$PATH

Creating a Project and Example¶

New Job > Freestyle project
Add a name and short description
Probably a good idea to Discard old builds and choose a number of builds to keep/how long to keep them
If using git, select Git and enter the URL - self explanatory
If you don't have any example software with tests available, here is a simple example using py.test. I also include a jenkins_build.sh example here.
Tell git to check out the repository on build with Additional Behaviours > Add > Check out to a sub-directory: Leaving this blank is fine as the code wil be checked out into the project workspace.
Select Build Trigger > 'Poll SCM' and 'Build when a change is pushed to GitHub' to only build the project if the git repository has changed. You'll still need to set a schedule for how often it will check - see the question mark by schedule for help.
- One possible solution could be to poll every 5 minutes to see if new commits have been pushed, with
```
  H/5 * * * *
```
- Note: post-commit hooks can also be set up for this, but difficult if Jenkins is behind a firewall that GitHub cannot successfully send packets through.
Set a build command - in the case of the example repository above, this would be

Execute shell: ./Jenkins_build.sh

Ensure any build file appropriately cleans the build environment to avoid subsequent build problems. In my case, I create a conda environment, install and test the module there and then delete the conda environment
Pytest error reports can be published on the build status page by added Post-build Actions > publish JUnit test result report > Test report XLMs: filename.xml and adding the junitxml flag to py.test in Jenkins_build.sh:

py.test --junitxml=filename.xml

To test if the system has worked without waiting for the next poll, click on the project and then Build Now

Configuration using Vagrant VM¶

A Virtual Machine may be useful for Jenkins servers running on shared clusters where admin priveleges are not given.

Vagrant Virtual Machine and Provisioning:¶

I'm going to describe the process for setting up an Ubuntu

vagrant init Ubuntu/xenial64

Now need to edit the vagrant file to allow port forwarding, i.e. uncomment/edit the line

config.vm.network "forwarded_port", guest: 8080, host: [Host Port number]

Host Port number here should be a memorable number that is not in use

netstat -a may help to show which ports are in use
network bridging may be a better idea here

Enable shell script provisioning in Vagrantfile, i.e. add the following line before end

config.vm.provision "shell", path: "setup.sh", privileged: true

Add a setup.sh bash file so that we have a software test-bed (I suppose this will require a python environment - Conda?, C compilers, C++ compiler, Fortran, cmake) plus the Jenkins dependencies (Java) and installation. Another provisioner would be better in general, however if you plan to run tests on a Debian machine, this is ok. My bash file contents in setup.sh are:

------------------------------------------------------------------------------------------------
# Guest machine setup script for a Jenkins server and software test-bed
# @author Paul Chambers
# @date 11/01/2016

apt-get update

# Misc tools
apt-get -y install cmake, git, hg

# Install compilers
apt-get -y install build-essential, gfortran 

# Install Anaconda python distribution to /home/vagrant/anaconda2
wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda2-2.4.1-Linux-x86_64.sh
chmod +x Anaconda2-2.4.1-Linux-x86_64.sh
VGR_HOME=/home/vagrant

CONDA_PATH=$VGR_HOME'/anaconda2'

BASHRC=$VGR_HOME'/.bashrc'

./Anaconda2-2.4.1-Linux-x86_64.sh -b -p $CONDA_PATH

echo '# Added during vagrant setup to correct Anaconda python path:' >> $BASHRC

echo 'export PATH='$CONDA_PATH'/bin:$PATH' >> $BASHRC

echo '' >> $BASHRC

source $BASHRC

conda update conda
conda update anaconda

# Install java (required for jenkins)
apt-get -y install default-jdk

# Jenkins specific setup:
wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
apt-get update
apt-get -y install jenkins
------------------------------------------------------------------------------------------------

make it executable...

chmod +x setup.sh

Note: Other settings such as RAM, CPU number and virtual hard drive size can and probably should also be set in the vagrant file.

Start the vagrant machine :

vagrant up

To control the machine later, ssh into the vagrant machine with

vagrant ssh

And use the normal vagrant commands to control the virtual machine once created i.e.

vagrant suspend/halt # suspend/shut down (vagrant up to recover)

vagrant destroy # delete the machine

Usage¶

Jenkins should now be available at domain_name:<Host Port number>, (localhost:<Host Port number> if accessing from the host machine, otherwise something like computername.domain.com:<Host Port number>).

Now set up security and plugins as shown above for a personal machine

Notes¶

It seems security can be switched off directly in the $JENKINS_HOME/config.xml although this requires sudo access so not a huge risk
this might be useful for moving existing jenkins jobs from one computer to another
Master-slave builds could be useful to avoid crashing the master Jenkins node
Sizing the hard disk and RAM for vagrant machines should be important factors to consider for shared Jenkins servers
- Watch out for virtualbox breakouts