Getting Started With Concourse CI - Part 1

Introduction

I am a big fan of learning new things, and I was pointed in the direction of  Concourse CI as the next cool thing.  I wanted to give it a shot so this post will go over setting up a test instance of Concourse CI.  The best way to learn the different components and how they work together is to install it from scratch.

Pick How To Install

If you go to the link above, you can start as easy as five minutes with loading up a Vagrant VM with Concourse already installed.  What is the fun of that?  That is the easy button.  The other issue with the Vagrant VM route is that if you want to upgrade, you will have to recreate the VM as a requirement of the upgrade.  You will lose job history and submitted pipelines.

The other options are deploying as a BOSH release, a Docker image, or using the standalone binary.  In this example, I will be going over installing the standalone binary on a single Ubuntu server VM.  I will be referencing the Concourse CI documentation site for this type installation.

Prerequisites - Standalone Binary Install

There are five things we need to do before we get started on this Linux installation.

• Get the binary downloaded from the Concourse CI Downloads site.
• Linux kernel v3.19 or later. (Best case to meet this requirement is to use Ubuntu)
• PostgresSQL 9.3+ installed
• Create a Linux user to run the Concourse service
• Create a Concourse folder structure

Linux Kernel Version Check

To verify that you are at a compatible Linux kernel version, you can run either of these commands.

[user@linux_prompt]$ uname -r

[user@linux_prompt]$ uname -a

The "-r" flag will provide just the kernel version while the "-a" will provide more detailed information.  If you are not at the supported kernel version, you can perform an upgrade using Ubuntu's apt-get process.

[user@linux_prompt]$ sudo apt-get install --install-recommends linux-generic-lts-xenial

This upgrade will get your Ubuntu server to the latest kernel version based on 16.04 Xenial.

PostgresSQL Install

If you don't already have PostgresSQL installed, you can install it using Ubuntu's apt-get process.

[user@linux_prompt]$ sudo apt-get update

[user@linux_prompt]$ sudo apt-get install postgresql postgresql-contrib

Concourse Service Account

The Concourse CI Web service cannot be run as the root user so another account will have to be used.  If you do not have an account already setup, you can create a user for this purpose.  In this example, an account was created named "concourse".  The Concourse CI Worker service has to be run as root so a service account is not needed for that function.

Folder Structure Setup

Before running the command for the first time, it would be good to establish a folder structure for where the different components are going to live.  Here are some suggestions for these folders.

• Concourse Home: /opt/concourse
• Generated SSH keys: /opt/concourse/.ssh
• Concourse Standalone Binary and Scripts: /opt/concourse/bin
• Concourse Worker: /opt/concourse/worker
• Concourse Logging: /var/log/concourse

After the folders have been created, make sure your newly created service account has access to them.

Concourse Install - Generate SSH Keys

For the services to work, we need to generate some private keys and an authorized worker SSH keys file.  You can take the defaults for the Concourse documentation, but in this example, these commands were ran.  Make sure you are in the folder where you store your Concourse SSH keys.

[user@linux_prompt]$ cd /opt/concourse/.ssh

[user@linux_prompt]$ ssh-keygen -b 4096 -f id_session_rsa -N ""

[user@linux_prompt]$ ssh-keygen -b 4096 -f id_web_rsa -N ""

[user@linux_prompt]$ ssh-keygen -b 4096 -f id_worker_rsa -N ""

[user@linux_prompt]$ cp id_worker_rsa.pub authorized_worker_keys

Concourse Install - Create Concourse Database

Now we need to create the database that Concourse will use for this instance.  I will run through an explanation of the commands below:

• Sign into PostgresSQL using the postgres user
• Create a PostgresSQL user and password
• Create a new database
• Assign this new user as an owner to the database

[user@linux_prompt]$ sudo -u postgres psql -U postgres

[user@linux_prompt]$ CREATE USER username WITH PASSWORD 'password';

[user@linux_prompt]$ CREATE DATABASE concourse;

[user@linux_prompt]$ ALTER DATABASE concourse OWNER to username;

[user@linux_prompt]$ \q

The "\q" command will exit out of the psql interface and back to your logged in user.

Concourse Install - Service Scripts

So we made it, we can finally start up the service.  But we should create a service in Ubuntu so we don't have to manually start and stop the service.  Also, the service can be started when the server reboots.

The first thing is that you want to move the downloaded Concourse executable and place it in the /opt/concourse/bin folder.  I also renamed the file to just simply concourse.  Make sure to make that file executable and set permissions on it so the service account can run it.

Now we need to create a total of four scripts.  These scripts will be for the startup and shutdown scripts as well as the custom service itself.  I have created sample scripts and hosted them on GitHub (Sample Scripts).  Here is a description of each file.

• web_startup_example - This file needs to be placed inside of your /opt/concourse/bin folder.  It contains the variables needed to connect to the database, set the external web address, set a Concourse username and password, etc.  It also sets the log file and location for this service.
• worker_startup_example - This file needs to be placed inside of your /opt/concourse/bin folder as well.  Similar to the web_startup, it contains variables that can be set to start up this service.
• concourse_shutdown - This file needs to be placed inside of /opt/concourse/bin folder.  It will shutdown both Concourse web and worker services when you execute it.
• concourse-service-example - This file needs to be placed inside of /etc/init.d folder.  This script calls the other three scripts and is the custom service file that will be registered to Ubuntu.

Remember to rename all the files except the concourse-service-example with a ".sh" file extension.  Also, make sure the Concourse service account that you created has access to the concourse-service script in the /etc/init.d folder.  To register the custom service with Ubuntu, make sure the file name is "concourse-service", and that you are located in the /etc/init.d folder before you run the following:

[user@linux_prompt]$ update-rc.d concourse-service enable

[user@linux_prompt]$ update-rc.d concourse-service defaults

Now you can start your service with just ./concourse-service start and stop with ./concourse-service stop.  You will now be able to go to your Concourse instance from any browser using http://concourse-hostname:8080.

Summary

I didn't set out to make this a multi-part post.  It sort of just happened.  The next post will go over running your first pipeline and upgrading Concourse CI.  I want to stress that this setup example should only be used for testing functionality and getting familiar with Concourse CI.  There are things like having passwords in the startup scripts that I would not suggest to do outside of a proof of concept instance.  IT security would have a field day with it if you did.

~RRRII