Building Software with Jenkins and Gitlab

Software development is a long and complex process, more often than not, involving multiple programmers working on the same software. Which is why one of the most important tools when coming to developing software as a company is to have a virtual service platform that can host the shared source code online, serve as a central spot for issue management and act as a hub to hold the various builds and patches throughout the development process.

As a start-up company, budget and cost plays a crucial factor to determine our future and success. So it is paramount to search for an economical solution that can provide for such a need (the service platform). 

Thankfully, there are such solutions! Some of the most famous service platforms on the internet are GitHub, Bitbucket, and JIRA. Atlassian1 has the most success in fully implementing scrum methodology throughout its ecosystem and products. GitHub on the other hand is the top leader for open source projects. Many software companies set up it’s own GitHub account with the motive of either contributing the project to the online community or for showcasing their products. In fact, until today, GitHub still is a value-added in resumes as a developer when going for interviews.

Choosing Gitlab 

However, when FILPAL was searching for such a service platform back in 2015, one particular service caught our eyes. Introducing GitLab2. The advantage of choosing Gitlab is that it allows hosting of private projects for free3

Another important feature that makes it outstanding is the exclusive CI/CD pipeline build feature where none of the other services integrate within their platform back in 2015/2016. (We will look into this deeper later on in the article). In this article, we hope to give a glimpse of how we in FILPAL build our softwares that is now released to the public (EDS HF and Touchstone Viewer).

The Need for Automated Builds

Aside from source code control, one of the points of concern when developing a software is building and deploying the product. It is a tedious process if done manually every time there is a code change or new release. When done manually, developers would need to compile the source code and upload to a shared location before a 3rd party is able to access it. Considering the number of builds that could be released daily, this is a repetitive and very boring chore for developers.

Internally, there is also a common requirement among software developers where testers should be able to get the latest build as soon as it happens. This is what we call a nightly build. It might be unstable but at least testers can grab everything the developer has fixed in the code as soon as it happens in the pipeline. This has been a crucial requirement in scrum methodology nowadays. The faster we can close the bug, the faster we can close the gap, and meet the deadline. This all means that having an automated building process would effectively hasten software development.

Using Jenkins Together with GitLab

Figure 1: Jenkins Dashboard

So the problem is how can we simplify and automate the process so that developers can always focus on the ‘real’ development work and for testers to be able to directly go to a dedicated place to download the binaries without bothering the developer?

The solution is to have an automated build server. The popular build servers are Jenkins, CircleCI, and AppVeyor but most of them are not free except Jenkins. Which is the reason why Jenkins is the first ever server we set up for this purpose. See how to set up Jenkins server.

With Jenkins, life is much easier. First, the developers will push the source code to GitLab, which acts as a hub holding the source code. Gitlab will then trigger and push the new version of the code to Jenkins to start the build process. Jenkins generates a unique ID every time it builds. This also eliminates the hassle of our versioning in case developers forgot to increase the version in assembly info.

Figure 2: Jenkins Workflow

But there’s a catch. Jenkins requires the same development environment installed on the build server to execute the compilation. In our case, this would be Visual Studio. This means a dedicated amount of space is needed for Jenkins to be able to perform all her functions. In fact, Jenkins keeps everything on the server, including artifacts and source code pulled from GitLab for every single build. After some time, there would be the problem of lack of space on the server. Which is why once in a while, we have to log in to the server to purge the artifacts when the hard disk is out of space4

As time went by, we realized that it is possible to streamline the size of the Jenkins build server, and it can be more easy to set up. We removed the huge Visual Studio from the server, and replaced it with a lighter Build Tools for Visual Studio (MS Build) instead which is a significant change in saving space. However, this still does not solve the fundamental problem.

From Jenkins to Gitlab Runner

Figure 3: GitLab Workflow

That’s when Gitlab introduced the CI/CD pipeline feature. By setting up a runner on a local server, with an initial build script (.gitlab-ci.yml) at project root folder with a specified upload path, the build process can be consolidated in GitLab itself. In simple words, we can move out of Jenkins and only use GitLab as the hub for our build process.

GitLab provides very good documentation and setup steps. You can find it on any of your GitLab project > settings > CI/CD or check out GitLab Runner Doc

So how does it work? Everytime a developer pushes a new code, it will automatically trigger the runner on our server. When the job is done, it will upload the artifact back to GitLab server again. Check out figure 3 for the workflow.

Figure 4: Sample yml to include uploaded artifacts

By specifying the upload path in .gitlab-ci.yml, team members can download the build from the CI/CD > pipeline after the job is done. Each pipeline has a unique ID generated, and tied to git hash to make it identical. GitLab also labels each build in the pipeline. With one glance, a developer will be able to know from which branch of code is the build from and whether it is the latest update.

Figure 5: Gitlab Pipelines

So now, by replacing Jenkins with GitLab runner, we can host our source code, and artifacts all at one place. After the changes, testers no longer need to keep switching between different platforms to verify if the build process is successful. Developers only need to directly check on whether the pipeline build is successful or failed on the CI/CD status on the same page in GitLab itself.

Besides, GitLab runner will not keep all the artifacts on the build server. It only keeps one copy of the source code per project build as well as one copy of the compiled code. This is more advanced and helps to keep the storage size compact on the server. Developers only need to care about the development setup, and required tools to build our code so the server can be as small as possible.

Alternative Gitlab Runner

Normally, the GitLab Runner is a personal runner installed in a local server that is tied to GitLab. However, If one choose not to install a personal runner to build the code, GitLab also provide the option to use their own in-house runner for the first 2,000 mins CI build time for free every month5. It is a runner that runs on a Linux docker. 

Unfortunately for us in FILPAL, EDS HF and our other softwares and applications are using WPF which requires a real Windows docker. This means the runner offered by GitLab is incompatible with our requirement. Still, for more information, refer to Test all the things in GitLab CI with Docker by example, and Building Dotnet with Gitlab CI.

But all hope is not loss. The development team in Gitlab is constantly upgrading their services. And until recently, GitLab started offering its Windows shared runner as beta. This gives us the opportunity to switch our existing runner to GitLab shared runner since they are using the same ‘Build Tools’ we use. After changing the runner tag in .gitlab-ci.yml, and enabling it under project settings, we are currently testing the shared runner in our development.

Figure 6: Blank Template to use GitLab shared Windows Runner

Downloading Artifacts

As mentioned earlier, one of the reasons developers come out with many builds is so that the internal testers can download and test the new iteration without much delay. But aside from that, and especially for some software startup companies, the builds are also intended to be released to the clients. 

To do that, the projects would need to be made public. In normal circumstances, most projects are set to hidden to prevent unwanted people accessing the intellectual property. Should companies want to set the project to public, it’s possible to set it up by GitLab > Settings > General > ‘Visibility, project features, permissions’ to configure the permission. 

A word of caution though, at this moment, GitLab does not allow customizing the settings where clients are allowed to download artifacts from the pipeline but have no access to the source code. Currently, GitLab makes these two as the same security level which means anybody with access to the pipeline has access to the source code.

If you are the subcontractor to the client, then this is probably fine because eventually the source code is also part of the intellectual property to your customer.

We hope this article gives some insight into the internal procedures when developing out softwares. And who knows? Perhaps you may want to give it a try and start using Gitlab for your own endeavors.

Further Reading

  1. https://semver.org/
  2. https://docs.gitlab.com/runner/
  3. https://docs.gitlab.com/ce/ci/yaml/

We would like to hear from you! Please leave your comment below if you have any idea or thoughts. Thanks.

Footnotes

  1. Atlassian company owned Bitbucket and JIRA.
  2. All the features discussed in this article are referring to the GitLab community. GitLab Enterprise edition provides more features.
  3. GitLab has set the market rule, force the competitors (i.e. GitHub, and Bitbucket) to make the same offering in recent years.
  4. This has been changed now you can set up a docker for Jenkins to keep your size smaller.
  5. Starting October 2020 GitLab reduces to the first 400 mins for free only see Customer FAQ.