Jenkins CI is well known open source continuous integration server, and a damn good one in my opinion. I guess the biggest issue is to get to know all the plugins available, a fun time indeed 🙂 Anyways, since the switch to Git/Gitlab, I needed a setup that would enable the team to use the CI environment in full. The idea was to allow CI environment to build all the branches, not just master (release, develop, whichever is your flavor of the day) branches. Manually setting up Jenkins projects for all the Git branches was out of question.
A little background first. The projects are mainly Java / Maven and there are a lot of dependencies. One rule that is most important for Jenkins environment was that the developers need to keep their tasks / features in separate Git branches. This would prevent clashes between developers, but still allow them to work through the entire stack if needed. Jenkins was to be used as continuous feature testing environment so all those branches had to find their place in the CI stack too.
Ideally, the solution was to satisfy the following requirements:
- a single branch that spans several projects should be built and referenced correctly (Maven dependencies)
- only master (develop) and releases branches are to be published on Artifactory, (short lived) features should not fill it up
- preferably a single Jenkins project, because of:
- resources when building on the same machine (e.g. building several branches for the same base project)
- no need to clutter the views
- build history should show the branch built
- should be able to build the entire feature stack across different slave nodes
So, we introduced the parameterized builds. Each Git project must have a single Jenkins project that is configured like this:
- This build is parameterized checked, with a single parameter “BRANCH_NAME_TO_BUILD”, and the default value “master” (or “develop” or what ever you use).
- Block build when upstream project is building checked – this is not really related to this workflow but is a good practice nevertheless, prevents building projects while dependencies are built
- Git repositories and branches set to track the Git repository, and to build branch from the above parameters setup. All using Git plugin.
- Deploy artifacts to Artifactory set to filter out snapshots
- Deploy artifacts to Maven repository
A specific thing about the setup, that might not work for you, is the Artifactory / Maven part. The policy is that only release artifacts are allowed in Artifactory. This reduces the noise and keeps the Artifactory slick (it is getting rather big anyway). The problem is with building dependencies for feature branches that are always snapshots. For this to work, and to be able to use different nodes, you still need something like Artifactory. If building on a single node, you can just put “install” as a Maven goal and you’d get the dependency on that computer. For multiple nodes, one idea was to tell all the nodes about each others maven repositories, but that seamed like too much maintenance.
So, the workaround was to create a webdav Apache folder that Jenkins could use to put all builds into. And, that same repository was referenced in Maven settings on each of the nodes, and on all of the developers machines too. This enabled Jenkins to “know” about all the feature branches artifacts, while not putting too much strain on Artifactory. And, that Maven repository can be cleaned periodically without peril.
This setup is pretty much it. With it you get to build a specified branch at any time. The feature branches build nicely throughout the entire stack and you have a single project on Jenkins that prevents concurrent builds of the same code base (so resource issue is no longer valid).
Still, nothing is perfect and there are a few gotchas:
- if you are using some wall plugin, you can’t really tell the status of the project since they show only the last build status, which can be feature or not – this can be a good thing if you decide to treat broken feature builds as a bad thing 🙂
- you get to build only one branch at the time, for many commits on the same project, you could wait a long time for Jenkins to build your commit
- Jenkins can’t really decide correctly on Maven dependencies since at one point in time a project might reference some feature snapshot (from pom.xml) whilst at other time it might reference the release version on the same project
The last point is the most problematic one. It will prevent Maven to correctly compute project dependencies which will directly influence the upstream/downstream build triggers in Jenkins. You’ll probably have to manually start builds for such situations or just commit changes again. If developers in your team are a bit disciplined, this might not be an issue after all. A nice idea on how to avoid this problem is to create a separate Jenkins project for each branch, automatically as noted here. I am in the process of enabling this support for Gitlab Jenkins plugin so stay tuned 🙂