GitLab Runner on LAN with self-signed certificates
The default SSL certificates used by GitLab server install will fail any GitLab Runner registration. This guide shows how to amend the SSL certificates used by GitLab server to allow Runner registration in LAN (Local Area Network).
For GitLat Runner Docker install, I used the installation optional 1. I also specified a custom volume mount path in case I want to install additional runners.
(Not 100% sure if a custom volume mount path is necessary for multiple Runners. Believe one single config path can be used? Runner(s) is just 'itemized' in the TOML file, and it's smart enough to figure out which entry to use? I have not tried. Sometimes it's better to be 'explicit' than 'implicit'.)
$ docker run -d --name gitlab-runner09 --restart always \
-v /srv/gitlab-runner09/config:/etc/gitlab-runner \
-v /srv/gitlab-runner/ssl:/usr/local/share/ca-certificates \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest
(Additionally, I have extra volume mount for SSL certificate. Which, I'll come back to later.)
Instead of spinning up a new container to do registration and write out the config file from the instructions, I entered the container and performed registration there.
I got the following error when attempting to register the Runner to GitLab server. My local GitLab server have DNS entry of 'gitlab', hence, 'https://gitlab/'.
(Run "docker container ls" to get the container ID. In my case, is "baf8665deb67".)
$ docker exec -it baf8665deb67 bash
root@baf8665deb67:/# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=41 revision=12335144 version=15.8.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab/
Enter the registration token:
***********
Enter a description for the runner:
[baf8665deb67]: dockersrv03-runner09
Enter tags for the runner (comma-separated):
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
ERROR: Registering runner... failed runner=******** status=couldn't execute POST against https://gitlab/api/v4/runners: Post "https://gitlab/api/v4/runners": x509: certificate relies on legacy Common Name field, use SANs instead
PANIC: Failed to register the runner.
(Note the Docker container name I used is 'gitlab-runner09' and during Runner registration is 'dockersrv03-runner09'. I wanted my Runner name show up in runner's list as <docker_host>-<runner_name>. So I know which Docker hosts the GitLab Runner.)
See the official documentation for more details on self-signed certificates.
Basically, the existing SSL certificates configured in GitLab server don't contain my local SAN host(+domain), or "subjectAltName". For LAN where DNS resolution is configured, GitLab enforces additional checks (it's just my own assumption; have not checked), and we need 'SAN certificates'.
On my GitLab server, here's how I manually generated self-signed certificates containing my GitLab server in the SAN section. I've included simple hostname and FQDN.
(I also run Hashicorp Vault that manages some of my certificates. For the sake of simplicity I didn't go that route and just did it on CLI.)
$ openssl genrsa -out ca.key 4096
$ openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
$ openssl req -newkey rsa:4096 -nodes -keyout gitlab.key -out gitlab.csr
$ openssl x509 -req -extfile <(printf "subjectAltName=DNS:gitlab,DNS:gitlab.mynetwork.local") -days 3650 -in gitlab.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out gitlab.crt
This will generate a bunch of files but we are only concerned with three files here:
GitLab Server
gitlab.key
gitlab.crt
GitLab Runner (explained in later steps)
- ca.crt
I moved directory '/etc/gitlab/ssl' to '/etc/gitlab/ssl_old'.
Create new folder '/etc/gitlab/ssl' and copied 'gitlab.key' and 'gitlab.crt' into it.
(Double-check and ensure gitlab.key file have file/group permission of '600'.)
Modify gitlab.rb config file (defaults to /etc/gitlab/gitlab.rb) and ensure the following two lines are set accordingly.
(I'm not going to cover the configuration of enforcing HTTPS here. You can see the guide here.)
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.key"
Reconfigure GitLab server.
$ gitlab-ctl reconfigure
Now if I try to register the GitLab Runner again, I'll get a different error.
root@baf8665deb67:/# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=65 revision=12335144 version=15.8.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab/
Enter the registration token:
***********
Enter a description for the runner:
[baf8665deb67]: dockersrv03-runner09
Enter tags for the runner (comma-separated):
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
ERROR: Registering runner... failed runner=******** status=couldn't execute POST against https://gitlab/api/v4/runners: Post "https://gitlab/api/v4/runners": x509: certificate signed by unknown authority
PANIC: Failed to register the runner.
This is happening because we need GitLab Runner to recognize the signed certificate chain during registration. In order to fix this, we need to do additional steps within the Runner container itself.
Remember when I first deployed the Runner container I specified extra volume mount path for SSL certificate? This is the place we place the CA certificate. So I fetched the 'ca.crt' generated on GitLab Server AND copy it to the Docker server. Then, I copy it to the volume path.
$ cp ca.crt /srv/gitlab-runner/ssl/
Now I just need to update container's system certificates. Within the Runner container:
root@baf8665deb67:/# ls /usr/local/share/ca-certificates/
ca.crt
root@baf8665deb67:/# update-ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Now I can properly register the GitLab Runner!
root@baf8665deb67:/# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=925 revision=12335144 version=15.8.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab/
Enter the registration token:
***********
Enter a description for the runner:
[baf8665deb67]: dockersrv03-runner09
Enter tags for the runner (comma-separated):
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
Registering runner... succeeded runner=********
Enter an executor: instance, docker, docker-ssh, parallels, shell, virtualbox, docker+machine, custom, ssh, docker-ssh+machine, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.7):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"
root@baf8665deb67:/# gitlab-runner verify
Runtime platform arch=amd64 os=linux pid=935 revision=12335144 version=15.8.0
Running in system-mode.
Verifying runner... is alive runner=xzSjgUo6
For executor, I used docker. For the Runner image, I used alpine:latest.
If we use 'ruby' image as prescribed from GitLab's documentation... every time the Runner is called it will spin up using the 'ruby' Docker image. Which, will be much larger than the plain vanilla "alpine" image. What we want is the leanest Runner and alternatively, custom Docker image can be specified and deployed from pipeline file.
Now I can see in my GitLab server the new Runner is registered!
I hope this guide clears up some confusion about how to register GitLab Runner on LAN where self-signed certicates are used.
As I was going through this process, I couldn't really find a clear-cut guide to do so. So, I wrote one up for myself as a reminder!