Tags: #gitlab #ci
16 September 2023
In all the CI providers we want do automate something for example, we want to run the tests on every commit or we want to check the lints whenever we update markdown files or we want to make sure the code compiles when we make changes in the code.
These all can be automated using different CI providers, generally all of them have their own way of defining how you'll write the automation logic so that their software can understand it. The general thought process behind running the CI remains the same that you want to automate something.
From a security perspective, it might be the case that people trust isolated runners more than your laptop. I'll consume an executable more easily that's built via github releases or gitlab pipelines than a tarball that's pushed by a developer via his laptop. It's good to have an automation that'll release things and no doubt, it's fast as well.
Now all the provider claims to be fast and offer unique selling points which are important if you are building something related to CI or working or an organisation that sells CI based products but in general for most of us who consume CI the basics remains the same.
In this blog, I'm going to understand all the basic concepts that are out there in gitlab CI and write about that.
- jobs are where you define what you want to do. It can be testing, building, pushing or anything.
run_tests: image: alpine before_script: - sudo apt install jq script: - kubectl get pods -o json | jq '.' build_image:
The above are two jobs configured for gitlab.
- for any job
before_scriptwill execute first followed by
variables and secrets
- Please note variables can be scoped to global level as well as to job level. For job level you'll use the following syntax.
build_image: variables: DOCKER_IMAGE: "ttl.sh/$(uuidgen)" script: - docker build -t $DOCKER_IMAGE .
- If you intend to define the variable at global level then you have to use root node. e.g.
variables: DOCKER_IMAGE: "ttl.sh/something/nginx" DOCKER_TAG: "v1" build_image: docker build -t $DOCKER_IMAGE:$DOCKER_TAG . push_image: docker push $DOCKER_IMAGE:$DOCKER_TAG
- Go to
settings -> CI/CD -> variables -> Add Variable
- for secret variables use
- For referencing the variable inside your
.gitlab-ci.ymluse the variable same as your bash variables. e.g. if you've defined a variable
DOCKER_TOKENthen you'll use the variable as
$DOCKER_TOKENinside the gitlab CI manifest.
To replicate docker in docker scenario in gitlab use the following syntax
some_random_job_name: image: docker:20:10:16 services: - docker:20:10:16-dind variables: DOCKER_TLS_CERTDIR: "/certs"
The services are like an adjacent container that are connected to the main container. If you want a database then you can start a service container and both will be linked at build and test time for your application.
DOCKER_TLS_CERTDIR is required so that both
docker client and server can communicate with each other.
Stages are something that we use to group together a number of jobs and run them in sequential order as per our requirement. Say you write three jobs for a simple application.
- build the container image
- test the container image
- sign the image
- push the container image
This is a very generic scenario that we deal with on a day to day basis. We build our image and test it in our pipeline and then if all succeeds then we sign and push the image to the registry.
Now if you define all of this as separate jobs in the following manner:
build_job: # build something test_job: # test something sign_job: # sign build artifacts push_job: # push artifacts
Now when you push the changes and all the four jobs will start running immediately and at least few of them will crash for sure because the image itself if not built yet. How can you push something that's not available as of now? You can't.
stages come in.
We define stages and control the order in which different jobs will execute.
To add a stage to a job you use the following syntax.
goodbye: stage: last
The above job will run when the
last stage will be reached. So, if you've the following stages:
stages: - build - deploy - last
Think for a moment, when will this
goodbye job execute?
The answer is after
deploy jobs are completed. Yes, all the jobs under last stage will run in parallel.
Everything that's inside
.gitlab-ci.yml combined together is called a pipeline. So, in general a pipeline is a collection of stages and jobs.
A pipeline is composed of independent jobs that run scripts, grouped into stages. Stages run in sequential order, but jobs within stages run in parallel.
The above statement is from gitlab pipeline automated template. The above describe the difference between jobs, stages and pipeline in a beautiful manner.
- Jobs are intended to run in parallel.
- We combine different jobs into stages to control the execution of tasks we intend to execute in order.
deploying on a Linux host
- create a secret variable that will contain the private ssh key for your machine.
- keep the type of the variable as
Filemeans it a variable of
The syntax will look like this:
deploy: stage: deploy script: - ssh -o StrictHostKeyChecking=no -i $SSH_KEY <username>@<ip-address>
Similar to GitHub actions marketplace, GitLab have templates that you can import within your projects. Some of them are maintained and offered by GitLab and some of them are community developed. To include a template in
.gitlab-ci.yml use the following syntax.
job: stage: last include: - template: <some-remote-reference-to-.gitlab-ci.
I'm feeling much better after understanding and reading
.gitlab-ci.yml files. I might write a second part of the blog if I find something and advanced or something that ends up wasting a lot of my time. If you're getting started then doing some hands on practice in pipeline editor will help. It's very helpful because you'll be able to visualize all your jobs and stages in the UI right next to the editor which is very helpful.