GitLab offers sophisticated abilities when it comes to organising your build. Understanding them well can give you faster runs and better feedback loop, making entire team more effective. It can be the difference between a CI which gets in the way and is “red” for most of the time — and a CI which helps in everyday work.

In this article, I assume you already had a try with GitLab or at least have some experience from other CI/CD systems like Jenkins, CircleCI etc. Keep the reference doc for .gitlab-ci.yml open and read more about each option, as we discuss them.

Getting familiar with GitLab nomenclature

In essence, there are three main ingredients to GitLab CI/CD system:

• Pipelines
• Stages
• Jobs

Let’s explain each of them, from the bottom of the list.

Jobs

Job is the smallest unit to run in GitLab CI/CD. It is often called a “build step”. It can be a build or compilation task; it can be running unit tests; it can be code quality check(s) like linting or code coverage thresholds checks; it can be a deployment task.

A single job can contain multiple commands (scripts) to run. The following is an example:

unit tests:
  script:
    - npm install
    - npm run test

It is worth noting that jobs can have constraints (which they often have): only run on a specific branch or tag, or when a particular condition is met.

Stages

Each job belongs to a single stage. Stage can contain zero, one or more jobs to execute. All jobs in a single stage run in parallel. The next stage is executed only if all jobs from previous stage complete successfully — or they are marked as allowed to fail.

GitLab out-of-the-box has defined the following three stages:

stages:
  - build
  - test
  - deploy

Here, when jobs from build stage complete with success, GitLab proceeds to the test stage, starting all jobs from that stage in parallel. Likewise, when the test stage completes (i.e. all jobs there finished running), the deploy stage will be executed. And so on.

If you need different stages, re-define the stages array with your items in .gitlab-ci.yml.

Hint: by default, when you don’t specify stage for the job, it belongs to the test stage.

Hint: if you want to allow job failure and proceed to the next stage despite, mark the job with allow_failure: true. GitLab will mark the entire stage as a success, but with yellow “warning” (see screenshot below).

Pipelines

A pipeline is an umbrella for your jobs and stages. Pipeline orchestrates and puts them all together. Pipeline runs when you push new commit or tag, executing all jobs in their stages in the right order. Entire pipeline config is stored in the .gitlab-ci.yml config file and, apart from jobs definition, can have some “global” settings like cache, environmental variables available in all jobs. See GitLab YAML reference for more details.

How NOT to structure your build steps

Let’s move to something practical. Imagine the following hypothetical CI build steps. It contains two jobs, with few pseudo scripts in each of them:

build:
  script:
    - run build
    - run linting
    - run unit tests
    - deploy pull request
    - run integration tests

stage:
  script:
    - run Docker build
    - deploy.sh

There are few problems with the above setup. For instance:

• When linting fails, nothing else gets executed. The developer might think: linting is not a biggie, I’ll quickly fix it later. However, when this step fails, anything after it is NOT executed. There’s no feedback about other steps. The developer does not know that it is not just linting, maybe the change also broke integration tests?
• When unit tests are failing, the next step, Merge Request deployment, is not executed. That prevents Developers, Product Owners and Designers collaborating and iterating quickly and seeing the new feature as it is being implemented.
• If anything fails in the earlier steps, the Developer is not aware that the new changes also affected Docker build. There might be a bitter disappointment when you think it’s just unit tests to fix, while in reality, there is much more work.
• Re-runs are slow. For instance, if your integration tests fail due to some external factors (e.g. API timeouts) and you want to re-run them quickly, you need to wait for the entire pipeline to run from the start. Waiting time is long and resources are wasted.

You get the drill.

Let’s talk about how, by organising your build steps better and splitting them more, you can mitigate all above and more.

Organise your build steps efficiently

Let’s highlight one thing: there is no single recipe for the perfect build setup. As the lawyers say: it all depends. However, there are “things to consider”. Here are few ideas I have learnt while using GitLab during past months.

Don’t split jobs too much

There’s an overhead in splitting jobs too much. GitLab by design runs them in fully distributed manners using remote “workers” (which is a good thing). Before the job starts, it has to spin a new Docker container in which the job is running, it has to pull the cache, uncompress it, fetch the artefacts (i.e. build results from previous jobs) and re-upload the cache after the job has finished.

You will need to find some reasonable balance here — see the following section.

Split the jobs just enough

Aim for fast “feedback loop”. It should be part of your Continuous Integration culture. The faster the developer gets feedback regarding what went right or wrong, the better.

What is essential for a developer to know, after he or she pushed a new change?

• Does the install, build and compilation process work? If it the code “didn’t see the compiler” — or the install process doesn’t work due to forgotten dependencies — there is perhaps no point in doing anything else. Put it as the first step in your build and wait for it to complete.
• As soon as you have the compile task completed, you have the artefacts available. Use them in the next stages. Fetching them is cheap and fast since the size of the compiled app is usually relatively small.
• If your project is a front-end app running in the browser, deploy it as soon as it is compiled (using GitLab environments and Review Apps). You’ll have a public URL posted in the Pull Request, so developers, designers and product owners can see the result of the work on some easily accessible URL, without the need to pull the project and test it locally.
• Consider leaving “audit checks” for later: application size budgets, code coverage thresholds, performance metrics etc.

Fine tune the cache

Cache pulling and pushing can affect build speed significantly. It deserves a separate article because there is a bit to cover. For now, in most of the projects, I settled on a default, global cache configuration with policy: pull. I override it to push-pull only on jobs which contribute to the cache (e.g. a build job, where all project dependencies are fetched/installed). You can also switch off cache entirely if the job only needs build artefacts by setting cache: {} for a particular job.

Your GitLab setup and number of workers matter

I only have experience with self-hosted GitLab. The number of available workers matters. If you have just one or two workers (which you can set to run many jobs in parallel), don’t put many CPU-intensive jobs in the same stage. They will all kick in at the same time, and the actual result, in fact, might be slow.

Re-use data, move less critical checks into late stages

• Can your build process generate data for application size analysis? Enable it, add results to artefacts. Though, consider analysing, generating reports or failing the checks in a separate job, which runs late (so it doesn’t block other stages from running and giving you valuable feedback).
• Can unit tests generate test coverage reports? Good, store them in artefacts, and consider moving coverage threshold or coverage diffs checks to another stage. Again, you’ll get feedback if your tests are passing or not, early on. Whether they meet some acceptance criteria is kinda another thing. Also, there’s a difference in feedback like “your tests are failing” vs “your tests are passing, you didn’t break anything, just write a bit more tests”.

Aim for production-like setup

Are you doing End-2-End (E22) testing? Is Docker build part of your pipeline? Start that Docker container you have built earlier on and test against it, instead of other “local” environment.

Continuously Deploying to some public URL? Consider adding a late step with some smoke-tests. You might use the same E2E tests you already have written. Parametrise them, if needed (so that they can work on different environments, not just development one).

Promote your builds to next environments

Can you easily “promote” application which has been built, which has been well tested, from one environment into another? Don’t throw it away then. Perhaps a few injected environmental variables into the Docker container can do the job? Split your deployment jobs wisely, consider adding jobs with when: manual constraint for such things, which you then trigger from GitLab interface. Might save you a lot of resources and help do rapid deployments.

Summary

It is impossible to come up with a perfect setup in one go. Give it some time and be patient. Over time you will come up with a good one. As you observe your builds, you will discover bottlenecks and ways to improve overall pipelines performance.

Feel free to share how you organise your builds. Would love to learn about your strategies. Let’s discuss in the comments!

TL;DR; This is a highly non-technical blog post about software testing and automation, with lots of digressions, stories and anecdotal evidence about why it is a good thing to do. Too long for your taste? Have a look at “What’s in it for me” paragraph, that’s the only one I’d like you to read. I’ll be writing soon more useful and technical posts about testing and things around, so please come back.

Until 7 years ago, I didn’t do any automated testing. I didn’t have any Continuous Integration (CI) setup, not to mention more fancy stuff like Continuous Delivery or Continuous Deployment. At this point, I had been in web development for 10 years. I had worked on large scale projects for top companies in Poland, some of our productions were used by millions of users daily. I was aware of the benefits of automated testing, but I was not doing it.

Sure enough, back then the concepts of automation, Test Driven Development (TDD) and in general Extreme Programming concepts were not that common. Or at least they were harder to implement in practice. I was aware of them, I was reading a lot about them, it was giving me shivers on my back from excitement, but still, I wasn’t doing much about it. I was in a position where I could put it in place, yet, I didn’t move my finger to actually do something about it. Seemingly more pressing issues were consuming all my time.

I can hear you: this guy is a total looser, with so much apparent experience, he should have figured it out. Fair enough, if I’d be you, I’d stop reading right now, but bear with me for a minute more.

I was that guy which Stephen R. Covey talks about in The 7 Habits of Highly Effective People, in the Sharpen Your Saw chapter. I was madly cutting the trees all these years, one after another, trying to deliver fast, on time and with the highest quality. But I never stopped to sharpen my tools. My axe was dull, squared-off actually. I was telling myself: “hey man, stop, sharpen that knife, add some testing, add some automation, make your life easier, sleep better at night”. The answer I was getting from my head was: “I know I know, but sorry, I’m too busy, I have many trees to chop down. Don’t have time for this right now”. Well, that approach did cost me a lot back then.

Hint 1: find the time to sharpen your tools. Everything just works that much better when your utensils are in the best shape. The good news is: the more you do that, the easier it gets. And I’m talking about two things here: first, the sharpening skill itself; second, identifying when to take a moment to sharpen the tools. You learn to not postpone it too long, you learn that it’s not that scary to take some time off to do that.

7 years ago I moved to the UK, where I landed a sweet job in a small, independent, but profitable financial company, benchmarking performance of companies in IT field. They needed help with their platform, until then developed by an external dev house. The platform delivered content to high profile subscribers (bankers, M&A guys, VCs, PEs etc), with loads of financial metrics and benchmarks, integrations with external data providers etc. Nothing too huge, but more than enough to keep you challenged. I was on my own, officially hired as a lead developer (leading group of one, myself), with a promise of new hires soon. I was very passionate about the project (and actually interested in the platform’s content itself) — something very important to me, to really be involved and give 100%.

Hint 2: It feels so good to be passionate about the project you’re working on. It stops being a “work” and transcends into a “fun activity to do”.

Hint 3: be careful about Hint 2, though. It’s easy to loose that thing which everyone talks about, so-called “work — life” balance. Since it is no longer a “work”, remember? It would be now more a “pleasure — life” balance, which we tend not to see as problematic… until we hit that addiction centre or we spot divorce papers in our inbox.

While I was moving my entire life to the UK, I was going through plenty of changes in my personal life, so I decided to add one to my career too: I’m going to finally apply some of the principles of eXtreme Programming, esp. testing and automation.

I wrote 10k tests in 3 years

I did it. I was so religious about it, that 3 years later I had over 10,000 tests (assertions) written in the code. I primarily focused on unit and functional testing, but I also had fair amount of integration (aka E2E) tests. The platform our team (of one) has built was old’s cool server-side rendered app shipping HTML to the browser, so focus on unit and functional worked out pretty well. I also committed to a decent deployment setup, initially scripted with bash, later with Vagrant and later later on, with Docker obviously (full stack containerised, all on production, when Docker just came out from alpha to beta phase. What can you do when there’s no other right tool for the job?).

Hint 4: playing with new tech is risky and very time-consuming. If you decide to do so, make sure your girlfriend/boyfriend/kid(s)/friends/family are happy about it, and you somehow compensate for it. Don’t stretch too much. I just got reminded of this one no longer than 10min ago…

No QA department needed

I spent 5 years in the above-mentioned company, most of it playing solo developer, owning front-end, back-end and infrastructure. I have been wearing each of that hat before, at least for a period of a few years in a row, so I was somehow able to juggle things. The company grew from just a few people on board to roughly 20 of us and quite a few external analysts and researchers, with a decent revenue in millions of £. All that time: no QA department, no dedicated tester role, not much manual testing apart from some company lazybones analyst checking things after more significant deployments. And then we went for beers.

I was meticulous about making sure things (finances, metrics) are calculated and rendered the way they should. I was pain-in-the-ass, questioning all “promises” we all hear often (i.e. “that thing will never happen”, “you don’t need to worry about it”, “this case is probably OK-ish to skip for now”). Knowing upfront about “things-I-don’t-need-to-worry-about” is good, because it allows you prepare for them better.

Just to give you a context: we also had one business/marketing person coming up with requirements, specs and copy; plus an external UX/design agency; plus an external experienced IT consultant available, shall I need help with anything – so of course I wasn’t that alone and had a bit of support.

Hint 5: Don’t believe these falsy promises that “this thing will never happen”. Especially when you deal with management with proven track of mis-managing. When you hear “you don’t need to worry about this” that’s when you in fact need to start worrying. That is similar to the situation in England when someone says “Hey, I don’t mean to disrespect you but…” – that’s usually the moment they going to disrespect you.

Use your judgment and experience to decide, and try to balance the above cases well. Often it’s so much easier to add something straight away and make the app that much more robust — and your life that much easier. Very often that never-will-happen thing is the first thing that actually happen. Sometimes you have that gut feeling about it, but you are just being mis-managed.

Sometimes you could add that will-never-ever-happen-I-promise-you case to your design/architecture right away with relatively small effort (or at least make room for it), which in the long-term would save you days and your company tons of precious dollars.

Having automated testing in place tremendously helps to cover such scenarios in a fast and efficient manner.

So how did it go? We were doing very well. Sure, bus factor was nice and low, equal to 1.0. But… look ma! there were no bugs, no regressions, there were rapid, frequent, nearly automated deployments. I mean: you press the button and you go outside grab your beer/lunch with your girl. And steady, predictable development speed of new business features. The company learnt to trust its agile IT department (a group of one), and even huge deployments were “business as usual”. Of course, we had some little fuck-ups somewhere on the road. For instance, someone in the team (of one) screwed Memcache implementation and most of the content on different pages was the same. Or when Rackspace was restarting the servers overnight and some automated updates kicked in, screwing up some stuff. The lead developer apparently had better things to do than reading boring Rackspace emails and follow up on them — smoking was his habit and a beautiful park with trees and flowers was nearby. But otherwise, for years, days were pretty much awesome, stress-less, seemingly easy-peasy.

Hint 6: That is the serenity of mind and confidence you get when you have automated testing and repeatable environments in place.

Hint 7: Go outside for a walk sometimes (even if you don’t smoke/vape/inhale/whatever-is-your-thing). Do that despite a long list of tickets on JIRA stressing you out. It helps with coming up with better code and testing strategies, too.

One day we discovered a bug, and it was a bit shocking to everyone. That rarely used functionality just didn’t work! Yeah, I was shocked too. How come it passed the tests? By then I developed some sort of cockiness and overconfidence in my testing infrastructure. I quickly figured out what happened: that part of the system wasn’t covered in the tests.

Hint 8: That’s what happens to your app or feature when you don’t test it. They say: what’s not tested [with automated testing], doesn’t work. In my experience this definitely is true.

Hint 9: Mistakes will happen. You will screw things up, you will make mistakes, you will get distracted and miss that case, you will forget about sth you know you should have done – and that’s OK. So don’t even sweat over it, just have a good contingency plan. You also don’t need 100% testing coverage, there are actually more healthy metrics out there (to be covered in another article). And even if you have that magic number, that still doesn’t guarantee a bugs-free app.

By the way, don’t get me wrong. I’m not saying QA is not needed or anything like that. They bring lots of value which cannot be delivered in any other ways. You’ll do much better working closely with your QA people and paying respect to what they do and say. The point I’m making is that automated testing brings a plethora of benefits. OK, the point I’m also making is that you don’t need over-grown QA team. There’s better bang for company’s dollars somewhere else. Automated testing can do an excellent job and will work extremely well with well-sized QA team (which, nota bene, should have things automated too).

After nearly 5 years of being the only dev there, myself wearing all possible hats, the company hired a CTO and, shortly after, bigger dev team of three (I decided to move on, so we had to implement some changes). The new team wasn’t that religious about automated testing, rapid and confident releases stopped being possible. Sure, things got a bit more complicated, we started moving to the new stack (incl. the new Angular, yeah), but that’s just another excuse. There was so much time invested into discussions and meetings about what to release and when. Magic word “deadline” was used to keep heretics from testing camp quiet. Complicated planning was enforced to avoid fuck-ups. After a couple of months (I was still staying in the company for a while, helping with on-boarding new devs), the new CTO, great guy with hands-on experience, came to me saying: “Marcin, how the heck were you able to deliver all that, consistently, for so many years? The business now has some unreasonably high expectations regarding the new team efficiency, delivery capacity – and is somehow not happy with our current pace”. Not that much longer the company had a QA guy too.

You waste money when you don’t do it

I bet you figured out by now where I’m heading with all this. You can consistently deliver high-quality code and new features with good speed. With regressions nearing zero. Without new hires just for bug fixing. Without anyone spending months and being bored with just bug fixing. Without wasting your life and company’s precious dollars on bug fixing. But you can only do that if you practise automated testing. It pays off to do so. The ROI is much higher than some of us realise. Sometimes you see the return of your investment as soon as the end of sprint.

The excuse to not to do testing is… well, an excuse. And, an “excuse” — as I’m sure you agree — is often a euphemism for “bullshit”. It is also a synonym to “I have so much fucking dollars/pounds/whatever-currency-you-use that I don’t care”.

Hint 10: Having testing in place saves your (company) money, often as soon as the end of sprint.

The opposite is also true: without testing you loose money. For many reasons (to be written).

What shall I do?

Setting up testing and at least minimal CI/CD setup should be the first thing you do. If you’re not in the position to set it up yourself, you should strongly insist on getting it done. Both, when starting a new project or attending an existing one. Even if it’s a side project. No exceptions. Why would you rip yourself off of all benefits it brings?

It’s similar to when you have good eating habits, and you usually eat healthily, and then you go travel. You don’t all of the sudden switch to crap food, do you? You know it would rip you off of all the benefits of feeling great, performing well, and being strong and being able to climb that f*** steep mountain ahead of you. Sure, you might indulge a bit here or there, or the logistics will get in the way (e.g. that favourite steakhouse you get used to having a salad from doesn’t have a presence where you currently at) but still, you strive for the best available food, right? Let’s do similar with the development. Strive for the best possible setup.

What’s in it for me

Perhaps nothing makes a more complete developer than writing tests. Tests are a merciless teacher and they will pinpoint each code smell. That’s a good thing. It’s hard to test poorly-written code. It would require lots of even more poorly-written testing code. Luckily, by then you usually realise something is wrong and you start improving the original code. Small, single responsibility functions are must-haves. Good abstraction is an another must-have. You check the requirements with stakeholders. Quite naturally a process of coming up with better ideas, with better code is triggered. The code gets leaner: more declarative, explicit and concise, easier to understand and maintain.

You become an artist. The world becomes a better place to live, to code. Your (girl|boy)?friend(s)? love you even more. Your scalp loves you more too, as you stop pulling your hair out the next time you look at your code. Beautiful, isn’t it?

Commitment to testing, apart from already discussed all benefits, can even give you a bit of recognition (a.k.a fame) to the table, as you can see on the attached image. Black Swan, the data science company where I‘m lucky to work with some awesome and smart folks on exciting products, seems to value testing, amongst of other skills (BTW, we’re hiring, and we work on groovy stuff, and we’re cool and our tech stack is pretty awesome too).

My bosses are very generous, as you can tell. To be honest, I might come across (sometimes, rarely, almost never) a bit too pushy towards “best practises”, testing, linting, CI, automation, developer experience and why “good enough” is actually not that good at all. But there’s a reason for that.

And by “good” I don’t mean just good code and good code coverage. I actually mean good digital experience delivered to the end user — because that is what we ultimately do, that is what matters at the end. And often smelly code somehow — don’t ask me how — has a magical tendency to find a way to surface on that final product. In the form of unexpected bugs, clunky interface, slow performance, or something else. That’s why the quality of your work as a developer matters so much.

The time factor is often brought to attention when we talk about testing. That’s a big topic. I will often ask my team members not to give a damn about deadlines if it sacrifices quality. Because too short deadlines are short… sighted. Especially if we talk about those artificial, end-of-sprint deadlines when nothing really bad happens if you don’t deliver on the dot.

Hint 11: Do care about deadlines. But also know which one are the important ones.

Hint 12: There are perhaps better ways to deal with short deadlines, than just cutting corners on quality side — which, as we covered well, long-term is not sustainable. You can, for instance, sacrifice scope (i.e. do less).

Bottom line

When you do invest in testing, you will do better than just achieving end-of-sprint goals. You will shift the focus from urgent to important. You will deliver faster the final product — which is more like a marathon, not just a 2-week long sprint. Heck, you will sometimes deliver on time, which is quite unseen in this industry. Sometimes you will even deliver Ahead of Time, something you can only read in fantasy books (and in the Angular documentation, if you happen to be a diligent student).

I’m trying to make it all sounding a bit easy and funny. But it’s not. It won’t always be funny at that very moment. It won’t always be that easy. It takes a bit of dedication. You’ll need to learn about testing if you haven’t done so yet. It is a significant investment — like everything good in this world, it comes with a price tag. It’ll be painful and frustrating. Sometimes you’ll need courage to fight off a jerk or two, who can’t do the simple math themselves. Sometimes you’ll get fired for that. And that’s OK too. You don’t want to waste your precious life time in companies who do things which don’t actually make sense, do you?

Automated testing plays a crucial role in being an efficient developer and writing robust applications. Luckily, things got better in recent years and we see more and more teams doing that. I hope sharing here a few personal struggles on my journey will inspire someone not to give up too quickly when obstacles arrive. As Ryan Holiday says in his book, the obstacle is the way. It really does work, pays off high dividends and it’s worth an effort. And if you haven’t tried it yet, I really encourage you to do so.

PS. Some of the “hints” might seem enigmatic, unrealistic and/or even brutal. That’s intentional. I’ll try to cover some of the topics in separate articles in the nearest future.

PPS. Big kudos and thanks to Ashnita who did the first proof-reading of this post and helped me making it so much better.