Releasing a new version of a tool should be boring. No ceremony, no planning meetings, not even a discussion. It should be transparent, effortless, reliable, and even informative – more on that later.

However, as a software consultant who helps teams implement DevOps solutions, I often find that this is not the case. What I do find varies wildly in quality, so instead of telling you what I don’t like, let’s focus on how I like to structure my pipelines in order to onboard new repositories quickly and easily.

To understand my perspective, I'd like to share how my views on the topic have evolved over the years. Currently, I apply a process that involves arranging “LEGO blocks” of modular workflows with minor tweaks, such as parameter changes. Ideally, I aim for defaults that allow me to simply copy and paste the same building blocks from one project to another when they share the same type. For example, the build process for Rust library X is probably about the same as Rust library Y, and Node server foo is probably just about the same as Node server bar. Etc.

Over the years, I attempted to assemble open-source components to achieve this, but I never found quite the right combination. I built my own solutions finally, and I’ll share those with you soon.

But first…

I realize versioning and releasing aren’t the most exciting topics; however, it’s a problem that every project faces, even when vibe-coded (probably especially!). So let’s talk about how to do it right. In my opinion, of course.

Doing it right will make your project look much more professional and communicate changes to its users. This establishes more trust in your projects by helping developers understand and adopt new changes in a clear and consistent way.

In the beginning…

Back at the beginning of my software engineering career, releases were quite the event. They were the byproduct of milestones, achieved through a series of sprints, planned, to the best of everyone’s ability, in a series of planning meetings where the backlog was scoured through, and tasks that were worthy were chosen. We’d do our best to write out a spec of a plan, and estimate a Fibonacci number or T-Shirt size, or in worst cases, a number of days. Of course, there were always some bug fixes to be included to fix the previous releases as well.

The release was the culmination of this process. The deliverable of deliverables, shipped to the real users. As the culmination of this process, and high stakes for getting it right, the release was an emotional moment.

Exciting?

Scary?

Maybe a few you’ve triggered have got your heart racing as they set the machines in motion.

There’s also the detail and drama of deciding a version number.

Is this a major release? A whole new version?? Minor? Patch?

Sometimes marketing dictated the version… 2.0!

At this point in my career, it was largely not my opinion that mattered on this topic of version numbers, and looking back, it probably should have been no one's (you’ll see what I mean later).

I was not responsible for releases in the beginning… However… that would not last forever.

Into Open Source

At some point in my career, I became interested in contributing to open source. Generally, in the form of npm modules, as I was very into Node at the time.

It was my turn to be in charge of releases.

I can’t remember how, maybe through some newsletter or maybe part of the NYC NodeJS meetup that Matt Walters and I used to run together – I came across the library semantic-release. Semantic Release implemented a principle, which I vibed with as a web developer, of a separation of concerns based on semantics.

Before you go run off and start using semantic-release, there are some limitations I encountered with it… And I’ll get to those in a moment.

First, at a high level, the basic idea that semantic-release preached was that you’d follow a specific format for your commit messages, and based on the messages, the version could be calculated.

It spoke of removing the emotion from releases, making them robotic, based on what has actually changed. This had the benefit of allowing the version number change to communicate what had changed.

If there was a new major version release – v2 to v3 for example, that meant there was a breaking change.

To signify this in your commit message you’d include BREAKING CHANGE: in the body of the commit.

feat: revamp user auth flow

BREAKING CHANGE: Per Chad's "game-changer" vision in the 3am Slack rant, we've ditched the old auth system for a blockchain-based solution because "passwords are so 2024." Update your clients or enjoy the 500 errors!

Minor version bumps, like v2.0.0, to v2.1.0 signified there was a new feature added.

feat: add dark mode toggle

Per the 47-comment thread in the "urgent" ticket, users can now save their retinas. Dark mode added, but brace for the inevitable "make it darker" feedback.

Patch versions signaled fixes, or refactors, or basically most anything else that should trigger a new release.

fix: revert "fix" by AI that skipped the breaking tests to avoid the failure

And finally, some commits should not trigger a release at all. “Chores” as they were introduced to me. No-ops, as I call them.

chore: update release pipeline version from v3.1.0 to v3.1.1

This specific format of commit messages was known as Conventional Commits, though there are others. You can check our their page for the full spec, but I wanted to share the gist of it.

Furthermore, this information contained in the commits could also be surfaced as a change log — such as highlighting a breaking change. For example:

What's new in v2.0.0

* feat: remove deprecated API

  BREAKING CHANGE: the FooBar API that was deprecated. To upgrade, you can use the new BazBar API. 

So, for a while, semantic-release was great for me. For Node projects, it still is… However…

Into DevOps

I was always interested in the idea of how to build scalable distributed systems, and so it was something I pursued for quite some time in my career. Long story short, I eventually learned how to do it well, but had a new problem: how to deploy all the pieces.

These weren’t npm modules, they were applications, services, databases, and queues!

I loved the simplicity of semantic-release, but it only worked well for Node.js projects. Sure, you could, and I did a few times, add a package.json file to non-node projects, but that was messy and hacky at best.

Around this time, I became deeply involved in DevOps, during the Jenkins era, as Docker was starting to gain traction. I wrote about my early Devops Journey and discovery of Docker for deployment. I learned to write pipelines with Jenkins, and deploy everything with Docker Swarm!

I was also embracing trunk based development at that time, and committing to master. My more recent and still relevant and accurate Guide to Git with Trunk Based Development article explains the why and how of that if you’re interested.

I always wanted these projects that I worked with to be versioned and released like semantic-release projects were, but I never found a solution that worked as well. Many wouldn’t be able to handle generating changelogs correctly, and some would sporadically bump the version wrong.

It wasn’t the most important thing on my list, so I generally called it good enough and lived with the flaws.

Over time, as a DevOps consultant, I have encountered and released various projects in multiple languages and frameworks, and every single one has required versioning, releases, and changelogs.

Another headache I often encountered was large, monolithic pipelines. Coming from the Jenkins era, I get it, this is how I used to write pipelines too. I‘d try to have the whole release flow in a big series of steps starting with a push to main. If you pushed to main, various quality checks would run, and a new version would be built, published, tagged and released.

This isn’t terrible per-say and also probably “good enough” in most cases, however, as a DevOps consultant, I have to do it repeatedly. Breaking things down to smaller more modular pieces, allows those pieces individually to serve more projects. I first heard this described by the infamous James Halliday, AKA substack, as the coverage of ideas, an idea that comes from embracing UNIX philosophies.

For example, a Node Quality pipeline can be the same in every Node project, assuming you follow the standard npm script conventions, and so a generic pipeline for running linting, building, and unit testing can be shared across most, if not all, Node.js projects. The pipeline steps are simply calling npm run lint --if-present and etc, and the scripts section of the package.json file determines what happens when that lint command is run.

High-Level Pipelines For Versioning and Releasing

At a high level, every project that needs to be released will first be tagged with a new version number. Then, a release can be created that is linked to that version number, and artifacts will be produced as part of its associated pipelines.

I use GitHub Actions most frequently, as I think pipelines are commodities, and it’s a simple one to get started with. I’ve used many, but they are all essentially the same — perform a series of steps with a shared volume. That shared volume generally contains the checked-out source code.

Versioning itself, if we are following the standard of conventional commits being tied to a semantic version increase, is not a language-specific thing. It has no dependencies on the contents of the code; instead, it’s concerned with the history of versions and commits.

For a long time, I tied releases to versions, thinking they were part of the same thing, when in reality they are two distinct entities joined together. A tag, and a release.

Splitting them apart made the versioning part of the pipeline its own modular piece, which then triggered a build of that version and subsequent release.

Let me break it down… For each project, in Github Actions, I have two pipelines:

1. On commit to the trunk branch, version and tag
   When a commit occurs in the trunk branch (usually main or master), trigger a pipeline that runs quality checks, and, if they pass, calculates a new version number. It then creates and pushes a tag with that new version number. We can also use this opportunity to update that version number in the code base if necessary, and generate a change log from the commit data that was used to calculate the new version number.
2. When a tag is created, build new versions and release them.
   This step is simplified by the previous step doing the version bump. Everything’s already been bumped to the new version. We can just run build and release using the tag as the base.

As I mentioned earlier, after trying many times to piece together open-source components from the GitHub Actions Marketplace, I eventually gave in and wrote my own tool that handles step one. I also wrote some workflows, which I’ll share with you as well.

Introducing vnext

vnext is a fast Rust CLI tool that uses conventional commit messages to compute your next semantic version, automating major, minor, or patch bumps for streamlined releases.

Here’s how to use it.

NEXT_VERSION=v`vnext`

vnext --changelog > CHANGELOG.md

I think it’s pretty straightforward! Let me know what you think.

It is entirely based on the project’s git history.

It does not matter what language or tool you are releasing; the versioning process is basically the same. Get a new version, update it in some files, create a changelog, tag, and push. The only difference is which files need to be updated — this is accounted for in the workflow I’ll share in a moment.

vnext is quick and easy to install via ubi — the “Universal Binary Installer”. Once configured, you can run:

ubi --project unbounded-tech/vnext

I’ve also created a shared GitHub Actions workflow that you can call, which handles all those details.

Here’s how to use the shared workflow:

name: On Push Main, Version and Tag

on:
  push:
    branches:
      - main
      - master

permissions:
  packages: write
  contents: write

jobs:

  version-and-tag:
    uses: unbounded-tech/workflow-vnext-tag/.github/workflows/workflow.yaml@v1
    with:
      useDeployKey: true
      changelog: true

Most projects will also need to configure how and which files to update with the new version number.

Currently, vnext supports two generic methods (yqPatches and regexPatches) and a few language-specific options.

The language-specific options are the simplest to use. For example, setting with.node to true uses npm to update the project’s package files.

yqPatches uses the tool yq to patch specific fields in a YAML file. I often use this for updating version numbers in Helm charts:

  version-and-tag:
    uses: unbounded-tech/workflow-vnext-tag/.github/workflows/workflow.yaml@v1.13.0
    secrets:
      GH_PAT: ${{ secrets.YOUR_ORG_SECRET_PAT }}
    with:
      usePAT: true
      changelog: true
      yqPatches: |
        patches:
          - filePath: helm/values.yaml
            selector: .image.tag
            valuePrefix: "v"
          - filePath: helm/Chart.yaml
            selector: .version
            valuePrefix: "v"

I also took this opportunity to share how to use a Personal Access Token instead of a Deploy Key.

regexPatches uses sed to find and replace a string with the new version number inserted. For example:

regexPatches: |
  patches:
    - filePath: package/composition.yaml
      regex: /ghcr.io/org-name/package-name:(.*)/g
      valuePrefix: ghcr.io/org-name/package-name:v
    - filePath: README.md
      regex: /Current version: v[0-9]+\.[0-9]+\.[0-9]+/g
      valuePrefix: Current version: v

You can use a combination of options as well.

A nuance about GitHub Actions

When you run an action on GitHub Actions, by default, the worker generates a temporary GitHub authentication token. This token has permission to perform a few basic tasks; however, it is never allowed to trigger other pipelines.

But our next step after tagging a new version was to trigger another pipeline with that tag!

Do not fret, this is by design — simply a measure by GitHub to prevent unintentionally spinning up a bunch of runners. You are allowed to do so, but you need to be intentional about it.

There are a few ways to be intentional about it — I’ve already shown an example of two:

1. Using a Deploy Key and corresponding GitHub Actions Secret
2. Using a Personal Access Token as a GitHub Actions Secret
3. Creating a GitHub Application (theoretically — I haven’t had a need for this option)

Using a Personal Access Token is convenient if you are a paying customer in the GitHub ecosystem, as organization-wide secrets are available.

On the free tier, that’s not an option, so I’ll show you how to set up a deploy key instead.

In fact, I made it very simple by building it into the vnext CLI. It is a great option, as when configured this way, no human needs to know it, and it can easily be rotated by rerunning the same command.

For each repository you want to set up vnext to release with a deploy key, in the directory of the project…

First, get an auth token using gh with permission to manage keys:

gh auth refresh -h github.com -s admin:public_key -s admin:ssh_signing_key

export GITHUB_TOKEN=$(gh auth token)

And then run:

vnext generate-deploy-key

You may be wondering, can’t this go into a pipeline itself? Unfortunately, I can’t find a way to automate this step in Github Actions without creating a Personal Access Token, as the default token can not modify Secrets. This defeats the pupose of using a Deploy Key instead in the free tier, as you’d need to create it in every repo. This requires some sort of secret management to make it easy, and thus makes it more work than using a deploy key. I have thought of it. :)

With your Deploy Key, or Personal Access Token (PAT), in place, and the version and tag workflow running on pushes to your trunk branch, the next step is to release!

This will vary per project, but they are all triggered by the version being tagged from the previous step. These pipelines will generally look something like this:

name: On Version Tag, Trigger GitHub Release

on:
  push:
    tags:
      - 'v*.*.*'

permissions:
  contents: write

jobs:
  release:
    uses: unbounded-tech/workflow-simple-release/.github/workflows/workflow.yaml@v1
    with:
      tag: ${{ github.ref_name }}
      name: ${{ github.ref_name }}

Or, if it were a Rust project, maybe something like this:

name: On Version Tagged, Build and Publish Rust Binaries
on:
  push:
    tags:
    - "v*.*.*"

permissions:
  contents: write

jobs:
  build-and-release:
    uses: unbounded-tech/workflows-rust/.github/workflows/release.yaml@v1
    with:
      binary_name: ${{ github.event.repository.name }}
      build_args: "--release --features vendored"

Or if it were an application with a Dockerfile, and k8s resource definitions for Gitops deployments, maybe something like this:

name: promote
on:
  push:
    tags:
    - v*.*.*

permissions:
  contents: write
  packages: write
  issues: write
  pull-requests: write

jobs:

  publish:
    uses: unbounded-tech/workflows-containers/.github/workflows/publish.yaml@v1.1.1

  release:
    needs: publish
    uses: unbounded-tech/workflow-simple-release/.github/workflows/workflow.yaml@v1
    with:
      tag: ${{ github.ref_name }}
      name: ${{ github.ref_name }}

  promote:
    needs: release
    uses: unbounded-tech/workflows-gitops/.github/workflows/helm-promote.yaml@v1
    secrets:
      GH_PAT: ${{ secrets.GH_PAT }}
    with:
      environment_repository: your-org/staging-env
      path: .gitops/deploy
      project: staging

Note: A deploy key can’t push to other repos, such as in the gitops promote step, so you might consider using a PAT everywhere for this repository. If you have lots of workloads like this, upgrading to a paid org might be worth it for organizational wide secrets rather than configuring every repo separately.

Generally, every release is a combination of some pieces of these similar modules, which do pretty much the same things but for their specific tools or languages.

Usually, organizations at least try to follow many of the same patterns across applications and services they build, and so breaking down the workflows into shared modular pieces allows devs to pretty much just copy and paste their flavor of these shared workflow calls to each project of that type.

Consistent Releases with Changelogs

You may have noticed I kind of glanced over the with.changelog flag in the Version step. I wanted to circle back to that, as it’s a significant part of a mature release process. The changelog is how you can communicate the ways in which your project has changed to other developers.

Along with bumping versions, vnext also uses the commit messages to construct this changelog and saves it into a file named CHANGELOG.md, which is committed along with the version tag and version bumps.

Hidden in those release workflows I shared above, this file is used as the body of the GitHub Release. For example, from the vnext project itself:

This release was a patch version, as it contained a fix, in the form of a dependency version bump, and a number of other chores.

These release notes are then surfaced in other tools, such as Renovate, a tool that makes PRs to keep your dependencies up to date.

When merging Pull Requests, I like to use the Squash and Merge feature of GitHub in order to have the opportunity to write a nice final commit message title and body that will be used in the changelog.

For example:

As shown in the example, you can put full-fledged Markdown as the commit body. It will be rendered appropriately in the Release Notes!

Conclusion

I know versioning and releasing isn’t the sexiest topic; it’s rather boring — or rather, it should be!

However, this isn’t always the case. That’s why I created vnext and a bunch of shared workflows! Now I can easily onboard most projects very quickly!

Hopefully it’s helpful for you too!

Please let me know if you give vnext and some shared workflows a try! I’d appreciate some stars on GitHub and a share if you find them useful!

Releasing New Versions Should Be Boring. Really Boring. was originally published in codeburst on Medium, where people are continuing the conversation by highlighting and responding to this story.

I went to a party a couple of weeks ago, and out of an entire table of food, not a single thing was from Texas.

The next time you walk through a grocery store, take a look — what’s local? Even in a high end market like Central Market, or Wheatsville, a co-op store in the area, or a Sprouts, a natural food health based grocery chain… Almost nothing is from where you live. Around here you’ll maybe find some local watermelon or peaches this time of year.

We are so disconnected from our food systems, that when a group of people get together and bring food, it almost entirely comes from many hundreds to thousands of miles away.

Food contains light information — biophotons. Animals, and us, create biophotons as well. Infrared light as a byproduct of Oxidative Phosphorylation, which is the primary process our body uses to create energy. We also produce UV biophotons in response to cold exposure.

When we eat food that is local to us, we are getting light information about the environment we live in which helps us adapt to it. When we eat food that was shipped from the tropics when we live in the north, we are sending our bodies the wrong signals. Also consider that these biophotons decay over time.

There’s a Native American tribe’s story about how fire is stored sunlight inside of plants, and the process of making a coal through friction is releasing that stored sunlight energy. In addition to getting energy from food through Oxidative Phosphorylation, we also store electrons from sunlight, as well as collect them from the Earth when grounded, and by generating piezoelectric energy through movement.

Electrons are funneled through the cell’s Electron Transport Chain, a series of protein complexes and electron carriers embedded in the inner mitochondrial membrane. Electrons are passed along the chain through a series of reduction or oxidation reactions (redox), moving from higher to lower energy levels. Each complex in the chain has different affinities for electrons, causing them to be transferred from one molecule to another in a specific sequence of oxidation and reduction. The energy released during these transfers is used to pump protons across a membrane (inner mitochondrial membrane in eukaryotes), creating a higher concentration of protons on one side. This electrochemical gradient represents potential energy. ATP synthase, embedded in the membrane, acts like a channel for protons to flow down their concentration gradient back across the membrane. This flow of protons provides the energy to rotate a part of the ATP synthase molecule, which then converts ADP and inorganic phosphate into ATP. ATP is the so called “energy molecule” used throughout our bodies.

We essentially have a store of electrons that we collect from being grounded on the Earth, swimming in it’s waters, bathing in it’s sunlight, climbing it’s mountains, and eating it’s fresh electron rich food, and arguably the aether, you know, if that exists. We use these electrons to power trillions of cells, which collectively produce more energy every second than a bolt of lightning. They are conducted through our coherent water based liquid crystal fascia to every part of our body.

My truck about a year ago had an issue with a component that was responsible for distributing energy to the different electronics of the vehicles. This started as issues with the touchscreen no longer responding to touch, and eventually led to the vehicle not starting, and the piece needing to be replaced. A vehicle is a system, and the failure of a single part can cause the failure of the whole system.

We are also systems — ones much more complex than a truck. If we have issues getting energy to the various subsystems of our bodies, whether through conductivity issues, or collection issues — not having enough, or not being able to get it where it needs to go, then we run into trouble. After time this can lead to cascading issues. Cells throughout the body panic and enter the cell danger response, shutting down essential health time duties in exchange for batoning down the hatches and survival mode. This manifests as many types of disease and issues throughout the body, such as the inability to detox properly, or evict parasites. Yes, that’s right, I’m saying parasites and heavy metals and nutrient imbalances, defeciencies, and dependencies are symptoms, not root causes. Therefore, detoxing before or without addressing redox potential, or the bodies ability to conduct electrons throughout the body, are very large missing pieces of the puzzle for many people.

Hell, most people didn’t even realize they were electric.

It’s a hard thing to generalize what exactly makes someone successful, but, over many years, I’ve landed on a framework that works well for me.

I’ll get to that in a moment, but first, let me share a quick story that illuminates the point I am trying to make.

In 2019, one of my best friends died. This was due to a congenital heart defect, and a resulting build-up of scar tissue over the years. As this was something he was born with, knew about, and saw a specialist to monitor yearly about, I’m not suggesting there was a lot that could have been done differently, yet, regardless of that, it certainly made me consider my own mortality.

“You can just die at 30 years old…” I thought to myself.

Up until this point, I have to admit, I felt pretty invincible, but the sudden death of a close friend brought that belief crashing down around me.

I took a good long look at myself, and my health, and had to admit that I was far from where I needed to be…

It was time to make some drastic changes… but what changes, exactly?

There are a plethora of differing opinions on the internet about what to do to be healthy. Luckily, I had learned some very poignant advice from one of my mentors, Russell Brunson…

“If you want results, find someone who has already achieved those results, and do what they say! Don’t take advice from someone who hasn’t achieved those results themselves!”

So that’s what I did. I found a dude who was jacked to be my mentor in getting in shape and did what they said. This person was Dr. John Jaquish.

Dr. Jaquish had previously spent years in the gym with very lackluster results but recently had figured out a formula that worked for him, and suddenly he was unstoppable. He clearly had the results I wanted, so, I figured I should listen to his advice.

This advice consisted of a few key pieces:

1. Variable resistance strength training
2. Carnivore nutrition
3. Fasting

All I had to do was be disciplined enough to follow the advice consistently, and it should work for me too!

So I did.

In the first three months, I went from 210 lbs to 180 lbs. In the 9 months following, I leaned out all the way down to 160 lbs.

In just one year, by following the advice of Russell, to follow the advice of someone who had the results I wanted, Dr. Jaquish, I was healthier than I had been in years! Not only that, but the results came quickly and easily.

This wasn’t the first time I’d tried to reach these goals, but in the past I just went off on my own, reading conflicting opinions from different people, and trying to figure out what worked or did not on my own.

If someone has already been down all of the paths and has seen what worked, and what did not, you really don’t need to do that as well!

And so, that leads to my key point – the fastest path to success is the one that has already been proven to get the results that you are looking for! Instead of wandering down many paths yourself, find someone who’s done that already and ask them “What is the best path to take?”

Yet, this seemingly obvious path is not one that many people consider. We’ve been trained from a young age to “figure it out” through our schooling system — to seek the “how?” of a solution when all along there’s been a better question to ask. “Who?”

This simple advice, “who, not how”, is something that the highest achieving entrepreneurs are intimately aware of.

The first time I became aware of this concept was when I read the famous book “Think and Grow Rich” by Napoleon Hill. The premise of this book is simple — think about what you are going to do before you do it and you’ll get much better results.

For those who are not aware of this cult-classic book, let me explain a bit…

Napoleon Hill was Andrew Carnegie’s right-hand man. He was tasked by Carnegie to go out and research — how does one become “successful”? He leveraged Carnegie’s network of successful people by interviewing them all and compiling everything he learned into a book. This book became “Think and Grow Rich”. Napoleon became fascinated with the subject and went on to write many other extremely influential books as well, such as “Outwitting the Devil”.

In “Think and Grow Rich”, one of the key concepts that Napoleon Hill outlines is something he called a “mastermind”. A mastermind is a meeting of the minds of the various masters of a subject. This concept is extremely popular nowadays, with every Guru leading a mastermind of their own, and for good reason! Getting the best and brightest together to share what paths they’ve taken, which ones paid off, and which ones did not, is an ENORMOUS time saver.

Napoleon Hill’s Think and Grow Rich is a highly recommended read. It’s widely revered for a reason. Mr. Hill was a legend of his time. A first edition of a rare book set of his was recently sold for $1.5M to give you some idea.

So there you have it, my secret framework for success — if you want to be successful, find someone who’s gotten the results you’re looking for, and do what they say!

Through my consulting business, I equip technology organizations with the latest and greatest in DevOps tooling to increase the velocity of their development teams.

Part of this means abandoning outdated workflows like “GitFlow” and replacing them with more streamlined processes — Trunk Based Development and Continuous Deployment.

We’ll get more into “Trunk Based Development” soon — but it’s important for this conversation to define what “trunk” is, as I’ve seen this misused as well.

The “trunk” is the main or master branch of your repository. Trunk is a constant. In trunk based development, you either commit to the trunk branch, or make branches and pull requests against the trunk branch. There are no long lived alternative branches to merge against such as development.

I’ve seen teams create a new branch and call it the new “trunk” every few weeks. Although I applaud the effort, it was missing the mark by quite a lot.

We’ll get more into what this process looks like end-to-end later on… First, let me spend a minute backing up my claims of GitFlow being outdated, and explain the goals of moving to a different approach.

Some of the best programming advice you’ll ever receive is from Eric Evan’s required reading, Domain Driven Design. Domain Driven Design, for the uninitiated, is a methodology for modeling business domains in code. The advice is the following:

“Make implicit concepts explicit” — Eric Evans, Domain Driven Design

What exactly does this mean? Also, how does this apply to Git or GitFlow?

First, let’s define what an “implicit concept” means. An implicit concept is something that happens as a side effect of something else. For example, a side effect of committing to the development using GitFlow is often to tie that to deploying a version of that service to an environment such as staging.

The explicit action taken is committing to a branch. The implicit action caused by that action is running a deployment process. We didn’t explicitly change our staging environment — we changed our application’s codebase on a specific branch and it updated the environment as a side effect.

We then often see the same thing committing against the “trunk” (main/master) branch. The explicit action is committing to master - the implicit action is tying some sort of deployment process to this explicit action. Expand that to several repositories and now you’ve got yourself a real headache.

So, ask yourself — how can we make the currently implicit concepts of releasing new versions of our software, explicit?

The answer: by making those concepts a part of our model. The reason for the awkwardness we are feeling is because the model doesn’t actually model our environments. Our environments in this model are entirely implicit — and as you can imagine, things like production environments are quite important pieces — one that should be modeled explicitly, for sure.

To do so, we can create repositories that explicitly model each of our environments. Meaning instead of tying the implicit concept of releasing and deploying code to our commits, we can instead explicitly define the requirements of such a release and deployment in a codebase.

For example, our staging environment’s repository might look something like this:

dependencies:
- name: content
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.75
- name: mongodb-replicaset
  repository: https://kubernetes-charts.storage.googleapis.com/
  version: 3.15.1
- name: person-model
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.20
- name: www
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.55

What version of each application is in staging?

The answer is no longer “whatever is in the development branch for each project”. It’s 0.0.75, 3.15.1, 0.0.20, and 0.0.55. It’s explicitly defined in an environment repository.

Meanwhile your production environment might be on www version 0.0.50 and content on 0.0.74. Also explicitly defined in code.

Much better than “whatever is in trunk of each project? 🤷‍♂️”

To accomplish this we’ll need to approach things a bit differently than you’re probably used to.

We’ll still want to tie some actions to commits, but not so many different concepts — we only want to do things that are related to the codebase we are working on. Instead we can limit that action to releasing a new versioned artifact of the project we are working on.

More explicitly, this means the first commit to the trunk branch will release version 0.0.1 of that project, bundled and ready to go – finalized and immutable. The second commit to trunk will result in release 0.0.2, and the third 0.0.3, etc. We can also run some automated tests as part of that release process, so if commit four failed those tests for example, 0.0.4 would not be released until the 5th commit which makes the tests pass again.

What exactly those versioned artifacts are will vary by the type of project you are working on, but are usually Docker images, chart repositories, or various language specific packages like NPM or Maven releases.

Next, we can hook into the release process and with each new release, caused by each verified commit to Trunk, to explicitly update the staging environment

Then, once the explicit combination of versioned artifacts are verified in the running staging an environment, they can all be moved into production in a single explicit commit

So that’s pretty much modern Trunk Based Development in a nutshell.

Not so bad, right?

You can use a tool like Jenkins X to get this set up for you out of the box. Using JX can be a bit overwhelming as it is based on Kubernetes and several tools in the Kubernetes ecosystem. In my course, DevOps Bliss, I walk through setting up a system like this end to end, as well as get into other topics like SSO, load testing, and more! Check it out if you’re interested!

What exactly is in git history?

With our Trunk Based Development workflow defined, it’s worth taking a moment to explain how to think about what your project’s git history really is.

The history of your repository is a series of commits in an ordered list. To move to different spots in that history, pointers are used. These pointers are called branches and tags.

Check out this image pulled from the Git — Branches in a Nutshell documentation.

It’s helpful to remember this as you are working with Git.

This whole structure, the commits, the pointers, etc, are all copied to your machine when you clone a repo. They are not kept up to date automatically though.

You are probably used to using git pull at this point to get the changes from the origin.

Instead, I want to show you a few alternative commands to help you with your Trunk Based Development workflow.

General Workflow

In this next section I want to capture the “day in the life” decision making and git workflows you’ll need to employ when working on a project that follows Trunk Based Development.

First of all, whether you are working directly against the Trunk branch (yes, this is ok), or on a branch that will be merged to the trunk branch, you’ll need to know what’s happening back on Github — information like, have other developers changed the code since I’ve last checked?

I’m assuming you are familiar with the basic git workflow of add, commit, push, and pull, and here is where you might think pull. I want to give you a more granular way of thinking about it instead.

To do that, we’ll start with a git fetch.

git fetch

As you are working away on your next fix or feature, either directly on the Trunk branch, or on a short-lived branch that will eventually be merged into the trunk branch, so are other developers.

They are pushing and pulling from the same remote on Github named origin as you are. Adding new commits to the log, and new pointers in the form of branches and tags.

The command git fetch simply fetches the information about these changes. It does not update your local copy with the remote changes, it simply makes git aware that those changes exist.

So, as you’re busy working, every once in awhile, run:

git fetch --all -p

The --all flag tells git to fetch from any remote you’ve added, origin or otherwise. The -p flag stands for prune which tells git it can cleanup deleted branch pointers and etc.

Then you’ll know if there are any updates you need to bring into your local copy.

Here is an example of what that looks like:

> git fetch --all -p

Fetching origin
From github.com:servicebus/kafkabus
 - [deleted]         (none)     -> origin/renovate/mocha-7.x
remote: Enumerating objects: 150, done.
remote: Counting objects: 100% (150/150), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 198 (delta 125), reused 97 (delta 92), pack-reused 48
Receiving objects: 100% (198/198), 362.90 KiB | 1.31 MiB/s, done.
Resolving deltas: 100% (144/144), completed with 6 local objects.
   98169c3..6b4ca35  master                  -> origin/master
 * [new branch]      renovate/commitizen-4.x -> origin/renovate/commitizen-4.x
 * [new branch]      renovate/kafkajs-1.x    -> origin/renovate/kafkajs-1.x
 * [new tag]         v2.0.1                  -> v2.0.1
 * [new tag]         v2.0.2                  -> v2.0.2
 * [new tag]         v2.0.3                  -> v2.0.3
 * [new tag]         v2.0.4                  -> v2.0.4

In response to the git fetch call, I can see that master has been updated on origin compared to my local version of git history: 98169c3..6b4ca35. There are also a few new branches and tags.

Running git log will verify I am still on the 98169c3 commit locally.

> git log

commit 98169c3fabf3052dd89fe0c6900bc3c11a0252a4 (HEAD -> master, tag: v2.0.3)
Author: Patrick Lee Scott <pat@patscott.io>
Date:   Sat May 23 13:48:00 2020 -0400

    fix: high throughput test already uses no transactions

# ... more commits ...

In the parenthesis we can see a few pointers for this commit: HEAD, master, and tag: v2.0.3.

With that information in mind, we’ll want to rebase our changes on to the version of history pointed to by origin/master, which was the latter, 6b4ca35, from the info about the fetch (98169c3..6b4ca35). That is to say, the changes in Github made it to Github first, before our changes, so our changes should be moved on top of it.

Depending on our situation, we’ll either want to use a regular, or an interactive rebase.

Working on a fix directly against the Trunk branch

If you are working on a fix that can be done easily, there is no issue, in my opinion, with committing directly against the trunk branch. I’ve worked on teams with 20 engineers doing this, and I promise you, it’s fine, so long as you have a solid CI/CD process. If this process fails, the worse that happens is nothing. The build is marked as failing, nothing is released or promoted.

The only rule is if you break the build it’s now your job to fix it, and that’s the top priority!

If you’re afraid of this, it’s ok to use a branch, and I’ll cover that workflow next.

So, when you are working directly on the trunk, you’ve fetched info from Github and there are new commits to incorporate, it’s now time to rebase.

git rebase

To rebase our changes on top of origin/master we run the command:

git rebase origin/master

This only works AFTER a fetch, otherwise, git doesn’t know the information about new commits and branches from the remote named origin. Make sense? Use fetch to get info, and rebase to then use that info.

All of the commits you’ve made after the initial commit, which is 98169c3 in this case, will be picked up, placed to the side, and then your history will be updated to match origin/master. This means your log of history will match the origin version of history - ending in 6b4ca35 in our example. Then your commits that were placed aside, will be placed on the end of the log.

> git rebase origin/master

First, rewinding head to replay your work on top of it...
Fast-forwarded master to origin/master.

Now, when I run git log you’ll see the latest commit has been updated to the latter commit identified by fetch earlier (98169c3..6b4ca35).

> git log

# ... your commits ...

commit 6b4ca353091a7d6a9eeba8ee5b1978112a81cabf (HEAD -> master, origin/master, origin/HEAD)
Author: Renovate Bot <bot@renovateapp.com>
Date:   Wed Nov 4 06:34:00 2020 +0000

    chore(deps): update dependency jest to v26.6.3

If there are no conflicts, this will be a pretty simple command.

If there are conflicts, they must be resolved first. To resolve conflicts, in your IDE, fix the conflicts, and when you are ready, save the changes, add them to git’s staging area, and then run git rebase --continue.

To learn more about rebase, make sure you read the Git’s docs - Git - Rebasing.

To learn more about resolving conflicts, I recommend learning the VSCode Merge tool: Version Control in Visual Studio Code

Working on a branch that will be merged into Trunk via a Pull Request

If you are doing something more than a quick fix, such as working on a new feature, you’ll likely want a branch to work on.

git rebase -i origin/master

The above git rebase works well when we are on master working on a single commit for a fix. When we are on a branch, preparing for a change to be merged into master via a Pull Request, it’s often expected that we make things nice and clean. Maintainers of open source projects will ask you to “squash and rebase” your commits into a single commit to be merged.

This keeps the history of the master branch clean and rollback’s easy to perform.

People will debate on the merits of doing this vs. not doing so, but let’s put all those reasons aside for now, and just focus on how to do it.

If you specify the -i flag or —interactive when you run the rebase command, instead of a regular rebase, you will be performing an “interactive rebase”.

This allows you to reword commits in your little segment of history, or squash multiple commits into a single commit.

Doing so drops you into an editor in your Terminal called vim, and for this reason, knowing the basics of VIM are required in order to perform the interactive rebase.

Here’s an example from the git documentation (Git — Rewriting History):

pick f7f3f6d Change my name a bit
pick 310154e Update README formatting and add blame
pick a5f4a0d Add cat-file

# Rebase 710f0f8..a5f4a0d onto 710f0f8
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

In order to squash all of the commits, use VIM to edit the messages that say pick to say s or squash for the 2nd and 3rd commits:

pick f7f3f6d Change my name a bit
squash 310154e Update README formatting and add blame
s a5f4a0d Add cat-file

This will result in a single commit, Change my name a bit, that consists of the changes from all three commits.

Say you wanted to change the commit message as well, instead you might edit the commits like this:

r f7f3f6d Change my name a bit
s 310154e Update README formatting and add blame
s a5f4a0d Add cat-file

Saving this change would then prompt you with another VIM window that allows you to type a new commit message for the r or reword flag.

git push origin/<branch> —force

After a rebase, you’ve essentially modified the history of commits in the log. If you try to push these changes to origin you will get an error! This is expected.

You need to tell git that you have intended to change the history by using the —force flag. This will replace origin’s log of commits with your new rebased log of commits.

WARNING: You should not be force pushing to master, just branches!

Rebasing when you have unsaved changes

Need to rebase but have uncommitted changes?

If you try to do this, you will receive an error saying you must commit or stash your changes in order to rebase!

> git rebase origin/master
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.

If you are ready to commit them go ahead and do so. If not, you can stash them, perform the rebase, and then retrieve your changes from the stash again.

git stash

To put all of your changes aside in a temporary space, run:

git stash

Then, you can complete the rebase:

git fetch --all -p
git rebase origin/master

After the rebase is complete, you can bring back all of the changes you were working on from your stash:

git stash pop

--autostash

The workflow above is common enough that rebase actually supports a flag, --autostash that combines all the steps into one!

git rebase origin/master --autostash

Conclusion

Although it can be more steps than git pull, sometimes pulling results in unexpected merge commits - using fetch, stash and rebase gives you more explicit control when working with git, which is very helpful when working with Trunk Based Development!

Hopefully you understand the benefits of making our previously implicit operations explicit and how you can use a few new git commands to ease that process!

If you’re looking to implement Continuous Deployment in your organization and are looking for help, reach out to me through my website patscott.io. If you’re a DIY-type of engineer like me, then check out my course DevOps Bliss instead!

Last year I wrote a five part series of articles that built a React boilerplate using Parcel. It demonstrated how to use streaming server side rendering, automatically enforce code quality, achieve 100% code coverage, explored using Docker for development, and covered multi-stage Dockerfiles for production.

Unfortunately, as time has passed, the boilerplate has begun to get stale. Some of the libraries I chose have changed their APIs, and as a result people following the tutorials as written have encountered some errors as they are getting later versions when following along and typing npm i.

It’s come time to update this boilerplate, but in my typical fashion, I’ve decided to turn it into a learning experience by showing how to let bots do the heavy lifting for me.

In this article we will use a particularly useful bot called “ Renovate” which will keep track of the project’s dependencies and make Pull Requests every time an update needs to be made. Renovate was recently acquired by WhiteSource and was previously named RenovateBot. Because we have a bunch of automated tests and code quality enforcement at each step of the way we should be able to quickly see if everything is still passing or if a particular update causes any breaks.

At the end, we will also explore what extra steps could be taken to give us even more confidence in each change.

Let’s get started.

Step 1: Enable Renovate

My repository is on Github, and therefore we will head over to the Github Marketplace where we can enable the Renovate Application. According to their website, there is also support for GitLab, as well as a tool that can be run with some manual set up in other situations such as On Premises. I’ve been using Renovate Bot for over a year now, with great success, but I’ve only tried the GitHub version.

Developer time is expensive, best to let bots do as much as possible!

The setup is pretty straightforward, just press “Install it for free” down at the bottom. If you need more help, I defer to checking their website, as the process may change after this article is written.

Step 2: Configure Renovate

Once Renovate has been installed, you can configure it from the “Settings” page of your account or organization in the “Applications” tab.

You’ll have the choice to enable it for all of your repositories, or select repositories. Seems how I have a lot of repos, I decided to only enable it for certain ones. This choice is up to you.

Shortly after giving Renovate access to your repository you should receive a Pull Request titled “Configure Renovate” which will add a file named renovate.json to your project.

If we take a look inside we can see that the config is pretty simple, but allows for lots of customization.

{
  "extends": [
    "config:base"
  ]
}

Some customizations you may want to make are things like enabling automerge if you have a high enough confidence in your automated test coverage and code quality enforcement. This will allow renovate to automatically update packages for you if all checks pass. It’s also possible to only enable automerge for minor versions, just devDependencies and much more. I’ll leave it to you to look through all of the configuration options and decide what is best for your team and project.

In this initial Pull Request, Renovate will detail which package files it has detected and what Pull Requests with what updates you can expect to see once it’s been merged.

One of the reason’s I switched from using Greenkeeper to Renovate is it’s support for much more than Node.js packages. For our purposes, it will also keep Dockerfiles, docker-compose, and Helm chart files up to date, but also supports much more such as Gradle. The full list can be found in the project’s documentation.

To finish the configuring your repository for Renovate simply merge the Pull Request.

Step 3: Automatically Run Tests For Every Pull Request with Travis CI

In the previous articles we set up a bunch of Code Quality enforcement as well as configured automated tests with 100% code coverage. Before we start merging changes that could potentially break our project, we’ll want to make sure that all of these checks are run with every Pull Request.

There are many options to do this, but the quickest and easiest for open source projects is probably Travis CI.

Enabling Travis CI is just as easy as Renovate, just head on over to the Application in the Github Marketplace, select the free “Open Source” version, and press Install.

Again, you’ll have the option to enable it for all repositories, or select repositories. I’m again choosing to limit it to just my selected repos.

In order for Travis to kick in, we’ll need to add a .travis.yml file to the project with the appropriate settings. The minimum we will need is to set the language of the project. Supported languages can be found in their docs.

language: node_js

Here’s the PR adding the file to the project: https://github.com/patrickleet/streaming-ssr-react-styled-components/pull/10

Travis notices right away and starts running our builds.

By default, Travis CI does not know what version of Node we are supposed to be running, however, and uses the default version of 0.10.48 which is probably about a decade old by now. We will need to make a small update to our config file to tell it what we want instead. At the time of writing this article Node 11 was the latest version, and was chosen.

Let’s go ahead and tell Travis to run the build in Node 11, as well as the latest version 12.

language: node_js
node_js:
  - 11
  - 12

Travis knows to run tests as part of its default Node.js script, and for many libraries or applications, this will be enough. This particular application, however, runs some tests against the built version that is outputted by Parcel, and for that reason, we will need to customize the script to make sure our Parcel build runs before the tests.

Let’s update our .travis.yml file again to account for this build step before the script runs.

language: node_js
node_js:
  - 11
  - 12

before_script:
  - npm run build

And with that, we have passing builds!

Step 4: Enable Code Coverage Reporting

Alright, we have passing builds, and our tests are outputting code coverage, but it’s a bit hard to get at. We want to ensure that we are not losing coverage with a new PR, so it’d be really helpful if we had that information as part of the Pull Request. For this, we can use a tool called CodeCov from the Github Marketplace.

Just as before, select the “Open Source” plan and press Install.

Again, I’ve chosen to only give permission for specific repositories, as I have a lot.

In order to finish setting up CodeCov we can add a couple of steps to our .travis.yml file. We can do this in theafter_successlifecycle step.

Here’s our new .travis.yml file:

language: node_js
node_js:
  - 11
  - 12

before_script:
  - npm run build

after_success:
  - npm i -g codecov
  - codecov

Now with each PR, a bot will comment with changes in coverage. Here’s the PR where CodeCov was enabled.

The question marks are simply because we didn’t have coverage data in the master branch yet. Once this PR is merged, future report will include the difference between the PR and the master branch.

Step 5: Pin Dependencies

While we were setting up Travis and CodeCov, Renovate made some Pull Requests. The first of which was setting all of the dependencies to a fixed version. This eliminates the reliance on SemVer, or semantic versioning. SemVer allows a range of versions and can be useful to get minor and patch updates each time you runnpm install. Seems how we have a bot monitoring our packages for us, we will rely on it to do this work instead of SemVer. This results in more reproducible builds and less possible variance between each developer’s machine. A more detailed discussion of the reasoning behind this can be found in Renovate’s blog.

We’d like to see how each of the PRs affect test coverage and ensure they are still passing, though. Renovate is pretty smart, and can easily rebase the PR’s it has made. Given enough time it will usually do so automatically, but we can help it along by clicking the checkbox in the PR’s description that says “If you want to rebase/retry this PR, check this box”.

Within a few moments a rebase has been performed at the Travis build kicks off, and a couple of minutes later we have passing builds and tests, as well as a code coverage report showing no change in coverage — still at 100%.

Pinning dependencies is a rather safe operation, as all that is happening is the explicit versions that are already installed in the package-lock.json file are being reflected in package.json. We can also see all of the tests are still passing, and code coverage has not been affected.

Step 6: Testing our Dockerfiles

If you’re paying close attention, you’ve noticed that we are testing our Node application, but not our Dockerfiles. In the previous article we created a process to run and test our Dockerfiles but we are not taking advantage of that process in Travis. The problem is that Renovate will make PRs to keep our Dockerfile’s up to date, and if we don’t automatically run them we may miss something that works on Travis but not in the container.

Let’s fix that. To do so we can add a new section to our .travis.yml file called jobs that define the docker build jobs.

language: node_js
node_js:
  - 11
  - 12

before_script:
  - npm run build

after_success:
  - npm i -g codecov
  - codecov

jobs:
  include:
    - services:
        - docker
      script:
        - docker build . -t ssr
    - services:
        - docker
      script:
        - docker build . -f nginx/Dockerfile -t nginx

We have two Dockerfiles, because this project specifically demonstrates two different alternatives — one with SSR, and one with Nginx. Therefore, we have two new jobs defined.

In the Travis UI we can see that we now have four jobs — the original Node 11 and Node 12 builds, as well as two more that run thedocker buildcommands.

Clicking inside of “ 25.3” and “ 25.4” we can see the output for each of the docker build commands indicating their success.

Step 7: Package updates

With the pinned dependencies out of the way, we will start receiving Pull Requests for our other packages that need updating. The updates will either be a minor version update, which are generally pretty safe so long as the package author is using semver appropriately, or major version updates, which should receive a bit more scrutiny. Make sure if you had any open before the Travis and CodeCov changes you trigger a rebase of those branches. We’ll also get updates to our Dockerfiles, and thanks to the last step, we know those are being built as well.

Renovate will only make two Pull Requests per hour, and not open more than twenty total pull requests at a time.

Let’s take a look at the PRs I’ve received so far, and talk about how to handle each of them.

There are five of them so far, and a range of different types of examples. #9 and #11 are both update to Dockerfiles, and they are both passing, including actually running the builds so we know that the updates still result in a working Dockerfile. It’d be nice to run them as well and have a preview environment created automatically for each environment, but we’ll need more tools and a more complicated set up to get that. It’s totally possible, and something I usually set up for my clients, but this article is getting long already.

#12 is a major version update. In version 1 of react-helmet-async there is a breaking change, and our 100% code coverage correctly detected that something went wrong. If automerge were enabled, this would not have made it through, exactly as planned.

#14 is a major version update as well, but our build and tests are working, so we can be pretty confident we are all good here. Again, a preview environment that actually runs this build in the cloud for us would get us to that next level of confidence. We can click the down arrow next to “Release Notes” in the PRs description to take a deeper look.

If an author is diligent they should detail what breaking change warranted the major version bump. Unfortunately in this case, the author did not provide this information. Not knowing what the reason was increases the risk, and decreases developer confidence. If you’re a library author, I strongly suggest using tools like commitizen and semantic-release to make sure users of your package are kept informed. I wrote up a process of how to do this in a previous article: These 6 essential tools will release, version, and maintain your NPM modules for you.

Lastly, #16 is a minor version update, but demonstrates Renovate’s ability to group related packages. Some libraries choose to version all components of the library together. Renovate contains some preset, popular libraries in this manner, such as babel as we are seeing here.

I like living on the wild side, and hate clicking buttons that a bot could click for me, so instead of going through and merging all of these, I’m just gonna go ahead and enable automerge for Renovate. I’m pretty confident that the 100% coverage of the tests do a good job in catching breaking changes as they have demonstrated with PR #12.

To enable automerge, update yourrenovate.jsonfile like so:

{
  "extends": [
    "config:base"
  ],
  "automerge": true
}

Here’s the PR that enables automerge.

However, without preview environments, there’s too much that could potentially go wrong with major versions. Let’s make one more update so major versions will require a manual review and merge:

{
  "extends": [
    "config:base"
  ],
  "automerge": true,
  "major": {
    "automerge": false
  }
}

Automerge will take some time to kick in, so I’ll let it do it’s thing for awhile.

While my bots get to work, let’s take a look at the failing PR and see if we can find out what went wrong there.

Step 8: Investigating failing builds

The failing build is caused by updating the library “react-helmet-async” to from version 0.2.0 to 1.0.0. Unfortunately, when we expand the release notes, the library author did not give us any information about what the breaking change may have been.

Renovate does allow us to click through via the “Compare Source” link to see what commits and code changes have occurred though, and of course, looking at the failing tests may give show us what went wrong.

From clicking Compare Source for V1.0.0 we can see that the breaking change is moving the default export to a named export.

The error message in the tests was pretty unclear, so thankfully looking at the “Compare Source” link helped us find the change pretty quickly.

To fix it, I’m going to go ahead and check out the branch that Renovate made and simply add a new commit to it which changes the import to use the new named import. Here’s the commit containing the fix.

And with that, our tests are now passing again.

Step 9: Testing major version changes

PR #14 is a great example of why we need to test major version changes. If we had a preview environment we’d find that although the tests are all passing, there are run time errors related toreact-imported-componentAPI changes. Seems how we do not, I had to discover this by checking out the branch that renovate created and running it manually.

The library has changed in order to be inline with React Suspense and React.lazy. Looking through the docs shows us how to refactor to use the new API. If you’re interested, the diff can be seen here.

Conclusion

Developer time is expensive, and maintenance is important as package updates often contain security fixes. Staying up to date will make sure any known bugs in the packages you are using get fixed as the author makes updates. Due to the time required to keep everything up to date, these small updates often fall by the wayside and over the months and years your projects can get painfully out of date. By employing a set of bots to do the heavy lifting for you, it’s much easier to keep everything up to date.

That’s all for now! Happy coding!

Check out the other articles in this series! This was part 6.

• Part 1: Move over Next.js and Webpack
• Part 2: A Better Way to Develop Node.js with Docker
• Part 3: Enforcing Code Quality for Node.js
• Part 4: The 100% Code Coverage Myth
• Part 5: A Tale of Two (Docker Multi-Stage Build) Layers

Interested in going deeper, and getting things like Preview Environments on every Pull Request? In my new masterclass, Cloud Native Entrepreneur, you’ll do just that by engineering end-to-end marketing systems with microservice backends and running them in production with Kubernetes! All while learning marketing and entrepreneurship along the way! Check my Hackernoon profile to find how to register for a free info session!

Best,
Patrick Lee Scott

Originally published at https://hackernoon.com on January 23, 2020.

Hey everyone!

Thanks for all the support I’ve gotten on Medium over the years!

1600 followers, 100’s of thousands of views, literally over a year of time spent reading my articles about DevOps, Microservices, Full Stack Engineering, and Entrepreneurship.

And this year, voted HackerNoon’s Contributing Writer of the Year!?

Amazing.

Thank you.

I’ve really enjoyed making content, and helping engineers like you succeed.

Now, I’m taking that to the next level.

I want you to not only be a Cloud Native Engineer, but I also want to help you become a Cloud Native Entrepreneur.

Which is why I’m excited to announce I’m launching my first masterclass, Cloud Native Entrepreneur, where you will learn marketing and business systems through engineering them!

But first, you know I’m one for telling a good story.

Check out my story of becoming a Cloud Native Entrepreneur here.

Best,
Patrick Lee Scott

If you’re starting a new business, launching a new idea, or doing anything at all really, you’re going to be getting a lot of feedback. People are going to tell you if they love it. And you’re inevitably going to hear some verison of:

“That’s not a good idea.”

The fact is, you’re probably going to hear that one a lot, and from a lot of different people. Some of them perfectly well-meaning.

So, how do we determine which feedback matters the most? When we’re overwhelmed with messages about our ideas being too risky or not good enough, how do we filter out the noise and get to the feedback that really matters?

How do we know whether to listen to the people saying yes, or the people saying no?

I’ve dealt with this problem a lot. Over the course of my entrepreneurial journey, I’ve had more than a few ideas.

Some of them were winners.

Many, not so much.

Some I never really got the chance to find out if they would be successful until it was too late.

Some I left behind because I listened to the wrong feedback.

This is the story of an idea that I wish I’d done more with, and how it taught me which feedback to listen to, and which to ignore.

Looking back at this, I realize what I should have done differently (and that I potentially could have made a lot of money!) had I only known then what I’m telling you now.

One weekend, my buddy Krish managed to get us a spot in a lean startup machine workshop. Normally, this would have cost us a few thousand dollars. Somehow though, he managed to get us in for free. For three days, we’d be attending this workshop and working on our own ideas for a startup.

I was pretty excited for this opportunity. I’d recently read The Lean Startup, and was interested in getting more hands on with it.

Inside the workshop, we divided up into groups of three to five people. Each group was tasked with coming up with a potential startup idea. Once we had one, it was time to hit the streets.

Having the idea wasn’t enough. We needed proof that our idea had value, and we needed to get that by getting people to sign onto it in one form or another.

We needed to hear from our potential customers.

Now, a lot of the people in my group lived in the upper east side. This workshop? It was in downtown Manhattan.

This was before the days of Uber. We had two options for getting there. Either we cough up the money for a very expensive cab ride, or we use the subway for a much cheaper, but also much slower, commute.

This was the problem that we tackled with our startup idea.

We called our solution “CabPool.” A very clever twist on the phrase “carpool.” People would use an app to find others going to the same place as them and share a cab, splitting the cost.

If you’re drawing parallels between what we were thinking about and rideshare apps? Trust me, I’ve been doing that for a while, too.

When it came time to test our idea, we planted ourselves in front of a Whole Foods market. The plan was simple. We’d wait to see someone hailing a cab and ask them if they’d be interested in a program like ours. We even had a little application.

When someone said they were interested, we’d ask them where they were going so we could find a match for them.

For the purposes of this test, I was the person waiting for the ride, and trust me, I just so happened to be going the same direction. They’d be matched up with me, I’d hop in the cab, and off we’d go. We made small talk along the way, and then, at the end of the ride, we split the cost.

The people I talked to seemed to genuinely enjoy this idea. Not surprising, right? Who doesn’t want cheaper transportation?

Whether that was the idea itself or just my winning personality, you be the judge.

As we did this, we got people to give us money. So, when it came time for the judges to determine which group had the best idea? We were pretty sure that we were going to win.

We did not, in fact, win.

The judges told us that, despite the fact that we’d made some money, our plan wasn’t a good idea. It simply wasn’t going to work.

The winner of the competition had proposed an app that would let people know when a parking spot opened up and where it was. I’m sure it’s a complete surprise that a lot of people in NYC were thinking about traffic, commuting, and parking.

Their group hadn’t gotten anyone to give them money, like ours had. Instead, they had gotten people to sign a letter of intent to use the app. The judges decided this was the better approach.

Now, I don’t want to harp on their idea, but I do feel like it is worth pointing out that a parking spot in NYC is probably only open for about 60 seconds on a good day — according to the team whom presented their findings… By the time you check an app, find an open spot, and drive to it? That spot isn’t empty anymore.

That’s probably the reason we’re not seeing this kind of app around much.

Nevertheless, we took the advice of the judges.

After all, they were experts on these things, right?

This had only been a weekend idea for us, not a serious venture. And the judges had told us it hadn’t been good enough. Why continue on with it?

Obviously, given the success of rideshare apps, we probably should have ignored them.

The judges, the experts on the issue? They were the feedback we chose to listen to. They weren’t the feedback we should have been listening to, though.

The customers that we had over the course of the trial had been giving us feedback too. They’d been doing it with their wallets, and they’d been doing it in a positive way.

That was the feedback we should have been listening to.

Our customers voted with their wallets, and we decided to listen to the people who had no financial interest in the matter.

When you launch a new idea, that idea is always going to have some detractors. Those detractors might even be the experts. They might tell you that your idea isn’t a good one. That there isn’t a market for what you’re thinking of doing.

That data? It really isn’t all that useful.

The useful data comes from your customers, and how they choose to vote with their wallets. We had people voting yes, putting their dollars down on our idea. We didn’t listen to them. We listened to the experts instead.

That was our mistake. Don’t let it be yours.

When you’re starting something new, the main feedback you need to listen to is the one that comes from the customers. Are they willing to spend money on what you’re proposing? That answer matters more than any other, and is going to really tell you if your idea can work or not.

When people vote with their wallets, listen to them.

Do you have an idea or product that you want to turn into a full-fledged business? Click here to learn more about me and my agency, Unbounded, and how I can put my experience to work for you.

Originally published at https://hackernoon.com on July 25, 2019.

A few days ago I was peeling an egg, like an idiot. My girlfriend, Angelly, was like — “why are you peeling the egg like that?”

She picked up another hard-boiled egg, smacked it on the cutting board, and then rolled it while pressing down firmly. The egg’s shell was off in about 3 seconds total.

Here I was chipping away tiny piece by tiny piece.

My point is: Working hard is overrated.

The best solutions are the simple, effective solutions.

Is it hard to take rust off of a screw? Probably, right? Unless you know that coca-cola removes rust from screws.

Now, is it hard? No. All you need to do is drop it in a cup and wait.

If you do not know the simple techniques, you cannot use them. You get experimentation instead of implementation. Exploration instead of replication.

When a technique in software engineering is used over and over again because it makes the problem you are solving simpler, it becomes a pattern, or “best practice”.

Sometimes, they have complex, scary-sounding names — like Command Query Responsibility Segregation (CQRS), or Event Sourcing (ES) — but these patterns exist to solve problems. Specifically, problems found in building sane, distributed systems.

When we look at the coding context as a whole there are more universal patterns: “Keep it Simple, Stupid” (KISS), and “Don’t Repeat Yourself” (DRY), for example.

I want to talk about these same sorts of patterns and principles as applied to DevOps. DevOps is often represented as the promised land where the birds are singing and the sun is shining. Without the right techniques, though, it will be a pointed spiky hell, like my tiny eggshell shards.

On top of the universal patterns like KISS and DRY, there are other “principles” that I’ve found building DevOps Systems that aren’t quite patterns, yet. These principles can help you move from picking away eggshells piece by piece to having them off in no time flat. They are, in no specific order:

1. Don’t repeat others.
2. Empower developers to be productive as possible.
3. There is no production.
4. Put everything in the cluster and backup the whole thing.
5. VPNs are not simple, and there are simple solutions to solve the same things.
6. Codify and Automate Everything.

Lesson 1 — Don’t repeat others

If you can buy it off the shelf or there is an easy to use open source version of a tool you need, use it.

Don’t reinvent the wheel. Buy the wheel.

Did you know you can use the same mail server that craigslist uses? Did you know you can use it for free? Need a mail server? Use that! Don’t build it, build around that!

I love awesome-self-hosted for finding things, as well as searching for it for use with Helm on their hub.

Although Helm is a pretty new tool for finding and sharing software, a v3 website is now up: https://v3.helm.sh/

Helm is really great for getting wheels.

Let me take you back to when I started programming. The first “real” language I learned was QBasic in 9th grade. I’d been making websites with HTML and CSS for a few years already. Back then, the internet was new. While people could create packages, they weren’t really sharing them around like we are nowadays. My standard mechanism for storage was a floppy disk — which I wish I could find so I could extract my 11th-grade brick break game I made in Java Swing…

When I programmed, I used standard libraries, baby. That was all there really was. That I knew of at least. I was like 13. I’m sure some of you more experienced Java Pros (Hi Vlad and Nick!) were killin’ it. ;)

Not today. Hell, today, you can get entire functioning UI components all bundled up nicely and ready to go. You can get a library for dealing elegantly and easily with dates with a single command.

Wouldn’t it be great if there were something that allowed you to use and install entire functioning subsystems? Need a database? install database

Good news! There is! You can now also get entire functioning subsystems all bundled up and ready to go, using Helm.

It’s the same concept we’ve embraced via NPM and Gradle, but the packages it manages are known as Charts.

A chart maps containers to run in different ways inside of Kubernetes.

Using a chart is kinda like buying a wheel. Except the wheels are containers mapped to run inside of Kubernetes.

The cool part about Charts is that you can easily package up your own services to run easily in any Kubernetes cluster, and even share it with others if you’d like.

This means that you could describe entire environments in a codified manner:

dependencies:
- name: backup
  repository: http://jenkins-x-chartmuseum:8080
  version: 0.0.2
- name: monitor
  repository: http://jenkins-x-chartmuseum:8080
  version: 0.0.3
- name: marketing-site
  repository: http://jenkins-x-chartmuseum:8080
  version: 1.1.10
- name: denormalizer-service
  repository: http://jenkins-x-chartmuseum:8080
  version: 1.0.0
- name: mongo
  repository: https://kubernetes-charts.storage.googleapis.com/
  version: 1.0.0

Need to update the marketing site to 1.2.0? Just change it, and commit. Which brings us to Lesson Two.

Lesson 2 — Empower developers to be as productive as possible

So there I was, sitting at my desk, face in my code, tracking down a bug. Users had been complaining about this for weeks and I found some free time and decided to be a hero.

Badda bing, badda boom! Found! Fixed!

I shouted over my half-height cubicle in the sun-lit room — “I fixed the bug!”

Next Tuesday, when the release went out, users were gonna love it! That is, after all of us got together in a room and pushed the boulder of the release up and over the top of another small mountain… err, if we were able to push it over, that was… if nothing goes wrong along the way…

Okay, next Tuesday, if the release went out — users were gonna love it!

This was my experience deploying when I was a junior software engineer at my first programming job out of college.

Things have come a long way since then.

Now, I practice trunk based development and deploy many times a day. When I make pull requests, a little bot posts a comment with an ephemeral pull request preview environment after a successful build and passing tests.

In today’s environment, you don’t need to shout over the half-height cubicle.

The more you can empower engineers to be able to control the parts of the infrastructure that they need, the simpler your job becomes as a DevOps engineer becomes.

In lesson one, we saw it was possible to update a production environment by literally changing a number and committing.

In the ideal world, where we are also packaging all of our applications into Charts, we are also empowering each engineer on the team to modify how that service runs in production. A chart is literally a mapping of containers to Kubernetes, and for that purpose, it exposes things like resource requirements.

Seems how I’m not great at guessing how much memory or CPU settings a new service will need — and I’m guessing you aren’t either — I also like to deploy monitoring and alerting with some rules that will inform me and my team via Slack when those settings need adjusting. This means it will guide you to correct settings once you deploy it. I used to spend hours at a time running Prometheus queries and adjusting, just like I used to spend too much time peeling my eggs. Now, I’ve learned to do it the smart way.

I can already hear you saying: “That sounds complicated.” Nope. Just install the chart.

Anything that can be automated or simplified should be. For example, what if simply deploying a service with a label that said “expose: true” would map it to a DNS path? This is where operators come in. This is a more advanced Kubernetes helper tool, and something worth understanding, but let’s not get lost in those details right now.

And that brings us to Lesson Three.

Lesson 3 — There is no production

This was one of those epiphany moments for me. It took me looking at things from a new perspective, so try to follow me for a minute.

For over a decade, I’d thought there were only a handful of environments. At it’s simplest, there was a Staging Environment and a Production environment. First, you deployed to staging, tested things, and then deployed the next thing, tested it, etc. Once everything was deployed together — “integrated” — then we could continue the process and repeat the process to production.

This is how I did things for years and thought nothing of it. It’s the way it was.

This is another one of those peeling apart eggshells piece by piece type of moments.

A chart is a dependency graph. Use the chart to represent the environment, and then the process of deploying to production is just deploying a single chart!

If each team or project or bounded context had its own chart, you could have many production environments that allowed you to group and update sets of services together in a single transaction.

So instead of thinking of production as this big, all-encompassing environment, realize there are actually many little productions.

Big changes are scary and dangerous. By keeping many small production environments you isolate changes and make the rest of the system more resilient.

Also, because everything is just in Charts all the way down, every setting is exposed and tweakable in a uniform manner. Meaning, things like running a less powerful Kafka cluster in staging where it’s not needed is just a simple config change.

Lesson 4 — Put everything in the cluster and back up the whole thing

Ok, so we can run things easily in production, but what about databases? What about Kafka? Is that safe?

Well, if you’ve been paying attention, you’ll remember I mentioned databases as a thing that can be packaged into a chart.

In Kubernetes, there is now an API specifically for running stateful applications like databases in highly available setups, called StatefulSets.

Pretty much the entire point of Kubernetes is to run containers reliably.

Paired with a tool called Velero that backs up an entire Kubernetes cluster, installed via Helm Chart, you can now backup the entire state of a Kubernetes cluster, as well as all of the attached volumes, like the ones created by the StatefulSet, and restore everything with a single command. Backups are also easy to configure on a schedule.

With backups and restores just a single command away, coupled with managed Kubernetes, spinning up an entirely new cluster and restoring it to your backed up version can now be done in just two commands. Three if you want a brand new backup first.

Instead of thinking of just servers as cattle, entire clusters start to become disposable.

Is staging being annoying? Throw it away. Just a simple backup, restore, and DNS change away.

Lesson 5 — VPNs are not simple, and there are simple solutions to solve the same things

Have you ever enjoyed using a VPN?

I mean, honestly.

Has anyone?

Google tried. Then, a couple of years back, announced that their company would no longer use them. I guess they didn’t enjoy using them, either.

Instead, they rely on trustless networks.

Instead of a master key that lets you access all the networks, a key is placed in front of each service via a Single Sign On screen. Need to access the monitoring service? Just log in with your authorized company credentials.

A small shift is all that’s required. Instead of using a VPN for authentication, use a proxy.

A VPN also places the Kubernetes management URL on a private network where an SSO gateway does not. However, they also provide an alternative authentication mechanism. For Google Cloud and AWS that is IAM authentication, and you can also whitelist IP addresses.

If you can maintain a lot less infrastructure with the same benefits, do it. Be like Google: ditch VPNs for trustless networks.

Lesson 6 — Codify and Automate Everything

Something takes a bit more effort to codify? I don’t care! It will save hours, days, or even weeks of effort later on. Codify it.

It’s the only reliably repeatable way to recreate infrastructure.

If every setting can be modified by changing and committing something to git, then your technology organization as a whole is essentially declarative. Any dev with git access can easily maintain any system.

To do so, use lots of little repositories, then tie them together later. Monorepos lead to people taking shortcuts and a dependency on an artificially important structure. Instead, use lots of little repositories, and tie them together later.

My friend Matt and I make a tool for doing this in development, it’s called meta. Helm is a tool for doing this in production: everything is just a dependency graph!

Conclusion

Don’t peel an egg piece by piece. Smack and roll.

Interested in having me automate your DevOps pains away? I’ve got the smack and roll on lock.

Check out my website to learn more about me, and my agency, Unbounded to schedule a free 15 minute Discovery call. I’m also available as an advisor on AdvisoryCloud.

Just Getting Started with DevOps? Check out My Journey to Acheiving DevOps Bliss, Without Useless AWS Certifications on HackerNoon and find a free 21 day email course at the bottom!

What Does Peeling An Egg Have To Do With DevOps? was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.