[{"content":"Over the past few weeks I’ve been developing and using a personal command-line tool called gwt (Git Worktree) to manage Git repositories using worktrees. This article explains what the tool does, how it evolved, and how I used GitHub Copilot CLI to develop it (in fact the idea of building the script was also to test the tool).\nThe Problem: Managing Multiple BranchesI was working on a project with multiple active branches, including orphans; the regular branches are for fixes or features, while the orphans are used to keep copies of remote documents or store processed versions of those documents.\nThe project also uses a special orphan branch that contains the scripts and the CI/CD configuration to store and process the external documents (it is on a separate branch to avoid mixing its operation with the main project code).\nThe plan is trigger a pipeline against the special branch from remote projects to create or update the doc branch for it in our git repository, retrieving artifacts from the remote projects to get the files and put them on an orphan branch (initially I added new commits after each update, but I changed the system to use force pushes and keep only one commit, as the history is not really needed).\nThe original documents have to be changed, so, after ingesting them, we run a script that modifies them and adds or updates another branch with the processed version; the contents of that branch are used by the main branch build process (there we use git fetch and git archive to retrieve its contents).\nWhen working on the scripts to manage the orphan branches I discovered the worktree feature of git, a functionality that allows me to keep multiple branches checked out in parallel using a single .git folder, removing the need to use git switch and git stash when changing between branches (until now I’ve been a heavy user of those commands).\nReading about it I found that a lot of people use worktrees with the help of a wrapper script to simplify the management. After looking at one or two posts and the related scripts I decided to create my own using a specific directory structure to simplify things.\nThat’s how I started to work on the gwt script; as I also wanted to test copilot I decided to build it using its help (I have a pro license at work and wanted to play with the cli version instead of integrated into an editor, as I didn’t want to learn a lot of new keyboard shortcuts).\nThe gwt Philosophy: Opinionated and Transparentgwt enforces a simple, filesystem-visible model:\nExactly one bare repository named bare.git (treated as an implementation detail)One worktree directory per branch where the directory name matches the branch nameSingle responsibility: gwt doesn’t try to be a general git wrapper; it only handles operations that map cleanly to this layout The repository structure looks like this:\nmy-repo/ +-- bare.git/ # the Git repository (internal) +-- main/ # worktree for branch \u0026#34;main\u0026#34; +-- feature/api/ # worktree for branch \u0026#34;feature/api\u0026#34; +-- fix/docs/ # worktree for branch \u0026#34;fix/docs\u0026#34; +-- orphan-history/ # worktree for the \u0026#34;orphan-history\u0026#34; branch The tool follows five core design principles:\nExplicit over clever: Git commands are not hidden or reinterpretedTransparent execution: Every operation is printed before it happensSafe, preview-first operations: Destructive commands default to preview, confirmation, then applyShell-agnostic core: The script never changes the caller’s working directory (shell wrappers handle that)Opinionated but minimal: Only commands that fit the layout model are included Core CommandsThe script provides these essential commands:\ngwt init \u0026lt;url\u0026gt; — Clone a repository and set up the gwt layoutgwt convert \u0026lt;dir\u0026gt; — Convert an existing Git checkout to the gwt layoutgwt add [--orphan] \u0026lt;branch\u0026gt; [\u0026lt;base\u0026gt;] — Create a new worktree (optionally orphaned)gwt remove \u0026lt;branch\u0026gt; — Remove a worktree and unregister it (asks the user to remove the local branch too, useful when removing already merged branches)gwt rename \u0026lt;old\u0026gt; \u0026lt;new\u0026gt; — Rename a branch AND its worktree directorygwt list — List all worktreesgwt default [\u0026lt;branch\u0026gt;] — Get or set the default branchgwt current — Print the current worktree or branch name Except init and convert all of the commands work inside a directory structure that follows the gwt layout, which looks for the bare.git folder to find the root folder of the structure.\nAs I don’t want to hide which commands are really used by the wrapper, all git and filesystem operations pass through a single run shell function that prints each command before executing it. This gives complete visibility into what the tool is doing.\nAlso, destructive operations (remove, rename) default to preview mode:\n$ gwt remove feature-old --dry-run + git -C bare.git branch -d feature-old + git -C bare.git worktree remove feature-old/ Apply these changes? [y/N]: The user sees exactly what will happen, can verify it’s correct, and only then confirm execution.\nIncremental Development with CopilotThe gwt script has grown from 597 lines in its original version (git-wt) to 1,111 lines when writing the first draft of this post.\nThis growth happened through incremental, test-driven development, with each feature being refined based on real usage patterns.\nWhat follows is a little history of the script evolution written with the help of git log.\nInitial versionFirst I wrote a design document and asked copilot to create the initial version of the git-wt script with the original core commands.\nI started to use the tool with a remote repostory (I made copies of the branches in some cases to avoid missing work) and fixed bugs (trivial ones with neovim, larger ones asking copilot to fix the issues for me, so I had less typing to do).\nNote: As I used copilot I noticed that when you make manual changes it is important to tell the tool about them, otherwise it gets confused and sometimes tries to remove manual changes.\nFirst command updateOne of the first commands I had to enhance was rename:\nas I normally use branches with / on their name and my tool checks out the worktrees using the branch name as the path inside the gwt root folder (i.e. a fix/rename branch creates the fix directory and checks the branch inside the fix/rename folder) the rename command had to clean up the empty parent directorieswhen renaming a worktree we move the folders and fix the references using the worktree repair command to make things work locally, but the rename also affects the remote branch reference, to avoid surprises the command unsets the remote branch reference so it can be pushed again using the new name (of course, the user is responsible of managing the old remote branch, as the gwt can’t guess what it should do with it). Integration with the shellAs I use zsh with the Powerlevel10k theme I asked copilot to help me add visual elements to the prompt when working with gwt folders, something that I would have never tried without help, as it would have required a lot of digging on my part on how to do it, as I never looked into it.\nThe initial version of the code was on an independent file that I sourced from my .zshrc file and it prints on the right part of the prompt when we are inside a gwt folder (note that if the folder is a worktree we see the existing git integration text right before it, so we have the previous behavior and we see that it is a gwt friendly repo) and if we are on the root folder or the bare.git folder we see gwt or bare (I added the text because there are no git promts on those folders).\nI also asked copilot to create zsh autocompletion functions (I only use zsh, so I didn’t add autocompletion for other shells). The good thing here is that I wouldn’t have done that manually, as it would have required some reading to get it right, but the output of copilot worked and I can update things using it or manually if I need to.\nOne thing I was missing from the script was the possibility of changing the working directory easily, so I wrote a gwt wrapper function for zsh that intercepts commands that require shell cooperation (changing the working directory) and delegates everything else to the core script.\nCurrently the function supports the following enhanced commands:\ncd [\u0026lt;branch\u0026gt;]: change into a worktree or the default one if missingconvert \u0026lt;dir\u0026gt;: convert a checkout, then cd into the initial worktreeadd [--orphan] \u0026lt;branch\u0026gt; [\u0026lt;base\u0026gt;]: create a worktree, then cd into it on successrename \u0026lt;old\u0026gt; \u0026lt;new\u0026gt;: rename a worktree, then cd into it if we were inside it Note that the cd command will not work on other shells or if the user does not load my wrapper, but the rest will still work without the working directory changes.\nRenaming the commandAs I felt that git-wt was a long name I renamed the tool to gwt, I could have done it by hand, but using copilot I didn’t have to review all files by myself and it did it right (note that I have it configured to always ask me before doing changes, as it sometimes tries to do something I don’t want and I like to check its changes …​ as I have the files in git repos, I manually add the files when I like the status and if the cli output is not clear I allow it to apply it and check the effects with git diff so I can validate or revert what was done).\nThe convert commandAfter playing with one repo I added the convert subcommand for migrating existing checkouts, it seemed a simple task at first, but it took multiple iterations to get it right, as I found multiple issues while testing (in fact I did copies of the existing checkouts to be able to re-test each update, as some of the iterations broke them).\nThe version of the function when this post was first edited had the following comment explaining what it does:\n# --------------------------------------------------------------------------- # convert - convert an existing checkout into the gwt layout # --------------------------------------------------------------------------- # # Must be run from the parent directory of \u0026lt;dir\u0026gt;. # # Steps: # 1. Read branch from the checkout\u0026#39;s HEAD # 2. Rename \u0026lt;dir\u0026gt; to \u0026lt;dir\u0026gt;.wt.tmp (sibling, same filesystem) # 3. Create \u0026lt;dir\u0026gt;/ as the new gwt root # 4. Move \u0026lt;dir\u0026gt;.wt.tmp/.git to \u0026lt;dir\u0026gt;/bare.git; set core.bare = true # 5. Fix fetch refspec (bare clone default maps refs directly, no remotes/) # 6. Add a --no-checkout worktree so git wires up the metadata and # creates \u0026lt;dir\u0026gt;/\u0026lt;branch\u0026gt;/.git (the only file in that dir) # 7. Move that .git file into the real working tree (\u0026lt;dir\u0026gt;.wt.tmp) # 8. Remove the now-empty placeholder directory # 9. Move the real working tree into place as \u0026lt;dir\u0026gt;/\u0026lt;branch\u0026gt; # 10. Reset the index to HEAD so git status is clean # (--no-checkout leaves the index empty) # 11. Create \u0026lt;dir\u0026gt;/.git -\u0026gt; bare.git symlink so plain git commands work # from the root without --git-dir # # The .git file ends up at the same absolute path git recorded in step 5, # so no worktree repair is needed. Working tree files are never modified. The .git link was added when I noticed that I could run commands that don’t need the checked out files on the root of the gwt structure, which is handy sometimes (i.e. a git fetch or a git log, that shows the log of the branch marked as default).\nAfter playing with commands that used the bare.git folder I updated the init and convert commands to keep the origin refs, ensuring that the remote tracking works correctly.\nImproving the add commandWhile playing with the tool on more repos I noticed that I also had to enhance the add command to better handle worktree creation, depending on my needs.\nRight now the tool supports the following use cases:\nif the branch exists locally or on origin, it just checks it out.if the branch does not exist, we create it using the given base branch or, if no base is given, the current worktree (if we are in the root folder or bare.git the command fails).as I needed it for my project, I added a --orphan option to be able to create orphan branches directly. Moving to a single fileEventually I decided to make the tool self contained; I removed the design document (I moved the content to comments on the top of the script and details to comments on each function definition) and added a pair of commands to print the code to source for the p10k and zsh integration (autocompletion \u0026amp; functions), leaving everything in a single file.\nNow my .zshrc file adds the following to source both things:\n# After loading the p10k configuration if type gwt \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then source \u0026lt;(gwt p10k) fi [...] # After loading autocompletion if type gwt \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then source \u0026lt;(gwt zsh) fi VersioningAs I modified the script I found interesting to use CalVer-based versioning (the version variable has the format YYYY.mm.dd-r#) so I added a subcommand to show its value or bump it using the current date and computing the right revision number.\nAbout the use of copilotAlthough I’ve never been a fan of AI tools I have to admit that the copilot CLI has been very useful for building the tool:\nRapid prototyping: Each commit represented a small feature or fix that I could implement, test immediately in my actual workflow, and iterate on based on the resultEdge case handling: Rather than trying to anticipate every scenario upfront, I could ask Copilot how to handle edge cases as they appeared in real usageScript refinement: Questions like \u0026#34;how do I clean up empty directories after a rename\u0026#34; or \u0026#34;how do I detect if I’m inside a specific worktree\u0026#34; were quickly answered with working codeShell integration: The Zsh wrapper and completion system grew from simple prototypes to sophisticated features, with each iteration informed by how I actually used the tool For example, the convert command started as a simple rename operation, but evolved to also create a .git symlink and intelligently handle various migration scenarios—all because I used it repeatedly and refined the implementation each time.\nSelf-Contained and Opinionatedgwt is deliberately opinionated:\nZsh \u0026amp; Powerlevel10k Integration: The tool includes built-in Zsh shell integration, accessed via source \u0026lt;(gwt zsh) and supports adding a prompt segment when using p10k, as described earlier.Directory Structure: The bare.git directory name is non-negotiable. This is how gwt discovers the repository root from any subdirectory, and how the tool knows whether a directory is a gwt repository. The simplicity of this marker means the discovery mechanism is foolproof and requires no configuration.No Configuration Files: gwt deliberately has no configuration. There are no .gwtrc files or config directories. This makes it portable; the tool works the same way everywhere, and repositories can be shared across systems without synchronizing configuration. From Script to SystemWhat started as a small helper script for managing worktrees has become a complete system:\nCore script (gwt): 1,111 lines of pure shell, no external dependenciesShell integration: Zsh functions and completionsPrompt integration: Powerlevel10k segmentDocumentation: Built-in help and design philosophy documentation The script is self-contained, everything needed for the tool to work is in a single file.\nThis makes it trivial to update (just replace the script) or audit (no hidden dependencies).\nDevelopment with AI supportDeveloping gwt with copilot taught me some things:\nIncremental refinement works well for small tools: Each iteration informed the next, resulting in a tool that handles real use cases elegantlyTransparency is a feature: Making operations visible builds confidence and is easier to debugOpinionated tools can be powerful: By constraining the problem space (one bare repo, one worktree per branch), the solution becomes simpler and more robustShell integration matters: The same core commands are easier to use when they can automatically change directories and provide completionsReal-world testing is essential: I wouldn’t have discovered the need for automatic directory cleanup or context-aware cd behavior without actually using the tool daily What was next?The tool is stable and handles my daily workflow well, so my guess is that I would keep using it and fixing issues if or when I found them, but I do not plan to include additional features unless I find a use case that justifies it (i.e. I never added support for some of the worktree subcommands, as it is easier to use the git versions if I ever needed them).\nWhat really happenedWhile editing this post I discovered that I needed to add another command to it and fixed a bug (see below).\nWith those changes and the inclusion of a license and copyright notice (just in case I distribute it at some point) now the script is 1,217 lines long instead of the 1,111 it had when I started to write this entry.\nSubmodule SupportWhen I converted this blog repository to the gwt format and tried to preview the post using docker compose, it failed because the worktree I was on didn’t have the Git submodule initialized.\nMy blog theme is included on the repository as a submodule, and when I used gwt to check out different branches in worktrees, the submodule was not initialized in the new worktrees.\nThis led me to add new internal function and a gwt submodule command to handle submodule initialization; the internal function is called from convert and add (when converting a repo or adding a worktree) and the public command is useful to update the submodules on existing branches.\nPath Handling with Branch Names Containing SlashesThe second discovery was a bug in how the tool handled branch names containing slashes (e.g., feature/new-api, docs/user-guide), the worktree directories are created with the branch name as the path, so a branch like feature/new-api would create two nested folders (feature and new-api inside it).\nHowever, there was a mismatch in how the zsh wrapper function resolved worktree paths (initially it used shell parameter expansion, i.e. rel=\u0026#34;${cwd#\u0026#34;$REPO_ROOT\u0026#34;/}\u0026#34;), versus how the core script calculated them, causing the cd command to fail or navigate to the wrong location when branch names contained slashes.\nThe fix involved ensuring consistent path resolution throughout the script and wrapper (now it uses a function that processes the git worktree list output), so that gwt cd feature/new-api correctly navigates to the worktree directory regardless of path depth.\nConclusiongwt is a tool that solves a real problem: managing multiple Git branches simultaneously without context-switching overhead.\nI’m sure I’m going to keep using it for my projects, as it simplifies some workflows, although I’ll also use switch and stash in some cases, but I like the use of multiple worktrees in parallel.\nIn fact I converted this blog repository checkout to the gwt format to work on a separate branch as it felt the right approach even if I’m the only one using the repo now, and it helped me improve the tool, as explained before.\nAlso, it was a good example of how to use AI tools like copilot to develop a simple tool and keep it evolving while using it.\nIn any case, although I find the copilot useful and has saved me time, I don’t trust it to work without supervision, it worked well, but got stuck some times and didn’t do the things as I wanted in multiple occasions.\nI also have an additional problem now …​ I’ve been reading about it, but I don’t really know which models to use or how the premium requests are computed (I’ve only been playing with it since last month and I ran out of requests the last day of the month on purpose, just to see what happened …​ it stops working …​ ;).\nOn my work machine I’ve been using a specific user account with a GitHub Copilot Business subscription and I only used the Anthropic Claude Sonnet 4.6 model and with my personal account I configured the Anthropic Claude Haiku 4.5 model, but I’ve only used that to create the initial draft of this post (I ended up rewriting most of it manually anyway) and to review the final version (I’m not a native speaker and it was useful for finding typos and improving the style in some parts).\nI guess I’ll try other models with copilot in the future and check other command line tools like aider or claude-code, but probably only using free accounts unless I get a payed account at work, as I have with GitHub Copilot.\nTo be fair, what I will love to be able to do is to use local models (aider can do it), but the machines I have are not powerful enough. I tried to run a simple test and it felt really slow, but when I have the time or the need I’ll try again, just in case.\n","permalink":"https://blogops.mixinet.net/posts/developing_a_git_worktree_helper_with_copilot/","summary":"\u003csection id=\"preamble\" aria-label=\"Preamble\"\u003e\u003cp\u003eOver the past few weeks I’ve been developing and using a personal command-line\ntool called \u003ccode\u003egwt\u003c/code\u003e (\u003cem\u003eGit Worktree\u003c/em\u003e) to manage Git repositories using worktrees.\nThis article explains what the tool does, how it evolved, and how I used\n\u003ca href=\"https://github.com/features/copilot/cli/\" target=\"_blank\" rel=\"noopener\"\u003eGitHub Copilot CLI\u003c/a\u003e to develop it (in\nfact the idea of building the script was also to test the tool).\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_the_problem_managing_multiple_branches\"\u003eThe Problem: Managing Multiple Branches\u003c/h2\u003e\u003cp\u003eI was working on a project with multiple active branches, including orphans; the\nregular branches are for fixes or features, while the orphans are used to keep\ncopies of remote documents or store processed versions of those documents.\u003c/p\u003e\n\u003cp\u003eThe project also uses a special orphan branch that contains the scripts and the\nCI/CD configuration to store and process the external documents (it is on a\nseparate branch to avoid mixing its operation with the main project code).\u003c/p\u003e\n\u003cp\u003eThe plan is trigger a pipeline against the special branch from remote projects\nto create or update the doc branch for it in our git repository, retrieving\nartifacts from the remote projects to get the files and put them on an orphan\nbranch (initially I added new commits after each update, but I changed the\nsystem to use force pushes and keep only one commit, as the history is not\nreally needed).\u003c/p\u003e\n\u003cp\u003eThe original documents have to be changed, so, after ingesting them, we run a\nscript that modifies them and adds or updates another branch with the processed\nversion; the contents of that branch are used by the \u003ccode\u003emain\u003c/code\u003e branch build process\n(there we use \u003ccode\u003egit fetch\u003c/code\u003e and \u003ccode\u003egit archive\u003c/code\u003e to retrieve its contents).\u003c/p\u003e\n\u003cp\u003eWhen working on the scripts to manage the orphan branches I discovered the\n\u003ca href=\"https://git-scm.com/docs/git-worktree\" target=\"_blank\" rel=\"noopener\"\u003eworktree\u003c/a\u003e feature of \u003ccode\u003egit\u003c/code\u003e, a\nfunctionality that allows me to keep multiple branches checked out in parallel\nusing a single \u003ccode\u003e.git\u003c/code\u003e folder, removing the need to use \u003ccode\u003egit switch\u003c/code\u003e and \u003ccode\u003egit\nstash\u003c/code\u003e when changing between branches (until now I’ve been a heavy user of those\ncommands).\u003c/p\u003e\n\u003cp\u003eReading about it I found that a lot of people use worktrees with the help of a\nwrapper script to simplify the management. After looking at one or two posts\nand the related scripts I decided to create my own using a specific directory\nstructure to simplify things.\u003c/p\u003e\n\u003cp\u003eThat’s how I started to work on the \u003ccode\u003egwt\u003c/code\u003e script; as I also wanted to test\n\u003ccode\u003ecopilot\u003c/code\u003e I decided to build it using its help (I have a pro license at work and\nwanted to play with the cli version instead of integrated into an editor, as I\ndidn’t want to learn a lot of new keyboard shortcuts).\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_the_gwt_philosophy_opinionated_and_transparent\"\u003eThe gwt Philosophy: Opinionated and Transparent\u003c/h2\u003e\u003cp\u003e\u003ccode\u003egwt\u003c/code\u003e enforces a simple, filesystem-visible model:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003e\u003cstrong\u003eExactly one bare repository\u003c/strong\u003e named \u003ccode\u003ebare.git\u003c/code\u003e (treated as an implementation\ndetail)\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eOne worktree directory per branch\u003c/strong\u003e where the directory name matches the\nbranch name\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eSingle responsibility\u003c/strong\u003e: \u003ccode\u003egwt\u003c/code\u003e doesn’t try to be a general \u003ccode\u003egit\u003c/code\u003e wrapper; it\nonly handles operations that map cleanly to this layout\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Developing a Git Worktree Helper with Copilot"},{"content":"When I configured forgejo-actions I used a docker-compose.yaml file to execute the runner and a dind container configured to run using privileged mode to be able to build images with it; as mentioned on my post about my setup, the use of the privileged mode is not a big issue for my use case, but reduces the overall security of the installation.\nOn a work chat the other day someone mentioned that the GitLab documentation about using kaniko says it is no longer maintained (see the kaniko issue #3348) so we should look into alternatives for kubernetes clusters.\nI never liked kaniko too much, but it works without privileged mode and does not need a daemon, which is a good reason to use it, but if it is deprecated it makes sense to look into alternatives, and today I looked into some of them to use with my forgejo-actions setup.\nI was going to try buildah and podman but it seems that they need to adjust things on the systems running them:\nWhen I tried to use buildah inside a docker container in Ubuntu I found the problems described on the buildah issue #1901 so I moved on.Reading the podman documentation I saw that I need to export the fuse device to run it inside a container and, as I found other option, I also skipped it. As my runner was already configured to use dind I decided to look into sysbox as a way of removing the privileged flag to make things more secure but have the same functionality.\nInstalling the sysbox packageAs I use Debian and Ubuntu systems I used the .deb packages distributed from the sysbox release page to install it (in my case I used the one from the 0.6.7 version).\nOn the machine running forgejo (a Debian 12 server) I downloaded the package, stopped the running containers (it is needed to install the package and the only ones running where the ones started by the docker-compose.yaml file) and installed the sysbox-ce_0.6.7.linux_amd64.deb package using dpkg.\nUpdating the docker-compose.yaml fileTo run the dind container without setting the privileged mode we set sysbox-runc as the runtime on the dind container definition and set the privileged flag to false (it is the same as removing the key, as it defaults to false):\n--- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,7 +2,9 @@ services: dind: image: docker:dind container_name: \u0026#39;dind\u0026#39; - privileged: \u0026#39;true\u0026#39; + # use sysbox-runc instead of using privileged mode + runtime: \u0026#39;sysbox-runc\u0026#39; + privileged: \u0026#39;false\u0026#39; command: [\u0026#39;dockerd\u0026#39;, \u0026#39;-H\u0026#39;, \u0026#39;unix:///dind/docker.sock\u0026#39;, \u0026#39;-G\u0026#39;, \u0026#39;$RUNNER_GID\u0026#39;] restart: \u0026#39;unless-stopped\u0026#39; volumes: Testing the changesAfter applying the changes to the docker-compose.yaml file we start the containers and to test things we re-run previously executed jobs to see if things work as before.\nIn my case I re-executed the build-image-from-tag workflow #18 from the oci project and everything worked as expected.\nConclusionFor my current use case (docker + dind) seems that sysbox is a good solution but I’m not sure if I’ll be installing it on kubernetes anytime soon unless I find a valid reason to do it (last time we talked about it my co workers said that they are evaluating buildah and podman for kubernetes and probably we will use them to replace kaniko in our gitlab-ci pipelines and for those tools the use of sysbox seems an overkill).\n","permalink":"https://blogops.mixinet.net/posts/forgejo/running_dind_with_sysbox/","summary":"\u003cp\u003eWhen I configured \u003ccode\u003eforgejo-actions\u003c/code\u003e I used a \u003ccode\u003edocker-compose.yaml\u003c/code\u003e file to execute the \u003ccode\u003erunner\u003c/code\u003e and a \u003ccode\u003edind\u003c/code\u003e container\nconfigured to run using privileged mode to be able to build images with it; as mentioned on my\n\u003ca href=\"https://blogops.mixinet.net/posts/forgejo/configuring_forgejo_actions/#_starting_it_with_docker_compose\" target=\"_blank\" rel=\"noopener\"\u003epost about my\nsetup\u003c/a\u003e, the use of the privileged mode is not a big issue for my use case, but reduces the overall security of the\ninstallation.\u003c/p\u003e\n\u003cp\u003eOn a work chat the other day someone mentioned that the \u003cstrong\u003eGitLab\u003c/strong\u003e documentation about\n\u003ca href=\"https://docs.gitlab.com/ci/docker/using_kaniko/\" target=\"_blank\" rel=\"noopener\"\u003eusing kaniko\u003c/a\u003e says it is no longer maintained (see the \u003ccode\u003ekaniko\u003c/code\u003e issue\n\u003ca href=\"https://github.com/GoogleContainerTools/kaniko/issues/3348\"\u003e#3348\u003c/a\u003e) so we should look into alternatives for kubernetes\nclusters.\u003c/p\u003e\n\u003cp\u003eI never liked \u003ccode\u003ekaniko\u003c/code\u003e too much, but it works without privileged mode and does not need a daemon, which is a good reason\nto use it, but if it is deprecated it makes sense to look into alternatives, and today I looked into some of them to use\nwith my \u003ccode\u003eforgejo-actions\u003c/code\u003e setup.\u003c/p\u003e\n\u003cp\u003eI was going to try \u003ca href=\"https://buildah.io/\" target=\"_blank\" rel=\"noopener\"\u003ebuildah\u003c/a\u003e and \u003ca href=\"https://podman.io/\" target=\"_blank\" rel=\"noopener\"\u003epodman\u003c/a\u003e but it seems that they need to adjust\nthings on the systems running them:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003eWhen I tried to use \u003ccode\u003ebuildah\u003c/code\u003e inside a \u003ccode\u003edocker\u003c/code\u003e container in \u003cstrong\u003eUbuntu\u003c/strong\u003e I found the problems described on the \u003ccode\u003ebuildah\u003c/code\u003e\nissue \u003ca href=\"https://github.com/containers/buildah/issues/1901\" target=\"_blank\" rel=\"noopener\"\u003e#1901\u003c/a\u003e so I moved on.\u003c/li\u003e\u003cli\u003eReading the \u003ccode\u003epodman\u003c/code\u003e documentation I saw that I need to export the \u003ccode\u003efuse\u003c/code\u003e device to run it inside a container and, as\nI found other option, I also skipped it.\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Running dind with sysbox"},{"content":"After my previous posts related to Argo CD (one about argocd-autopilot and another with some usage examples) I started to look into Kluctl (I also plan to review Flux, but I’m more interested on the kluctl approach right now).\nWhile reading an entry on the project blog about Cluster API somehow I ended up on the vCluster site and decided to give it a try, as it can be a valid way of providing developers with on demand clusters for debugging or run CI/CD tests before deploying things on common clusters or even to have multiple debugging virtual clusters on a local machine with only one of them running at any given time.\nOn this post I will deploy a vcluster using the k3d_argocd kubernetes cluster (the one we created on the posts about argocd) as the host and will show how to:\nuse its ingress (in our case traefik) to access the API of the virtual one (removes the need of having to use the vcluster connect command to access it with kubectl),publish the ingress objects deployed on the virtual cluster on the host ingress, anduse the sealed-secrets of the host cluster to manage the virtual cluster secrets. Note: On this post we will use the k3d cluster used to work with argocd, but it is about playing with vcluster, so I’ll be deploying things using kubectl directly.\nIn any case, if wanted things could be done using the GitOps approach once the vcluster is available.\nCreating the virtual clusterInstalling the vcluster applicationTo create the virtual clusters we need the vcluster command, we can install it with arkade:\n❯ arkade get vcluster The vcluster.yaml fileTo create the cluster we are going to use the following vcluster.yaml file (you can find the documentation about all its options here):\ncontrolPlane: proxy: # Extra hostnames to sign the vCluster proxy certificate for extraSANs: - my-vcluster-api.lo.mixinet.net exportKubeConfig: context: my-vcluster_k3d-argocd server: https://my-vcluster-api.lo.mixinet.net:8443 secret: name: my-vcluster-kubeconfig sync: toHost: ingresses: enabled: true serviceAccounts: enabled: true fromHost: ingressClasses: enabled: true nodes: enabled: true clearImageStatus: true secrets: enabled: true mappings: byName: # sync all Secrets from the \u0026#39;my-vcluster-default\u0026#39; namespace to the # virtual \u0026#34;default\u0026#34; namespace. \u0026#34;my-vcluster-default/*\u0026#34;: \u0026#34;default/*\u0026#34; # We could add other namespace mappings if needed, i.e.: # \u0026#34;my-vcluster-kube-system/*\u0026#34;: \u0026#34;kube-system/*\u0026#34; On the controlPlane section we’ve added the proxy.extraSANs entry to add an extra host name to make sure it is added to the cluster certificates if we use it from an ingress.\nNote: It is possible to create the ingress under the controlPlane key as described in the documentation, but the generated object uses annotations for the nginx-ingress controller to use the TLS certificates of the original service that don’t work for us (we are using traefik as our ingress controller).\nTo solve the issue we will create an IngressRouteTCP object manually after deploying the virtual cluster.\nThe exportKubeConfig section creates a kubeconfig secret on the virtual cluster namespace using the provided host name; the secret can be used by GitOps tools or we can dump it to a file to connect from our machine.\nOn the sync section we enable the synchronization of Ingress objects and ServiceAccounts from the virtual to the host cluster:\nWe copy the ingress definitions to use the ingress server that runs on the host to make them work from the outside world.The service account synchronization is not really needed, but we enable it because if we test this configuration with EKS it would be useful if we use IAM roles for the service accounts. On the opposite direction (from the host to the virtual cluster) we synchronize:\nThe IngressClass objects, to be able to use the host ingress server(s).The Nodes (we are not using the info right now, but it could be interesting if we want to have the real information of the nodes running pods of the virtual cluster).The Secrets from the my-vcluster-default host namespace to the default of the virtual cluster; that synchronization allows us to deploy SealedSecrets on the host that generate secrets that are copied automatically to the virtual one. Initially we only copy secrets for one namespace but if the virtual cluster needs others we can add namespaces on the host and their mappings to the virtual one on the vcluster.yaml file. Creating the virtual clusterTo create the virtual cluster we run the following command:\nvcluster create my-vcluster --namespace my-vcluster --upgrade --connect=false \\ --values vcluster.yaml It creates the virtual cluster on the my-vcluster namespace using the vcluster.yaml file shown before without connecting to the cluster from our local machine (if we don’t pass that option the command adds an entry on our kubeconfig and launches a proxy to connect to the virtual cluster that we don’t plan to use).\nAdding an ingress TCP route to connect to the vcluster apiAs explained before, we need to create an IngressTcpRoute object to be able to connect to the vcluster API, we use the following definition:\napiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteTCP metadata: name: my-vcluster-api namespace: my-vcluster spec: entryPoints: - websecure routes: - match: HostSNI(`my-vcluster-api.lo.mixinet.net`) services: - name: my-vcluster port: 443 tls: passthrough: true Once we apply those changes the cluster API will be available on the https://my-cluster-api.lo.mixinet.net:8443 URL using its own self signed certificate (we have enabled TLS passthrough) that includes the hostname we use (we adjusted it on the vcluster.yaml file, as explained before).\nGetting the kubeconfig for the vclusterOnce the vcluster is running we will have its kubeconfig available on the my-vcluster-kubeconfig secret on its namespace on the host cluster.\nTo dump it to the ~/.kube/my-vcluster-config we can do the following:\n❯ kubectl get -n my-vcluster secret/my-vcluster-kubeconfig \\ --template=\u0026#34;{{.data.config}}\u0026#34; | base64 -d \u0026gt; ~/.kube/my-vcluster-config Once available we can define the vkubectl alias to adjust the KUBECONFIG variable to access it:\nalias vkubectl=\u0026#34;KUBECONFIG=~/.kube/my-vcluster-config kubectl\u0026#34; Or we can merge the configuration with the one on the KUBECONFIG variable and use kubectx or a similar tool to change the context (for our vcluster the context will be my-vcluster_k3d-argocd). If the KUBECONFIG variable is defined and only has the PATH to a single file the merge can be done running the following:\nKUBECONFIG=\u0026#34;$KUBECONFIG:~/.kube/my-vcluster-config\u0026#34; kubectl config view \\ --flatten \u0026gt;\u0026#34;$KUBECONFIG.new\u0026#34; mv \u0026#34;$KUBECONFIG.new\u0026#34; \u0026#34;$KUBECONFIG\u0026#34; On the rest of this post we will use the vkubectl alias when connecting to the virtual cluster, i.e. to check that it works we can run the cluster-info subcommand:\n❯ vkubectl cluster-info Kubernetes control plane is running at https://my-vcluster-api.lo.mixinet.net:8443 CoreDNS is running at https://my-vcluster-api.lo.mixinet.net:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use \u0026#39;kubectl cluster-info dump\u0026#39;. Installing the dummyhttpd applicationTo test the virtual cluster we are going to install the dummyhttpd application using the following kustomization.yaml file:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0 # Add the config map configMapGenerator: - name: dummyhttp-configmap literals: - CM_VAR=\u0026#34;Vcluster Test Value\u0026#34; behavior: create options: disableNameSuffixHash: true patches: # Change the ingress host name - target: kind: Ingress name: dummyhttp patch: |- - op: replace path: /spec/rules/0/host value: vcluster-dummyhttp.lo.mixinet.net # Add reloader annotations -- it will only work if we install reloader on the # virtual cluster, as the one on the host cluster doesn\u0026#39;t see the vcluster # deployment objects - target: kind: Deployment name: dummyhttp patch: |- - op: add path: /metadata/annotations value: reloader.stakater.com/auto: \u0026#34;true\u0026#34; reloader.stakater.com/rollout-strategy: \u0026#34;restart\u0026#34; It is quite similar to the one we used on the Argo CD examples but uses a different DNS entry; to deploy it we run kustomize and vkubectl:\n❯ kustomize build . | vkubectl apply -f - configmap/dummyhttp-configmap created service/dummyhttp created deployment.apps/dummyhttp created ingress.networking.k8s.io/dummyhttp created We can check that everything worked using curl:\n❯ curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ | jq -cM . {\u0026#34;c\u0026#34;: \u0026#34;Vcluster Test Value\u0026#34;,\u0026#34;s\u0026#34;: \u0026#34;\u0026#34;} The objects available on the vcluster now are:\n❯ vkubectl get all,configmap,ingress NAME READY STATUS RESTARTS AGE pod/dummyhttp-55569589bc-9zl7t 1/1 Running 0 24s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dummyhttp ClusterIP 10.43.51.39 \u0026lt;none\u0026gt; 80/TCP 24s service/kubernetes ClusterIP 10.43.153.12 \u0026lt;none\u0026gt; 443/TCP 14m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/dummyhttp 1/1 1 1 24s NAME DESIRED CURRENT READY AGE replicaset.apps/dummyhttp-55569589bc 1 1 1 24s NAME DATA AGE configmap/dummyhttp-configmap 1 24s configmap/kube-root-ca.crt 1 14m NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/dummyhttp traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 24s While we have the following ones on the my-vcluster namespace of the host cluster:\n❯ kubectl get all,configmap,ingress -n my-vcluster NAME READY STATUS RESTARTS AGE pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 18m pod/dummyhttp-55569589bc-9zl7t-x-default-x-my-vcluster 1/1 Running 0 45s pod/my-vcluster-0 1/1 Running 0 19m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dummyhttp-x-default-x-my-vcluster ClusterIP 10.43.51.39 \u0026lt;none\u0026gt; 80/TCP 45s service/kube-dns-x-kube-system-x-my-vcluster ClusterIP 10.43.91.198 \u0026lt;none\u0026gt; 53/UDP,53/TCP,9153/TCP 18m service/my-vcluster ClusterIP 10.43.153.12 \u0026lt;none\u0026gt; 443/TCP,10250/TCP 19m service/my-vcluster-headless ClusterIP None \u0026lt;none\u0026gt; 443/TCP 19m service/my-vcluster-node-k3d-argocd-agent-1 ClusterIP 10.43.189.188 \u0026lt;none\u0026gt; 10250/TCP 18m NAME READY AGE statefulset.apps/my-vcluster 1/1 19m NAME DATA AGE configmap/coredns-x-kube-system-x-my-vcluster 2 18m configmap/dummyhttp-configmap-x-default-x-my-vcluster 1 45s configmap/kube-root-ca.crt 1 19m configmap/kube-root-ca.crt-x-default-x-my-vcluster 1 11m configmap/kube-root-ca.crt-x-kube-system-x-my-vcluster 1 18m configmap/vc-coredns-my-vcluster 1 19m NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 45s As shown, we have copies of the Service, Pod, Configmap and Ingress objects, but there is no copy of the Deployment or ReplicaSet.\nCreating a sealed secret for dummyhttpdTo use the hosts sealed secrets controller with the virtual cluster we will create the my-vcluster-default namespace and add there the sealed secrets we want to have available as secrets on the default namespace of the virtual cluster:\n❯ kubectl create namespace my-vcluster-default ❯ echo -n \u0026#34;Vcluster Boo\u0026#34; | kubectl create secret generic \u0026#34;dummyhttp-secret\u0026#34; \\ --namespace \u0026#34;my-vcluster-default\u0026#34; --dry-run=client \\ --from-file=SECRET_VAR=/dev/stdin -o yaml \u0026gt;dummyhttp-secret.yaml ❯ kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml ❯ kubectl apply -f dummyhttp-sealed-secret.yaml ❯ rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml After running the previous commands we have the following objects available on the host cluster:\n❯ kubectl get sealedsecrets.bitnami.com,secrets -n my-vcluster-default NAME STATUS SYNCED AGE sealedsecret.bitnami.com/dummyhttp-secret True 34s NAME TYPE DATA AGE secret/dummyhttp-secret Opaque 1 34s And we can see that the secret is also available on the virtual cluster with the content we expected:\n❯ vkubectl get secrets NAME TYPE DATA AGE dummyhttp-secret Opaque 1 34s ❯ vkubectl get secret/dummyhttp-secret --template=\u0026#34;{{.data.SECRET_VAR}}\u0026#34; \\ | base64 -d Vcluster Boo But the output of the curl command has not changed because, although we have the reloader controller deployed on the host cluster, it does not see the Deployment object of the virtual one and the pods are not touched:\n❯ curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ | jq -cM . {\u0026#34;c\u0026#34;: \u0026#34;Vcluster Test Value\u0026#34;,\u0026#34;s\u0026#34;: \u0026#34;\u0026#34;} Installing the reloader applicationTo make reloader work on the virtual cluster we just need to install it as we did on the host using the following kustomization.yaml file:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: kube-system resources: - github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2 patches: # Add flags to reload workloads when ConfigMaps or Secrets are created or deleted - target: kind: Deployment name: reloader-reloader patch: |- - op: add path: /spec/template/spec/containers/0/args value: - \u0026#39;--reload-on-create=true\u0026#39; - \u0026#39;--reload-on-delete=true\u0026#39; - \u0026#39;--reload-strategy=annotations\u0026#39; We deploy it with kustomize and vkubectl:\n❯ kustomize build . | vkubectl apply -f - serviceaccount/reloader-reloader created clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created deployment.apps/reloader-reloader created As the controller was not available when the secret was created the pods linked to the Deployment are not updated, but we can force things removing the secret on the host system; after we do that the secret is re-created from the sealed version and copied to the virtual cluster where the reloader controller updates the pod and the curl command shows the new output:\n❯ kubectl delete -n my-vcluster-default secrets dummyhttp-secret secret \u0026#34;dummyhttp-secret\u0026#34; deleted ❯ sleep 2 ❯ vkubectl get pods NAME READY STATUS RESTARTS AGE dummyhttp-78bf5fb885-fmsvs 1/1 Terminating 0 6m33s dummyhttp-c68684bbf-nx8f9 1/1 Running 0 6s ❯ curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ | jq -cM . {\u0026#34;c\u0026#34;:\u0026#34;Vcluster Test Value\u0026#34;,\u0026#34;s\u0026#34;:\u0026#34;Vcluster Boo\u0026#34;} If we change the secret on the host systems things get updated pretty quickly now:\n❯ echo -n \u0026#34;New secret\u0026#34; | kubectl create secret generic \u0026#34;dummyhttp-secret\u0026#34; \\ --namespace \u0026#34;my-vcluster-default\u0026#34; --dry-run=client \\ --from-file=SECRET_VAR=/dev/stdin -o yaml \u0026gt;dummyhttp-secret.yaml ❯ kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml ❯ kubectl apply -f dummyhttp-sealed-secret.yaml ❯ rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml ❯ curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ | jq -cM . {\u0026#34;c\u0026#34;:\u0026#34;Vcluster Test Value\u0026#34;,\u0026#34;s\u0026#34;:\u0026#34;New secret\u0026#34;} Pause and restore the vclusterThe status of pods and statefulsets while the virtual cluster is active can be seen using kubectl:\n❯ kubectl get pods,statefulsets -n my-vcluster NAME READY STATUS RESTARTS AGE pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 127m pod/dummyhttp-587c7855d7-pt9b8-x-default-x-my-vcluster 1/1 Running 0 4m39s pod/my-vcluster-0 1/1 Running 0 128m pod/reloader-reloader-7f56c54d75-544gd-x-kube-system-x-my-vcluster 1/1 Running 0 60m NAME READY AGE statefulset.apps/my-vcluster 1/1 128m Pausing the vclusterIf we don’t need to use the virtual cluster we can pause it and after a small amount of time all Pods are gone because the statefulSet is scaled down to 0 (note that other resources like volumes are not removed, but all the objects that have to be scheduled and consume CPU cycles are not running, which can translate in a lot of savings when running on clusters from cloud platforms or, in a local cluster like the one we are using, frees resources like CPU and memory that now can be used for other things):\n❯ vcluster pause my-vcluster 11:20:47 info Scale down statefulSet my-vcluster/my-vcluster... 11:20:48 done Successfully paused vcluster my-vcluster/my-vcluster ❯ kubectl get pods,statefulsets -n my-vcluster NAME READY AGE statefulset.apps/my-vcluster 0/0 130m Now the curl command fails:\n❯ curl -s https://vcluster-dummyhttp.localhost.mixinet.net:8443 404 page not found Although the ingress is still available (it returns a 404 because there is no pod behind the service):\n❯ kubectl get ingress -n my-vcluster NAME CLASS HOSTS ADDRESS PORTS AGE dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 120m In fact, the same problem happens when we try to connect to the vcluster API; the error shown by kubectl is related to the TLS certificate because the 404 page uses the wildcard certificate instead of the self signed one:\n❯ vkubectl get pods Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority ❯ curl -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/ 404 page not found ❯ curl -v -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/ 2\u0026gt;\u0026amp;1 | grep subject * subject: CN=lo.mixinet.net * subjectAltName: host \u0026#34;my-vcluster-api.lo.mixinet.net\u0026#34; matched cert\u0026#39;s \u0026#34;*.lo.mixinet.net\u0026#34; Resuming the vclusterWhen we want to use the virtual cluster again we just need to use the resume command:\n❯ vcluster resume my-vcluster 12:03:14 done Successfully resumed vcluster my-vcluster in namespace my-vcluster Once all the pods are running the virtual cluster goes back to its previous state, although all of them were started, of course.\nCleaning upThe virtual cluster can be removed using the delete command:\n❯ vcluster delete my-vcluster 12:09:18 info Delete vcluster my-vcluster... 12:09:18 done Successfully deleted virtual cluster my-vcluster in namespace my-vcluster 12:09:18 done Successfully deleted virtual cluster namespace my-vcluster 12:09:18 info Waiting for virtual cluster to be deleted... 12:09:50 done Virtual Cluster is deleted That removes everything we used on this post except the sealed secrets and secrets that we put on the my-vcluster-default namespace because it was created by us.\nIf we delete the namespace all the secrets and sealed secrets on it are also removed:\n❯ kubectl delete namespace my-vcluster-default namespace \u0026#34;my-vcluster-default\u0026#34; deleted ConclusionsI believe that the use of virtual clusters can be a good option for two of the proposed use cases that I’ve encountered in real projects in the past:\nneed of short lived clusters for developers or teams,execution of integration tests from CI pipelines that require a complete cluster (the tests can be run on virtual clusters that are created on demand or paused and resumed when needed). For both cases things can be set up using the Apache licensed product, although maybe evaluating the vCluster Platform offering could be interesting.\nIn any case when everything is not done inside kubernetes we will also have to check how to manage the external services (i.e. if we use databases or message buses as SaaS instead of deploying them inside our clusters we need to have a way of creating, deleting or pause and resume those services).\n","permalink":"https://blogops.mixinet.net/posts/k8s/playing_with_vcluster/","summary":"\u003cp\u003eAfter my previous posts related to \u003ca href=\"https://argo-cd.readthedocs.io/\" target=\"_blank\" rel=\"noopener\"\u003eArgo CD\u003c/a\u003e (one about\n\u003ca href=\"https://blogops.mixinet.net/posts/gitops/argocd-autopilot/\" target=\"_blank\" rel=\"noopener\"\u003eargocd-autopilot\u003c/a\u003e and another with some\n\u003ca href=\"https://blogops.mixinet.net/posts/gitops/argocd_usage_examples/\" target=\"_blank\" rel=\"noopener\"\u003eusage examples\u003c/a\u003e) I started to look into\n\u003ca href=\"https://kluctl.io/\" target=\"_blank\" rel=\"noopener\"\u003eKluctl\u003c/a\u003e (I also plan to review \u003ca href=\"https://fluxcd.io/\" target=\"_blank\" rel=\"noopener\"\u003eFlux\u003c/a\u003e, but I’m more interested on the \u003ccode\u003ekluctl\u003c/code\u003e\napproach right now).\u003c/p\u003e\n\u003cp\u003eWhile reading an entry on the project blog about \u003ca href=\"https://kluctl.io/blog/2024/03/13/cluster-api-kluctl/\" target=\"_blank\" rel=\"noopener\"\u003eCluster API\u003c/a\u003e\nsomehow I ended up on the \u003ca href=\"https://www.vcluster.com/\" target=\"_blank\" rel=\"noopener\"\u003evCluster\u003c/a\u003e site and decided to give it a try, as it can be a valid\nway of providing developers with on demand clusters for debugging or run CI/CD tests before deploying things on common\nclusters or even to have multiple debugging virtual clusters on a local machine with only one of them running at any\ngiven time.\u003c/p\u003e\n\u003cp\u003eOn this post I will deploy a \u003ccode\u003evcluster\u003c/code\u003e using the \u003ccode\u003ek3d_argocd\u003c/code\u003e kubernetes cluster (the one we created on the posts about\n\u003ccode\u003eargocd\u003c/code\u003e) as the host and will show how to:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003euse its ingress (in our case \u003ccode\u003etraefik\u003c/code\u003e) to access the API of the virtual one (removes the need of having to use the\n\u003ccode\u003evcluster connect\u003c/code\u003e command to access it with \u003ccode\u003ekubectl\u003c/code\u003e),\u003c/li\u003e\u003cli\u003epublish the ingress objects deployed on the virtual cluster on the host ingress, and\u003c/li\u003e\u003cli\u003euse the \u003ccode\u003esealed-secrets\u003c/code\u003e of the host cluster to manage the virtual cluster secrets.\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Playing with vCluster"},{"content":"As a followup of my post about the use of argocd-autopilot I’m going to deploy various applications to the cluster using Argo CD from the same repository we used on the previous post.\nFor our examples we are going to test a solution to the problem we had when we updated a ConfigMap used by the argocd-server (the resource was updated but the application Pod was not because there was no change on the argocd-server deployment); our original fix was to kill the pod manually, but the manual operation is something we want to avoid.\nThe proposed solution to this kind of issues on the helm documentation is to add annotations to the Deployments with values that are a hash of the ConfigMaps or Secrets used by them, this way if a file is updated the annotation is also updated and when the Deployment changes are applied a roll out of the pods is triggered.\nOn this post we will install a couple of controllers and an application to show how we can handle Secrets with argocd and solve the issue with updates on ConfigMaps and Secrets, to do it we will execute the following tasks:\nDeploy the Reloader controller to our cluster. It is a tool that watches changes in ConfigMaps and Secrets and does rolling upgrades on the Pods that use them from Deployment, StatefulSet, DaemonSet or DeploymentConfig objects when they are updated (by default we have to add some annotations to the objects to make things work).Deploy a simple application that can use ConfigMaps and Secrets and test that the Reloader controller does its job when we add or update a ConfigMap.Install the Sealed Secrets controller to manage secrets inside our cluster, use it to add a secret to our sample application and see that the application is reloaded automatically. Creating the test project for argocd-autopilotAs we did our installation using argocd-autopilot we will use its structure to manage the applications.\nNote: For all examples we are going to assume that the GIT_REPO and GIT_TOKEN variables are defined:\nexport GIT_REPO=\u0026#34;https://forgejo.mixinet.net/blogops/argocd.git\u0026#34;; export GIT_TOKEN=\u0026#34;$(pass mixinet.net/argocd@forgejo/repository-write)\u0026#34;; And the PATH includes the arkade installation directory:\ncase \u0026#34;:${PATH}:\u0026#34; in *:\u0026#34;${HOME}/.arkade/bin\u0026#34;:*) ;; *) export PATH=\u0026#34;${PATH}:${HOME}/.arkade/bin\u0026#34; ;; esac The first thing to do is to create a project (we will name it test) as follows:\n❯ argocd-autopilot project create test INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git Enumerating objects: 18, done. Counting objects: 100% (18/18), done. Compressing objects: 100% (16/16), done. Total 18 (delta 1), reused 0 (delta 0), pack-reused 0 INFO using revision: \u0026#34;\u0026#34;, installation path: \u0026#34;/\u0026#34; INFO pushing new project manifest to repo INFO project created: \u0026#39;test\u0026#39; Now that the test project is available we will use it on our argocd-autopilot invocations when creating applications.\nInstalling the reloader controllerTo add the reloader application to the test project as a kustomize application and deploy it on the tools namespace with argocd-autopilot we do the following:\n❯ argocd-autopilot app create reloader \\ --app \u0026#39;github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2\u0026#39; \\ --project test --type kustomize --dest-namespace tools INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git Enumerating objects: 19, done. Counting objects: 100% (19/19), done. Compressing objects: 100% (18/18), done. Total 19 (delta 2), reused 0 (delta 0), pack-reused 0 INFO using revision: \u0026#34;\u0026#34;, installation path: \u0026#34;/\u0026#34; INFO created \u0026#39;application namespace\u0026#39; file at \u0026#39;/bootstrap/cluster-resources/in-cluster/tools-ns.yaml\u0026#39; INFO committing changes to gitops repo... INFO installed application: reloader That command creates four files on the argocd repository:\nOne to create the tools namespace:\nbootstrap/cluster-resources/in-cluster/tools-ns.yaml apiVersion: v1 kind: Namespace metadata: annotations: argocd.argoproj.io/sync-options: Prune=false creationTimestamp: null name: tools spec: {} status: {}Another to include the reloader base application from the upstream repository:\napps/reloader/base/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2The kustomization.yaml file for the test project (by default it includes the same configuration used on the base definition, but we could make other changes if needed):\napps/reloader/overlays/test/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: tools resources: - ../../baseThe config.json file used to define the application on argocd for the test project (it points to the folder that includes the previous kustomization.yaml file):\napps/reloader/overlays/test/config.json { \u0026#34;appName\u0026#34;: \u0026#34;reloader\u0026#34;, \u0026#34;userGivenName\u0026#34;: \u0026#34;reloader\u0026#34;, \u0026#34;destNamespace\u0026#34;: \u0026#34;tools\u0026#34;, \u0026#34;destServer\u0026#34;: \u0026#34;https://kubernetes.default.svc\u0026#34;, \u0026#34;srcPath\u0026#34;: \u0026#34;apps/reloader/overlays/test\u0026#34;, \u0026#34;srcRepoURL\u0026#34;: \u0026#34;https://forgejo.mixinet.net/blogops/argocd.git\u0026#34;, \u0026#34;srcTargetRevision\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;labels\u0026#34;: null, \u0026#34;annotations\u0026#34;: null } Note: The config.json file is used to build the Argo CD Application from the ApplicationSet defined for the project (the definition for the test project can be reviewed here), the idea is that we use the config.json or the config_dir.json file to adjust the application parameters to generate the CRD, which is a lot shorter and easier that providing the YAML version directly.\nWe can check that the application is working using the argocd command line application:\n❯ argocd app get argocd/test-reloader -o tree Name: argocd/test-reloader Project: test Server: https://kubernetes.default.svc Namespace: tools URL: https://argocd.lo.mixinet.net:8443/applications/test-reloader Source: - Repo: https://forgejo.mixinet.net/blogops/argocd.git Target: Path: apps/reloader/overlays/test SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: Synced to (2893b56) Health Status: Healthy KIND/NAME STATUS HEALTH MESSAGE ClusterRole/reloader-reloader-role Synced ClusterRoleBinding/reloader-reloader-role-binding Synced ServiceAccount/reloader-reloader Synced serviceaccount/reloader-reloader created Deployment/reloader-reloader Synced Healthy deployment.apps/reloader-reloader created └─ReplicaSet/reloader-reloader-5b6dcc7b6f Healthy └─Pod/reloader-reloader-5b6dcc7b6f-vwjcx Healthy Adding flags to the reloader serverThe runtime configuration flags for the reloader server are described on the project README.md file, in our case we want to adjust three values:\nWe want to enable the option to reload a workload when a ConfigMap or Secret is created,We want to enable the option to reload a workload when a ConfigMap or Secret is deleted,We want to use the annotations strategy for reloads, as it is the recommended mode of operation when using argocd. To pass them we edit the apps/reloader/overlays/test/kustomization.yaml file to patch the pod container template, the text added is the following:\npatches: # Add flags to reload workloads when ConfigMaps or Secrets are created or deleted - target: kind: Deployment name: reloader-reloader patch: |- - op: add path: /spec/template/spec/containers/0/args value: - \u0026#39;--reload-on-create=true\u0026#39; - \u0026#39;--reload-on-delete=true\u0026#39; - \u0026#39;--reload-strategy=annotations\u0026#39; After committing and pushing the updated file the system launches the application with the new options.\nThe dummyhttp applicationTo do a quick test we are going to deploy the dummyhttp web server using an image generated using the following Dockerfile:\n# Image to run the dummyhttp application \u0026lt;https://github.com/svenstaro/dummyhttp\u0026gt; # This arg could be passed by the container build command (used with mirrors) ARG OCI_REGISTRY_PREFIX # Latest tested version of alpine FROM ${OCI_REGISTRY_PREFIX}alpine:3.21.3 # Tool versions ARG DUMMYHTTP_VERS=1.1.1 # Download binary RUN ARCH=\u0026#34;$(apk --print-arch)\u0026#34; \u0026amp;\u0026amp; \\ VERS=\u0026#34;$DUMMYHTTP_VERS\u0026#34; \u0026amp;\u0026amp; \\ URL=\u0026#34;https://github.com/svenstaro/dummyhttp/releases/download/v$VERS/dummyhttp-$VERS-$ARCH-unknown-linux-musl\u0026#34; \u0026amp;\u0026amp; \\ wget \u0026#34;$URL\u0026#34; -O \u0026#34;/tmp/dummyhttp\u0026#34; \u0026amp;\u0026amp; \\ install /tmp/dummyhttp /usr/local/bin \u0026amp;\u0026amp; \\ rm -f /tmp/dummyhttp # Set the entrypoint to /usr/local/bin/dummyhttp ENTRYPOINT [ \u0026#34;/usr/local/bin/dummyhttp\u0026#34; ] The kustomize base application is available on a monorepo that contains the following files:\nA Deployment definition that uses the previous image but uses /bin/sh -c as its entrypoint (command in the k8s Pod terminology) and passes as its argument a string that runs the eval command to be able to expand environment variables passed to the pod (the definition includes two optional variables, one taken from a ConfigMap and another one from a Secret):\napiVersion: apps/v1 kind: Deployment metadata: name: dummyhttp labels: app: dummyhttp spec: selector: matchLabels: app: dummyhttp template: metadata: labels: app: dummyhttp spec: containers: - name: dummyhttp image: forgejo.mixinet.net/oci/dummyhttp:1.0.0 command: [ \u0026#34;/bin/sh\u0026#34;, \u0026#34;-c\u0026#34; ] args: - \u0026#39;eval dummyhttp -b \\\u0026#34;{\\\\\\\u0026#34;c\\\\\\\u0026#34;: \\\\\\\u0026#34;$CM_VAR\\\\\\\u0026#34;, \\\\\\\u0026#34;s\\\\\\\u0026#34;: \\\\\\\u0026#34;$SECRET_VAR\\\\\\\u0026#34;}\\\u0026#34;\u0026#39; ports: - containerPort: 8080 env: - name: CM_VAR valueFrom: configMapKeyRef: name: dummyhttp-configmap key: CM_VAR optional: true - name: SECRET_VAR valueFrom: secretKeyRef: name: dummyhttp-secret key: SECRET_VAR optional: trueA Service that publishes the previous Deployment (the only relevant thing to mention is that the web server uses the port 8080 by default):\napiVersion: v1 kind: Service metadata: name: dummyhttp spec: selector: app: dummyhttp ports: - name: http port: 80 targetPort: 8080An Ingress definition to allow access to the application from the outside:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: dummyhttp annotations: traefik.ingress.kubernetes.io/router.tls: \u0026#34;true\u0026#34; spec: rules: - host: dummyhttp.localhost.mixinet.net http: paths: - path: / pathType: Prefix backend: service: name: dummyhttp port: number: 80And the kustomization.yaml file that includes the previous files:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml - ingress.yaml Deploying the dummyhttp application from argocdWe could create the dummyhttp application using the argocd-autopilot command as we’ve done on the reloader case, but we are going to do it manually to show how simple it is.\nFirst we’ve created the apps/dummyhttp/base/kustomization.yaml file to include the application from the previous repository:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0 Note: As we are not using github.com to publish the application we add the resource using the full https URI, splitting the PATH from the repository base address and including the tag we want to use.\nAs a second step we create the apps/dummyhttp/overlays/test/kustomization.yaml file to include the previous file:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base And finally we add the apps/dummyhttp/overlays/test/config.json file to configure the application as the ApplicationSet defined by argocd-autopilot expects:\n{ \u0026#34;appName\u0026#34;: \u0026#34;dummyhttp\u0026#34;, \u0026#34;userGivenName\u0026#34;: \u0026#34;dummyhttp\u0026#34;, \u0026#34;destNamespace\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;destServer\u0026#34;: \u0026#34;https://kubernetes.default.svc\u0026#34;, \u0026#34;srcPath\u0026#34;: \u0026#34;apps/dummyhttp/overlays/test\u0026#34;, \u0026#34;srcRepoURL\u0026#34;: \u0026#34;https://forgejo.mixinet.net/blogops/argocd.git\u0026#34;, \u0026#34;srcTargetRevision\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;labels\u0026#34;: null, \u0026#34;annotations\u0026#34;: null } Once we have the three files we commit and push the changes and argocd deploys the application; we can check that things are working using curl:\n❯ curl -s https://dummyhttp.lo.mixinet.net:8443/ | jq -M . { \u0026#34;c\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;s\u0026#34;: \u0026#34;\u0026#34; } Patching the applicationNow we will add patches to the apps/dummyhttp/overlays/test/kustomization.yaml file:\nOne to add annotations for reloader (one to enable it and another one to set the roll out strategy to restart to avoid touching the deployments, as that can generate issues with argocd).Another to change the ingress hostname (not really needed, but something quite reasonable for a specific project). The file diff is as follows:\n--- a/apps/dummyhttp/overlays/test/kustomization.yaml +++ b/apps/dummyhttp/overlays/test/kustomization.yaml @@ -2,3 +2,22 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base +patches: +# Add reloader annotations +- target: + kind: Deployment + name: dummyhttp + patch: |- + - op: add + path: /metadata/annotations + value: + reloader.stakater.com/auto: \u0026#34;true\u0026#34; + reloader.stakater.com/rollout-strategy: \u0026#34;restart\u0026#34; +# Change the ingress host name +- target: + kind: Ingress + name: dummyhttp + patch: |- + - op: replace + path: /spec/rules/0/host + value: test-dummyhttp.lo.mixinet.net After committing and pushing the changes we can use the argocd cli to check the status of the application:\n❯ argocd app get argocd/test-dummyhttp -o tree Name: argocd/test-dummyhttp Project: test Server: https://kubernetes.default.svc Namespace: default URL: https://argocd.lo.mixinet.net:8443/applications/test-dummyhttp Source: - Repo: https://forgejo.mixinet.net/blogops/argocd.git Target: Path: apps/dummyhttp/overlays/test SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: Synced to (fbc6031) Health Status: Healthy KIND/NAME STATUS HEALTH MESSAGE Deployment/dummyhttp Synced Healthy deployment.apps/dummyhttp configured └─ReplicaSet/dummyhttp-55569589bc Healthy └─Pod/dummyhttp-55569589bc-qhnfk Healthy Ingress/dummyhttp Synced Healthy ingress.networking.k8s.io/dummyhttp configured Service/dummyhttp Synced Healthy service/dummyhttp unchanged ├─Endpoints/dummyhttp └─EndpointSlice/dummyhttp-x57bl As we can see, the Deployment and Ingress where updated, but the Service is unchanged.\nTo validate that the ingress is using the new hostname we can use curl:\n❯ curl -s https://dummyhttp.lo.mixinet.net:8443/ 404 page not found ❯ curl -s https://test-dummyhttp.lo.mixinet.net:8443/ {\u0026#34;c\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;s\u0026#34;: \u0026#34;\u0026#34;} Adding a ConfigMapNow that the system is adjusted to reload the application when the ConfigMap or Secret is created, deleted or updated we are ready to add one file and see how the system reacts.\nWe modify the apps/dummyhttp/overlays/test/kustomization.yaml file to create the ConfigMap using the configMapGenerator as follows:\n--- a/apps/dummyhttp/overlays/test/kustomization.yaml +++ b/apps/dummyhttp/overlays/test/kustomization.yaml @@ -2,6 +2,14 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base +# Add the config map +configMapGenerator: +- name: dummyhttp-configmap + literals: + - CM_VAR=\u0026#34;Default Test Value\u0026#34; + behavior: create + options: + disableNameSuffixHash: true patches: # Add reloader annotations - target: After committing and pushing the changes we can see that the ConfigMap is available, the pod has been deleted and started again and the curl output includes the new value:\n❯ kubectl get configmaps,pods NAME READY STATUS RESTARTS AGE configmap/dummyhttp-configmap 1 11s configmap/kube-root-ca.crt 1 4d7h NAME DATA AGE pod/dummyhttp-779c96c44b-pjq4d 1/1 Running 0 11s pod/dummyhttp-fc964557f-jvpkx 1/1 Terminating 0 2m42s ❯ curl -s https://test-dummyhttp.lo.mixinet.net:8443 | jq -M . { \u0026#34;c\u0026#34;: \u0026#34;Default Test Value\u0026#34;, \u0026#34;s\u0026#34;: \u0026#34;\u0026#34; } Using helm with argocd-autopilotRight now there is no direct support in argocd-autopilot to manage applications using helm (see the issue #38 on the project), but we want to use a chart in our next example.\nThere are multiple ways to add the support, but the simplest one that allows us to keep using argocd-autopilot is to use kustomize applications that call helm as described here.\nThe only thing needed before being able to use the approach is to add the kustomize.buildOptions flag to the argocd-cm on the bootstrap/argo-cd/kustomization.yaml file, its contents now are follows:\nbootstrap/argo-cd/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 configMapGenerator: - behavior: merge literals: # Enable helm usage from kustomize (see https://github.com/argoproj/argo-cd/issues/2789#issuecomment-960271294) - kustomize.buildOptions=\u0026#34;--enable-helm\u0026#34; - | repository.credentials=- passwordSecret: key: git_token name: autopilot-secret url: https://forgejo.mixinet.net/ usernameSecret: key: git_username name: autopilot-secret name: argocd-cm # Disable TLS for the Argo Server (see https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#traefik-v30) - behavior: merge literals: - \u0026#34;server.insecure=true\u0026#34; name: argocd-cmd-params-cm kind: Kustomization namespace: argocd resources: - github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.4.19 - ingress_route.yaml On the following section we will explain how the application is defined to make things work.\nInstalling the sealed-secrets controllerTo manage secrets in our cluster we are going to use the sealed-secrets controller and to install it we are going to use its chart.\nAs we mentioned on the previous section, the idea is to create a kustomize application and use that to deploy the chart, but we are going to create the files manually, as we are not going import the base kustomization files from a remote repository.\nAs there is no clear way to override helm Chart values using overlays we are going to use a generator to create the helm configuration from an external resource and include it from our overlays (the idea has been taken from this repository, which was referenced from a comment on the kustomize issue #38 mentioned earlier).\nThe sealed-secrets applicationWe have created the following files and folders manually:\napps/sealed-secrets/ ├── helm │ ├── chart.yaml │ └── kustomization.yaml └── overlays └── test ├── config.json ├── kustomization.yaml └── values.yaml The helm folder contains the generator template that will be included from our overlays.\nThe kustomization.yaml includes the chart.yaml as a resource:\napps/sealed-secrets/helm/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - chart.yaml And the chart.yaml file defines the HelmChartInflationGenerator:\napps/sealed-secrets/helm/chart.yaml apiVersion: builtin kind: HelmChartInflationGenerator metadata: name: sealed-secrets releaseName: sealed-secrets name: sealed-secrets namespace: kube-system repo: https://bitnami-labs.github.io/sealed-secrets version: 2.17.2 includeCRDs: true # Add common values to all argo-cd projects inline valuesInline: fullnameOverride: sealed-secrets-controller # Load a values.yaml file from the same directory that uses this generator valuesFile: values.yaml For this chart the template adjusts the namespace to kube-system and adds the fullnameOverride on the valuesInline key because we want to use those settings on all the projects (they are the values expected by the kubeseal command line application, so we adjust them to avoid the need to add additional parameters to it).\nWe adjust global values as inline to be able to use a the valuesFile from our overlays; as we are using a generator the path is relative to the folder that contains the kustomization.yaml file that calls it, in our case we will need to have a values.yaml file on each overlay folder (if we don’t want to overwrite any values for a project we can create an empty file, but it has to exist).\nFinally, our overlay folder contains three files, a kustomization.yaml file that includes the generator from the helm folder, the values.yaml file needed by the chart and the config.json file used by argocd-autopilot to install the application.\nThe kustomization.yaml file contents are:\napps/sealed-secrets/overlays/test/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Uncomment if you want to add additional resources using kustomize #resources: #- ../../base generators: - ../../helm The values.yaml file enables the ingress for the application and adjusts its hostname:\napps/sealed-secrets/overlays/test/values.yaml ingress: enabled: true hostname: test-sealed-secrets.lo.mixinet.net And the config.json file is similar to the ones used with the other applications we have installed:\napps/sealed-secrets/overlays/test/config.json { \u0026#34;appName\u0026#34;: \u0026#34;sealed-secrets\u0026#34;, \u0026#34;userGivenName\u0026#34;: \u0026#34;sealed-secrets\u0026#34;, \u0026#34;destNamespace\u0026#34;: \u0026#34;kube-system\u0026#34;, \u0026#34;destServer\u0026#34;: \u0026#34;https://kubernetes.default.svc\u0026#34;, \u0026#34;srcPath\u0026#34;: \u0026#34;apps/sealed-secrets/overlays/test\u0026#34;, \u0026#34;srcRepoURL\u0026#34;: \u0026#34;https://forgejo.mixinet.net/blogops/argocd.git\u0026#34;, \u0026#34;srcTargetRevision\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;labels\u0026#34;: null, \u0026#34;annotations\u0026#34;: null } Once we commit and push the files the sealed-secrets application is installed in our cluster, we can check it using curl to get the public certificate used by it:\n❯ curl -s https://test-sealed-secrets.lo.mixinet.net:8443/v1/cert.pem -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- The dummyhttp-secretTo create sealed secrets we need to install the kubeseal tool:\n❯ arkade get kubeseal Now we create a local version of the dummyhttp-secret that contains some value on the SECRET_VAR key (the easiest way for doing it is to use kubectl):\n❯ echo -n \u0026#34;Boo\u0026#34; | kubectl create secret generic dummyhttp-secret \\ --dry-run=client --from-file=SECRET_VAR=/dev/stdin -o yaml \\ \u0026gt;/tmp/dummyhttp-secret.yaml The secret definition in yaml format is:\napiVersion: v1 data: SECRET_VAR: Qm9v kind: Secret metadata: creationTimestamp: null name: dummyhttp-secret To create a sealed version using the kubeseal tool we can do the following:\n❯ kubeseal -f /tmp/dummyhttp-secret.yaml -w /tmp/dummyhttp-sealed-secret.yaml That invocation needs to have access to the cluster to do its job and in our case it works because we modified the chart to use the kube-system namespace and set the controller name to sealed-secrets-controller as the tool expects.\nIf we need to create the secrets without credentials we can connect to the ingress address we added to retrieve the public key:\n❯ kubeseal -f /tmp/dummyhttp-secret.yaml -w /tmp/dummyhttp-sealed-secret.yaml \\ --cert https://test-sealed-secrets.lo.mixinet.net:8443/v1/cert.pem Or, if we don’t have access to the ingress address, we can save the certificate on a file and use it instead of the URL.\nThe sealed version of the secret looks like this:\napiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: creationTimestamp: null name: dummyhttp-secret namespace: default spec: encryptedData: SECRET_VAR: [...] template: metadata: creationTimestamp: null name: dummyhttp-secret namespace: default This file can be deployed to the cluster to create the secret (in our case we will add it to the argocd application), but before doing that we are going to check the output of our dummyhttp service and get the list of Secrets and SealedSecrets in the default namespace:\n❯ curl -s https://test-dummyhttp.lo.mixinet.net:8443 | jq -M . { \u0026#34;c\u0026#34;: \u0026#34;Default Test Value\u0026#34;, \u0026#34;s\u0026#34;: \u0026#34;\u0026#34; } ❯ kubectl get sealedsecrets,secrets No resources found in default namespace. Now we add the SealedSecret to the dummyapp copying the file and adding it to the kustomization.yaml file:\n--- a/apps/dummyhttp/overlays/test/kustomization.yaml +++ b/apps/dummyhttp/overlays/test/kustomization.yaml @@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base +- dummyhttp-sealed-secret.yaml # Create the config map value configMapGenerator: - name: dummyhttp-configmap Once we commit and push the files Argo CD creates the SealedSecret and the controller generates the Secret:\n❯ kubectl apply -f /tmp/dummyhttp-sealed-secret.yaml sealedsecret.bitnami.com/dummyhttp-secret created ❯ kubectl get sealedsecrets,secrets NAME STATUS SYNCED AGE sealedsecret.bitnami.com/dummyhttp-secret True 3s NAME TYPE DATA AGE secret/dummyhttp-secret Opaque 1 3s If we check the command output we can see the new value of the secret:\n❯ curl -s https://test-dummyhttp.lo.mixinet.net:8443 | jq -M . { \u0026#34;c\u0026#34;: \u0026#34;Default Test Value\u0026#34;, \u0026#34;s\u0026#34;: \u0026#34;Boo\u0026#34; } Using sealed-secrets in production clustersIf you plan to use sealed-secrets look into its documentation to understand how it manages the private keys, how to backup things and keep in mind that, as the documentation explains, you can rotate your sealed version of the secrets, but that doesn’t change the actual secrets.\nIf you want to rotate your secrets you have to update them and commit the sealed version of the updates (as the controller also rotates the encryption keys your new sealed version will also be using a newer key, so you will be doing both things at the same time).\nFinal remarksOn this post we have seen how to deploy applications using the argocd-autopilot model, including the use of helm charts inside kustomize applications and how to install and use the sealed-secrets controller.\nIt has been interesting and I’ve learnt a lot about argocd in the process, but I believe that if I ever want to use it in production I will also review the native helm support in argocd using a separate repository to manage the applications, at least to be able to compare it to the model explained here.\n","permalink":"https://blogops.mixinet.net/posts/gitops/argocd_usage_examples/","summary":"\u003cp\u003eAs a followup of my post about the use of \u003ca href=\"https://blogops.mixinet.net/posts/gitops/argocd-autopilot/\" target=\"_blank\" rel=\"noopener\"\u003eargocd-autopilot\u003c/a\u003e\nI’m going to deploy various applications to the cluster using \u003ca href=\"https://argo-cd.readthedocs.io/\" target=\"_blank\" rel=\"noopener\"\u003eArgo CD\u003c/a\u003e from the same\nrepository we used on the previous post.\u003c/p\u003e\n\u003cp\u003eFor our examples we are going to test a solution to the problem we had when we updated a \u003ccode\u003eConfigMap\u003c/code\u003e used by the\n\u003ccode\u003eargocd-server\u003c/code\u003e (the resource was updated but the application Pod was not because there was no change on the\n\u003ccode\u003eargocd-server\u003c/code\u003e deployment); our original fix was to kill the pod manually, but the manual operation is something we\nwant to avoid.\u003c/p\u003e\n\u003cp\u003eThe proposed solution to this kind of issues on the\n\u003ca href=\"https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments\" target=\"_blank\" rel=\"noopener\"\u003ehelm documentation\u003c/a\u003e is to add\nannotations to the \u003ccode\u003eDeployments\u003c/code\u003e with values that are a hash of the \u003ccode\u003eConfigMaps\u003c/code\u003e or \u003ccode\u003eSecrets\u003c/code\u003e used by them, this way if\na file is updated the annotation is also updated and when the \u003ccode\u003eDeployment\u003c/code\u003e changes are applied a roll out of the pods is\ntriggered.\u003c/p\u003e\n\u003cp\u003eOn this post we will install a couple of controllers and an application to show how we can handle \u003ccode\u003eSecrets\u003c/code\u003e with\n\u003ccode\u003eargocd\u003c/code\u003e and solve the issue with updates on \u003ccode\u003eConfigMaps\u003c/code\u003e and \u003ccode\u003eSecrets\u003c/code\u003e, to do it we will execute the following tasks:\u003c/p\u003e\n\u003cdiv class=\"olist arabic\"\u003e\u003col class=\"arabic\"\u003e\u003cli\u003eDeploy the \u003ca href=\"https://github.com/stakater/Reloader\" target=\"_blank\" rel=\"noopener\"\u003eReloader\u003c/a\u003e controller to our cluster. It is a tool that watches\nchanges in \u003ccode\u003eConfigMaps\u003c/code\u003e and \u003ccode\u003eSecrets\u003c/code\u003e and does rolling upgrades on the \u003ccode\u003ePods\u003c/code\u003e that use them from \u003ccode\u003eDeployment\u003c/code\u003e,\n\u003ccode\u003eStatefulSet\u003c/code\u003e, \u003ccode\u003eDaemonSet\u003c/code\u003e or \u003ccode\u003eDeploymentConfig\u003c/code\u003e objects when they are updated (by default we have to add some\nannotations to the objects to make things work).\u003c/li\u003e\u003cli\u003eDeploy a simple application that can use \u003ccode\u003eConfigMaps\u003c/code\u003e and \u003ccode\u003eSecrets\u003c/code\u003e and test that the \u003ccode\u003eReloader\u003c/code\u003e controller does its\njob when we add or update a \u003ccode\u003eConfigMap\u003c/code\u003e.\u003c/li\u003e\u003cli\u003eInstall the \u003ca href=\"https://github.com/bitnami-labs/sealed-secrets\" target=\"_blank\" rel=\"noopener\"\u003eSealed Secrets\u003c/a\u003e controller to manage secrets inside our\ncluster, use it to add a secret to our sample application and see that the application is reloaded automatically.\u003c/li\u003e\u003c/ol\u003e\u003c/div\u003e","title":"Argo CD Usage Examples"},{"content":"For a long time I’ve been wanting to try GitOps tools, but I haven’t had the chance to try them for real on the projects I was working on.\nAs now I have some spare time I’ve decided I’m going to play a little with Argo CD, Flux and Kluctl to test them and be able to use one of them in a real project in the future if it looks appropriate.\nOn this post I will use Argo-CD Autopilot to install argocd on a k3d local cluster installed using OpenTofu to test the autopilot approach of managing argocd and test the tool (as it manages argocd using a git repository it can be used to test argocd as well).\nInstalling tools locally with arkadeRecently I’ve been using the arkade tool to install kubernetes related applications on Linux servers and containers, I usually get the applications with it and install them on the /usr/local/bin folder.\nFor this post I’ve created a simple script that checks if the tools I’ll be using are available and installs them on the $HOME/.arkade/bin folder if missing (I’m assuming that docker is already available, as it is not installable with arkade):\n#!/bin/sh # TOOLS LIST ARKADE_APPS=\u0026#34;argocd argocd-autopilot k3d kubectl sops tofu\u0026#34; # Add the arkade binary directory to the path if missing case \u0026#34;:${PATH}:\u0026#34; in *:\u0026#34;${HOME}/.arkade/bin\u0026#34;:*) ;; *) export PATH=\u0026#34;${PATH}:${HOME}/.arkade/bin\u0026#34; ;; esac # Install or update arkade if command -v arkade \u0026gt;/dev/null; then echo \u0026#34;Trying to update the arkade application\u0026#34; sudo arkade update else echo \u0026#34;Installing the arkade application\u0026#34; curl -sLS https://get.arkade.dev | sudo sh fi echo \u0026#34;\u0026#34; echo \u0026#34;Installing tools with arkade\u0026#34; echo \u0026#34;\u0026#34; for app in $ARKADE_APPS; do app_path=\u0026#34;$(command -v $app)\u0026#34; || true if [ \u0026#34;$app_path\u0026#34; ]; then echo \u0026#34;The application \u0026#39;$app\u0026#39; already available on \u0026#39;$app_path\u0026#39;\u0026#34; else arkade get \u0026#34;$app\u0026#34; fi done cat \u0026lt;\u0026lt;EOF Add the ~/.arkade/bin directory to your PATH if tools have been installed there EOF The rest of scripts will add the binary directory to the PATH if missing to make sure things work if something was installed there.\nNote: The arkade command has support to install applications using helm or YAML manifests besides downloading applications and it can probably be useful for quick tests, but I prefer to manage the deployments myself, so I have not used that part.\nCreating a k3d cluster with opentofuAlthough using k3d directly will be a good choice for the creation of the cluster, I’m using tofu to do it because that will probably be the tool used to do it if we were working with Cloud Platforms like AWS or Google.\nThe main.tf file is as follows:\nterraform { required_providers { k3d = { source = \u0026#34;moio/k3d\u0026#34; version = \u0026#34;0.0.12\u0026#34; } sops = { source = \u0026#34;carlpett/sops\u0026#34; version = \u0026#34;1.2.0\u0026#34; } } } data \u0026#34;sops_file\u0026#34; \u0026#34;secrets\u0026#34; { source_file = \u0026#34;secrets.yaml\u0026#34; } resource \u0026#34;k3d_cluster\u0026#34; \u0026#34;argocd_cluster\u0026#34; { name = \u0026#34;argocd\u0026#34; servers = 1 agents = 2 image = \u0026#34;rancher/k3s:v1.31.5-k3s1\u0026#34; network = \u0026#34;argocd\u0026#34; token = data.sops_file.secrets.data[\u0026#34;token\u0026#34;] port { host_port = 8443 container_port = 443 node_filters = [ \u0026#34;loadbalancer\u0026#34;, ] } k3d { disable_load_balancer = false disable_image_volume = false } kubeconfig { update_default_kubeconfig = true switch_current_context = true } runtime { gpu_request = \u0026#34;all\u0026#34; } } The k3d configuration is quite simple, as I plan to use the default traefik ingress controller with TLS I publish the 443 port on the hosts 8443 port, I’ll explain how I add a valid certificate on the next step.\nI’ve prepared the following script to initialize and apply the changes:\n#!/bin/sh set -e # VARIABLES # Default token for the argocd cluster K3D_CLUSTER_TOKEN=\u0026#34;argocdToken\u0026#34; # Relative PATH to install the k3d cluster using terr-iaform K3D_TF_RELPATH=\u0026#34;k3d-tf\u0026#34; # Secrets yaml file SECRETS_YAML=\u0026#34;secrets.yaml\u0026#34; # Relative PATH to the workdir from the script directory WORK_DIR_RELPATH=\u0026#34;..\u0026#34; # Compute WORKDIR SCRIPT=\u0026#34;$(readlink -f \u0026#34;$0\u0026#34;)\u0026#34; SCRIPT_DIR=\u0026#34;$(dirname \u0026#34;$SCRIPT\u0026#34;)\u0026#34; WORK_DIR=\u0026#34;$(readlink -f \u0026#34;$SCRIPT_DIR/$WORK_DIR_RELPATH\u0026#34;)\u0026#34; # Update the PATH to add the arkade bin directory # Add the arkade binary directory to the path if missing case \u0026#34;:${PATH}:\u0026#34; in *:\u0026#34;${HOME}/.arkade/bin\u0026#34;:*) ;; *) export PATH=\u0026#34;${PATH}:${HOME}/.arkade/bin\u0026#34; ;; esac # Go to the k3d-tf dir cd \u0026#34;$WORK_DIR/$K3D_TF_RELPATH\u0026#34; || exit 1 # Create secrets.yaml file and encode it with sops if missing if [ ! -f \u0026#34;$SECRETS_YAML\u0026#34; ]; then echo \u0026#34;token: $K3D_CLUSTER_TOKEN\u0026#34; \u0026gt;\u0026#34;$SECRETS_YAML\u0026#34; sops encrypt -i \u0026#34;$SECRETS_YAML\u0026#34; fi # Initialize terraform tofu init # Apply the configuration tofu apply Note: The main.tf file requires a secrets.yaml file ciphered with sops, that is why the script tries to create it with a generic value if it can’t be found.\nNote that the script is assuming that there is a .sops.yaml file on the user HOME directory prepared to cipher secrets (for development I use an age key as described on the sops documentation but you can use GPG instead).\nAdding a wildcard certificate to the k3d ingressAs an optional step, after creating the k3d cluster I’m going to add a default wildcard certificate for the traefik ingress server to be able to use everything with HTTPS without certificate issues.\nAs I manage my own DNS domain I’ve created the lo.mixinet.net and *.lo.mixinet.net DNS entries on my public and private DNS servers (both return 127.0.0.1 and ::1) and I’ve created a TLS certificate for both entries using Let’s Encrypt with Certbot.\nThe certificate is updated automatically on one of my servers and when I need it I copy the contents of the fullchain.pem and privkey.pem files from the /etc/letsencrypt/live/lo.mixinet.net server directory to the local files lo.mixinet.net.crt and lo.mixinet.net.key.\nAfter copying the files I run the following file to install or update the certificate and configure it as the default for traefik:\n#!/bin/sh # Script to update the secret=\u0026#34;lo-mixinet-net-ingress-cert\u0026#34; cert=\u0026#34;${1:-lo.mixinet.net.crt}\u0026#34; key=\u0026#34;${2:-lo.mixinet.net.key}\u0026#34; if [ -f \u0026#34;$cert\u0026#34; ] \u0026amp;\u0026amp; [ -f \u0026#34;$key\u0026#34; ]; then kubectl -n kube-system create secret tls $secret \\ --key=$key \\ --cert=$cert \\ --dry-run=client --save-config -o yaml | kubectl apply -f - kubectl apply -f - \u0026lt;\u0026lt; EOF apiVersion: traefik.containo.us/v1alpha1 kind: TLSStore metadata: name: default namespace: kube-system spec: defaultCertificate: secretName: $secret EOF else cat \u0026lt;\u0026lt;EOF To add or update the traefik TLS certificate the following files are needed: - cert: \u0026#39;$cert\u0026#39; - key: \u0026#39;$key\u0026#39; Note: you can pass the paths as arguments to this script. EOF fi Once it is installed if I connect to https://foo.lo.mixinet.net:8443/ I get a 404 but the certificate is valid.\nInstalling argocd with argocd-autopilotCreating a repository and a token for autopilotI’ll be using a project on my forgejo instance to manage argocd, the repository I’ve created is on the URL https://forgejo.mixinet.net/blogops/argocd and I’ve created a private user named argocd that only has write access to that repository.\nLogging as the argocd user on forgejo I’ve created a token with permission to read and write repositories that I’ve saved on my pass password store on the mixinet.net/argocd@forgejo/repository-write entry.\nBootstrapping the installationTo bootstrap the installation I’ve used the following script (it uses the previous GIT_REPO and GIT_TOKEN values):\n#!/bin/sh set -e # VARIABLES # Relative PATH to the workdir from the script directory WORK_DIR_RELPATH=\u0026#34;..\u0026#34; # Compute WORKDIR SCRIPT=\u0026#34;$(readlink -f \u0026#34;$0\u0026#34;)\u0026#34; SCRIPT_DIR=\u0026#34;$(dirname \u0026#34;$SCRIPT\u0026#34;)\u0026#34; WORK_DIR=\u0026#34;$(readlink -f \u0026#34;$SCRIPT_DIR/$WORK_DIR_RELPATH\u0026#34;)\u0026#34; # Update the PATH to add the arkade bin directory # Add the arkade binary directory to the path if missing case \u0026#34;:${PATH}:\u0026#34; in *:\u0026#34;${HOME}/.arkade/bin\u0026#34;:*) ;; *) export PATH=\u0026#34;${PATH}:${HOME}/.arkade/bin\u0026#34; ;; esac # Go to the working directory cd \u0026#34;$WORK_DIR\u0026#34; || exit 1 # Set GIT variables if [ -z \u0026#34;$GIT_REPO\u0026#34; ]; then export GIT_REPO=\u0026#34;https://forgejo.mixinet.net/blogops/argocd.git\u0026#34; fi if [ -z \u0026#34;$GIT_TOKEN\u0026#34; ]; then GIT_TOKEN=\u0026#34;$(pass mixinet.net/argocd@forgejo/repository-write)\u0026#34; export GIT_TOKEN fi argocd-autopilot repo bootstrap --provider gitea The output of the execution is as follows:\n❯ bin/argocd-bootstrap.sh INFO cloning repo: https://forgejo.mixinet.net/blogops/argocd.git INFO empty repository, initializing a new one with specified remote INFO using revision: \u0026#34;\u0026#34;, installation path: \u0026#34;\u0026#34; INFO using context: \u0026#34;k3d-argocd\u0026#34;, namespace: \u0026#34;argocd\u0026#34; INFO applying bootstrap manifests to cluster... namespace/argocd created customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created serviceaccount/argocd-application-controller created serviceaccount/argocd-applicationset-controller created serviceaccount/argocd-dex-server created serviceaccount/argocd-notifications-controller created serviceaccount/argocd-redis created serviceaccount/argocd-repo-server created serviceaccount/argocd-server created role.rbac.authorization.k8s.io/argocd-application-controller created role.rbac.authorization.k8s.io/argocd-applicationset-controller created role.rbac.authorization.k8s.io/argocd-dex-server created role.rbac.authorization.k8s.io/argocd-notifications-controller created role.rbac.authorization.k8s.io/argocd-redis created role.rbac.authorization.k8s.io/argocd-server created clusterrole.rbac.authorization.k8s.io/argocd-application-controller created clusterrole.rbac.authorization.k8s.io/argocd-applicationset-controller created clusterrole.rbac.authorization.k8s.io/argocd-server created rolebinding.rbac.authorization.k8s.io/argocd-application-controller created rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created rolebinding.rbac.authorization.k8s.io/argocd-dex-server created rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created rolebinding.rbac.authorization.k8s.io/argocd-redis created rolebinding.rbac.authorization.k8s.io/argocd-server created clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created clusterrolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created clusterrolebinding.rbac.authorization.k8s.io/argocd-server created configmap/argocd-cm created configmap/argocd-cmd-params-cm created configmap/argocd-gpg-keys-cm created configmap/argocd-notifications-cm created configmap/argocd-rbac-cm created configmap/argocd-ssh-known-hosts-cm created configmap/argocd-tls-certs-cm created secret/argocd-notifications-secret created secret/argocd-secret created service/argocd-applicationset-controller created service/argocd-dex-server created service/argocd-metrics created service/argocd-notifications-controller-metrics created service/argocd-redis created service/argocd-repo-server created service/argocd-server created service/argocd-server-metrics created deployment.apps/argocd-applicationset-controller created deployment.apps/argocd-dex-server created deployment.apps/argocd-notifications-controller created deployment.apps/argocd-redis created deployment.apps/argocd-repo-server created deployment.apps/argocd-server created statefulset.apps/argocd-application-controller created networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created networkpolicy.networking.k8s.io/argocd-redis-network-policy created networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created networkpolicy.networking.k8s.io/argocd-server-network-policy created secret/autopilot-secret created INFO pushing bootstrap manifests to repo INFO applying argo-cd bootstrap application INFO pushing bootstrap manifests to repo INFO applying argo-cd bootstrap application application.argoproj.io/autopilot-bootstrap created INFO running argocd login to initialize argocd config Context \u0026#39;autopilot\u0026#39; updated INFO argocd initialized. password: XXXXXXX-XXXXXXXX INFO run: kubectl port-forward -n argocd svc/argocd-server 8080:80 Now we have the argocd installed and running, it can be checked using the port-forward and connecting to https://localhost:8080/ (the certificate will be wrong, we are going to fix that in the next step).\nUpdating the argocd installation in gitNow that we have the application deployed we can clone the argocd repository and edit the deployment to disable TLS for the argocd server (we are going to use TLS termination with traefik and that needs the server running as insecure, see the Argo CD documentation)\n❯ ssh clone ssh://git@forgejo.mixinet.net/blogops/argocd.git ❯ cd argocd ❯ edit bootstrap/argo-cd/kustomization.yaml ❯ git commit -m \u0026#39;Disable TLS for the argocd-server\u0026#39; The changes made to the kustomization.yaml file are the following:\n--- a/bootstrap/argo-cd/kustomization.yaml +++ b/bootstrap/argo-cd/kustomization.yaml @@ -11,6 +11,11 @@ configMapGenerator: key: git_username name: autopilot-secret name: argocd-cm + # Disable TLS for the Argo Server (see https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#traefik-v30) +- behavior: merge + literals: + - \u0026#34;server.insecure=true\u0026#34; + name: argocd-cmd-params-cm kind: Kustomization namespace: argocd resources: Once the changes are pushed we sync the argo-cd application manually to make sure they are applied:\nAs a test we can download the argocd-cmd-params-cm ConfigMap to make sure everything is OK:\napiVersion: v1 data: server.insecure: \u0026#34;true\u0026#34; kind: ConfigMap metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {\u0026#34;apiVersion\u0026#34;:\u0026#34;v1\u0026#34;,\u0026#34;data\u0026#34;:{\u0026#34;server.insecure\u0026#34;:\u0026#34;true\u0026#34;},\u0026#34;kind\u0026#34;:\u0026#34;ConfigMap\u0026#34;,\u0026#34;metadata\u0026#34;:{\u0026#34;annotations\u0026#34;:{},\u0026#34;labels\u0026#34;:{\u0026#34;app.kubernetes.io/instance\u0026#34;:\u0026#34;argo-cd\u0026#34;,\u0026#34;app.kubernetes.io/name\u0026#34;:\u0026#34;argocd-cmd-params-cm\u0026#34;,\u0026#34;app.kubernetes.io/part-of\u0026#34;:\u0026#34;argocd\u0026#34;},\u0026#34;name\u0026#34;:\u0026#34;argocd-cmd-params-cm\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;argocd\u0026#34;}} creationTimestamp: \u0026#34;2025-04-27T17:31:54Z\u0026#34; labels: app.kubernetes.io/instance: argo-cd app.kubernetes.io/name: argocd-cmd-params-cm app.kubernetes.io/part-of: argocd name: argocd-cmd-params-cm namespace: argocd resourceVersion: \u0026#34;16731\u0026#34; uid: a460638f-1d82-47f6-982c-3017699d5f14 As this simply changes the ConfigMap we have to restart the argocd-server to read it again, to do it we delete the server pods so they are re-created using the updated resource:\n❯ kubectl delete pods -n argocd -l app.kubernetes.io/name=argocd-server After doing this the port-forward command is killed automatically, if we run it again the connection to get to the argocd-server has to be done using HTTP instead of HTTPS.\nInstead of testing that we are going to add an ingress definition to be able to connect to the server using HTTPS and GRPC against the address argocd.lo.mixinet.net using the wildcard TLS certificate we installed earlier.\nTo do it we to edit the bootstrap/argo-cd/kustomization.yaml file to add the ingress_route.yaml file to the deployment:\n--- a/bootstrap/argo-cd/kustomization.yaml +++ b/bootstrap/argo-cd/kustomization.yaml @@ -20,3 +20,4 @@ kind: Kustomization namespace: argocd resources: - github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.4.19 +- ingress_route.yaml The ingress_route.yaml file contents are the following:\napiVersion: traefik.io/v1alpha1 kind: IngressRoute metadata: name: argocd-server namespace: argocd spec: entryPoints: - websecure routes: - kind: Rule match: Host(`argocd.lo.mixinet.net`) priority: 10 services: - name: argocd-server port: 80 - kind: Rule match: Host(`argocd.lo.mixinet.net`) \u0026amp;\u0026amp; Header(`Content-Type`, `application/grpc`) priority: 11 services: - name: argocd-server port: 80 scheme: h2c tls: certResolver: default After pushing the changes and waiting a little bit the change is applied and we can access the server using HTTPS and GRPC, the first way can be tested from a browser and the GRPC using the command line interface:\n❯ argocd --grpc-web login argocd.lo.mixinet.net:8443 Username: admin Password: \u0026#39;admin:login\u0026#39; logged in successfully Context \u0026#39;argocd.lo.mixinet.net:8443\u0026#39; updated ❯ argocd app list -o name argocd/argo-cd argocd/autopilot-bootstrap argocd/cluster-resources-in-cluster argocd/root So things are working fine …​ and that is all on this post, folks!\n","permalink":"https://blogops.mixinet.net/posts/gitops/argocd-autopilot/","summary":"\u003cp\u003eFor a long time I’ve been wanting to try GitOps tools, but I haven’t had the chance to try them for real on the projects\nI was working on.\u003c/p\u003e\n\u003cp\u003eAs now I have some spare time I’ve decided I’m going to play a little with \u003ca href=\"https://argo-cd.readthedocs.io/\" target=\"_blank\" rel=\"noopener\"\u003eArgo CD\u003c/a\u003e,\n\u003ca href=\"https://fluxcd.io/\" target=\"_blank\" rel=\"noopener\"\u003eFlux\u003c/a\u003e and \u003ca href=\"https://kluctl.io/\" target=\"_blank\" rel=\"noopener\"\u003eKluctl\u003c/a\u003e to test them and be able to use one of them in a real project\nin the future if it looks appropriate.\u003c/p\u003e\n\u003cp\u003eOn this post I will use \u003ca href=\"https://argocd-autopilot.readthedocs.io/\" target=\"_blank\" rel=\"noopener\"\u003eArgo-CD Autopilot\u003c/a\u003e to install \u003ccode\u003eargocd\u003c/code\u003e on a\n\u003ca href=\"https://k3d.io/\" target=\"_blank\" rel=\"noopener\"\u003ek3d\u003c/a\u003e local cluster installed using \u003ca href=\"https://opentofu.org/\" target=\"_blank\" rel=\"noopener\"\u003eOpenTofu\u003c/a\u003e to test the autopilot approach of\nmanaging \u003ccode\u003eargocd\u003c/code\u003e and test the tool (as it manages \u003ccode\u003eargocd\u003c/code\u003e using a git repository it can be used to test \u003ccode\u003eargocd\u003c/code\u003e as\nwell).\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_installing_tools_locally_with_arkade\"\u003eInstalling tools locally with \u003ccode\u003earkade\u003c/code\u003e\u003c/h2\u003e\u003cp\u003eRecently I’ve been using the \u003ca href=\"https://github.com/alexellis/arkade\" target=\"_blank\" rel=\"noopener\"\u003earkade\u003c/a\u003e tool to install \u003ccode\u003ekubernetes\u003c/code\u003e related\napplications on Linux servers and containers, I usually get the applications with it and install them on the\n\u003ccode\u003e/usr/local/bin\u003c/code\u003e folder.\u003c/p\u003e\n\u003cp\u003eFor this post I’ve created a simple script that checks if the tools I’ll be using are available and installs them on the\n\u003ccode\u003e$HOME/.arkade/bin\u003c/code\u003e folder if missing (I’m assuming that \u003ccode\u003edocker\u003c/code\u003e is already available, as it is not installable with\n\u003ccode\u003earkade\u003c/code\u003e):\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003e\u003cspan class=\"c\"\u003e#!/bin/sh\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# TOOLS LIST\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eARKADE_APPS\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;argocd argocd-autopilot k3d kubectl sops tofu\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Add the arkade binary directory to the path if missing\u003c/span\u003e\n\u003cspan class=\"k\"\u003ecase\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;:\u003c/span\u003e\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePATH\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e:\u0026#34;\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e*\u003c/span\u003e:\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eHOME\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e/.arkade/bin\u0026#34;\u003c/span\u003e:\u003cspan class=\"k\"\u003e*\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e;;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e*\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"nb\"\u003eexport \u003c/span\u003e\u003cspan class=\"nv\"\u003ePATH\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePATH\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e:\u003c/span\u003e\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eHOME\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e/.arkade/bin\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e;;\u003c/span\u003e\n\u003cspan class=\"k\"\u003eesac\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Install or update arkade\u003c/span\u003e\n\u003cspan class=\"k\"\u003eif \u003c/span\u003e\u003cspan class=\"nb\"\u003ecommand\u003c/span\u003e \u003cspan class=\"nt\"\u003e-v\u003c/span\u003e arkade \u003cspan class=\"o\"\u003e\u0026gt;\u003c/span\u003e/dev/null\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  \u003c/span\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Trying to update the arkade application\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"nb\"\u003esudo \u003c/span\u003earkade update\n\u003cspan class=\"k\"\u003eelse\n  \u003c/span\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Installing the arkade application\u0026#34;\u003c/span\u003e\n  curl \u003cspan class=\"nt\"\u003e-sLS\u003c/span\u003e https://get.arkade.dev | \u003cspan class=\"nb\"\u003esudo \u003c/span\u003esh\n\u003cspan class=\"k\"\u003efi\n\n\u003c/span\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Installing tools with arkade\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003efor \u003c/span\u003eapp \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"nv\"\u003e$ARKADE_APPS\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\n  \u003c/span\u003e\u003cspan class=\"nv\"\u003eapp_path\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003e\u003cspan class=\"nb\"\u003ecommand\u003c/span\u003e \u003cspan class=\"nt\"\u003e-v\u003c/span\u003e \u003cspan class=\"nv\"\u003e$app\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e||\u003c/span\u003e \u003cspan class=\"nb\"\u003etrue\n  \u003c/span\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$app_path\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n    \u003c/span\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;The application \u0026#39;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$app\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#39; already available on \u0026#39;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$app_path\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#39;\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003eelse\n    \u003c/span\u003earkade get \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$app\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003efi\ndone\n\n\u003c/span\u003e\u003cspan class=\"nb\"\u003ecat\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026lt;\u0026lt;\u003c/span\u003e\u003cspan class=\"no\"\u003eEOF\u003c/span\u003e\u003cspan class=\"sh\"\u003e\n\nAdd the ~/.arkade/bin directory to your PATH if tools have been installed there\n\nEOF\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"ArgoCD Autopilot"},{"content":"As promised on my previous post, on this entry I’ll explain how I’ve set up forgejo actions on the source repository of this site to build it using a runner instead of doing it on the public server using a webhook to trigger the operation.\nSetting up the systemThe first thing I’ve done is to disable the forgejo webhook call that was used to publish the site, as I don’t want to run it anymore.\nNote: For now I’ve just removed the Active flag from the webhook, just in case I want to use it again in the future.\nI’ve left the system based on the json2file server running as it does nothing if no webhook is called, if we want to use it again it would be a good idea to disable actions to avoid conflicts if something is pushed, but it can be executed manually if needed and nothing will break, as both systems use the same directories to publish things).\nAfter that I added a new workflow to the repository that does the following things:\nbuild the site using my hugo-adoc image.push the result to a branch that contains the generated site (we do this because the server is already configured to work with the git repository and we can use force pushes to keep only the last version of the site, removing the need of extra code to manage package uploads and removals).uses curl to send a notification to an instance of the webhook server installed on the remote server that triggers a script that updates the site using the git branch. Note: As explained on my original post on the production setup section, I could have used the json2file server to process the notification, but I’ve decided to use the webhook server instead because it is simpler to set up and allows me to show a different approach to the same problem.\nThe script that updates the site is executed by a user that has permissions to clone the repository using a ssh key.\nSetting up the webhook serviceOn the server machine we have installed and configured the webhook service to run a script that updates the site.\nTo install the application and setup the configuration we have used the following script:\n#!/bin/sh set -e # --------- # VARIABLES # --------- ARCH=\u0026#34;$(dpkg --print-architecture)\u0026#34; WEBHOOK_VERSION=\u0026#34;2.8.2\u0026#34; DOWNLOAD_URL=\u0026#34;https://github.com/adnanh/webhook/releases/download\u0026#34; WEBHOOK_TGZ_URL=\u0026#34;$DOWNLOAD_URL/$WEBHOOK_VERSION/webhook-linux-$ARCH.tar.gz\u0026#34; WEBHOOK_SERVICE_NAME=\u0026#34;webhook\u0026#34; # Files WEBHOOK_SERVICE_FILE=\u0026#34;/etc/systemd/system/$WEBHOOK_SERVICE_NAME.service\u0026#34; WEBHOOK_SOCKET_FILE=\u0026#34;/etc/systemd/system/$WEBHOOK_SERVICE_NAME.socket\u0026#34; WEBHOOK_TML_TEMPLATE=\u0026#34;/srv/blogops/action/webhook.yml.envsubst\u0026#34; WEBHOOK_YML=\u0026#34;/etc/webhook.yml\u0026#34; # Config file values WEBHOOK_USER=\u0026#34;$(id -u)\u0026#34; WEBHOOK_GROUP=\u0026#34;$(id -g)\u0026#34; WEBHOOK_LISTEN_STREAM=\u0026#34;172.31.31.1:4444\u0026#34; # ---- # MAIN # ---- # Install binary from releases (on Debian only version 2.8.0 is available, but # I need the 2.8.2 version to support the systemd activation mode). curl -fsSL -o \u0026#34;/tmp/webhook.tgz\u0026#34; \u0026#34;$WEBHOOK_TGZ_URL\u0026#34; tar -C /tmp -xzf /tmp/webhook.tgz sudo install -m 755 \u0026#34;/tmp/webhook-linux-$ARCH/webhook\u0026#34; /usr/local/bin/webhook rm -rf \u0026#34;/tmp/webhook-linux-$ARCH\u0026#34; /tmp/webhook.tgz # Service file sudo sh -c \u0026#34;cat \u0026gt;\u0026#39;$WEBHOOK_SERVICE_FILE\u0026#39;\u0026#34; \u0026lt;\u0026lt;EOF [Unit] Description=Webhook server [Service] Type=exec ExecStart=webhook -nopanic -hooks $WEBHOOK_YML User=$WEBHOOK_USER Group=$WEBHOOK_GROUP EOF # Socket config sudo sh -c \u0026#34;cat \u0026gt;\u0026#39;$WEBHOOK_SOCKET_FILE\u0026#39;\u0026#34; \u0026lt;\u0026lt;EOF [Unit] Description=Webhook server socket [Socket] # Set FreeBind to listen on missing addresses (the VPN can be down sometimes) FreeBind=true # Set ListenStream to the IP and port you want to listen on ListenStream=$WEBHOOK_LISTEN_STREAM [Install] WantedBy=multi-user.target EOF # Config file BLOGOPS_TOKEN=\u0026#34;$(uuid)\u0026#34; \\ envsubst \u0026lt;\u0026#34;$WEBHOOK_TML_TEMPLATE\u0026#34; | sudo sh -c \u0026#34;cat \u0026gt;$WEBHOOK_YML\u0026#34; chmod 0640 \u0026#34;$WEBHOOK_YML\u0026#34; chwon \u0026#34;$WEBHOOK_USER:$WEBHOOK_GROUP\u0026#34; \u0026#34;$WEBHOOK_YML\u0026#34; # Restart and enable service sudo systemctl daemon-reload sudo systemctl stop \u0026#34;$WEBHOOK_SERVICE_NAME.socket\u0026#34; sudo systemctl start \u0026#34;$WEBHOOK_SERVICE_NAME.socket\u0026#34; sudo systemctl enable \u0026#34;$WEBHOOK_SERVICE_NAME.socket\u0026#34; # ---- # vim: ts=2:sw=2:et:ai:sts=2 As seen on the code, we’ve installed the application using a binary from the project repository instead of a package because we needed the latest version of the application to use systemd with socket activation.\nThe configuration file template is the following one:\n- id: \u0026#34;update-blogops\u0026#34; execute-command: \u0026#34;/srv/blogops/action/bin/update-blogops.sh\u0026#34; command-working-directory: \u0026#34;/srv/blogops\u0026#34; trigger-rule: match: type: \u0026#34;value\u0026#34; value: \u0026#34;$BLOGOPS_TOKEN\u0026#34; parameter: source: \u0026#34;header\u0026#34; name: \u0026#34;X-Blogops-Token\u0026#34; The version on /etc/webhook.yml has the BLOGOPS_TOKEN adjusted to a random value that has to exported as a secret on the forgejo project (see later).\nNote: As we are going to be connecting to this server using a VPN we don’t need to enable TLS on the webhook server, but if we want to do it we just need to change the /etc/systemd/system/webhook.service file to include the -secure, -cert and -key options with the right paths to the certificate and key files on the ExecStart line:\n[Service] ExecStart=webhook -nopanic -hooks /etc/webhook.yaml \\ -secure -cert PATH_TO_WEBHOOK_CRT -key PATH_TO_WEBHOOK_KEY Once the service is started each time the action is executed the webhook daemon will get a notification and will run the following update-blogops.sh script to publish the updated version of the site:\n#!/bin/sh set -e # --------- # VARIABLES # --------- # Values REPO_URL=\u0026#34;ssh://git@forgejo.mixinet.net/mixinet/blogops.git\u0026#34; REPO_BRANCH=\u0026#34;html\u0026#34; REPO_DIR=\u0026#34;public\u0026#34; MAIL_PREFIX=\u0026#34;[BLOGOPS-UPDATE-ACTION] \u0026#34; # Address that gets all messages, leave it empty if not wanted MAIL_TO_ADDR=\u0026#34;blogops@mixinet.net\u0026#34; # Directories BASE_DIR=\u0026#34;/srv/blogops\u0026#34; PUBLIC_DIR=\u0026#34;$BASE_DIR/$REPO_DIR\u0026#34; NGINX_BASE_DIR=\u0026#34;$BASE_DIR/nginx\u0026#34; PUBLIC_HTML_DIR=\u0026#34;$NGINX_BASE_DIR/public_html\u0026#34; ACTION_BASE_DIR=\u0026#34;$BASE_DIR/action\u0026#34; ACTION_LOG_DIR=\u0026#34;$ACTION_BASE_DIR/log\u0026#34; # Files OUTPUT_BASENAME=\u0026#34;$(date +%Y%m%d-%H%M%S.%N)\u0026#34; ACTION_LOGFILE_PATH=\u0026#34;$ACTION_LOG_DIR/$OUTPUT_BASENAME.log\u0026#34; # --------- # Functions # --------- action_log() { echo \u0026#34;$(date -R) $*\u0026#34; \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; } action_check_directories() { for _d in \u0026#34;$ACTION_BASE_DIR\u0026#34; \u0026#34;$ACTION_LOG_DIR\u0026#34;; do [ -d \u0026#34;$_d\u0026#34; ] || mkdir \u0026#34;$_d\u0026#34; done } action_clean_directories() { # Try to remove empty dirs for _d in \u0026#34;$ACTION_LOG_DIR\u0026#34; \u0026#34;$ACTION_BASE_DIR\u0026#34;; do if [ -d \u0026#34;$_d\u0026#34; ]; then rmdir \u0026#34;$_d\u0026#34; 2\u0026gt;/dev/null || true fi done } mail_success() { to_addr=\u0026#34;$MAIL_TO_ADDR\u0026#34; if [ \u0026#34;$to_addr\u0026#34; ]; then subject=\u0026#34;OK - updated blogops site\u0026#34; mail -s \u0026#34;${MAIL_PREFIX}${subject}\u0026#34; \u0026#34;$to_addr\u0026#34; \u0026lt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; fi } mail_failure() { to_addr=\u0026#34;$MAIL_TO_ADDR\u0026#34; if [ \u0026#34;$to_addr\u0026#34; ]; then subject=\u0026#34;KO - failed to update blogops site\u0026#34; mail -s \u0026#34;${MAIL_PREFIX}${subject}\u0026#34; \u0026#34;$to_addr\u0026#34; \u0026lt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; fi exit 1 } # ---- # MAIN # ---- ret=\u0026#34;0\u0026#34; # Check directories action_check_directories # Go to the base directory cd \u0026#34;$BASE_DIR\u0026#34; # Remove the old build dir if present if [ -d \u0026#34;$PUBLIC_DIR\u0026#34; ]; then rm -rf \u0026#34;$PUBLIC_DIR\u0026#34; fi # Update the repository checkout action_log \u0026#34;Updating the repository checkout\u0026#34; git fetch --all \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then action_log \u0026#34;Failed to update the repository checkout\u0026#34; mail_failure fi # Get it from the repo branch \u0026amp; extract it action_log \u0026#34;Downloading and extracting last site version using \u0026#39;git archive\u0026#39;\u0026#34; git archive --remote=\u0026#34;$REPO_URL\u0026#34; \u0026#34;$REPO_BRANCH\u0026#34; \u0026#34;$REPO_DIR\u0026#34; \\ | tar xf - \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; # Fail if public dir was missing if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ] || [ ! -d \u0026#34;$PUBLIC_DIR\u0026#34; ]; then action_log \u0026#34;Failed to download or extract site\u0026#34; mail_failure fi # Remove old public_html copies action_log \u0026#39;Removing old site versions, if present\u0026#39; find $NGINX_BASE_DIR -mindepth 1 -maxdepth 1 -name \u0026#39;public_html-*\u0026#39; -type d \\ -exec rm -rf {} \\; \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then action_log \u0026#34;Removal of old site versions failed\u0026#34; mail_failure fi # Switch site directory TS=\u0026#34;$(date +%Y%m%d-%H%M%S)\u0026#34; if [ -d \u0026#34;$PUBLIC_HTML_DIR\u0026#34; ]; then action_log \u0026#34;Moving \u0026#39;$PUBLIC_HTML_DIR\u0026#39; to \u0026#39;$PUBLIC_HTML_DIR-$TS\u0026#39;\u0026#34; mv \u0026#34;$PUBLIC_HTML_DIR\u0026#34; \u0026#34;$PUBLIC_HTML_DIR-$TS\u0026#34; \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; fi if [ \u0026#34;$ret\u0026#34; -eq \u0026#34;0\u0026#34; ]; then action_log \u0026#34;Moving \u0026#39;$PUBLIC_DIR\u0026#39; to \u0026#39;$PUBLIC_HTML_DIR\u0026#39;\u0026#34; mv \u0026#34;$PUBLIC_DIR\u0026#34; \u0026#34;$PUBLIC_HTML_DIR\u0026#34; \u0026gt;\u0026gt;\u0026#34;$ACTION_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; fi if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then action_log \u0026#34;Site switch failed\u0026#34; mail_failure else action_log \u0026#34;Site updated successfully\u0026#34; mail_success fi # ---- # vim: ts=2:sw=2:et:ai:sts=2 The hugo-adoc workflowThe workflow is defined in the .forgejo/workflows/hugo-adoc.yml file and looks like this:\nname: hugo-adoc # Run this job on push events to the main branch on: push: branches: - \u0026#39;main\u0026#39; jobs: build-and-push: if: ${{ vars.BLOGOPS_WEBHOOK_URL != \u0026#39;\u0026#39; \u0026amp;\u0026amp; secrets.BLOGOPS_TOKEN != \u0026#39;\u0026#39; }} runs-on: docker container: image: forgejo.mixinet.net/oci/hugo-adoc:latest # Allow the job to write to the repository (not really needed on forgejo) permissions: contents: write steps: - name: Checkout the repo uses: actions/checkout@v4 with: submodules: \u0026#39;true\u0026#39; - name: Build the site shell: sh run: | rm -rf public hugo - name: Push compiled site to html branch shell: sh run: | # Set the git user git config --global user.email \u0026#34;blogops@mixinet.net\u0026#34; git config --global user.name \u0026#34;BlogOps\u0026#34; # Create a new orphan branch called html (it was not pulled by the # checkout step) git switch --orphan html # Add the public directory to the branch git add public # Commit the changes git commit --quiet -m \u0026#34;Updated site @ $(date -R)\u0026#34; public # Push the changes to the html branch git push origin html --force # Switch back to the main branch git switch main - name: Call the blogops update webhook endpoint shell: sh run: | HEADER=\u0026#34;X-Blogops-Token: ${{ secrets.BLOGOPS_TOKEN }}\u0026#34; curl --fail -k -H \u0026#34;$HEADER\u0026#34; ${{ vars.BLOGOPS_WEBHOOK_URL }} The only relevant thing is that we have to add the BLOGOPS_TOKEN variable to the project secrets (its value is the one included on the /etc/webhook.yml file created when installing the webhook service) and the BLOGOPS_WEBHOOK_URL project variable (its value is the URL of the webhook server, in my case http://172.31.31.1:4444/hooks/update-blogops); note that the job includes the -k flag on the curl command just in case I end up using TLS on the webhook server in the future, as discussed previously.\nConclusionNow that I have forgejo actions on my server I no longer need to build the site on the public server as I did initially, a good thing when the server is a small OVH VPS that only runs a couple of containers and a web server directly on the host.\nI’m still using a notification system to make the server run a script to update the site because that way the forgejo server does not need access to the remote machine shell, only the webhook server which, IMHO, is a more secure setup.\n","permalink":"https://blogops.mixinet.net/posts/forgejo/using_actions_to_build_this_site/","summary":"\u003cp\u003eAs promised on my previous \u003ca href=\"https://blogops.mixinet.net/posts/forgejo/configuring_forgejo_actions/\" target=\"_blank\" rel=\"noopener\"\u003epost\u003c/a\u003e, on this entry I’ll explain how I’ve set up forgejo\nactions on the source repository of this site to build it using a runner instead of doing it on the public server using\na webhook to trigger the operation.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_setting_up_the_system\"\u003eSetting up the system\u003c/h2\u003e\u003cp\u003eThe first thing I’ve done is to disable the forgejo webhook call that was used to publish the site, as I don’t want to\nrun it anymore.\u003c/p\u003e\n\u003caside class=\"admonition-block note\" role=\"note\"\u003e\u003ch6 class=\"block-title label-only\"\u003e\u003cspan class=\"title-label\"\u003eNote: \u003c/span\u003e\u003c/h6\u003e\u003cp\u003eFor now I’ve just removed the \u003cem\u003eActive\u003c/em\u003e flag from the webhook, just in case I want to use it again in the future.\u003c/p\u003e\n\u003cp\u003eI’ve left the system based on the \u003ccode\u003ejson2file\u003c/code\u003e server running as it does nothing if no webhook is called, if we want\nto use it again it would be a good idea to disable actions to avoid conflicts if something is pushed, but it can be\nexecuted manually if needed and nothing will break, as both systems use the same directories to publish things).\u003c/p\u003e\u003c/aside\u003e\n\u003cp\u003eAfter that I added a new workflow to the repository that does the following things:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003ebuild the site using my \u003ca href=\"https://forgejo.mixinet.net/oci/images/src/branch/main/hugo-adoc/Dockerfile\" target=\"_blank\" rel=\"noopener\"\u003ehugo-adoc\u003c/a\u003e image.\u003c/li\u003e\u003cli\u003epush the result to a branch that contains the generated site (we do this because the server is already configured\nto work with the git repository and we can use force pushes to keep only the last version of the site, removing the\nneed of extra code to manage package uploads and removals).\u003c/li\u003e\u003cli\u003euses \u003ccode\u003ecurl\u003c/code\u003e to send a notification to an instance of the \u003ca href=\"https://github.com/adnanh/webhook\" target=\"_blank\" rel=\"noopener\"\u003ewebhook\u003c/a\u003e server installed on the\nremote server that triggers a script that updates the site using the git branch.\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Using actions to build this site"},{"content":"Last week I decided I wanted to try out forgejo actions to build this blog instead of using webhooks, so I looked the documentation and started playing with it until I had it working as I wanted.\nThis post is to describe how I’ve installed and configured a forgejo runner, how I’ve added an oci organization to my instance to build, publish and mirror container images and added a couple of additional organizations (actions and docker for now) to mirror interesting actions.\nThe changes made to build the site using actions will be documented on a separate post, as I’ll be using this entry to test the new setup on the blog project.\nInstalling the runnerThe first thing I’ve done is to install a runner on my server, I decided to use the OCI image installation method, as it seemed to be the easiest and fastest one.\nThe commands I’ve used to setup the runner are the following:\n$ cd /srv $ git clone https://forgejo.mixinet.net/blogops/forgejo-runner.git $ cd forgejo-runner $ sh ./bin/setup-runner.sh The setup-runner.sh script does multiple things:\ncreate a forgejo-runner user and groupcreate the necessary directories for the runnercreate a .runner file with a predefined secret and the docker label The setup-runner.sh code is available here.\nAfter running the script the runner has to be registered with the forgejo server, it can be done using the following command:\n$ forgejo forgejo-cli actions register --name \u0026#34;$RUNNER_NAME\u0026#34; \\ --secret \u0026#34;$FORGEJO_SECRET\u0026#34; The RUNNER_NAME variable is defined on the setup-runner.sh script and the FORGEJO_SECRET must match the value used on the .runner file.\nNote: The secret was pre-created on the setup-runner.sh script using openssl, but the runner can also be created using the forgejo server web interface; in that case the setup-runner.sh script would have to be modified to use the secret provided by the web interface.\nStarting it with docker-composeTo launch the runner I’m going to use a docker-compose.yml file that starts two containers, a docker in docker service to run the containers used by the workflow jobs and another one that runs the forgejo-runner itself.\nThe initial version used a TCP port to communicate with the dockerd server from the runner, but when I tried to build images from a workflow I noticed that the containers launched by the runner were not going to be able to execute another dockerd inside the dind one and, even if they were, it was going to be expensive computationally.\nTo avoid the issue I modified the dind service to use a unix socket on a shared volume that can be used by the runner service to communicate with the daemon and also re-shared with the job containers so the dockerd server can be used from them to build images.\nWarning: The use of the same docker server that runs the jobs from them has security implications, but this instance is for a home server where I am the only user, so I am not worried about it and this way I can save some resources (in fact, I could use the host docker server directly instead of using a dind service, but just in case I want to run other containers on the host I prefer to keep the one used for the runner isolated from it).\nFor those concerned about sharing the same server an alternative would be to launch a second dockerd only for the jobs (i.e. actions-dind) using the same approach (the volume with its socket will have to be shared with the runner service so it can be re-shared, but the runner does not need to use it).\nThe final docker-compose.yaml file is as follows:\nservices: dind: image: docker:dind container_name: \u0026#39;dind\u0026#39; privileged: \u0026#39;true\u0026#39; command: [\u0026#39;dockerd\u0026#39;, \u0026#39;-H\u0026#39;, \u0026#39;unix:///dind/docker.sock\u0026#39;, \u0026#39;-G\u0026#39;, \u0026#39;$RUNNER_GID\u0026#39;] restart: \u0026#39;unless-stopped\u0026#39; volumes: - ./dind:/dind runner: image: \u0026#39;data.forgejo.org/forgejo/runner:6.2.2\u0026#39; links: - dind depends_on: dind: condition: service_started container_name: \u0026#39;runner\u0026#39; environment: DOCKER_HOST: \u0026#39;unix:///dind/docker.sock\u0026#39; user: $RUNNER_UID:$RUNNER_GID volumes: - ./config.yaml:/config.yaml - ./data:/data - ./dind:/dind restart: \u0026#39;unless-stopped\u0026#39; command: \u0026#39;/bin/sh -c \u0026#34;sleep 5; forgejo-runner daemon -c /config.yaml\u0026#34;\u0026#39; There are multiple things to comment about this file:\nThe dockerd server is started with the -H unix:///dind/docker.sock flag to use the unix socket to communicate with the daemon instead of using a TCP port (as said, it is faster and allows us to share the socket with the containers started by the runner).We are running the dockerd daemon with the RUNNER_GID group so the runner can communicate with it (the socket gets that group which is the same used by the runner).The runner container mounts three volumes: the data directory, the dind folder where docker creates the unix socket and a config.yaml file used by us to change the default runner configuration. The config.yaml file was originally created using the forgejo-runner:\n$ docker run --rm data.forgejo.org/forgejo/runner:6.2.2 \\ forgejo-runner generate-config \u0026gt; config.yaml The changes to it are minimal, the runner capacity has been increased to 2 (that allows it to run two jobs at the same time) and the /dind/docker.sock value has been added to the valid_volumes key to allow the containers launched by the runner to mount it when needed; the diff against the default version is as follows:\n@@ -13,7 +13,8 @@ # Where to store the registration result. file: .runner # Execute how many tasks concurrently at the same time. - capacity: 1 + # STO: Allow 2 concurrent tasks + capacity: 2 # Extra environment variables to run jobs. envs: A_TEST_ENV_NAME_1: a_test_env_value_1 @@ -87,7 +88,9 @@ # If you want to allow any volume, please use the following configuration: # valid_volumes: # - \u0026#39;**\u0026#39; - valid_volumes: [] + # STO: Allow to mount the /dind/docker.sock on the containers + valid_volumes: + - /dind/docker.sock # overrides the docker client host with the specified one. # If \u0026#34;-\u0026#34; or \u0026#34;\u0026#34;, an available docker host will automatically be found. # If \u0026#34;automount\u0026#34;, an available docker host will automatically be found and ... To start the runner we export the RUNNER_UID and RUNNER_GID variables and call docker-compose up to start the containers on the background:\n$ RUNNER_UID=\u0026#34;$(id -u forgejo-runner)\u0026#34; RUNNER_GID=\u0026#34;$(id -g forgejo-runner)\u0026#34; \\ docker compose up -d If the server was configured right we are now able to start using actions with this runner.\nPreparing the system to run things locallyTo avoid unnecessary network traffic we are going to create a multiple organizations in our forgejo instance to maintain our own actions and container images and mirror remote ones.\nThe rationale behind the mirror use is that we reduce a lot the need to connect to remote servers to download the actions and images, which is good for performance and security reasons.\nIn fact, we are going to build our own images for some things to install the tools we want without needing to do it over and over again on the workflow jobs.\nMirrored actionsThe actions we are mirroring are on the actions and docker organizations, we have created the following ones for now (the mirrors were created using the forgejo web interface and we have disabled manually all the forgejo modules except the code one for them):\nactions/checkout: Action for checking out a repo.docker/login-action: Action to login against a Docker registry.docker/setup-buildx-action: Action to set up Docker Buildx.docker/build-push-action: Action to build and push Docker images with Buildx. To use our actions by default (i.e., without needing to add the server URL on the uses keyword) we have added the following section to the app.ini file of our forgejo server:\n[actions] ENABLED = true DEFAULT_ACTIONS_URL = https://forgejo.mixinet.net Setting up credentials to push imagesTo be able to push images to the oci organization I’ve created a token with package:write permission for my own user because I’m a member of the organization and I’m authorized to publish packages on it (a different user could be created, but as I said this is for personal use, so there is no need to complicate things for now).\nTo allow the use of those credentials on the actions I have added a secret (REGISTRY_PASS) and a variable (REGISTRY_USER) to the oci organization to allow the actions to use them.\nI’ve also logged myself on my local docker client to be able to push images to the oci group by hand, as I it is needed for bootstrapping the system (as I’m using local images on the worflows I need to push them to the server before running the ones that are used to build the images).\nLocal and mirrored imagesOur images will be stored on the packages section of a new organization called oci, inside it we have created two projects that use forgejo actions to keep things in shape:\nimages: contains the source files used to generate our own images and the actions to build, tag and push them to the oci organization group.mirrors: contains a configuration file for the regsync tool to mirror containers and an action to run it. On the next sections we are going to describe the actions and images we have created and mirrored from those projects.\nThe oci/images projectThe images project is a monorepo that contains the source files for the images we are going to build and a couple of actions.\nThe image sources are on sub directories of the repository, to be considered an image the folder has to contain a Dockerfile that will be used to build the image.\nThe repository has two workflows:\nbuild-image-from-tag: Workflow to build, tag and push an image to the oci organizationmulti-semantic-release: Workflow to create tags for the images using the multi-semantic-release tool. As the workflows are already configured to use some of our images we pushed some of them from a checkout of the repository using the following commands:\nregistry=\u0026#34;forgejo.mixinet.net/oci\u0026#34; for img in alpine-mixinet node-mixinet multi-semantic-release; do docker build -t $registry/$img:1.0.0 $img docker tag $registry/$img:1.0.0 $registry/$img:latest docker push $registry/$img:1.0.0 docker push $registry/$img:latest done On the next sub sections we will describe what the workflows do and will show their source code.\nbuild-image-from-tag workflowThis workflow uses a docker client to build an image from a tag on the repository with the format image-name-v[0-9].[0-9].[0-9]+.\nAs the runner is executed on a container (instead of using lxc) it seemed unreasonable to run another dind container from that one, that is why, after some tests, I decided to share the dind service server socket with the runner container and enabled the option to mount it also on the containers launched by the runner when needed (I only do it on the build-image-from-tag action for now).\nThe action was configured to run using a trigger or when new tags with the right format were created, but when the tag is created by multi-semantic-release the trigger does not work for some reason, so now it only runs the job on triggers and checks if it is launched for a tag with the right format on the job itself.\nThe source code of the action is as follows:\nname: build-image-from-tag on: workflow_dispatch: jobs: build: # Don\u0026#39;t build the image if the registry credentials are not set, the ref is not a tag or it doesn\u0026#39;t contain \u0026#39;-v\u0026#39; if: ${{ vars.REGISTRY_USER != \u0026#39;\u0026#39; \u0026amp;\u0026amp; secrets.REGISTRY_PASS != \u0026#39;\u0026#39; \u0026amp;\u0026amp; startsWith(github.ref, \u0026#39;refs/tags/\u0026#39;) \u0026amp;\u0026amp; contains(github.ref, \u0026#39;-v\u0026#39;) }} runs-on: docker container: image: forgejo.mixinet.net/oci/node-mixinet:latest # Mount the dind socket on the container at the default location options: -v /dind/docker.sock:/var/run/docker.sock steps: - name: Extract image name and tag from git and get registry name from env id: job_data run: | echo \u0026#34;::set-output name=img_name::${GITHUB_REF_NAME%%-v*}\u0026#34; echo \u0026#34;::set-output name=img_tag::${GITHUB_REF_NAME##*-v}\u0026#34; echo \u0026#34;::set-output name=registry::$( echo \u0026#34;${{ github.server_url }}\u0026#34; | sed -e \u0026#39;s%https://%%\u0026#39; )\u0026#34; echo \u0026#34;::set-output name=oci_registry_prefix::$( echo \u0026#34;${{ github.server_url }}/oci\u0026#34; | sed -e \u0026#39;s%https://%%\u0026#39; )\u0026#34; - name: Checkout the repo uses: actions/checkout@v4 - name: Export build dir and Dockerfile id: build_data run: | img=\u0026#34;${{ steps.job_data.outputs.img_name }}\u0026#34; build_dir=\u0026#34;$(pwd)/${img}\u0026#34; dockerfile=\u0026#34;${build_dir}/Dockerfile\u0026#34; if [ -f \u0026#34;$dockerfile\u0026#34; ]; then echo \u0026#34;::set-output name=build_dir::$build_dir\u0026#34; echo \u0026#34;::set-output name=dockerfile::$dockerfile\u0026#34; else echo \u0026#34;Couldn\u0026#39;t find the Dockerfile for the \u0026#39;$img\u0026#39; image\u0026#34; exit 1 fi - name: Login to the Container Registry uses: docker/login-action@v3 with: registry: ${{ steps.job_data.outputs.registry }} username: ${{ vars.REGISTRY_USER }} password: ${{ secrets.REGISTRY_PASS }} - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Build and Push uses: docker/build-push-action@v6 with: push: true tags: | ${{ steps.job_data.outputs.oci_registry_prefix }}/${{ steps.job_data.outputs.img_name }}:${{ steps.job_data.outputs.img_tag }} ${{ steps.job_data.outputs.oci_registry_prefix }}/${{ steps.job_data.outputs.img_name }}:latest context: ${{ steps.build_data.outputs.build_dir }} file: ${{ steps.build_data.outputs.dockerfile }} build-args: | OCI_REGISTRY_PREFIX=${{ steps.job_data.outputs.oci_registry_prefix }}/ Some notes about this code:\nThe if condition of the build job is not perfect, but it is good enough to avoid wrong uses as long as nobody uses manual tags with the wrong format and expects things to work (it checks if the REGISTRY_USER and REGISTRY_PASS variables are set, if the ref is a tag and if it contains the -v string).To be able to access the dind socket we mount it on the container using the options key on the container section of the job (this only works if supported by the runner configuration as explained before).We use the job_data step to get information about the image from the tag and the registry URL from the environment variables, it is executed first because all the information is available without checking out the repository.We use the job_data step to get the build dir and Dockerfile paths from the repository (right now we are assuming fixed paths and checking if the Dockerfile exists, but in the future we could use a configuration file to get them, if needed).As we are using a docker daemon that is already running there is no need to use the docker/setup-docker-action to install it.On the build and push step we pass the OCI_REGISTRY_PREFIX build argument to the Dockerfile to be able to use it on the FROM instruction (we are using it in our images). multi-semantic-release workflowThis workflow is used to run the multi-semantic-release tool on pushes to the main branch.\nIt is configured to create the configuration files on the fly (it prepares things to tag the folders that contain a Dockerfile using a couple of template files available on the repository’s .forgejo directory) and run the multi-semantic-release tool to create tags and push them to the repository if new versions are to be built.\nInitially we assumed that the tag creation pushed by multi-semantic-release would be enough to run the build-tagged-image-task action, but as it didn’t work we removed the rule to run the action on tag creation and added code to trigger the action using an api call for the newly created tags (we get them from the output of the multi-semantic-release execution).\nThe source code of the action is as follows:\nname: multi-semantic-release on: push: branches: - \u0026#39;main\u0026#39; jobs: multi-semantic-release: runs-on: docker container: image: forgejo.mixinet.net/oci/multi-semantic-release:latest steps: - name: Checkout the repo uses: actions/checkout@v4 - name: Generate multi-semantic-release configuration shell: sh run: | # Get the list of images to work with (the folders that have a Dockerfile) images=\u0026#34;$(for img in */Dockerfile; do dirname \u0026#34;$img\u0026#34;; done)\u0026#34; # Generate a values.yaml file for the main packages.json file package_json_values_yaml=\u0026#34;.package.json-values.yaml\u0026#34; echo \u0026#34;images:\u0026#34; \u0026gt;\u0026#34;$package_json_values_yaml\u0026#34; for img in $images; do echo \u0026#34; - $img\u0026#34; \u0026gt;\u0026gt;\u0026#34;$package_json_values_yaml\u0026#34; done echo \u0026#34;::group::Generated values.yaml for the project\u0026#34; cat \u0026#34;$package_json_values_yaml\u0026#34; echo \u0026#34;::endgroup::\u0026#34; # Generate the package.json file validating that is a good json file with jq tmpl -f \u0026#34;$package_json_values_yaml\u0026#34; \u0026#34;.forgejo/package.json.tmpl\u0026#34; | jq . \u0026gt; \u0026#34;package.json\u0026#34; echo \u0026#34;::group::Generated package.json for the project\u0026#34; cat \u0026#34;package.json\u0026#34; echo \u0026#34;::endgroup::\u0026#34; # Remove the temporary values file rm -f \u0026#34;$package_json_values_yaml\u0026#34; # Generate the package.json file for each image for img in $images; do tmpl -v \u0026#34;img_name=$img\u0026#34; -v \u0026#34;img_path=$img\u0026#34; \u0026#34;.forgejo/ws-package.json.tmpl\u0026#34; | jq . \u0026gt; \u0026#34;$img/package.json\u0026#34; echo \u0026#34;::group::Generated package.json for the \u0026#39;$img\u0026#39; image\u0026#34; cat \u0026#34;$img/package.json\u0026#34; echo \u0026#34;::endgroup::\u0026#34; done - name: Run multi-semantic-release shell: sh run: | multi-semantic-release | tee .multi-semantic-release.log - name: Trigger builds shell: sh run: | # Get the list of tags published on the previous steps tags=\u0026#34;$( sed -n -e \u0026#39;s/^\\[.*\\] \\[\\(.*\\)\\] .* Published release \\([0-9]\\+\\.[0-9]\\+\\.[0-9]\\+\\) on .*$/\\1-v\\2/p\u0026#39; \\ .multi-semantic-release.log )\u0026#34; rm -f .multi-semantic-release.log if [ \u0026#34;$tags\u0026#34; ]; then # Prepare the url for building the images workflow=\u0026#34;build-image-from-tag.yaml\u0026#34; dispatch_url=\u0026#34;${{ github.api_url }}/repos/${{ github.repository }}/actions/workflows/$workflow/dispatches\u0026#34; echo \u0026#34;$tags\u0026#34; | while read -r tag; do echo \u0026#34;Triggering build for tag \u0026#39;$tag\u0026#39;\u0026#34; curl \\ -H \u0026#34;Content-Type:application/json\u0026#34; \\ -H \u0026#34;Authorization: token ${{ secrets.GITHUB_TOKEN }}\u0026#34; \\ -d \u0026#34;{\\\u0026#34;ref\\\u0026#34;:\\\u0026#34;$tag\\\u0026#34;}\u0026#34; \u0026#34;$dispatch_url\u0026#34; done fi Notes about this code:\nThe use of the tmpl tool to process the multi-semantic-release configuration templates comes from previous uses, but on this case we could use a different approach (i.e. envsubst could be used) but we left it because it keeps things simple and can be useful in the future if we want to do more complex things with the template files.We use tee to show and dump to a file the output of the multi-semantic-release execution.We get the list of pushed tags using sed against the output of the multi-semantic-release execution and for each one found we use curl to call the forgejo API to trigger the build job; as the call is against the same project we can use the GITHUB_TOKEN generated for the workflow to do it, without creating a user token that has to be shared as a secret. The .forgejo/package.json.tmpl file is the following one:\n{ \u0026#34;name\u0026#34;: \u0026#34;multi-semantic-release\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;0.0.0-semantically-released\u0026#34;, \u0026#34;private\u0026#34;: true, \u0026#34;multi-release\u0026#34;: { \u0026#34;tagFormat\u0026#34;: \u0026#34;${name}-v${version}\u0026#34; }, \u0026#34;workspaces\u0026#34;: {{ .images | toJson }} } As can be seen it only needs a list of paths to the images as argument (the file we generate contains the names and paths, but it could be simplified).\nAnd the .forgejo/ws-package.json.tmpl file is the following one:\n{ \u0026#34;name\u0026#34;: \u0026#34;{{ .img_name }}\u0026#34;, \u0026#34;license\u0026#34;: \u0026#34;UNLICENSED\u0026#34;, \u0026#34;release\u0026#34;: { \u0026#34;plugins\u0026#34;: [ [ \u0026#34;@semantic-release/commit-analyzer\u0026#34;, { \u0026#34;preset\u0026#34;: \u0026#34;conventionalcommits\u0026#34;, \u0026#34;releaseRules\u0026#34;: [ { \u0026#34;breaking\u0026#34;: true, \u0026#34;release\u0026#34;: \u0026#34;major\u0026#34; }, { \u0026#34;revert\u0026#34;: true, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;feat\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;minor\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;fix\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;perf\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; } ] } ], [ \u0026#34;semantic-release-replace-plugin\u0026#34;, { \u0026#34;replacements\u0026#34;: [ { \u0026#34;files\u0026#34;: [ \u0026#34;{{ .img_path }}/msr.yaml\u0026#34; ], \u0026#34;from\u0026#34;: \u0026#34;^version:.*$\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;version: ${nextRelease.version}\u0026#34;, \u0026#34;allowEmptyPaths\u0026#34;: true } ] } ], [ \u0026#34;@semantic-release/git\u0026#34;, { \u0026#34;assets\u0026#34;: [ \u0026#34;msr.yaml\u0026#34; ], \u0026#34;message\u0026#34;: \u0026#34;ci(release): {{ .img_name }}-v${nextRelease.version}\\n\\n${nextRelease.notes}\u0026#34; } ] ], \u0026#34;branches\u0026#34;: [ \u0026#34;main\u0026#34; ] } } The oci/mirrors projectThe repository contains a template for the configuration file we are going to use with regsync (regsync.envsubst.yml) to mirror images from remote registries using a workflow that generates a configuration file from the template and runs the tool.\nThe initial version of the regsync.envsubst.yml file is prepared to mirror alpine containers from version 3.21 to 3.29 (we explicitly remove version 3.20) and needs the forgejo.mixinet.net/oci/node-mixinet:latest image to run (as explained before it was pushed manually to the server):\nversion: 1 creds: - registry: \u0026#34;$REGISTRY\u0026#34; user: \u0026#34;$REGISTRY_USER\u0026#34; pass: \u0026#34;$REGISTRY_PASS\u0026#34; sync: - source: alpine target: $REGISTRY/oci/alpine type: repository tags: allow: - \u0026#34;latest\u0026#34; - \u0026#34;3\\\\.2\\\\d+\u0026#34; - \u0026#34;3\\\\.2\\\\d+.\\\\d+\u0026#34; deny: - \u0026#34;3\\\\.20\u0026#34; - \u0026#34;3\\\\.20.\\\\d+\u0026#34; mirror workflowThe mirror workflow creates a configuration file replacing the value of the REGISTRY environment variable (computed by removing the protocol from the server_url), the REGISTRY_USER organization value and the REGISTRY_PASS secret using the envsubst command and running the regsync tool to mirror the images using the configuration file.\nThe action is configured to run daily, on push events when the regsync.envsubst.yml file is modified on the main branch and can also be triggered manually.\nThe source code of the action is as follows:\n.forgejo/workflows/mirror.yaml name: mirror on: schedule: - cron: \u0026#39;@daily\u0026#39; push: branches: - main paths: - \u0026#39;regsync.envsubst.yml\u0026#39; workflow_dispatch: jobs: mirror: if: ${{ vars.REGISTRY_USER != \u0026#39;\u0026#39; \u0026amp;\u0026amp; secrets.REGISTRY_PASS != \u0026#39;\u0026#39; }} runs-on: docker container: image: forgejo.mixinet.net/oci/node-mixinet:latest steps: - name: Checkout uses: actions/checkout@v4 - name: Sync images run: | REGISTRY=\u0026#34;$(echo \u0026#34;${{ github.server_url }}\u0026#34; | sed -e \u0026#39;s%https://%%\u0026#39;)\u0026#34; \\ REGISTRY_USER=\u0026#34;${{ vars.REGISTRY_USER }}\u0026#34; \\ REGISTRY_PASS=\u0026#34;${{ secrets.REGISTRY_PASS }}\u0026#34; \\ envsubst \u0026lt;regsync.envsubst.yml \u0026gt;.regsync.yml regsync --config .regsync.yml once rm -f .regsync.yml ConclusionWe have installed a forgejo-runner and configured it to run actions for our own server and things are working fine.\nThis approach allows us to have a powerful CI/CD system on a modest home server, something very useful for maintaining personal projects and playing with things without needing SaaS platforms like github or gitlab.\n","permalink":"https://blogops.mixinet.net/posts/forgejo/configuring_forgejo_actions/","summary":"\u003cp\u003eLast week I decided I wanted to try out \u003ca href=\"https://forgejo.org/docs/latest/admin/actions/\" target=\"_blank\" rel=\"noopener\"\u003eforgejo actions\u003c/a\u003e to build this blog instead of using\n\u003cem\u003ewebhooks\u003c/em\u003e, so I looked the documentation and started playing with it until I had it working as I wanted.\u003c/p\u003e\n\u003cp\u003eThis post is to describe how I’ve installed and configured a \u003ca href=\"https://forgejo.org/docs/latest/admin/runner-installation/\" target=\"_blank\" rel=\"noopener\"\u003eforgejo runner\u003c/a\u003e, how I’ve added an\n\u003ca href=\"https://forgejo.mixinet.net/oci/\" target=\"_blank\" rel=\"noopener\"\u003eoci\u003c/a\u003e organization to my instance to build, publish and mirror container images and added a couple of\nadditional organizations (\u003ca href=\"https://forgejo.mixinet.net/actions/\" target=\"_blank\" rel=\"noopener\"\u003eactions\u003c/a\u003e and \u003ca href=\"https://forgejo.mixinet.net/docker/\" target=\"_blank\" rel=\"noopener\"\u003edocker\u003c/a\u003e for now) to mirror interesting\nactions.\u003c/p\u003e\n\u003cp\u003eThe changes made to build the site using actions will be documented on a separate post, as I’ll be using this entry to\ntest the new setup on the blog project.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_installing_the_runner\"\u003eInstalling the runner\u003c/h2\u003e\u003cp\u003eThe first thing I’ve done is to install a runner on my server, I decided to use the\n\u003ca href=\"https://forgejo.org/docs/latest/admin/runner-installation/#oci-image-installation\"\u003eOCI image installation method\u003c/a\u003e, as it seemed to be the easiest and fastest\none.\u003c/p\u003e\n\u003cp\u003eThe commands I’ve used to setup the runner are the following:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003e\u003cspan class=\"nv\"\u003e$ \u003c/span\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e /srv\n\u003cspan class=\"nv\"\u003e$ \u003c/span\u003egit clone https://forgejo.mixinet.net/blogops/forgejo-runner.git\n\u003cspan class=\"nv\"\u003e$ \u003c/span\u003e\u003cspan class=\"nb\"\u003ecd \u003c/span\u003eforgejo-runner\n\u003cspan class=\"nv\"\u003e$ \u003c/span\u003esh ./bin/setup-runner.sh\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Configuring forgejo actions"},{"content":"With all the recent buzz about DeepSeek and its capabilities, I’ve decided to give it a try using Ollama and Open WebUI on my work laptop which has an NVIDIA GPU:\n$ lspci | grep NVIDIA 0000:01:00.0 3D controller: NVIDIA Corporation GA107GLM [RTX A2000 8GB Laptop GPU] (rev a1) For the installation I initially I looked into the approach suggested on this article, but after reviewing it I decided to go for a docker only approach, as it leaves my system clean and updates are easier.\nStep 0: Install dockerI already had it on my machine, so nothing to do here.\nStep 1: Install the nvidia-container-toolkit packageAs it is needed to use the NVIDIA GPU with docker I followed the instructions to install the package using apt from the NVIDIA website.\nStep 2: Run the Open WebUI container bundled with OllamaI could install ollama directly on linux or run it on docker, but I found out that there is a container with Open WebUI bundled with Ollama, so I decided to use it instead.\nTo start the container I’ve executed the following command:\ndocker run -d \\ -e OLLAMA_HOST=\u0026#34;0.0.0.0:11434\u0026#34; -p 127.0.0.1:11434:11434 \\ -p 127.0.0.1:3000:8080 \\ -v ollama:/root/.ollama \\ -v open-webui:/app/backend/data \\ --gpus=all --name open-webui --restart always \\ ghcr.io/open-webui/open-webui:ollama Once the container is up and running, I can access the Open WebUI on http://localhost:3000/ and the Ollama API on http://localhost:11434/ (I added the OLLAMA_HOST environment variable to be bind the server to all the container IPs, as it is needed to export it to the host).\nStep 3: Install DeepSeek-R1 model from Open WebUIAfter creating an admin user on the Open WebUI, I went to the select a model button and asked to download the deepseek-r1 model (at the time of writing this it matches version deepseek-r1:7b, which uses 4.7GB of space).\nAfter the download was completed I was able to start chatting with the model on my local machine.\nStep 4: Ask a questionThe first question I asked was Can you tell me what is Debian? and the answer I got was:\nDebian is one of the most popular Linux distributions used worldwide. It is an open-source operating system that has been around since 1993 and is known for its stability, security, and reliability. Debian is widely used in servers, data centers, and enterprise environments because it provides a robust base platform with excellent documentation, tools, and community support. ### Key Features of Debian: 1. **History**: - Debian started as a project called \u0026#34;Pop!!\u0026#34; (Pootle) led by Lars Knutsen, a developer from Denmark. - It was initially forked from the Linux distribution Red Hat\u0026#39;s RH-Debian to gain independence and reduce reliance on Red Hat. 2. **Use Cases**: - Debian is commonly used as an operating system for servers due to its reliability and ease of use in enterprise environments. - It is also used for individual user machines (laptops, desktops) because it offers a smooth and stable experience. - Debian provides the foundation for many popular derivatives like Ubuntu, which are designed for consumer use. 3. **Supported Architectures**: - Debian supports all major Linux architectures, including 32-bit and 64-bit systems, and runs on a wide range of hardware from old servers to modern smartphones. 4. **Package Management**: - Debian comes with a robust package management system called **Debian Policy Language (DPL)**, which allows users to easily install, update, and manage software packages. - This makes it easy for users to keep their systems up to date without needing to download multiple repositories or deal with version conflicts. 5. **Security Focus**: - Debian is known for its strong security practices. It has a reputation for being one of the most stable Linux distributions, making it a favorite among organizations that prioritize security. - The distribution regularly updates its software packages to patch vulnerabilities and protect against common attacks. 6. **Compatibility with Servers**: - Debian is widely used in data centers because it provides a consistent and reliable environment for running servers and applications. - Many open-source projects are compatible with Debian, making it an ideal choice for developers who want to work on servers or distributed systems. 7. **Popularity**: - Debian has a large and active community of users and contributors. - It is one of the most widely used Linux distributions in the world, with many enterprise organizations relying on it as their primary operating system. ### Comparison with Other Distributions: - **Ubuntu**: Ubuntu is based on Debian but adds graphical user interfaces (like Gutsy Gibbon), modern tools (like GIMP and LibreOffice), and integrates with Microsoft services. - **RHEL (Red Hat Enterprise Linux)**: RHEL is a derivative of Red Hat\u0026#39;s distribution, which forked from Debian. It focuses more heavily on enterprise security and support compared to Debian. ### Why Debian? Debian is popular because: - It is free and open-source, making it accessible to anyone. - It provides excellent documentation and tools for system administrators. - It has a strong focus on stability and reliability. - It supports a wide range of hardware and software needs. If you\u0026#39;re looking for a reliable and flexible Linux distribution that works well in both personal and enterprise environments, Debian is an excellent choice. Haven’t tried the same question with ChatGPT, but the History, Package Management and Comparison with Other Distributions sections are hilarious, but they provided me with important data:\nNow I know that APT is the real Debian Project Leader, what don’t know is why we have elections each year …​ ;)Linux Distributions are recursive …​ Debian was a fork of Red Hat, which was a fork of Debian, which was a fork of Red Hat, …​ As everybody is testing the model I will not talk more about the chat and the results, I just thought that this answer was really funny.\nStep 5: Install the DeepSeek Coder and DeepSeek Coder v2 models from Open WebUIAs done before, to download the models I went to the select a model button and asked to download the deepseek-coder and deepseek-coder-v2 models (the default version of version one is said to be really quick and small, while version two is supposed to be better but slower and bigger, so I decided to install both for testing).\nStep 6: Integrate Ollama with NeovimSince some months ago I’ve been using Github Copilot with Neovim; I don’t feel it has been very helpful in the general case, but I wanted to try it and it comes handy when you need to perform repetitive tasks when programming.\nIt seems that there are multiple neovim plugins that support ollama, for now I’ve installed and configured the codecompanion plugin on my config.lua file using packer:\nrequire(\u0026#39;packer\u0026#39;).startup(function() [...] -- Codecompanion plugin use { \u0026#34;olimorris/codecompanion.nvim\u0026#34;, requires = { \u0026#34;nvim-lua/plenary.nvim\u0026#34;, \u0026#34;nvim-treesitter/nvim-treesitter\u0026#34;, } } [...] end) [...] -- -------------------------------- -- BEG: Codecompanion configuration -- -------------------------------- -- Module setup local codecompanion = require(\u0026#39;codecompanion\u0026#39;).setup({ adapters = { ollama = function() return require(\u0026#39;codecompanion.adapters\u0026#39;).extend(\u0026#39;ollama\u0026#39;, { schema = { model = { default = \u0026#39;deepseek-coder-v2:latest\u0026#39;, } }, }) end, }, strategies = { chat = { adapter = \u0026#39;ollama\u0026#39;, }, inline = { adapter = \u0026#39;ollama\u0026#39;, }, }, }) -- -------------------------------- -- END: Codecompanion configuration -- -------------------------------- I’ve tested it a little bit and it seems to work fine, but I’ll have to test it more to see if it is really useful, I’ll try to do it on future projects.\nConclusionAt a personal level I don’t like nor trust AI systems, but as long as they are treated as tools and not as a magical thing you must trust they have their uses and I’m happy to see that open source tools like Ollama and models like DeepSeek available for everyone to use.\n","permalink":"https://blogops.mixinet.net/posts/testing_deepseek_with_ollama_and_open_webui/","summary":"\u003cp\u003eWith all the recent buzz about \u003ca href=\"https://www.deepseek.com/\" target=\"_blank\" rel=\"noopener\"\u003eDeepSeek\u003c/a\u003e and its capabilities, I’ve decided to give it a try\nusing \u003ca href=\"https://ollama.com/\" target=\"_blank\" rel=\"noopener\"\u003eOllama\u003c/a\u003e and \u003ca href=\"https://openwebui.com/\" target=\"_blank\" rel=\"noopener\"\u003eOpen WebUI\u003c/a\u003e on my work laptop which has an NVIDIA GPU:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"console\"\u003e\u003cspan class=\"gp\"\u003e$\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003elspci | \u003cspan class=\"nb\"\u003egrep \u003c/span\u003eNVIDIA\n\u003cspan class=\"go\"\u003e0000:01:00.0 3D controller: NVIDIA Corporation GA107GLM [RTX A2000 8GB Laptop GPU]\n             (rev a1)\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\u003cp\u003eFor the installation I initially I looked into the approach suggested on this\n\u003ca href=\"https://www.theregister.com/2025/01/26/deepseek_r1_ai_cot/\" target=\"_blank\" rel=\"noopener\"\u003earticle\u003c/a\u003e, but after reviewing it I decided to go for a\n\u003ccode\u003edocker\u003c/code\u003e only approach, as it leaves my system clean and updates are easier.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_step_0_install_docker\"\u003eStep 0: Install docker\u003c/h2\u003e\u003cp\u003eI already had it on my machine, so nothing to do here.\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_step_1_install_the_nvidia_container_toolkit_package\"\u003eStep 1: Install the nvidia-container-toolkit package\u003c/h2\u003e\u003cp\u003eAs it is needed to use the NVIDIA GPU with \u003ccode\u003edocker\u003c/code\u003e I followed the instructions to install the package using\n\u003ca href=\"https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-apt\" target=\"_blank\" rel=\"noopener\"\u003eapt\u003c/a\u003e\nfrom the NVIDIA website.\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_step_2_run_the_open_webui_container_bundled_with_ollama\"\u003eStep 2: Run the Open WebUI container bundled with Ollama\u003c/h2\u003e\u003cp\u003eI could install ollama directly on \u003ca href=\"https://github.com/ollama/ollama/blob/main/docs/linux.md\" target=\"_blank\" rel=\"noopener\"\u003elinux\u003c/a\u003e or run it on\n\u003ca href=\"https://github.com/ollama/ollama/blob/main/docs/docker.md\" target=\"_blank\" rel=\"noopener\"\u003edocker\u003c/a\u003e, but I found out that there is a container with\n\u003ca href=\"https://openwebui.com/\" target=\"_blank\" rel=\"noopener\"\u003eOpen WebUI\u003c/a\u003e bundled with \u003ca href=\"https://ollama.com/\" target=\"_blank\" rel=\"noopener\"\u003eOllama\u003c/a\u003e, so I decided to use\n\u003ca href=\"https://docs.openwebui.com/#open-webui-bundled-with-ollama\" target=\"_blank\" rel=\"noopener\"\u003eit\u003c/a\u003e instead.\u003c/p\u003e\n\u003cp\u003eTo start the container I’ve executed the following command:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003edocker run \u003cspan class=\"nt\"\u003e-d\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"nt\"\u003e-e\u003c/span\u003e \u003cspan class=\"nv\"\u003eOLLAMA_HOST\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;0.0.0.0:11434\u0026#34;\u003c/span\u003e \u003cspan class=\"nt\"\u003e-p\u003c/span\u003e 127.0.0.1:11434:11434 \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"nt\"\u003e-p\u003c/span\u003e 127.0.0.1:3000:8080 \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"nt\"\u003e-v\u003c/span\u003e ollama:/root/.ollama \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"nt\"\u003e-v\u003c/span\u003e open-webui:/app/backend/data \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"nt\"\u003e--gpus\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003eall \u003cspan class=\"nt\"\u003e--name\u003c/span\u003e open-webui \u003cspan class=\"nt\"\u003e--restart\u003c/span\u003e always \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  ghcr.io/open-webui/open-webui:ollama\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Testing DeepSeek with Ollama and Open WebUI"},{"content":"Although I am a Debian Developer (not very active, BTW) I am using Ubuntu LTS (right now version 24.04.1) on my main machine; it is my work laptop and I was told to keep using Ubuntu on it when it was assigned to me, although I don’t believe it is really necessary or justified (I don’t need support, I don’t provide support to others and I usually test my shell scripts on multiple systems if needed anyway).\nInitially I kept using Debian Sid on my personal laptop, but I gave it to my oldest son as the one he was using (an old Dell XPS 13) was stolen from him a year ago.\nI am still using Debian stable on my servers (one at home that also runs LXC containers and another one on an OVH VPS), but I don’t have a Debian Sid machine anymore and while I could reinstall my work machine, I’ve decided I’m going to try to use a system container to run Debian Sid on it.\nAs I want to use a container instead of a VM I’ve narrowed my options to lxc or systemd-nspawn (I have docker and podman installed, but I don’t believe they are good options for running system containers).\nAs I will want to take snapshots of the container filesystem I’ve decided to try incus instead of systemd-nspawn (I already have experience with it and while it works well it has less features than incus).\nInstalling incusAs this is a personal system where I want to try things, instead of using the packages included with Ubuntu I’ve decided to install the ones from the zabbly incus stable repository.\nTo do it I’ve executed the following as root:\n# Get the zabbly repository GPG key curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc # Create the zabbly-incus-stable.sources file sh -c \u0026#39;cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/apt/sources.list.d/zabbly-incus-stable.sources Enabled: yes Types: deb URIs: https://pkgs.zabbly.com/incus/stable Suites: $(. /etc/os-release \u0026amp;\u0026amp; echo ${VERSION_CODENAME}) Components: main Architectures: $(dpkg --print-architecture) Signed-By: /etc/apt/keyrings/zabbly.asc EOF\u0026#39; Initially I only plan to use the command line tools, so I’ve installed the incus and the incus-extra packages, but once things work I’ll probably install the incus-ui-canonical package too, at least for testing it:\napt update apt install incus incus-extra Adding my personal user to the incus-admin groupTo be able to run incus commands as my personal user I’ve added it to the incus-admin group:\nsudo adduser \u0026#34;$(id -un)\u0026#34; incus-admin And I’ve logged out and in again of my desktop session to make the changes effective.\nInitializing the incus environmentTo configure the incus environment I’ve executed the incus admin init command and accepted the defaults for all the questions, as they are good enough for my current use case.\nCreating a Debian containerTo create a Debian container I’ve used the default debian/trixie image:\nincus launch images:debian/trixie debian This command downloads the image and creates a container named debian using the default profile.\nThe exec command can be used to run a root login shell inside the container:\nincus exec debian -- su -l Instead of exec we can use the shell alias:\nincus shell debian which does the same as the previous command.\nInside that shell we can try to update the machine to sid changing the /etc/apt/sources.list file and using apt:\nroot@debian:~# echo \u0026#34;deb http://deb.debian.org/debian sid main contrib non-free\u0026#34; \\ \u0026gt;/etc/apt/sources.list root@debian:~# apt update root@debian:~# apt dist-upgrade As my machine has docker installed the apt update command fails because the network does not work, to fix it I’ve executed the commands of the following section and re-run the apt update and apt dist-upgrade commands.\nMaking the incusbr0 bridge work with DockerTo avoid problems with docker networking we have to add rules for the incusbr0 bridge to the DOCKER-USER chain as follows:\nsudo iptables -I DOCKER-USER -i incusbr0 -j ACCEPT sudo iptables -I DOCKER-USER -o incusbr0 -m conntrack \\ --ctstate RELATED,ESTABLISHED -j ACCEPT That makes things work now, but to make things persistent across reboots we need to add them each time the machine boots.\nAs suggested by the incus documentation I’ve installed the iptables-persistent package (my command also purges the ufw package, as I was not using it) and saved the current rules when installing:\nsudo apt install iptables-persistent --purge Integrating the DNS resolution of the container with the hostTo make DNS resolution for the ictus containers work from the host I’ve followed the incus documentation.\nTo set up things manually I’ve run the following:\nbr=\u0026#34;incusbr0\u0026#34;; br_ipv4=\u0026#34;$(incus network get \u0026#34;$br\u0026#34; ipv4.address)\u0026#34;; br_domain=\u0026#34;$(incus network get \u0026#34;$br\u0026#34; dns.domain)\u0026#34;; dns_address=\u0026#34;${br_ipv4%/*}\u0026#34;; dns_domain=\u0026#34;${br_domain:=incus}\u0026#34;; resolvectl dns \u0026#34;$br\u0026#34; \u0026#34;${dns_address}\u0026#34;; resolvectl domain \u0026#34;$br\u0026#34; \u0026#34;~${dns_domain}\u0026#34;; resolvectl dnssec \u0026#34;$br\u0026#34; off; resolvectl dnsovertls \u0026#34;$br\u0026#34; off; And to make the changes persistent across reboots I’ve created the following service file:\nsh -c \u0026#34;cat \u0026lt;\u0026lt;EOF | sudo tee /etc/systemd/system/incus-dns-${br}.service [Unit] Description=Incus per-link DNS configuration for ${br} BindsTo=sys-subsystem-net-devices-${br}.device After=sys-subsystem-net-devices-${br}.device [Service] Type=oneshot ExecStart=/usr/bin/resolvectl dns ${br} ${dns_address} ExecStart=/usr/bin/resolvectl domain ${br} ~${dns_domain} ExecStart=/usr/bin/resolvectl dnssec ${br} off ExecStart=/usr/bin/resolvectl dnsovertls ${br} off ExecStopPost=/usr/bin/resolvectl revert ${br} RemainAfterExit=yes [Install] WantedBy=sys-subsystem-net-devices-${br}.device EOF\u0026#34; And enabled it:\nsudo systemctl daemon-reload sudo systemctl enable --now incus-dns-${br}.service If all goes well the DNS resolution works from the host:\n$ host debian.incus debian.incus has address 10.149.225.121 debian.incus has IPv6 address fd42:1178:afd8:cc2c:216:3eff:fe2b:5cea Using my host user and home dir inside the containerTo use my host user and home directory inside the container I need to add the user and group to the container.\nFirst I’ve added my user group with the same GID used on the host:\nincus exec debian -- addgroup --gid \u0026#34;$(id --group)\u0026#34; --allow-bad-names \\ \u0026#34;$(id --group --name)\u0026#34; Once I have the group I’ve added the user with the same UID and GID as on the host, without defining a password for it:\nincus exec debian -- adduser --uid \u0026#34;$(id --user)\u0026#34; --gid \u0026#34;$(id --group)\u0026#34; \\ --comment \u0026#34;$(getent passwd \u0026#34;$(id --user -name)\u0026#34; | cut -d \u0026#39;:\u0026#39; -f 5)\u0026#34; \\ --no-create-home --disabled-password --allow-bad-names \\ \u0026#34;$(id --user --name)\u0026#34; Once the user is created we can mount the home directory on the container (we add the shift option to make the container use the same UID and GID as we do on the host):\nincus config device add debian home disk source=$HOME path=$HOME shift=true We have the shell alias to log with the root account, now we can add another one to log into the container using the newly created user:\nincus alias add ush \u0026#34;exec @ARGS@ -- su -l $(id --user --name)\u0026#34; To log into the container as our user now we just need to run:\nincus ush debian To be able to use sudo inside the container we could add our user to the sudo group:\nincus exec debian -- adduser \u0026#34;$(id --user --name)\u0026#34; \u0026#34;sudo\u0026#34; But that requires a password and we don’t have one, so instead we are going to add a file to the /etc/sudoers.d directory to allow our user to run sudo without a password:\nincus exec debian -- \\ sh -c \u0026#34;echo \u0026#39;$(id --user --name) ALL = NOPASSWD: ALL\u0026#39; /etc/sudoers.d/user\u0026#34; Accessing the container using sshTo use the container as a real machine and log into it as I do on remote machines I’ve installed the openssh-server and authorized my laptop public key to log into my laptop (as we are mounting the home directory from the host that allows us to log in without password from the local machine).\nAlso, to be able to run X11 applications from the container I’ve adusted the $HOME/.ssh/config file to always forward X11 (option ForwardX11 yes for Host debian.incus) and installed the xauth package.\nAfter that I can log into the container running the command ssh debian.incus and start using it after installing other interesting tools like neovim, rsync, tmux, etc.\nTaking snapshots of the containerAs this is a system container we can take snapshots of it using the incus snapshot command; that can be specially useful to take snapshots before doing a dist-upgrade so we can rollback if something goes wrong.\nTo work with container snapshots we run use the incus snapshot command, i.e. to create a snapshot we use de create subcommand:\nincus snapshot create debian The snapshot sub commands include options to list the available snapshots, restore a snapshot, delete a snapshot, etc.\nConclusionSince last week I have a terminal running a tmux session on the Debian Sid container with multiple zsh windows open (I’ve changed the prompt to be able to notice easily where I am) and it is working as expected.\nMy plan now is to add some packages and use the container for personal projects so I can work on a Debian Sid system without having to reinstall my work machine.\nI’ll probably write more about it in the future, but for now, I’m happy with the results.\n","permalink":"https://blogops.mixinet.net/posts/incus/","summary":"\u003cp\u003eAlthough I am a Debian Developer (not very active, BTW) I am using Ubuntu LTS (right now version 24.04.1) on my main\nmachine; it is my work laptop and I was told to keep using Ubuntu on it when it was assigned to me, although I don’t\nbelieve it is really necessary or justified (I don’t need support, I don’t provide support to others and I usually test\nmy shell scripts on multiple systems if needed anyway).\u003c/p\u003e\n\u003cp\u003eInitially I kept using Debian Sid on my personal laptop, but I gave it to my oldest son as the one he was using (an old\nDell XPS 13) was stolen from him a year ago.\u003c/p\u003e\n\u003cp\u003eI am still using Debian stable on my servers (one at home that also runs LXC containers and another one on an OVH VPS),\nbut I don’t have a Debian Sid machine anymore and while I could reinstall my work machine, I’ve decided I’m going to try\nto use a system container to run Debian Sid on it.\u003c/p\u003e\n\u003cp\u003eAs I want to use a container instead of a VM I’ve narrowed my options to \u003ccode\u003elxc\u003c/code\u003e or \u003ccode\u003esystemd-nspawn\u003c/code\u003e (I have \u003ccode\u003edocker\u003c/code\u003e and\n\u003ccode\u003epodman\u003c/code\u003e installed, but I don’t believe they are good options for running system containers).\u003c/p\u003e\n\u003cp\u003eAs I will want to take snapshots of the container filesystem I’ve decided to try\n\u003ca href=\"https://linuxcontainers.org/incus/\" target=\"_blank\" rel=\"noopener\"\u003eincus\u003c/a\u003e instead of \u003ca href=\"https://wiki.debian.org/nspawn\" target=\"_blank\" rel=\"noopener\"\u003esystemd-nspawn\u003c/a\u003e (I already have\nexperience with it and while it works well it has less features than \u003ccode\u003eincus\u003c/code\u003e).\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_installing_incus\"\u003eInstalling incus\u003c/h2\u003e\u003cp\u003eAs this is a personal system where I want to try things, instead of using the packages included with Ubuntu I’ve decided\nto install the ones from the \u003ca href=\"https://github.com/zabbly/incus?tab=readme-ov-file#stable-repository\" target=\"_blank\" rel=\"noopener\"\u003ezabbly incus stable\nrepository\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eTo do it I’ve executed the following as \u003ccode\u003eroot\u003c/code\u003e:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003e\u003cspan class=\"c\"\u003e# Get the zabbly repository GPG key\u003c/span\u003e\ncurl \u003cspan class=\"nt\"\u003e-fsSL\u003c/span\u003e https://pkgs.zabbly.com/key.asc \u003cspan class=\"nt\"\u003e-o\u003c/span\u003e /etc/apt/keyrings/zabbly.asc\n\u003cspan class=\"c\"\u003e# Create the zabbly-incus-stable.sources file\u003c/span\u003e\nsh \u003cspan class=\"nt\"\u003e-c\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/apt/sources.list.d/zabbly-incus-stable.sources\nEnabled: yes\nTypes: deb\nURIs: https://pkgs.zabbly.com/incus/stable\nSuites: $(. /etc/os-release \u0026amp;\u0026amp; echo ${VERSION_CODENAME})\nComponents: main\nArchitectures: $(dpkg --print-architecture)\nSigned-By: /etc/apt/keyrings/zabbly.asc\nEOF\u0026#39;\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Running a Debian Sid on Ubuntu"},{"content":"For a long time I’ve been using the Terminator terminal emulator on Linux machines, but last week I read a LWN article about a new emulator called Ghostty that looked interesting and I decided to give it a try.\nThe author sells it as a fast, feature-rich and cross-platform terminal emulator that follows the zero configuration philosophy.\nInstallation and configurationI installed the debian package for Ubuntu 24.04 from the ghostty-ubuntu project and started playing with it.\nThe first thing I noticed is that the zero configuration part is true; I was able to use the terminal without a configuration file, although I created one to change the theme and the font size, but other than that it worked OK for me; my $HOME/.config/ghostty/config file is as simple as:\nfont-size=14 theme=/usr/share/ghostty/themes/iTerm2 Solarized Light Starting the terminal maximizedAfter playing a little bit with the terminal I was turned off by the fact that there was no option to start it maximized, but is seemed to me that someone should have asked for the feature, or, if not, I could ask for it.\nI did a quick search on the project and I found out that there was a merged PR that added the option, so I downloaded the source code, installed Zig and built the program on my machine.\nAs the change is going to be included on the next version on the package I replaced the binary with my version and started playing with the terminal.\nAccessing remote machinesThe first thing I noticed was that when logging into remote machines using ssh the terminal variable was not known, but on the help section of the project documentation there was an entry about how to fix it copying the terminfo configuration to remote machines, it is as simple as running the following:\ninfocmp -x | ssh YOUR-SERVER -- tic -x - Dead keys on UbuntuWith that sorted out everything looked good to me until I tried to add an accented character when editing a file and the terminal stopped working.\nAgain, I looked at the project issues and found one that matched what was happening to me, and it remembered me about one of the best things about actively maintained open source software.\nIt turns out that the issue is related to a bug on ibus, but other terminals were working right, so the ghostty developer was already working on a fix on the way the terminal handles the keyboard input on GTK, so I subscribed to the issue and stopped using ghostty until there was something new to try again (I use an Spanish keyboard map and I can’t use a terminal that does not support dead keys).\nYesterday I saw some messages about things being almost fixed, so I pulled the latest changes on my cloned repository, compiled it and writing accented characters works now, there is a small issue with the cursor (the dead key pressed is left on the block cursor unless you change the window focus), but that is something manageable for me.\nConclusionI think that ghostty is a good terminal emulator and I’m going to keep using it on my laptop unless I find something annoying that I can’t work with (i hope that the cursor issue will be fixed soon and I can live with it as the only thing I need to do to recover from it is changing the widow focus, and that can be done really quickly using keyboard shortcuts).\nAs it is actively maintained and the developer seems to be quite active I don’t expect problems and is nice to play with new things from time to time.\n","permalink":"https://blogops.mixinet.net/posts/user_tools/ghostty_terminal_emulator/","summary":"\u003cp\u003eFor a long time I’ve been using the \u003ca href=\"https://gnome-terminator.org/\" target=\"_blank\" rel=\"noopener\"\u003eTerminator\u003c/a\u003e terminal emulator on Linux machines, but\nlast week I read a \u003ca href=\"https://lwn.net/Articles/1004377/\" target=\"_blank\" rel=\"noopener\"\u003eLWN article\u003c/a\u003e about a new emulator called\n\u003ca href=\"https://ghostty.org/\" target=\"_blank\" rel=\"noopener\"\u003eGhostty\u003c/a\u003e that looked interesting and I decided to give it a try.\u003c/p\u003e\n\u003cp\u003eThe author sells it as a fast, feature-rich and cross-platform terminal emulator that follows the \u003cem\u003ezero configuration\nphilosophy\u003c/em\u003e.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_installation_and_configuration\"\u003eInstallation and configuration\u003c/h2\u003e\u003cp\u003eI installed the debian package for Ubuntu 24.04 from the \u003ca href=\"https://github.com/mkasberg/ghostty-ubuntu\" target=\"_blank\" rel=\"noopener\"\u003eghostty-ubuntu\nproject\u003c/a\u003e and started playing with it.\u003c/p\u003e\n\u003cp\u003eThe first thing I noticed is that the zero configuration part is true; I was able to use the terminal without a\nconfiguration file, although I created one to change the theme and the font size, but other than that it worked OK for\nme; my \u003ccode\u003e$HOME/.config/ghostty/config\u003c/code\u003e file is as simple as:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"toml\"\u003e\u003cspan class=\"py\"\u003efont-size\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"mi\"\u003e14\u003c/span\u003e\n\u003cspan class=\"py\"\u003etheme\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"err\"\u003e/usr/share/ghostty/themes/iTerm\u003c/span\u003e\u003cspan class=\"mi\"\u003e2\u003c/span\u003e \u003cspan class=\"err\"\u003eSolarized\u003c/span\u003e \u003cspan class=\"err\"\u003eLight\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Ghostty Terminal Emulator"},{"content":"I’ve always been a fan of template engines that work with text files, mainly to work with static site generators, but also to generate code, configuration files, and other text-based files.\nFor my own web projects I used to go with Jinja2, as all my projects were written in Python, while for static web sites I used the template engines included with the tools I was using, i.e. Liquid with Jekyll and Go Templates (based on the text/template and the html/template go packages) for Hugo.\nWhen I needed to generate code snippets or configuration files from shell scripts I used to go with sed and/or envsubst, but lately things got complicated and I started to use a command line application called tmpl that uses the Go Template Language with functions from the Sprig library.\ntmplI’ve been using my fork of the tmpl program to process templates on CI/CD pipelines (gitlab-ci) to generate configuration files and code snippets because it uses the same syntax used by helm (easier to use by other DevOps already familiar with the format) and the binary is small and can be easily included into the docker images used by the pipeline jobs.\nOne interesting feature of the tmpl tool is that it can read values from command line arguments and from multiple files in different formats (YAML, JSON, TOML, etc) and merge them into a single object that can be used to render the templates.\nThere are alternatives to the tmpl tool and I’ve looked at them (i.e. simple ones like go-template-cli or complex ones like gomplate), but I haven’t found one that fits my needs.\nFor my next project I plan to evaluate a move to a different tool or template format, as tmpl is not being actively maintained (as I said, I’m using my own fork) and it is not included on existing GNU/Linux distributions (I packaged it for Debian and Alpine, but I don’t want to maintain something like that without an active community and I’m not interested in being the upstream myself, as I’m trying to move to Rust instead of Go as the compiled programming language for my projects).\nMini JinjaLooking for alternate tools to process templates on the command line I found the minijinja rust crate, a minimal implementation of the Jinja2 template engine that also includes a small command line utility (minijinja-cli) and I believe I’ll give it a try on the future for various reasons:\nI’m already familiar with the Jinja2 syntax and it is widely used on the industry.On my code I can use the original Jinja2 module for Python projects and MiniJinja for Rust programs.The included command line utility is small and easy to use, and the binaries distributed by the project are good enough to add them to the docker container images used by CI/CD pipelines.As I want to move to Rust I can try to add functionalities to the existing command line client or create my own version of it if they are needed (don’t think so, but who knows). ","permalink":"https://blogops.mixinet.net/posts/user_tools/cli_template_tools/","summary":"\u003cp\u003eI’ve always been a fan of template engines that work with text files, mainly to work with static site generators, but\nalso to generate code, configuration files, and other text-based files.\u003c/p\u003e\n\u003cp\u003eFor my own web projects I used to go with \u003ca href=\"https://jinja.palletsprojects.com/\" target=\"_blank\" rel=\"noopener\"\u003eJinja2\u003c/a\u003e, as all my projects were written\nin \u003ca href=\"https://python.org/\" target=\"_blank\" rel=\"noopener\"\u003ePython\u003c/a\u003e, while for static web sites I used the template engines included with the tools I was\nusing, i.e. \u003ca href=\"https://shopify.github.io/liquid/\" target=\"_blank\" rel=\"noopener\"\u003eLiquid\u003c/a\u003e with \u003ca href=\"https://jekyllrb.com/\" target=\"_blank\" rel=\"noopener\"\u003eJekyll\u003c/a\u003e and\n\u003ca href=\"https://gohugo.io/templates/introduction/\" target=\"_blank\" rel=\"noopener\"\u003eGo Templates\u003c/a\u003e (based on the \u003ca href=\"https://pkg.go.dev/text/template\" target=\"_blank\" rel=\"noopener\"\u003etext/template\u003c/a\u003e\nand the \u003ca href=\"https://pkg.go.dev/html/template\" target=\"_blank\" rel=\"noopener\"\u003ehtml/template\u003c/a\u003e go packages) for \u003ca href=\"https://gohugo.io/\" target=\"_blank\" rel=\"noopener\"\u003eHugo\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhen I needed to generate code snippets or configuration files from shell scripts I used to go with\n\u003ca href=\"https://man7.org/linux/man-pages/man1/sed.1.html\" target=\"_blank\" rel=\"noopener\"\u003esed\u003c/a\u003e and/or\n\u003ca href=\"https://man7.org/linux/man-pages/man1/envsubst.1.html\" target=\"_blank\" rel=\"noopener\"\u003eenvsubst\u003c/a\u003e, but lately things got complicated and I started to use\na command line application called \u003ca href=\"https://github.com/sto/tmpl\" target=\"_blank\" rel=\"noopener\"\u003etmpl\u003c/a\u003e that uses the \u003ca href=\"https://pkg.go.dev/text/template\" target=\"_blank\" rel=\"noopener\"\u003eGo\nTemplate Language\u003c/a\u003e with functions from the \u003ca href=\"https://masterminds.github.io/sprig/\" target=\"_blank\" rel=\"noopener\"\u003eSprig\u003c/a\u003e library.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_tmpl\"\u003etmpl\u003c/h2\u003e\u003cp\u003eI’ve been using my fork of the \u003ca href=\"https://github.com/sto/tmpl\" target=\"_blank\" rel=\"noopener\"\u003etmpl\u003c/a\u003e program to process templates on CI/CD pipelines\n(\u003ccode\u003egitlab-ci\u003c/code\u003e) to generate configuration files and code snippets because it uses the same syntax used by\n\u003ca href=\"https://helm.sh\" target=\"_blank\" rel=\"noopener\"\u003ehelm\u003c/a\u003e (easier to use by other \u003cstrong\u003eDevOps\u003c/strong\u003e already familiar with the format) and the binary is small and\ncan be easily included into the docker images used by the pipeline jobs.\u003c/p\u003e\n\u003cp\u003eOne interesting feature of the \u003ccode\u003etmpl\u003c/code\u003e tool is that it can read values from command line arguments and from multiple\nfiles in different formats (YAML, JSON, TOML, etc) and merge them into a single object that can be used to render the\ntemplates.\u003c/p\u003e\n\u003cp\u003eThere are alternatives to the \u003ccode\u003etmpl\u003c/code\u003e tool and I’ve looked at them (i.e. simple ones like\n\u003ca href=\"https://github.com/bluebrown/go-template-cli\" target=\"_blank\" rel=\"noopener\"\u003ego-template-cli\u003c/a\u003e or complex ones like\n\u003ca href=\"https://docs.gomplate.ca/\"\u003egomplate\u003c/a\u003e), but I haven’t found one that fits my needs.\u003c/p\u003e\n\u003cp\u003eFor my next project I plan to evaluate a move to a different tool or template format, as \u003ccode\u003etmpl\u003c/code\u003e is not being actively\nmaintained (as I said, I’m using my own fork) and it is not included on existing GNU/Linux distributions (I packaged it\nfor \u003ccode\u003eDebian\u003c/code\u003e and \u003ccode\u003eAlpine\u003c/code\u003e, but I don’t want to maintain something like that without an active community and I’m not\ninterested in being the upstream myself, as I’m trying to move to \u003ca href=\"https://www.rust-lang.org/\" target=\"_blank\" rel=\"noopener\"\u003eRust\u003c/a\u003e instead of\n\u003ca href=\"https://go.dev/\" target=\"_blank\" rel=\"noopener\"\u003eGo\u003c/a\u003e as the compiled programming language for my projects).\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_mini_jinja\"\u003eMini Jinja\u003c/h2\u003e\u003cp\u003eLooking for alternate tools to process templates on the command line I found the \u003ca href=\"https://docs.rs/minijinja/\" target=\"_blank\" rel=\"noopener\"\u003eminijinja\u003c/a\u003e\nrust crate, a minimal implementation of the \u003cstrong\u003eJinja2\u003c/strong\u003e template engine that also includes a small command line utility\n(\u003ca href=\"https://docs.rs/crate/minijinja-cli/\" target=\"_blank\" rel=\"noopener\"\u003eminijinja-cli\u003c/a\u003e) and I believe I’ll give it a try on the future for various\nreasons:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003eI’m already familiar with the \u003cstrong\u003eJinja2\u003c/strong\u003e syntax and it is widely used on the industry.\u003c/li\u003e\u003cli\u003eOn my code I can use the original \u003cstrong\u003eJinja2\u003c/strong\u003e module for \u003cstrong\u003ePython\u003c/strong\u003e projects and \u003cstrong\u003eMiniJinja\u003c/strong\u003e for \u003cstrong\u003eRust\u003c/strong\u003e programs.\u003c/li\u003e\u003cli\u003eThe included command line utility is small and easy to use, and the binaries distributed by the project are good\nenough to add them to the docker container images used by CI/CD pipelines.\u003c/li\u003e\u003cli\u003eAs I want to move to \u003cstrong\u003eRust\u003c/strong\u003e I can try to add functionalities to the existing command line client or create my own\nversion of it if they are needed (don’t think so, but who knows).\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Command line tools to process templates"},{"content":"On recent weeks I’ve had some time to scratch my own itch on matters related to tools I use daily on my computer, namely the desktop / window manager and my text editor of choice.\nThis post is a summary of what I tried, how it worked out and my short and medium-term plans related to them.\nDesktop / WMOn the desktop / window manager front I’ve been using Cinnamon on Debian and Ubuntu systems since Gnome 3 was published (I never liked version 3, so I decided to move to something similar to Gnome 2, including the keyboard shortcuts).\nIn fact I’ve never been a fan of Desktop environments, before Gnome I used OpenBox and IceWM because they where a lot faster than desktop systems on my hardware at the time and I was using them only to place one or two windows on multiple workspaces using mainly the keyboard for my interactions (well, except for the web browsers and the image manipulation programs).\nAlthough I was comfortable using Cinnamon, some years ago I tried to move to i3, a tilling window manager for X11 that looked like a good choice for me, but I didn’t have much time to play with it and never used it enough to make me productive with it (I didn’t prepare a complete configuration nor had enough time to learn the new shortcuts, so I went back to Cinnamon and never tried again).\nAnyway, some weeks ago I updated my work machine OS (it was using Ubuntu 22.04 LTS and I updated it to the 24.04 LTS version) and the Cinnamon systray applet stopped working as it used to do (in fact I still have to restart Cinnamon after starting a session to make it work) and, as I had some time, I decided to try a tilling window manager again, but now I decided to go for SwayWM, as it uses Wayland instead of X11.\nSway configurationOn my ~/.config/sway/config I tuned some things:\nSet fuzzel as the application launcher.Installed manually the shikane application and created a configuration to be executed always when sway is started / reloaded (I adjusted my configuration with wdisplays and used shikanectl to save it).Added support for running the xdg-desktop-portal-wlr service.Enabled the swayidle command to lock the screen after some time of inactivity.Adjusted the keyboard to use the es key mapAdded some keybindings to make my life easier, including the use of grimm and swappy to take screenshotsConfigured waybar as the environment bar.Added a shell script to start applications when sway is started (it uses swaymsg to execute background commands and the i3toolwait script to wait for the\n#!/bin/sh # VARIABLES CHROMIUM_LOCAL_STATE=\u0026#34;$HOME/.config/google-chrome/Local State\u0026#34; I3_TOOLWAIT=\u0026#34;$HOME/.config/sway/scripts/i3-toolwait\u0026#34; # Functions chromium_profile_dir() { jq -r \u0026#34;.profile.info_cache|to_entries|map({(.value.name): .key})|add|.\\\u0026#34;$1\\\u0026#34; // \\\u0026#34;\\\u0026#34;\u0026#34; \u0026#34;$CHROMIUM_LOCAL_STATE\u0026#34; } # MAIN IGZ_PROFILE_DIR=\u0026#34;$(chromium_profile_dir \u0026#34;sergio.talens@intelygenz.com\u0026#34;)\u0026#34; OURO_PROFILE_DIR=\u0026#34;$(chromium_profile_dir \u0026#34;sergio.talens@nxr.global\u0026#34;)\u0026#34; PERSONAL_PROFILE_DIR=\u0026#34;$(chromium_profile_dir \u0026#34;stalens@gmail.com\u0026#34;)\u0026#34; # Common programs swaymsg \u0026#34;exec nextcloud --background\u0026#34; swaymsg \u0026#34;exec nm-applet\u0026#34; # Run spotify on the first workspace (it is mapped to the laptop screen) swaymsg -q \u0026#34;workspace 1\u0026#34; ${I3_TOOLWAIT} \u0026#34;spotify\u0026#34; # Run tmux on the swaymsg -q \u0026#34;workspace 2\u0026#34; ${I3_TOOLWAIT} -- foot tmux a -dt sto wp_num=\u0026#34;3\u0026#34; if [ \u0026#34;$OURO_PROFILE_DIR\u0026#34; ]; then swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} -m ouro-browser -- google-chrome --profile-directory=\u0026#34;$OURO_PROFILE_DIR\u0026#34; wp_num=\u0026#34;$((wp_num+1))\u0026#34; fi if [ \u0026#34;$IGZ_PROFILE_DIR\u0026#34; ]; then swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} -m igz-browser -- google-chrome --profile-directory=\u0026#34;$IGZ_PROFILE_DIR\u0026#34; wp_num=\u0026#34;$((wp_num+1))\u0026#34; fi if [ \u0026#34;$PERSONAL_PROFILE_DIR\u0026#34; ]; then swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} -m personal-browser -- google-chrome --profile-directory=\u0026#34;$PERSONAL_PROFILE_DIR\u0026#34; wp_num=\u0026#34;$((wp_num+1))\u0026#34; fi # Open the browser without setting the profile directory if none was found if [ \u0026#34;$wp_num\u0026#34; = \u0026#34;3\u0026#34; ]; then swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} google-chrome wp_num=\u0026#34;$((wp_num+1))\u0026#34; fi swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} evolution wp_num=\u0026#34;$((wp_num+1))\u0026#34; swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} slack wp_num=\u0026#34;$((wp_num+1))\u0026#34; # Open a private browser and a console in the last workspace swaymsg -q \u0026#34;workspace $wp_num\u0026#34; ${I3_TOOLWAIT} -- google-chrome --incognito ${I3_TOOLWAIT} foot # Go back to the second workspace for keepassxc swaymsg \u0026#34;workspace 2\u0026#34; ${I3_TOOLWAIT} keepassxc ConclusionAfter using Sway for some days I can confirm that it is a good choice for me, but some of the components needed to make it work as I want are too new and not available on the Ubuntu 24.04 LTS repositories, so I decided to go back to Cinnamon and try Sway again in the future, although I added more workspaces to my setup (now they are only available on the main monitor, the laptop screen is fixed while there is a big monitor connected), added some additional keyboard shortcuts and installed or updated some applets.\nText editorWhen I started using Linux many years ago I used vi/vim and emacs as my text editors (vi for plain text and emacs for programming and editing HTML/XML), but eventually I moved to vim as my main text editor and I’ve been using it since (well, I moved to neovim some time ago, although I kept my old vim configuration).\nTo be fair I’m not as expert as I could be with vim, but I’m productive with it and it has many plugins that make my life easier on my machines, while keeping my ability to edit text and configurations on any system that has a vi compatible editor installed.\nFor work reasons I tried to use Visual Studio Code last year, but I’ve never really liked it and almost everything I do with it I can do with neovim (i. e. I even use copilot with it). Besides, I’m a heavy terminal user (I use tmux locally and via ssh) and I like to be able to use my text editor on my shell sessions, and code does not work like that.\nThe only annoying thing about vim/neovim is its configuration (well, the problem is that I have a very old one and probably should spend some time fixing and updating it), but, as I said, it’s been working well for me for a long time, so I never really had the motivation to do it.\nAnyway, after finishing my desktop tests I saw that I had the Helix editor installed for some time but I never tried it, so I decided to give it a try and see if it could be a good replacement for neovim on my environments (the only drawback is that as it is not vi compatible, I would need to switch back to vi mode when working on remote systems, but I guess I could live with that).\nI ran the helix tutorial and I liked it, so I decided to configure and install the Language Servers I can probably take advantage of on my daily work on my personal and work machines and see how it works.\nLanguage server installationsA lot of manual installations are needed to get the language servers working what I did on my machines is more or less the following:\n# AWK sudo npm i -g \u0026#39;awk-language-server@\u0026gt;=0.5.2\u0026#39; # BASH sudo apt-get install shellcheck shfmt sudo npm i -g bash-language-server # C/C++ sudo apt-get install clangd # CSS, HTML, ESLint, JSON, SCS sudo npm i -g vscode-langservers-extracted # Docker sudo npm install -g dockerfile-language-server-nodejs # Docker compose sudo npm install -g @microsoft/compose-language-service # Helm app=\u0026#34;helm_ls_linux_amd64\u0026#34; url=\u0026#34;$( curl -s https://api.github.com/repos/mrjosh/helm-ls/releases/latest | jq -r \u0026#34;.assets[] | select(.name == \\\u0026#34;$app\\\u0026#34;) | .browser_download_url\u0026#34; )\u0026#34; curl -L \u0026#34;$url\u0026#34; --output /tmp/helm_ls sudo install /tmp/helm_ls /usr/local/bin rm /tmp/helm_ls # Markdown app=\u0026#34;marksman-linux-x64\u0026#34; url=\u0026#34;$( curl -s https://api.github.com/repos/artempyanykh/marksman/releases/latest | jq -r \u0026#34;.assets[] | select(.name == \\\u0026#34;$app\\\u0026#34;) | .browser_download_url\u0026#34; )\u0026#34; curl -L \u0026#34;$url\u0026#34; --output /tmp/marksman sudo install /tmp/marksman /usr/local/bin rm /tmp/marksman # Python sudo npm i -g pyright # Rust rustup component add rust-analyzer # SQL sudo npm i -g sql-language-server # Terraform sudo apt-get install terraform-ls # TOML cargo install taplo-cli --locked --features lsp # YAML sudo npm install --global yaml-language-server # JavaScript, TypeScript sudo npm install -g typescript-language-server typescript sudo npm install -g --save-dev --save-exact @biomejs/biome Helix configurationThe helix configuration is done on a couple of toml files that are placed on the ~/.config/helix directory, the config.toml file I used is this one:\ntheme = \u0026#34;solarized_light\u0026#34; [editor] line-number = \u0026#34;relative\u0026#34; mouse = false [editor.statusline] left = [\u0026#34;mode\u0026#34;, \u0026#34;spinner\u0026#34;] center = [\u0026#34;file-name\u0026#34;] right = [\u0026#34;diagnostics\u0026#34;, \u0026#34;selections\u0026#34;, \u0026#34;position\u0026#34;, \u0026#34;file-encoding\u0026#34;, \u0026#34;file-line-ending\u0026#34;, \u0026#34;file-type\u0026#34;] separator = \u0026#34;│\u0026#34; mode.normal = \u0026#34;NORMAL\u0026#34; mode.insert = \u0026#34;INSERT\u0026#34; mode.select = \u0026#34;SELECT\u0026#34; [editor.cursor-shape] insert = \u0026#34;bar\u0026#34; normal = \u0026#34;block\u0026#34; select = \u0026#34;underline\u0026#34; [editor.file-picker] hidden = false [editor.whitespace] render = \u0026#34;all\u0026#34; [editor.indent-guides] render = true character = \u0026#34;╎\u0026#34; # Some characters that work well: \u0026#34;▏\u0026#34;, \u0026#34;┆\u0026#34;, \u0026#34;┊\u0026#34;, \u0026#34;⸽\u0026#34; skip-levels = 1 And to configure the language servers I used the following language-servers.toml file:\n[[language]] name = \u0026#34;go\u0026#34; auto-format = true formatter = { command = \u0026#34;goimports\u0026#34; } [[language]] name = \u0026#34;javascript\u0026#34; language-servers = [ \u0026#34;typescript-language-server\u0026#34;, # optional \u0026#34;vscode-eslint-language-server\u0026#34;, ] [language-server.rust-analyzer.config.check] command = \u0026#34;clippy\u0026#34; [language-server.sql-language-server] command = \u0026#34;sql-language-server\u0026#34; args = [\u0026#34;up\u0026#34;, \u0026#34;--method\u0026#34;, \u0026#34;stdio\u0026#34;] [[language]] name = \u0026#34;sql\u0026#34; language-servers = [ \u0026#34;sql-language-server\u0026#34; ] [[language]] name = \u0026#34;hcl\u0026#34; language-servers = [ \u0026#34;terraform-ls\u0026#34; ] language-id = \u0026#34;terraform\u0026#34; [[language]] name = \u0026#34;tfvars\u0026#34; language-servers = [ \u0026#34;terraform-ls\u0026#34; ] language-id = \u0026#34;terraform-vars\u0026#34; [language-server.terraform-ls] command = \u0026#34;terraform-ls\u0026#34; args = [\u0026#34;serve\u0026#34;] [[language]] name = \u0026#34;toml\u0026#34; formatter = { command = \u0026#34;taplo\u0026#34;, args = [\u0026#34;fmt\u0026#34;, \u0026#34;-\u0026#34;] } [[language]] name = \u0026#34;typescript\u0026#34; language-servers = [ \u0026#34;typescript-language-server\u0026#34;, \u0026#34;vscode-eslint-language-server\u0026#34;, ] Neovim configurationAfter a little while I noticed that I was going to need some time to get used to helix and the most interesting thing for me was the easy configuration and the language server integrations, but as I am already comfortable with neovim and just had installed the language server support tools on my machines I just need to configure them for neovim and I can keep using it for a while.\nAs I said my configuration is old, to configure neovim I have the following init.vim file on my ~/.config/nvim folder:\nset runtimepath^=~/.vim runtimepath+=~/.vim/after let \u0026amp;packpath=\u0026amp;runtimepath source ~/.vim/vimrc \u0026#34; load lua configuration lua require(\u0026#39;config\u0026#39;) With that configuration I keep my old vimrc (it is a little bit messy, but it works) and I use a lua configuration file for the language servers and some additional neovim plugins on the ~/.config/nvim/lua/config.lua file:\n-- ----------------------- -- BEG: LSP Configurations -- ----------------------- -- AWS (awk_ls) require\u0026#39;lspconfig\u0026#39;.awk_ls.setup{} -- Bash (bashls) require\u0026#39;lspconfig\u0026#39;.bashls.setup{} -- C/C++ (clangd) require\u0026#39;lspconfig\u0026#39;.clangd.setup{} -- CSS (cssls) require\u0026#39;lspconfig\u0026#39;.cssls.setup{} -- Docker (dockerls) require\u0026#39;lspconfig\u0026#39;.dockerls.setup{} -- Docker Compose require\u0026#39;lspconfig\u0026#39;.docker_compose_language_service.setup{} -- Golang (gopls) require\u0026#39;lspconfig\u0026#39;.gopls.setup{} -- Helm (helm_ls) require\u0026#39;lspconfig\u0026#39;.helm_ls.setup{} -- Markdown require\u0026#39;lspconfig\u0026#39;.marksman.setup{} -- Python (pyright) require\u0026#39;lspconfig\u0026#39;.pyright.setup{} -- Rust (rust-analyzer) require\u0026#39;lspconfig\u0026#39;.rust_analyzer.setup{} -- SQL (sqlls) require\u0026#39;lspconfig\u0026#39;.sqlls.setup{} -- Terraform (terraformls) require\u0026#39;lspconfig\u0026#39;.terraformls.setup{} -- TOML (taplo) require\u0026#39;lspconfig\u0026#39;.taplo.setup{} -- Typescript (ts_ls) require\u0026#39;lspconfig\u0026#39;.ts_ls.setup{} -- YAML (yamlls) require\u0026#39;lspconfig\u0026#39;.yamlls.setup{ settings = { yaml = { customTags = { \u0026#34;!reference sequence\u0026#34; } } } } -- ----------------------- -- END: LSP Configurations -- ----------------------- -- --------------------------------- -- BEG: Autocompletion configuration -- --------------------------------- -- Ref: https://github.com/neovim/nvim-lspconfig/wiki/Autocompletion -- -- Pre requisites: -- -- # Packer -- git clone --depth 1 https://github.com/wbthomason/packer.nvim \\ -- ~/.local/share/nvim/site/pack/packer/start/packer.nvim -- -- # Start nvim and run :PackerSync or :PackerUpdate -- --------------------------------- local use = require(\u0026#39;packer\u0026#39;).use require(\u0026#39;packer\u0026#39;).startup(function() use \u0026#39;wbthomason/packer.nvim\u0026#39; -- Packer, useful to avoid removing it with PackerSync / PackerUpdate use \u0026#39;neovim/nvim-lspconfig\u0026#39; -- Collection of configurations for built-in LSP client use \u0026#39;hrsh7th/nvim-cmp\u0026#39; -- Autocompletion plugin use \u0026#39;hrsh7th/cmp-nvim-lsp\u0026#39; -- LSP source for nvim-cmp use \u0026#39;saadparwaiz1/cmp_luasnip\u0026#39; -- Snippets source for nvim-cmp use \u0026#39;L3MON4D3/LuaSnip\u0026#39; -- Snippets plugin end) -- Add additional capabilities supported by nvim-cmp local capabilities = require(\u0026#34;cmp_nvim_lsp\u0026#34;).default_capabilities() local lspconfig = require(\u0026#39;lspconfig\u0026#39;) -- Enable some language servers with the additional completion capabilities offered by nvim-cmp local servers = { \u0026#39;clangd\u0026#39;, \u0026#39;rust_analyzer\u0026#39;, \u0026#39;pyright\u0026#39;, \u0026#39;ts_ls\u0026#39; } for _, lsp in ipairs(servers) do lspconfig[lsp].setup { -- on_attach = my_custom_on_attach, capabilities = capabilities, } end -- luasnip setup local luasnip = require \u0026#39;luasnip\u0026#39; -- nvim-cmp setup local cmp = require \u0026#39;cmp\u0026#39; cmp.setup { snippet = { expand = function(args) luasnip.lsp_expand(args.body) end, }, mapping = cmp.mapping.preset.insert({ [\u0026#39;\u0026lt;C-u\u0026gt;\u0026#39;] = cmp.mapping.scroll_docs(-4), -- Up [\u0026#39;\u0026lt;C-d\u0026gt;\u0026#39;] = cmp.mapping.scroll_docs(4), -- Down -- C-b (back) C-f (forward) for snippet placeholder navigation. [\u0026#39;\u0026lt;C-Space\u0026gt;\u0026#39;] = cmp.mapping.complete(), [\u0026#39;\u0026lt;CR\u0026gt;\u0026#39;] = cmp.mapping.confirm { behavior = cmp.ConfirmBehavior.Replace, select = true, }, [\u0026#39;\u0026lt;Tab\u0026gt;\u0026#39;] = cmp.mapping(function(fallback) if cmp.visible() then cmp.select_next_item() elseif luasnip.expand_or_jumpable() then luasnip.expand_or_jump() else fallback() end end, { \u0026#39;i\u0026#39;, \u0026#39;s\u0026#39; }), [\u0026#39;\u0026lt;S-Tab\u0026gt;\u0026#39;] = cmp.mapping(function(fallback) if cmp.visible() then cmp.select_prev_item() elseif luasnip.jumpable(-1) then luasnip.jump(-1) else fallback() end end, { \u0026#39;i\u0026#39;, \u0026#39;s\u0026#39; }), }), sources = { { name = \u0026#39;nvim_lsp\u0026#39; }, { name = \u0026#39;luasnip\u0026#39; }, }, } -- --------------------------------- -- END: Autocompletion configuration -- --------------------------------- ConclusionI guess I’ll keep helix installed and try it again on some of my personal projects to see if I can get used to it, but for now I’ll stay with neovim as my main text editor and learn the shortcuts to use it with the language servers.\n","permalink":"https://blogops.mixinet.net/posts/user_tools/testing_new_user_tools/","summary":"\u003cp\u003eOn recent weeks I’ve had some time to \u003ca href=\"https://en.m.wiktionary.org/wiki/scratch_one%27s_own_itch\" target=\"_blank\" rel=\"noopener\"\u003escratch my own itch\u003c/a\u003e on\nmatters related to tools I use daily on my computer, namely the desktop / window manager and my text editor of choice.\u003c/p\u003e\n\u003cp\u003eThis post is a summary of what I tried, how it worked out and my short and medium-term plans related to them.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_desktop_wm\"\u003eDesktop / WM\u003c/h2\u003e\u003cp\u003eOn the desktop / window manager front I’ve been using \u003ca href=\"https://projects.linuxmint.com/cinnamon/\" target=\"_blank\" rel=\"noopener\"\u003eCinnamon\u003c/a\u003e on \u003cstrong\u003eDebian\u003c/strong\u003e\nand \u003cstrong\u003eUbuntu\u003c/strong\u003e systems since \u003cstrong\u003eGnome 3\u003c/strong\u003e was published (I never liked version 3, so I decided to move to something similar\nto \u003ca href=\"https://gnome.org/\" target=\"_blank\" rel=\"noopener\"\u003eGnome 2\u003c/a\u003e, including the keyboard shortcuts).\u003c/p\u003e\n\u003cp\u003eIn fact I’ve never been a fan of Desktop environments, before \u003cstrong\u003eGnome\u003c/strong\u003e I used \u003ca href=\"https://openbox.org/\" target=\"_blank\" rel=\"noopener\"\u003eOpenBox\u003c/a\u003e and\n\u003ca href=\"https://ice-wm.org/\" target=\"_blank\" rel=\"noopener\"\u003eIceWM\u003c/a\u003e because they where a lot faster than desktop systems on my hardware at the time and I was\nusing them only to place one or two windows on multiple workspaces using mainly the keyboard for my interactions (well,\nexcept for the web browsers and the image manipulation programs).\u003c/p\u003e\n\u003cp\u003eAlthough I was comfortable using \u003cstrong\u003eCinnamon\u003c/strong\u003e, some years ago I tried to move to \u003ca href=\"https://i3wm.org/\" target=\"_blank\" rel=\"noopener\"\u003ei3\u003c/a\u003e, a tilling window\nmanager for X11 that looked like a good choice for me, but I didn’t have much time to play with it and never used it\nenough to make me productive with it (I didn’t prepare a complete configuration nor had enough time to learn the new\nshortcuts, so I went back to \u003cstrong\u003eCinnamon\u003c/strong\u003e and never tried again).\u003c/p\u003e\n\u003cp\u003eAnyway, some weeks ago I updated my work machine OS (it was using \u003cstrong\u003eUbuntu 22.04 LTS\u003c/strong\u003e and I updated it to the \u003cstrong\u003e24.04\nLTS\u003c/strong\u003e version) and the \u003cstrong\u003eCinnamon systray applet\u003c/strong\u003e stopped working as it used to do (in fact I still have to restart\n\u003cstrong\u003eCinnamon\u003c/strong\u003e after starting a session to make it work) and, as I had some time, I decided to try a tilling window\nmanager again, but now I decided to go for \u003ca href=\"https://swaywm.org/\" target=\"_blank\" rel=\"noopener\"\u003eSwayWM\u003c/a\u003e, as it uses\n\u003ca href=\"https://wayland.freedesktop.org/\" target=\"_blank\" rel=\"noopener\"\u003eWayland\u003c/a\u003e instead of \u003ca href=\"https://x.org/\" target=\"_blank\" rel=\"noopener\"\u003eX11\u003c/a\u003e.\u003c/p\u003e\n\u003csection class=\"doc-section level-2\"\u003e\u003ch3 id=\"_sway_configuration\"\u003eSway configuration\u003c/h3\u003e\u003cp\u003eOn my \u003ccode\u003e~/.config/sway/config\u003c/code\u003e I tuned some things:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003eSet \u003ca href=\"https://codeberg.org/dnkl/fuzzel\"\u003efuzzel\u003c/a\u003e as the application launcher.\u003c/li\u003e\u003cli\u003eInstalled manually the \u003ca href=\"https://gitlab.com/w0lff/shikane\"\u003eshikane\u003c/a\u003e application and created a configuration to be\nexecuted always when \u003ccode\u003esway\u003c/code\u003e is started / reloaded (I adjusted my configuration with \u003ccode\u003ewdisplays\u003c/code\u003e and used \u003ccode\u003eshikanectl\u003c/code\u003e\nto save it).\u003c/li\u003e\u003cli\u003eAdded support for \u003ca href=\"https://github.com/emersion/xdg-desktop-portal-wlr/blob/master/README.md#running\"\u003erunning\u003c/a\u003e the\n\u003ccode\u003exdg-desktop-portal-wlr\u003c/code\u003e service.\u003c/li\u003e\u003cli\u003eEnabled the \u003ccode\u003eswayidle\u003c/code\u003e command to lock the screen after some time of inactivity.\u003c/li\u003e\u003cli\u003eAdjusted the keyboard to use the \u003ccode\u003ees\u003c/code\u003e key map\u003c/li\u003e\u003cli\u003eAdded some keybindings to make my life easier, including the use of \u003ccode\u003egrimm\u003c/code\u003e and \u003ccode\u003eswappy\u003c/code\u003e to take screenshots\u003c/li\u003e\u003cli\u003eConfigured \u003ccode\u003ewaybar\u003c/code\u003e as the environment bar.\u003c/li\u003e\u003cli\u003e\u003cp\u003eAdded a shell script to start applications when \u003ccode\u003esway\u003c/code\u003e is started (it uses \u003ccode\u003eswaymsg\u003c/code\u003e to execute background commands\nand the \u003ca href=\"https://gitlab.com/wef/dotfiles/-/blob/master/bin/i3-toolwait\"\u003e\u003ccode\u003ei3toolwait\u003c/code\u003e\u003c/a\u003e script to wait for the\u003c/p\u003e\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003e\u003cspan class=\"c\"\u003e#!/bin/sh\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# VARIABLES\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eCHROMIUM_LOCAL_STATE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$HOME\u003c/span\u003e\u003cspan class=\"s2\"\u003e/.config/google-chrome/Local State\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$HOME\u003c/span\u003e\u003cspan class=\"s2\"\u003e/.config/sway/scripts/i3-toolwait\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Functions\u003c/span\u003e\nchromium_profile_dir\u003cspan class=\"o\"\u003e()\u003c/span\u003e \u003cspan class=\"o\"\u003e{\u003c/span\u003e\n  jq \u003cspan class=\"nt\"\u003e-r\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;.profile.info_cache|to_entries|map({(.value.name): .key})|add|.\u003c/span\u003e\u003cspan class=\"se\"\u003e\\\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$1\u003c/span\u003e\u003cspan class=\"se\"\u003e\\\u0026#34;\u003c/span\u003e\u003cspan class=\"s2\"\u003e // \u003c/span\u003e\u003cspan class=\"se\"\u003e\\\u0026#34;\\\u0026#34;\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CHROMIUM_LOCAL_STATE\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"o\"\u003e}\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# MAIN\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eIGZ_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003echromium_profile_dir \u003cspan class=\"s2\"\u003e\u0026#34;sergio.talens@intelygenz.com\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eOURO_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003echromium_profile_dir \u003cspan class=\"s2\"\u003e\u0026#34;sergio.talens@nxr.global\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nv\"\u003ePERSONAL_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003echromium_profile_dir \u003cspan class=\"s2\"\u003e\u0026#34;stalens@gmail.com\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Common programs\u003c/span\u003e\nswaymsg \u003cspan class=\"s2\"\u003e\u0026#34;exec nextcloud --background\u0026#34;\u003c/span\u003e\nswaymsg \u003cspan class=\"s2\"\u003e\u0026#34;exec nm-applet\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Run spotify on the first workspace (it is mapped to the laptop screen)\u003c/span\u003e\nswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace 1\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;spotify\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Run tmux on the\u003c/span\u003e\nswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace 2\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"nt\"\u003e--\u003c/span\u003e foot tmux a \u003cspan class=\"nt\"\u003e-dt\u003c/span\u003e sto\n\n\u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;3\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OURO_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  \u003c/span\u003eswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"nt\"\u003e-m\u003c/span\u003e ouro-browser \u003cspan class=\"nt\"\u003e--\u003c/span\u003e google-chrome \u003cspan class=\"nt\"\u003e--profile-directory\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OURO_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003efi\n\nif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$IGZ_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  \u003c/span\u003eswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"nt\"\u003e-m\u003c/span\u003e igz-browser \u003cspan class=\"nt\"\u003e--\u003c/span\u003e google-chrome \u003cspan class=\"nt\"\u003e--profile-directory\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$IGZ_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003efi\n\nif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$PERSONAL_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  \u003c/span\u003eswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"nt\"\u003e-m\u003c/span\u003e personal-browser \u003cspan class=\"nt\"\u003e--\u003c/span\u003e google-chrome \u003cspan class=\"nt\"\u003e--profile-directory\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$PERSONAL_PROFILE_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Open the browser without setting the profile directory if none was found\u003c/span\u003e\n\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;3\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  \u003c/span\u003eswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e google-chrome\n  \u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003efi\n\u003c/span\u003eswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e evolution\n\u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\nswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e slack\n\u003cspan class=\"nv\"\u003ewp_num\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$((\u003c/span\u003ewp_num+1\u003cspan class=\"k\"\u003e))\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\n\u003cspan class=\"c\"\u003e# Open a private browser and a console in the last workspace\u003c/span\u003e\nswaymsg \u003cspan class=\"nt\"\u003e-q\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;workspace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$wp_num\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e \u003cspan class=\"nt\"\u003e--\u003c/span\u003e google-chrome \u003cspan class=\"nt\"\u003e--incognito\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e foot\n\n\u003cspan class=\"c\"\u003e# Go back to the second workspace for keepassxc\u003c/span\u003e\nswaymsg \u003cspan class=\"s2\"\u003e\u0026#34;workspace 2\u0026#34;\u003c/span\u003e\n\u003cspan class=\"k\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eI3_TOOLWAIT\u003c/span\u003e\u003cspan class=\"k\"\u003e}\u003c/span\u003e keepassxc\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Testing New User Tools"},{"content":"This post describes how I’m using semantic-release on gitlab-ci to manage versioning automatically for different kinds of projects following a simple workflow (a develop branch where changes are added or merged to test new versions, a temporary release/#.#.# to generate the release candidate versions and a main branch where the final versions are published).\nWhat is semantic-releaseIt is a Node.js application designed to manage project versioning information on Git Repositories using a Continuous integration system (in this post we will use gitlab-ci)\nHow does it workBy default semantic-release uses semver for versioning (release versions use the format MAJOR.MINOR.PATCH) and commit messages are parsed to determine the next version number to publish.\nIf after analyzing the commits the version number has to be changed, the command updates the files we tell it to (i.e. the package.json file for nodejs projects and possibly a CHANGELOG.md file), creates a new commit with the changed files, creates a tag with the new version and pushes the changes to the repository.\nWhen running on a CI/CD system we usually generate the artifacts related to a release (a package, a container image, etc.) from the tag, as it includes the right version number and usually has passed all the required tests (it is a good idea to run the tests again in any case, as someone could create a tag manually or we could run extra jobs when building the final assets …​ if they fail it is not a big issue anyway, numbers are cheap and infinite, so we can skip releases if needed).\nCommit messages and versioningThe commit messages must follow a known format, the default module used to analyze them uses the angular git commit guidelines, but I prefer the conventional commits one, mainly because it’s a lot easier to use when you want to update the MAJOR version.\nThe commit message format used must be:\n\u0026lt;type\u0026gt;(optional scope): \u0026lt;description\u0026gt; [optional body] [optional footer(s)] The system supports three types of branches: release, maintenance and pre-release, but for now I’m not using maintenance ones.\nThe branches I use and their types are:\nmain as release branch (final versions are published from there)develop as pre release branch (used to publish development and testing versions with the format #.#.#-SNAPSHOT.#)release/#.#.# as pre release branches (they are created from develop to publish release candidate versions with the format #.#.#-rc.# and once they are merged with main they are deleted) On the release branch (main) the version number is updated as follows:\nThe MAJOR number is incremented if a commit with a BREAKING CHANGE: footer or an exclamation (!) after the type/scope is found in the list of commits found since the last version change (it looks for tags on the same branch).The MINOR number is incremented if the MAJOR number is not going to be changed and there is a commit with type feat in the commits found since the last version change.The PATCH number is incremented if neither the MAJOR nor the MINOR numbers are going to be changed and there is a commit with type fix in the the commits found since the last version change. On the pre release branches (develop and release/#.#.#) the version and pre release numbers are always calculated from the last published version available on the branch (i. e. if we published version 1.3.2 on main we need to have the commit with that tag on the develop or release/#.#.# branch to get right what will be the next version).\nThe version number is updated as follows:\nThe MAJOR number is incremented if a commit with a BREAKING CHANGE: footer or an exclamation (!) after the type/scope is found in the list of commits found since the last released version.\nIn our example it was 1.3.2 and the version is updated to 2.0.0-SNAPSHOT.1 or 2.0.0-rc.1 depending on the branch.\nThe MINOR number is incremented if the MAJOR number is not going to be changed and there is a commit with type feat in the commits found since the last released version.\nIn our example the release was 1.3.2 and the version is updated to 1.4.0-SNAPSHOT.1 or 1.4.0-rc.1 depending on the branch.\nThe PATCH number is incremented if neither the MAJOR nor the MINOR numbers are going to be changed and there is a commit with type fix in the the commits found since the last version change.\nIn our example the release was 1.3.2 and the version is updated to 1.3.3-SNAPSHOT.1 or 1.3.3-rc.1 depending on the branch.\nThe pre release number is incremented if the MAJOR, MINOR and PATCH numbers are not going to be changed but there is a commit that would otherwise update the version (i.e. a fix on 1.3.3-SNAPSHOT.1 will set the version to 1.3.3-SNAPSHOT.2, a fix or feat on 1.4.0-rc.1 will set the version to 1.4.0-rc.2 an so on). How do we manage its configurationAlthough the system is designed to work with nodejs projects, it can be used with multiple programming languages and project types.\nFor nodejs projects the usual place to put the configuration is the project’s package.json, but I prefer to use the .releaserc file instead.\nAs I use a common set of CI templates, instead of using a .releaserc on each project I generate it on the fly on the jobs that need it, replacing values related to the project type and the current branch on a template using the tmpl command (lately I use a branch of my own fork while I wait for some feedback from upstream, as you will see on the Dockerfile).\nContainer used to run itAs we run the command on a gitlab-ci job we use the image built from the following Dockerfile:\nDockerfile # Semantic release image FROM golang:alpine AS tmpl-builder #RUN go install github.com/krakozaure/tmpl@v0.4.0 RUN go install github.com/sto/tmpl@v0.4.0-sto.2 FROM node:lts-alpine COPY --from=tmpl-builder /go/bin/tmpl /usr/local/bin/tmpl RUN apk update \u0026amp;\u0026amp;\\ apk upgrade \u0026amp;\u0026amp;\\ apk add curl git jq openssh-keygen yq zip \u0026amp;\u0026amp;\\ npm install --location=global\\ conventional-changelog-conventionalcommits@6.1.0\\ @qiwi/multi-semantic-release@7.0.0\\ semantic-release@21.0.7\\ @semantic-release/changelog@6.0.3\\ semantic-release-export-data@1.0.1\\ @semantic-release/git@10.0.1\\ @semantic-release/gitlab@9.5.1\\ @semantic-release/release-notes-generator@11.0.4\\ semantic-release-replace-plugin@1.2.7\\ semver@7.5.4\\ \u0026amp;\u0026amp;\\ rm -rf /var/cache/apk/* CMD [\u0026#34;/bin/sh\u0026#34;] Note: The versions of some of the components are not the latest ones, I try to review them from time to time but you know the saying: if it ain’t broken, don’t fix it.\nNote: The image includes some tools and modules I use on other projects like the qiwi/multi-semantic-release, a fork of semantic-release that allows to publish multiple packages from a single repository (monorepo).\nI’ll probably write about how I use it on a future post.\nHow and when is it executedThe job that runs semantic-release is executed when new commits are added to the develop, release/#.#.# or main branches (basically when something is merged or pushed) and after all tests have passed (we don’t want to create a new version that does not compile or passes at least the unit tests).\nThe job is something like the following:\nsemantic_release: image: $SEMANTIC_RELEASE_IMAGE rules: - if: \u0026#39;$CI_COMMIT_BRANCH =~ /^(develop|main|release\\/\\d+.\\d+.\\d+)$/\u0026#39; when: always stage: release before_script: - echo \u0026#34;Loading scripts.sh\u0026#34; - . $ASSETS_DIR/scripts.sh script: - sr_gen_releaserc_json - git_push_setup - semantic-release Where the SEMANTIC_RELEASE_IMAGE variable contains the URI of the image built using the Dockerfile above and the sr_gen_releaserc_json and git_push_setup are functions defined on the $ASSETS_DIR/scripts.sh file:\nThe sr_gen_releaserc_json function generates the .releaserc.json file using the tmpl command.The git_push_setup function configures git to allow pushing changes to the repository with the semantic-release command, optionally signing them with a SSH key. The sr_gen_releaserc_json functionThe code for the sr_gen_releaserc_json function is the following:\nsr_gen_releaserc_json() { # Use nodejs as default project_type project_type=\u0026#34;${PROJECT_TYPE:-nodejs}\u0026#34; # REGEX to match the rc_branch name rc_branch_regex=\u0026#39;^release\\/[0-9]\\+\\.[0-9]\\+\\.[0-9]\\+$\u0026#39; # PATHS on the local ASSETS_DIR assets_dir=\u0026#34;${CI_PROJECT_DIR}/${ASSETS_DIR}\u0026#34; sr_local_plugin=\u0026#34;${assets_dir}/local-plugin.cjs\u0026#34; releaserc_tmpl=\u0026#34;${assets_dir}/releaserc.json.tmpl\u0026#34; pipeline_runtime_values_yaml=\u0026#34;/tmp/releaserc_values.yaml\u0026#34; pipeline_values_yaml=\u0026#34;${assets_dir}/values_${project_type}_project.yaml\u0026#34; # Destination PATH releaserc_json=\u0026#34;.releaserc.json\u0026#34; # Create an empty pipeline_values_yaml if missing test -f \u0026#34;$pipeline_values_yaml\u0026#34; || : \u0026gt;\u0026#34;$pipeline_values_yaml\u0026#34; # Create the pipeline_runtime_values_yaml file echo \u0026#34;branch: ${CI_COMMIT_BRANCH}\u0026#34; \u0026gt;\u0026#34;$pipeline_runtime_values_yaml\u0026#34; echo \u0026#34;gitlab_url: ${CI_SERVER_URL}\u0026#34; \u0026gt;\u0026#34;$pipeline_runtime_values_yaml\u0026#34; # Add the rc_branch name if we are on an rc_branch if [ \u0026#34;$(echo \u0026#34;$CI_COMMIT_BRANCH\u0026#34; | sed -ne \u0026#34;/$rc_branch_regex/{p}\u0026#34;)\u0026#34; ]; then echo \u0026#34;rc_branch: ${CI_COMMIT_BRANCH}\u0026#34; \u0026gt;\u0026gt;\u0026#34;$pipeline_runtime_values_yaml\u0026#34; elif [ \u0026#34;$(echo \u0026#34;$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME\u0026#34; | sed -ne \u0026#34;/$rc_branch_regex/{p}\u0026#34;)\u0026#34; ]; then echo \u0026#34;rc_branch: ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}\u0026#34; \\ \u0026gt;\u0026gt;\u0026#34;$pipeline_runtime_values_yaml\u0026#34; fi echo \u0026#34;sr_local_plugin: ${sr_local_plugin}\u0026#34; \u0026gt;\u0026gt;\u0026#34;$pipeline_runtime_values_yaml\u0026#34; # Create the releaserc_json file tmpl -f \u0026#34;$pipeline_runtime_values_yaml\u0026#34; -f \u0026#34;$pipeline_values_yaml\u0026#34; \\ \u0026#34;$releaserc_tmpl\u0026#34; | jq . \u0026gt;\u0026#34;$releaserc_json\u0026#34; # Remove the pipeline_runtime_values_yaml file rm -f \u0026#34;$pipeline_runtime_values_yaml\u0026#34; # Print the releaserc_json file print_file_collapsed \u0026#34;$releaserc_json\u0026#34; # --*-- BEG: NOTE --*-- # Rename the package.json to ignore it when calling semantic release. # The idea is that the local-plugin renames it back on the first step of the # semantic-release process. # --*-- END: NOTE --*-- if [ -f \u0026#34;package.json\u0026#34; ]; then echo \u0026#34;Renaming \u0026#39;package.json\u0026#39; to \u0026#39;package.json_disabled\u0026#39;\u0026#34; mv \u0026#34;package.json\u0026#34; \u0026#34;package.json_disabled\u0026#34; fi } Almost all the variables used on the function are defined by gitlab except the ASSETS_DIR and PROJECT_TYPE; in the complete pipelines the ASSETS_DIR is defined on a common file included by all the pipelines and the project type is defined on the .gitlab-ci.yml file of each project.\nIf you review the code you will see that the file processed by the tmpl command is named releaserc.json.tmpl, its contents are shown here:\n{ \u0026#34;plugins\u0026#34;: [ {{- if .sr_local_plugin }} \u0026#34;{{ .sr_local_plugin }}\u0026#34;, {{- end }} [ \u0026#34;@semantic-release/commit-analyzer\u0026#34;, { \u0026#34;preset\u0026#34;: \u0026#34;conventionalcommits\u0026#34;, \u0026#34;releaseRules\u0026#34;: [ { \u0026#34;breaking\u0026#34;: true, \u0026#34;release\u0026#34;: \u0026#34;major\u0026#34; }, { \u0026#34;revert\u0026#34;: true, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;feat\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;minor\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;fix\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;perf\u0026#34;, \u0026#34;release\u0026#34;: \u0026#34;patch\u0026#34; } ] } ], {{- if .replacements }} [ \u0026#34;semantic-release-replace-plugin\u0026#34;, { \u0026#34;replacements\u0026#34;: {{ .replacements | toJson }} } ], {{- end }} \u0026#34;@semantic-release/release-notes-generator\u0026#34;, {{- if eq .branch \u0026#34;main\u0026#34; }} [ \u0026#34;@semantic-release/changelog\u0026#34;, { \u0026#34;changelogFile\u0026#34;: \u0026#34;CHANGELOG.md\u0026#34;, \u0026#34;changelogTitle\u0026#34;: \u0026#34;# Changelog\u0026#34; } ], {{- end }} [ \u0026#34;@semantic-release/git\u0026#34;, { \u0026#34;assets\u0026#34;: {{ if .assets }}{{ .assets | toJson }}{{ else }}[]{{ end }}, \u0026#34;message\u0026#34;: \u0026#34;ci(release): v${nextRelease.version}\\n\\n${nextRelease.notes}\u0026#34; } ], [ \u0026#34;@semantic-release/gitlab\u0026#34;, { \u0026#34;gitlabUrl\u0026#34;: \u0026#34;{{ .gitlab_url }}\u0026#34;, \u0026#34;successComment\u0026#34;: false } ] ], \u0026#34;branches\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;develop\u0026#34;, \u0026#34;prerelease\u0026#34;: \u0026#34;SNAPSHOT\u0026#34; }, {{- if .rc_branch }} { \u0026#34;name\u0026#34;: \u0026#34;{{ .rc_branch }}\u0026#34;, \u0026#34;prerelease\u0026#34;: \u0026#34;rc\u0026#34; }, {{- end }} \u0026#34;main\u0026#34; ] } The values used to process the template are defined on a file built on the fly (releaserc_values.yaml) that includes the following keys and values:\nbranch: the name of the current branchgitlab_url: the URL of the gitlab server (the value is taken from the CI_SERVER_URL variable)rc_branch: the name of the current rc branch; we only set the value if we are processing one because semantic-release only allows one branch to match the rc prefix and if we use a wildcard (i.e. release/*) but the users keep more than one release/#.#.# branch open at the same time the calls to semantic-release will fail for sure.sr_local_plugin: the path to the local plugin we use (shown later) The template also uses a values_${project_type}_project.yaml file that includes settings specific to the project type, the one for nodejs is as follows:\nreplacements: - files: - \u0026#34;package.json\u0026#34; from: \u0026#34;\\\u0026#34;version\\\u0026#34;: \\\u0026#34;.*\\\u0026#34;\u0026#34; to: \u0026#34;\\\u0026#34;version\\\u0026#34;: \\\u0026#34;${nextRelease.version}\\\u0026#34;\u0026#34; assets: - \u0026#34;CHANGELOG.md\u0026#34; - \u0026#34;package.json\u0026#34; The replacements section is used to update the version field on the relevant files of the project (in our case the package.json file) and the assets section includes the files that will be committed to the repository when the release is published (looking at the template you can see that the CHANGELOG.md is only updated for the main branch, we do it this way because if we update the file on other branches it creates a merge nightmare and we are only interested on it for released versions anyway).\nThe local plugin adds code to rename the package.json_disabled file to package.json if present and prints the last and next versions on the logs with a format that can be easily parsed using sed:\nlocal-plugin.cjs // Minimal plugin to: // - rename the package.json_disabled file to package.json if present // - log the semantic-release last \u0026amp; next versions function verifyConditions(pluginConfig, context) { var fs = require(\u0026#39;fs\u0026#39;); if (fs.existsSync(\u0026#39;package.json_disabled\u0026#39;)) { fs.renameSync(\u0026#39;package.json_disabled\u0026#39;, \u0026#39;package.json\u0026#39;); context.logger.log(`verifyConditions: renamed \u0026#39;package.json_disabled\u0026#39; to \u0026#39;package.json\u0026#39;`); } } function analyzeCommits(pluginConfig, context) { if (context.lastRelease \u0026amp;\u0026amp; context.lastRelease.version) { context.logger.log(`analyzeCommits: LAST_VERSION=${context.lastRelease.version}`); } } function verifyRelease(pluginConfig, context) { if (context.nextRelease \u0026amp;\u0026amp; context.nextRelease.version) { context.logger.log(`verifyRelease: NEXT_VERSION=${context.nextRelease.version}`); } } module.exports = { verifyConditions, analyzeCommits, verifyRelease } The git_push_setup functionThe code for the git_push_setup function is the following:\ngit_push_setup() { # Update global credentials to allow git clone \u0026amp; push for all the group repos git config --global credential.helper store cat \u0026gt;\u0026#34;$HOME/.git-credentials\u0026#34; \u0026lt;\u0026lt;EOF https://fake-user:${GITLAB_REPOSITORY_TOKEN}@gitlab.com EOF # Define user name, mail and signing key for semantic-release user_name=\u0026#34;$SR_USER_NAME\u0026#34; user_email=\u0026#34;$SR_USER_EMAIL\u0026#34; ssh_signing_key=\u0026#34;$SSH_SIGNING_KEY\u0026#34; # Export git user variables export GIT_AUTHOR_NAME=\u0026#34;$user_name\u0026#34; export GIT_AUTHOR_EMAIL=\u0026#34;$user_email\u0026#34; export GIT_COMMITTER_NAME=\u0026#34;$user_name\u0026#34; export GIT_COMMITTER_EMAIL=\u0026#34;$user_email\u0026#34; # Sign commits with ssh if there is a SSH_SIGNING_KEY variable if [ \u0026#34;$ssh_signing_key\u0026#34; ]; then echo \u0026#34;Configuring GIT to sign commits with SSH\u0026#34; ssh_keyfile=\u0026#34;/tmp/.ssh-id\u0026#34; : \u0026gt;\u0026#34;$ssh_keyfile\u0026#34; chmod 0400 \u0026#34;$ssh_keyfile\u0026#34; echo \u0026#34;$ssh_signing_key\u0026#34; | tr -d \u0026#39;\\r\u0026#39; \u0026gt;\u0026#34;$ssh_keyfile\u0026#34; git config gpg.format ssh git config user.signingkey \u0026#34;$ssh_keyfile\u0026#34; git config commit.gpgsign true fi } The function assumes that the GITLAB_REPOSITORY_TOKEN variable (set on the CI/CD variables section of the project or group we want) contains a token with read_repository and write_repository permissions on all the projects we are going to use this function.\nThe SR_USER_NAME and SR_USER_EMAIL variables can be defined on a common file or the CI/CD variables section of the project or group we want to work with and the script assumes that the optional SSH_SIGNING_KEY is exported as a CI/CD default value of type variable (that is why the keyfile is created on the fly) and git is configured to use it if the variable is not empty.\nWarning: Keep in mind that the variables GITLAB_REPOSITORY_TOKEN and SSH_SIGNING_KEY contain secrets, so probably is a good idea to make them protected (if you do that you have to make the develop, main and release/* branches protected too).\nWarning: The semantic-release user has to be able to push to all the projects on those protected branches, it is a good idea to create a dedicated user and add it as a MAINTAINER for the projects we want (the MAINTAINERS need to be able to push to the branches), or, if you are using a Gitlab with a Premium license you can use the api to allow the semantic-release user to push to the protected branches without allowing it for any other user.\nThe semantic-release commandOnce we have the .releaserc file and the git configuration ready we run the semantic-release command.\nIf the branch we are working with has one or more commits that will increment the version, the tool does the following (note that the steps are described are the ones executed if we use the configuration we have generated):\nIt detects the commits that will increment the version and calculates the next version number.Generates the release notes for the version.Applies the replacements defined on the configuration (in our example updates the version field on the package.json file).Updates the CHANGELOG.md file adding the release notes if we are going to publish the file (when we are on the main branch).Creates a commit if all or some of the files listed on the assets key have changed and uses the commit message we have defined, replacing the variables for their current values.Creates a tag with the new version number and the release notes.As we are using the gitlab plugin after tagging it also creates a release on the project with the tag name and the release notes. Notes about the git workflows and merges between branchesIt is very important to remember that semantic-release looks at the commits of a given branch when calculating the next version to publish, that has two important implications:\nOn pre release branches we need to have the commit that includes the tag with the released version, if we don’t have it the next version is not calculated correctly.It is a bad idea to squash commits when merging a branch to another one, if we do that we will lose the information semantic-release needs to calculate the next version and even if we use the right prefix for the squashed commit (fix, feat, …​) we miss all the messages that would otherwise go to the CHANGELOG.md file. To make sure that we have the right commits on the pre release branches we should merge the main branch changes into the develop one after each release tag is created; in my pipelines the fist job that processes a release tag creates a branch from the tag and an MR to merge it to develop.\nThe important thing about that MR is that is must not be squashed, if we do that the tag commit will probably be lost, so we need to be careful.\nTo merge the changes directly we can run the following code:\n# Set the SR_TAG variable to the tag you want to process SR_TAG=\u0026#34;v1.3.2\u0026#34; # Fetch all the changes git fetch --all --prune # Switch to the main branch git switch main # Pull all the changes git pull # Switch to the development branch git switch develop # Pull all the changes git pull # Create followup branch from tag git switch -c \u0026#34;followup/$SR_TAG\u0026#34; \u0026#34;$SR_TAG\u0026#34; # Change files manually \u0026amp; commit the changed files git commit -a --untracked-files=no -m \u0026#34;ci(followup): $SR_TAG to develop\u0026#34; # Switch to the development branch git switch develop # Merge the followup branch into the development one using the --no-ff option git merge --no-ff \u0026#34;followup/$SR_TAG\u0026#34; # Remove the followup branch git branch -d \u0026#34;followup/$SR_TAG\u0026#34; # Push the changes git push If we can’t push directly to develop we can create a MR pushing the followup branch after committing the changes, but we have to make sure that we don’t squash the commits when merging or it will not work as we want.\nNote: We haven’t discussed the release/#.#.# branches because our assumption is that they don’t exist after a release is published (the branch is deleted after merging it to main) and new release candidate branches will be created from develop and the commits included with the followup will already be present.\n","permalink":"https://blogops.mixinet.net/posts/gitlab-ci/semantic-release/","summary":"\u003cp\u003eThis post describes how I’m using\n\u003ca href=\"https://semantic-release.gitbook.io/\" target=\"_blank\" rel=\"noopener\"\u003esemantic-release\u003c/a\u003e on\n\u003ca href=\"https://docs.gitlab.com/ee/ci/\" target=\"_blank\" rel=\"noopener\"\u003egitlab-ci\u003c/a\u003e to manage versioning automatically\nfor different kinds of projects following a simple workflow (a \u003ccode\u003edevelop\u003c/code\u003e branch\nwhere changes are added or merged to test new versions, a temporary\n\u003ccode\u003erelease/#.#.#\u003c/code\u003e to generate the release candidate versions and a \u003ccode\u003emain\u003c/code\u003e branch\nwhere the final versions are published).\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_what_is_semantic_release\"\u003eWhat is \u003ccode\u003esemantic-release\u003c/code\u003e\u003c/h2\u003e\u003cp\u003eIt is a \u003ca href=\"https://nodejs.org/\" target=\"_blank\" rel=\"noopener\"\u003eNode.js\u003c/a\u003e application designed to manage project\nversioning information on \u003ca href=\"https://git-scm.com/\" target=\"_blank\" rel=\"noopener\"\u003eGit Repositories\u003c/a\u003e using a\n\u003ca href=\"https://en.wikipedia.org/wiki/Continuous_integration\" target=\"_blank\" rel=\"noopener\"\u003eContinuous integration\u003c/a\u003e\nsystem (in this post we will use \u003ccode\u003egitlab-ci\u003c/code\u003e)\u003c/p\u003e\u003c/section\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_how_does_it_work\"\u003eHow does it work\u003c/h2\u003e\u003cp\u003eBy default \u003ccode\u003esemantic-release\u003c/code\u003e uses \u003ca href=\"https://semver.org/\" target=\"_blank\" rel=\"noopener\"\u003esemver\u003c/a\u003e for versioning\n(release versions use the format \u003ccode\u003eMAJOR.MINOR.PATCH\u003c/code\u003e) and commit messages are\nparsed to determine the next version number to publish.\u003c/p\u003e\n\u003cp\u003eIf after analyzing the commits the version number has to be changed, the command\nupdates the files we tell it to (i.e. the \u003ccode\u003epackage.json\u003c/code\u003e file for \u003ccode\u003enodejs\u003c/code\u003e\nprojects and possibly a \u003ccode\u003eCHANGELOG.md\u003c/code\u003e file), creates a new commit with the\nchanged files, creates a tag with the new version and pushes the changes to the\nrepository.\u003c/p\u003e\n\u003cp\u003eWhen running on a CI/CD system we usually generate the artifacts related to a\nrelease (a package, a container image, etc.) from the tag, as it includes the\nright version number and usually has passed all the required tests (it is a good\nidea to run the tests again in any case, as someone could create a tag manually\nor we could run extra jobs when building the final assets …​ if they fail it is\nnot a big issue anyway, numbers are cheap and infinite, so we can skip releases\nif needed).\u003c/p\u003e\n\u003csection class=\"doc-section level-2\"\u003e\u003ch3 id=\"_commit_messages_and_versioning\"\u003eCommit messages and versioning\u003c/h3\u003e\u003cp\u003eThe commit messages must follow a known format, the default module used to\n\u003ca href=\"https://github.com/semantic-release/commit-analyzer\" target=\"_blank\" rel=\"noopener\"\u003eanalyze\u003c/a\u003e them uses the\n\u003ca href=\"https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#-git-commit-guidelines\" target=\"_blank\" rel=\"noopener\"\u003eangular\u003c/a\u003e\ngit commit guidelines, but I prefer the\n\u003ca href=\"https://www.conventionalcommits.org/\" target=\"_blank\" rel=\"noopener\"\u003econventional commits\u003c/a\u003e one, mainly because\nit’s a lot easier to use when you want to update the \u003ccode\u003eMAJOR\u003c/code\u003e version.\u003c/p\u003e\n\u003cp\u003eThe commit message format used must be:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode\u003e\u0026lt;type\u0026gt;(optional scope): \u0026lt;description\u0026gt;\n\n[optional body]\n\n[optional footer(s)]\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"GitLab CI/CD Tips: Automatic Versioning Using semantic-release"},{"content":"This post describes how to define and use rule templates with semantic names using extends or !reference tags, how to define manual jobs using the same templates and how to use gitlab-ci inputs as macros to give names to regular expressions used by rules.\nBasic rule templatesI keep my templates in a rules.yml file stored on a common repository used from different projects as I mentioned on my previous post, but they can be defined anywhere, the important thing is that the files that need them include their definition somehow.\nThe first version of my rules.yml file was as follows:\n.rules_common: # Common rules; we include them from others instead of forcing a workflow rules: # Disable branch pipelines while there is an open merge request from it - if: \u0026gt;- $CI_COMMIT_BRANCH \u0026amp;\u0026amp; $CI_OPEN_MERGE_REQUESTS \u0026amp;\u0026amp; $CI_PIPELINE_SOURCE != \u0026#34;merge_request_event\u0026#34; when: never .rules_default: # Default rules, we need to add the when: on_success to make things work rules: - !reference [.rules_common, rules] - when: on_success The main idea is that .rules_common defines a rule section to disable jobs as we can do on a workflow definition; in our case common rules only have if rules that apply to all jobs and are used to disable them. The example includes one that avoids creating duplicated jobs when we push to a branch that is the source of an open MR as explained here.\nTo use the rules in a job we have two options, use the extends keyword (we do that when we want to use the rule as is) or declare a rules section and add a !reference to the template we want to use as described here (we do that when we want to add additional rules to disable a job before evaluating the template conditions).\nAs an example, with the following definitions both jobs use the same rules:\njob_1: extends: - .rules_default [...] job_2: rules: - !reference [.rules_default, rules] [...] Manual jobs and rule templatesTo make the jobs manual we have two options, create a version of the job that includes when: manual and defines if we want it to be optional or not (allow_failure: true makes the job optional, if we don’t add that to the rule the job is blocking) or add the when: manual and the allow_failure value to the job (if we work at the job level the default value for allow_failure is false for when: manual, so it is optional by default, we have to add an explicit allow_failure = true it to make it blocking).\nThe following example shows how we define blocking or optional manual jobs using rules with when conditions:\n.rules_default_manual_blocking: # Default rules for optional manual jobs rules: - !reference [.rules_common, rules] - when: manual # allow_failure: false is implicit .rules_default_manual_optional: # Default rules for optional manual jobs rules: - !reference [.rules_common, rules] - when: manual allow_failure: true manual_blocking_job: extends: - .rules_default_manual_blocking [...] manual_optional_job: extends: - .rules_default_manual_optional [...] The problem here is that we have to create new versions of the same rule template to add the conditions, but we can avoid it using the keywords at the job level with the original rules to get the same effect; the following definitions create jobs equivalent to the ones defined earlier without creating additional templates:\nmanual_blocking_job: extends: - .rules_default when: manual allow_failure: false [...] manual_optional_job: extends: - .rules_default when: manual # allow_failure: true is implicit [...] As you can imagine, that is my preferred way of doing it, as it keeps the rules.yml file smaller and I see that the job is manual in its definition without problem.\nRules with allow_failure, changes, exists, needs or variablesUnluckily for us, for now there is no way to avoid creating additional templates as we did on the when: manual case when a rule is similar to an existing one but adds changes, exists, needs or variables to it.\nSo, for now, if a rule needs to add any of those fields we have to copy the original rule and add the keyword section.\nSome notes, though:\nwe only need to add allow_failure if we want to change its value for a given condition, in other cases we can set the value at the job level.if we are adding changes to the rule it is important to make sure that they are going to be evaluated as explained here.when we add a needs value to a rule for a specific condition and it matches it replaces the job needs section; when using templates I would use two different job names with different conditions instead of adding a needs on a single job. Defining rule templates with semantic namesI started to use rule templates to avoid repetition when defining jobs that needed the same rules and soon I noticed that giving them names with a semantic meaning they where easier to use and understand (we provide a name that tells us when we are going to execute the job, while the details of the variables names or values used on the rules are an implementation detail of the templates).\nWe are not going to define real jobs on this post, but as an example we are going to define a set of rules that can be useful if we plan to follow a scaled trunk based development workflow when developing, that is, we are going to put the releasable code on the main branch and use short-lived branches to test and complete changes before pushing things to main.\nUsing this approach we can define an initial set of rule templates with semantic names:\n.rules_mr_to_main: rules: - !reference [.rules_common, rules] - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == \u0026#39;main\u0026#39; .rules_mr_or_push_to_main: rules: - !reference [.rules_common, rules] - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == \u0026#39;main\u0026#39; - if: \u0026gt;- $CI_COMMIT_BRANCH == \u0026#39;main\u0026#39; \u0026amp;\u0026amp; $CI_PIPELINE_SOURCE != \u0026#39;merge_request_event\u0026#39; .rules_push_to_main: rules: - !reference [.rules_common, rules] - if: \u0026gt;- $CI_COMMIT_BRANCH == \u0026#39;main\u0026#39; \u0026amp;\u0026amp; $CI_PIPELINE_SOURCE != \u0026#39;merge_request_event\u0026#39; .rules_push_to_branch: rules: - !reference [.rules_common, rules] - if: \u0026gt;- $CI_COMMIT_BRANCH != \u0026#39;main\u0026#39; \u0026amp;\u0026amp; $CI_PIPELINE_SOURCE != \u0026#39;merge_request_event\u0026#39; .rules_push_to_branch_or_mr_to_main: rules: - !reference [.rules_push_to_branch, rules] - if: \u0026gt;- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME != \u0026#39;main\u0026#39; \u0026amp;\u0026amp; $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == \u0026#39;main\u0026#39; .rules_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG =~ /^([0-9a-zA-Z_.-]+-)?v\\d+.\\d+.\\d+$/ .rules_non_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG !~ /^([0-9a-zA-Z_.-]+-)?v\\d+.\\d+.\\d+$/ With those names it is clear when a job is going to be executed and when using the templates on real jobs we can add additional restrictions and make the execution manual if needed as described earlier.\nUsing inputs as macrosOn the previous rules we have used a regular expression to identify the release tag format and assumed that the general branches are the ones with a name different than main; if we want to force a format for those branch names we can replace the condition != \u0026#39;main\u0026#39; by a regex comparison (=~ if we look for matches, !~ if we want to define valid branch names removing the invalid ones).\nWhen testing the new gitlab-ci inputs my colleague Jorge noticed that if you keep their default value they basically work as macros.\nThe variables declared as inputs can’t hold YAML values, the truth is that their value is always a string that is replaced by the value assigned to them when including the file (if given) or by their default value, if defined.\nIf you don’t assign a value to an input variable when including the file that declares it its occurrences are replaced by its default value, making them work basically as macros; this is useful for us when working with strings that can’t managed as variables, like the regular expressions used inside if conditions.\nWith those two ideas we can add the following prefix to the rules.yaml defining inputs for both regular expressions and replace the rules that can use them by the ones shown here:\nspec: inputs: # Regular expression for branches; the prefix matches the type of changes # we plan to work on inside the branch (we use conventional commit types as # the branch prefix) branch_regex: default: \u0026#39;/^(build|ci|chore|docs|feat|fix|perf|refactor|style|test)\\/.+$/\u0026#39; # Regular expression for tags release_tag_regex: default: \u0026#39;/^([0-9a-zA-Z_.-]+-)?v\\d+.\\d+.\\d+$/\u0026#39; --- [...] .rules_push_to_changes_branch: rules: - !reference [.rules_common, rules] - if: \u0026gt;- $CI_COMMIT_BRANCH =~ $[[ inputs.branch_regex ]] \u0026amp;\u0026amp; $CI_PIPELINE_SOURCE != \u0026#39;merge_request_event\u0026#39; .rules_push_to_branch_or_mr_to_main: rules: - !reference [.rules_push_to_branch, rules] - if: \u0026gt;- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ $[[ inputs.branch_regex ]] \u0026amp;\u0026amp; $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == \u0026#39;main\u0026#39; .rules_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG =~ $[[ inputs.release_tag_regex ]] .rules_non_release_tag: rules: - !reference [.rules_common, rules] - if: $CI_COMMIT_TAG !~ $[[ inputs.release_tag_regex ]] Creating rules reusing existing onesI’m going to finish this post with a comment about how I avoid defining extra rule templates in some common cases.\nThe idea is simple, we can use !reference tags to fine tune rules when we need to add conditions to disable them simply adding conditions with when: never before referencing the template.\nAs an example, in some projects I’m using different job definitions depending on the DEPLOY_ENVIRONMENT value to make the job manual or automatic; as we just said we can define different jobs referencing the same rule adding a condition to check if the environment is the one we are interested in:\ndeploy_job_auto: rules: # Only deploy automatically if the environment is \u0026#39;dev\u0026#39; by skipping this job # for other values of the DEPLOY_ENVIRONMENT variable - if: $DEPLOY_ENVIRONMENT != \u0026#34;dev\u0026#34; when: never - !reference [.rules_release_tag, rules] [...] deploy_job_manually: rules: # Disable this job if the environment is \u0026#39;dev\u0026#39; - if: $DEPLOY_ENVIRONMENT == \u0026#34;dev\u0026#34; when: never - !reference [.rules_release_tag, rules] when: manual # Change this to `false` to make the deployment job blocking allow_failure: true [...] If you think about it the idea of adding negative conditions is what we do with the .rules_common template; we add conditions to disable the job before evaluating the real rules.\nThe difference in that case is that we reference them at the beginning because we want those negative conditions on all jobs and that is also why we have a .rules_default condition with an when: on_success for the jobs that only need to respect the default workflow (we need the last condition to make sure that they are executed if the negative rules don’t match).\n","permalink":"https://blogops.mixinet.net/posts/gitlab-ci/rule_templates/","summary":"\u003cp\u003eThis post describes how to define and use rule templates with \u003cem\u003esemantic names\u003c/em\u003e using \u003ccode\u003eextends\u003c/code\u003e or \u003ccode\u003e!reference\u003c/code\u003e tags, how\nto define manual \u003cem\u003ejobs\u003c/em\u003e using the same templates and how to use \u003ca href=\"https://docs.gitlab.com/ee/ci/yaml/inputs.html\" target=\"_blank\" rel=\"noopener\"\u003egitlab-ci\ninputs\u003c/a\u003e as macros to give names to regular expressions used by rules.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_basic_rule_templates\"\u003eBasic rule templates\u003c/h2\u003e\u003cp\u003eI keep my templates in a \u003ccode\u003erules.yml\u003c/code\u003e file stored on a common repository used from different projects as I mentioned on\nmy previous \u003ca href=\"../common_ci_repo_with_assets/\"\u003epost\u003c/a\u003e, but they can be defined anywhere, the important thing is that the\nfiles that need them include their definition somehow.\u003c/p\u003e\n\u003cp\u003eThe first version of my \u003ccode\u003erules.yml\u003c/code\u003e file was as follows:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"yaml\"\u003e\u003cspan class=\"na\"\u003e.rules_common\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"c1\"\u003e# Common rules; we include them from others instead of forcing a workflow\u003c/span\u003e\n  \u003cspan class=\"na\"\u003erules\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"c1\"\u003e# Disable branch pipelines while there is an open merge request from it\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003eif\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e\u0026gt;-\u003c/span\u003e\n        \u003cspan class=\"s\"\u003e$CI_COMMIT_BRANCH \u0026amp;\u0026amp;\u003c/span\u003e\n        \u003cspan class=\"s\"\u003e$CI_OPEN_MERGE_REQUESTS \u0026amp;\u0026amp;\u003c/span\u003e\n        \u003cspan class=\"s\"\u003e$CI_PIPELINE_SOURCE != \u0026#34;merge_request_event\u0026#34;\u003c/span\u003e\n      \u003cspan class=\"na\"\u003ewhen\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003enever\u003c/span\u003e\n\n\u003cspan class=\"na\"\u003e.rules_default\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"c1\"\u003e# Default rules, we need to add the when: on_success to make things work\u003c/span\u003e\n  \u003cspan class=\"na\"\u003erules\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"kt\"\u003e!reference\u003c/span\u003e \u003cspan class=\"pi\"\u003e[\u003c/span\u003e\u003cspan class=\"nv\"\u003e.rules_common\u003c/span\u003e\u003cspan class=\"pi\"\u003e,\u003c/span\u003e \u003cspan class=\"nv\"\u003erules\u003c/span\u003e\u003cspan class=\"pi\"\u003e]\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ewhen\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eon_success\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"GitLab CI/CD Tips: Using Rule Templates"},{"content":"This post describes how to handle files that are used as assets by jobs and pipelines defined on a common gitlab-ci repository when we include those definitions from a different project.\nProblem descriptionWhen a .giltlab-ci.yml file includes files from a different repository its contents are expanded and the resulting code is the same as the one generated when the included files are local to the repository.\nIn fact, even when the remote files include other files everything works right, as they are also expanded (see the description of how included files are merged for a complete explanation), allowing us to organise the common repository as we want.\nAs an example, suppose that we have the following script on the assets/ folder of the common repository:\ndumb.sh #!/bin/sh echo \u0026#34;The script arguments are: \u0026#39;$@\u0026#39;\u0026#34; If we run the following job on the common repository:\njob: script: - $CI_PROJECT_DIR/assets/dumb.sh ARG1 ARG2 the output will be:\nThe script arguments are: \u0026#39;ARG1 ARG2\u0026#39; But if we run the same job from a different project that includes the same job definition the output will be different:\n/scripts-23-19051/step_script: eval: line 138: d./assets/dumb.sh: not found The problem here is that we include and expand the YAML files, but if a script wants to use other files from the common repository as an asset (configuration file, shell script, template, etc.), the execution fails if the files are not available on the project that includes the remote job definition.\nSolutionsWe can solve the issue using multiple approaches, I’ll describe two of them:\nCreate files using scriptsDownload files from the common repository Create files using scriptsOne way to dodge the issue is to generate the non YAML files from scripts included on the pipelines using HERE documents.\nThe problem with this approach is that we have to put the content of the files inside a script on a YAML file and if it uses characters that can be replaced by the shell (remember, we are using HERE documents) we have to escape them (error prone) or encode the whole file into base64 or something similar, making maintenance harder.\nAs an example, imagine that we want to use the dumb.sh script presented on the previous section and we want to call it from the same PATH of the main project (on the examples we are using the same folder, in practice we can create a hidden folder inside the project directory or use a PATH like /tmp/assets-$CI_JOB_ID to leave things outside the project folder and make sure that there will be no collisions if two jobs are executed on the same place (i.e. when using a ssh runner).\nTo create the file we will use hidden jobs to write our script template and reference tags to add it to the scripts when we want to use them.\nHere we have a snippet that creates the file with cat:\n.file_scripts: create_dumb_sh: - | # Create dumb.sh script mkdir -p \u0026#34;${CI_PROJECT_DIR}/assets\u0026#34; cat \u0026gt;\u0026#34;${CI_PROJECT_DIR}/assets/dumb.sh\u0026#34; \u0026lt;\u0026lt;EOF #!/bin/sh echo \u0026#34;The script arguments are: \u0026#39;\\$@\u0026#39;\u0026#34; EOF chmod +x \u0026#34;${CI_PROJECT_DIR}/assets/dumb.sh\u0026#34; Note that to make things work we’ve added 6 spaces before the script code and escaped the dollar sign.\nTo do the same using base64 we replace the previous snippet by this:\n.file_scripts: create_dumb_sh: - | # Create dumb.sh script mkdir -p \u0026#34;${CI_PROJECT_DIR}/assets\u0026#34; base64 -d \u0026gt;\u0026#34;${CI_PROJECT_DIR}/assets/dumb.sh\u0026#34; \u0026lt;\u0026lt;EOF IyEvYmluL3NoCmVjaG8gIlRoZSBzY3JpcHQgYXJndW1lbnRzIGFyZTogJyRAJyIK EOF chmod +x \u0026#34;${CI_PROJECT_DIR}/assets/dumb.sh\u0026#34; Again, we have to indent the base64 version of the file using 6 spaces (all lines of the base64 output have to be indented) and to make changes we have to decode and re-code the file manually, making it harder to maintain.\nWith either version we just need to add a !reference before using the script, if we add the call on the first lines of the before_script we can use the downloaded file in the before_script, script or after_script sections of the job without problems:\njob: before_script: - !reference [.file_scripts, create_dumb_sh] script: - ${CI_PROJECT_DIR}/assets/dumb.sh ARG1 ARG2 The output of a pipeline that uses this job will be the same as the one shown in the original example:\nThe script arguments are: \u0026#39;ARG1 ARG2\u0026#39; Download the files from the common repositoryAs we’ve seen the previous solution works but is not ideal as it makes the files harder to read, maintain and use.\nAn alternative approach is to keep the assets on a directory of the common repository (in our examples we will name it assets) and prepare a YAML file that declares some variables (i.e. the URL of the templates project and the PATH where we want to download the files) and defines a script fragment to download the complete folder.\nOnce we have the YAML file we just need to include it and add a reference to the script fragment at the beginning of the before_script of the jobs that use files from the assets directory and they will be available when needed.\nThe following file is an example of the YAML file we just mentioned:\nbootstrap.yml variables: CI_TMPL_API_V4_URL: \u0026#34;${CI_API_V4_URL}/projects/common%2Fci-templates\u0026#34; CI_TMPL_ARCHIVE_URL: \u0026#34;${CI_TMPL_API_V4_URL}/repository/archive\u0026#34; CI_TMPL_ASSETS_DIR: \u0026#34;/tmp/assets-${CI_JOB_ID}\u0026#34; .scripts_common: bootstrap_ci_templates: - | # Downloading assets echo \u0026#34;Downloading assets\u0026#34; mkdir -p \u0026#34;$CI_TMPL_ASSETS_DIR\u0026#34; wget -q -O - --header=\u0026#34;PRIVATE-TOKEN: $CI_TMPL_READ_TOKEN\u0026#34; \\ \u0026#34;$CI_TMPL_ARCHIVE_URL?path=assets\u0026amp;sha=${CI_TMPL_REF:-main}\u0026#34; | tar --strip-components 2 -C \u0026#34;$ASSETS_DIR\u0026#34; -xzf - Note: The script fragment that downloads the folder uses wget and tar because both tools are available on alpine containers and on debian based distributions we need to install a tool to download files from the web anyway (if we use curl instead of wget we are forced to install the package on alpine images too).\nThe file defines the following variables:\nCI_TMPL_API_V4_URL: URL of the common project, in our case we are using the project ci-templates inside the common group (note that the slash between the group and the project is escaped, that is needed to reference the project by name, if we don’t like that approach we can replace the url encoded path by the project id, i.e. we could use a value like ${CI_API_V4_URL}/projects/31)CI_TMPL_ARCHIVE_URL: Base URL to use the gitlab API to download files from a repository, we will add the arguments path and sha to select which sub path to download and from which commit, branch or tag (we will explain later why we use the CI_TMPL_REF, for now just keep in mind that if it is not defined we will download the version of the files available on the main branch when the job is executed).CI_TMPL_ASSETS_DIR: Destination of the downloaded files. And uses variables defined in other places:\nCI_TMPL_READ_TOKEN: token that includes the read_api scope for the common project, we need it because the tokens created by the CI/CD pipelines of other projects can’t be used to access the api of the common one.\nWe define the variable on the gitlab CI/CD variables section to be able to change it if needed (i.e. if it expires)\nCI_TMPL_REF: branch or tag of the common repo from which to get the files (we need that to make sure we are using the right version of the files, i.e. when testing we will use a branch and on production pipelines we can use fixed tags to make sure that the assets don’t change between executions unless we change the reference).\nWe will set the value on the .gitlab-ci.yml file of the remote projects and will use the same reference when including the files to make sure that everything is coherent.\nThis is an example YAML file that defines a pipeline with a job that uses the script from the common repository:\npipeline.yml include: - /bootstrap.yaml stages: - test dumb_job: stage: test before_script: - !reference [.bootstrap_ci_templates, create_dumb_sh] script: - ${CI_TMPL_ASSETS_DIR}/dumb.sh ARG1 ARG2 To use it from an external project we will use the following gitlab ci configuration:\ngitlab-ci.yml include: - project: \u0026#39;common/ci-templates\u0026#39; ref: \u0026amp;ciTmplRef \u0026#39;main\u0026#39; file: \u0026#39;/pipeline.yml\u0026#39; variables: CI_TMPL_REF: *ciTmplRef Where we use a YAML anchor to ensure that we use the same reference when including and when assigning the value to the CI_TMPL_REF variable (as far as I know we have to pass the ref value explicitly to know which reference was used when including the file, the anchor allows us to make sure that the value is always the same in both places).\nThe reference we use is quite important for the reproducibility of the jobs, if we don’t use fixed tags or commit hashes as references each time a job that downloads the files is executed we can get different versions of them.\nFor that reason is not a bad idea to create tags on our common repo and use them as reference on the projects or branches that we want to behave as if their CI/CD configuration was local (if we point to a fixed version of the common repo the way everything is going to work is almost the same as having the pipelines directly in our repo).\nBut while developing pipelines using branches as references is a really useful option; it allows us to re-run the jobs that we want to test and they will download the latest versions of the asset files on the branch, speeding up the testing process.\nHowever keep in mind that the trick only works with the asset files, if we change a job or a pipeline on the YAML files restarting the job is not enough to test the new version as the restart uses the same job created with the current pipeline.\nTo try the updated jobs we have to create a new pipeline using a new action against the repository or executing the pipeline manually.\nConclusionFor now I’m using the second solution and as it is working well my guess is that I’ll keep using that approach unless giltab itself provides a better or simpler way of doing the same thing.\n","permalink":"https://blogops.mixinet.net/posts/gitlab-ci/common_ci_repo_with_assets/","summary":"\u003cp\u003eThis post describes how to handle files that are used as \u003cem\u003eassets\u003c/em\u003e by jobs and pipelines defined on a common gitlab-ci\nrepository when we include those definitions from a different project.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_problem_description\"\u003eProblem description\u003c/h2\u003e\u003cp\u003eWhen a \u003ccode\u003e.giltlab-ci.yml\u003c/code\u003e file \u003ca href=\"https://docs.gitlab.com/ee/ci/yaml/includes.html\" target=\"_blank\" rel=\"noopener\"\u003eincludes\u003c/a\u003e files from a different\nrepository its contents are expanded and the resulting code is the same as the one generated when the included files\nare local to the repository.\u003c/p\u003e\n\u003cp\u003eIn fact, even when the remote files include other files everything works right, as they are also expanded (see the\ndescription of how included files are \u003ca href=\"https://docs.gitlab.com/ee/ci/yaml/includes.html#merge-method-for-include\" target=\"_blank\" rel=\"noopener\"\u003emerged\u003c/a\u003e\nfor a complete explanation), allowing us to organise the common repository as we want.\u003c/p\u003e\n\u003cp\u003eAs an example, suppose that we have the following script on the \u003ccode\u003eassets/\u003c/code\u003e folder of the common repository:\u003c/p\u003e\n\u003cfigure class=\"listing-block\"\u003e\u003cfigcaption\u003e\u003ccode\u003edumb.sh\u003c/code\u003e\u003c/figcaption\u003e\n\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"shell\"\u003e\u003cspan class=\"c\"\u003e#!/bin/sh\u003c/span\u003e\n\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;The script arguments are: \u0026#39;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$@\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#39;\u0026#34;\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/figure\u003e\n\u003cp\u003eIf we run the following job on the common repository:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"yaml\"\u003e\u003cspan class=\"na\"\u003ejob\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003escript\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s\"\u003e$CI_PROJECT_DIR/assets/dumb.sh ARG1 ARG2\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"GitLab CI/CD Tips: Using a Common CI Repository with Assets"},{"content":"This post describes how to deploy cilium (and hubble) using docker on a Linux system with k3d or kind to test it as CNI and Service Mesh.\nI wrote some scripts to do a local installation and evaluate cilium to use it at work (in fact we are using cilium on an EKS cluster now), but I thought it would be a good idea to share my original scripts in this blog just in case they are useful to somebody, at least for playing a little with the technology.\nLinksAs there is no point on explaining here all the concepts related to cilium I’m providing some links for the reader interested on reading about it:\nWhat is CNI?What is Cilium?What is eBPF?What is Hubble?Why use Cilium with Kubernetes? All the scripts and configuration files discussed in this post are available on my cilium-docker git repository.\nInstallationFor each platform we are going to deploy two clusters on the same docker network; I’ve chosen this model because it allows the containers to see the addresses managed by metallb from both clusters (the idea is to use those addresses for load balancers and treat them as if they were public).\nThe installation(s) use cilium as CNI, metallb for BGP (I tested the cilium options, but I wasn’t able to configure them right) and nginx as the ingress controller (again, I tried to use cilium but something didn’t work either).\nTo be able to use the previous components some default options have been disabled on k3d and kind and, in the case of k3d, a lot of k3s options (traefik, servicelb, kubeproxy, network-policy, …​) have also been disabled to avoid conflicts.\nTo use the scripts we need to install cilium, docker, helm, hubble, k3d, kind, kubectl and tmpl in our system.\nAfter cloning the repository, the sbin/tools.sh script can be used to do that on a linux-amd64 system:\n$ git clone https://forgejo.mixinet.net/blogops/cilium-docker.git $ cd cilium-docker $ ./sbin/tools.sh apps Note: For the rest of the post we will assume that our working directory is the root of the repository checkout.\nOnce we have the tools, to install everything on k3d (for kind replace k3d by kind) we can use the sbin/cilium-install.sh script as follows:\n$ # Deploy first k3d cluster with cilium \u0026amp; cluster-mesh $ ./sbin/cilium-install.sh k3d 1 full [...] $ # Deploy second k3d cluster with cilium \u0026amp; cluster-mesh $ ./sbin/cilium-install.sh k3d 2 full [...] $ # The 2nd cluster-mesh installation connects the clusters If we run the command cilium status after the installation we should get an output similar to the one seen on the following screenshot:\nThe installation script uses the following templates:\ntmpl/k3d-config.yaml: configuration to deploy the k3d cluster.tmpl/kind-config.yaml: configuration to deploy the kind cluster.tmpl/metallb-crds.yaml and tmpl/ippols.yaml: configurations for the metallb deployment.tmpl/cilium.yaml: values to deploy the cilium using the helm chart. Once we have finished our tests we can remove the installation using the sbin/cilium-remove.sh script.\nSome notes about the configurationAs noted on the documentation, the cilium deployment needs to mount the bpffs on /sys/fs/bpf and cgroupv2 on /run/cilium/cgroupv2; that is done automatically on kind, but fails on k3d because the image does not include bash (see this issue).\nTo fix it we mount a script on all the k3d containers that is executed each time they are started (the script is mounted as /bin/k3d-entrypoint-cilium.sh because the /bin/k3d-entrypoint.sh script executes the scripts that follow the pattern /bin/k3d-entrypoint-*.sh before launching the k3s daemon).\nThe source code of the script is available here.\nWhen testing the multi-cluster deployment with k3d we have found issues with open files, looks like they are related to inotify (see this page on the kind documentation); adding the following to the /etc/sysctl.conf file fixed the issue:\n# fix inotify issues with docker \u0026amp; k3d fs.inotify.max_user_watches = 524288 fs.inotify.max_user_instances = 512Although the deployment theoretically supports it, we are not using cilium as the cluster ingress yet (it did not work, so it is no longer enabled) and we are also ignoring the gateway-api for now.The documentation uses the cilium cli to do all the installations, but I noticed that following that route the current version does not work right with hubble (it messes up the TLS support, there are some notes about the problems on this cilium issue), so we are deploying with helm right now.\nThe problem with the helm approach is that there is no official documentation on how to install the cluster mesh with it (there is a request for documentation here), so we are using the cilium cli for now and it looks that it does not break the hubble configuration.\nTestsTo test cilium we have used some scripts \u0026amp; additional config files that are available on the test sub directory of the repository:\ncilium-connectivity.sh: a script that runs the cilium connectivity test for one cluster or in multi cluster mode (for mesh testing).\nIf we export the variable HUBBLE_PF=true the script executes the command cilium hubble port-forward before launching the tests.\nhttp-sw.sh: Simple tests for cilium policies from the cilium demo; the script deploys the Star Wars demo application and allows us to add the L3/L4 policy or the L3/L4/L7 policy, test the connectivity and view the policies.ingress-basic.sh: This test is for checking the ingress controller, it is prepared to work against cilium and nginx, but as explained before the use of cilium as an ingress controller is not working as expected, so the idea is to call it with nginx always as the first argument for now.mesh-test.sh: Tool to deploy a global service on two clusters, change the service affinity to local or remote, enable or disable if the service is shared and test how the tools respond. Running the testsThe cilium-connectivity.sh executes the standard cilium tests:\n$ ./test/cilium-connectivity.sh k3d 12 ℹ️ Monitor aggregation detected, will skip some flow validation steps ✨ [k3d-cilium1] Creating namespace cilium-test for connectivity check... ✨ [k3d-cilium2] Creating namespace cilium-test for connectivity check... [...] ✅ All 33 tests (248 actions) successful, 2 tests skipped, 0 scenarios skipped. To test how the cilium policies work use the http-sw.sh script:\nkubectx k3d-cilium2 # (just in case) # Create test namespace and services ./test/http-sw.sh create # Test without policies (exaust-port fails by design) ./test/http-sw.sh test # Create and view L3/L4 CiliumNetworkPolicy ./test/http-sw.sh policy-l34 # Test policy (no access from xwing, exaust-port fails) ./test/http-sw.sh test # Create and view L7 CiliumNetworkPolicy ./test/http-sw.sh policy-l7 # Test policy (no access from xwing, exaust-port returns 403) ./test/http-sw.sh test # Delete http-sw test ./test/http-sw.sh delete And to see how the service mesh works use the mesh-test.sh script:\n# Create services on both clusters and test ./test/mesh-test.sh k3d create ./test/mesh-test.sh k3d test # Disable service sharing from cluster 1 and test ./test/mesh-test.sh k3d svc-shared-false ./test/mesh-test.sh k3d test # Restore sharing, set local affinity and test ./test/mesh-test.sh k3d svc-shared-default ./test/mesh-test.sh k3d svc-affinity-local ./test/mesh-test.sh k3d test # Delete deployment from cluster 1 and test ./test/mesh-test.sh k3d delete-deployment ./test/mesh-test.sh k3d test # Delete test ./test/mesh-test.sh k3d delete ","permalink":"https://blogops.mixinet.net/posts/testing_cilium_with_k3d_and_kind/","summary":"\u003cp\u003eThis post describes how to deploy \u003ca href=\"https://github.com/cilium/cilium\" target=\"_blank\" rel=\"noopener\"\u003ecilium\u003c/a\u003e (and\n\u003ca href=\"https://github.com/cilium/hubble\" target=\"_blank\" rel=\"noopener\"\u003ehubble\u003c/a\u003e) using \u003ccode\u003edocker\u003c/code\u003e on a Linux system with\n\u003ca href=\"https://k3d.io/\" target=\"_blank\" rel=\"noopener\"\u003ek3d\u003c/a\u003e or \u003ca href=\"https://kind.sigs.k8s.io/\" target=\"_blank\" rel=\"noopener\"\u003ekind\u003c/a\u003e to test it as\n\u003ca href=\"https://www.cni.dev/\" target=\"_blank\" rel=\"noopener\"\u003eCNI\u003c/a\u003e and\n\u003ca href=\"https://en.wikipedia.org/wiki/Service_mesh\" target=\"_blank\" rel=\"noopener\"\u003eService Mesh\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eI wrote some scripts to do a local installation and evaluate \u003ccode\u003ecilium\u003c/code\u003e to use it\nat \u003ca href=\"https://intelygenz.com/\" target=\"_blank\" rel=\"noopener\"\u003ework\u003c/a\u003e (in fact we are using \u003ccode\u003ecilium\u003c/code\u003e on an EKS\ncluster now), but I thought it would be a good idea to share my original\nscripts in this blog just in case they are useful to somebody, at least for\nplaying a little with the technology.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_links\"\u003eLinks\u003c/h2\u003e\u003cp\u003eAs there is no point on explaining here all the concepts related to \u003ccode\u003ecilium\u003c/code\u003e\nI’m providing some links for the reader interested on reading about it:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003eWhat is \u003ca href=\"https://www.cni.dev/\" target=\"_blank\" rel=\"noopener\"\u003eCNI\u003c/a\u003e?\u003c/li\u003e\u003cli\u003eWhat is\n\u003ca href=\"https://docs.cilium.io/en/stable/overview/intro/#what-is-cilium\" target=\"_blank\" rel=\"noopener\"\u003eCilium\u003c/a\u003e?\u003c/li\u003e\u003cli\u003eWhat is \u003ca href=\"https://ebpf.io/what-is-ebpf/\" target=\"_blank\" rel=\"noopener\"\u003eeBPF\u003c/a\u003e?\u003c/li\u003e\u003cli\u003eWhat is\n\u003ca href=\"https://docs.cilium.io/en/stable/overview/intro/#what-is-hubble\" target=\"_blank\" rel=\"noopener\"\u003eHubble\u003c/a\u003e?\u003c/li\u003e\u003cli\u003eWhy use \u003ca href=\"https://blog.palark.com/why-cilium-for-kubernetes-networking/\" target=\"_blank\" rel=\"noopener\"\u003eCilium\nwith Kubernetes\u003c/a\u003e?\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Testing cilium with k3d and kind"},{"content":"This entry explains how I have configured a linux bridge, dnsmasq and iptables to be able to run and communicate different virtualization systems and containers on laptops running Debian GNU/Linux.\nI’ve used different variations of this setup for a long time with VirtualBox and KVM for the Virtual Machines and Linux-VServer, OpenVZ, LXC and lately Docker or Podman for the Containers.\nRequired packagesI’m running Debian Sid with systemd and network-manager to configure the WiFi and Ethernet interfaces, but for the bridge I use bridge-utils with ifupdown (as I said this setup is old, I guess ifupdow2 and ifupdown-ng will work too).\nTo start and stop the DNS and DHCP services and add NAT rules when the bridge is brought up or down I execute a script that uses:\nip from iproute2 to get the network information,dnsmasq to provide the DNS and DHCP services (currently only the dnsmasq-base package is needed and it is recommended by network-manager, so it is probably installed),iptables to configure NAT (for now docker kind of forces me to keep using iptables, but at some point I’d like to move to nftables). To make sure you have everything installed you can run the following command:\nsudo apt install bridge-utils dnsmasq-base ifupdown iproute2 iptables Bridge configurationThe bridge configuration for ifupdow is available on the file /etc/network/interfaces.d/vmbr0:\n# Virtual servers NAT Bridge auto vmbr0 iface vmbr0 inet static address 10.0.4.1 network 10.0.4.0 netmask 255.255.255.0 broadcast 10.0.4.255 bridge_ports none bridge_maxwait 0 up /usr/local/sbin/vmbridge ${IFACE} start nat pre-down /usr/local/sbin/vmbridge ${IFACE} stop nat Warning: To use a separate file with ifupdown make sure that /etc/network/interfaces contains the line:\nsource /etc/network/interfaces.d/* or add its contents to /etc/network/interfaces directly, if you prefer.\nThis configuration creates a bridge with the address 10.0.4.1 and assumes that the machines connected to it will use the 10.0.4.0/24 network; you can change the network address if you want, as long as you use a private range and it does not collide with networks used in your Virtual Machines all should be OK.\nNote: All my configurations are using IPv4 for now, I was planning to move some things to IPv6 not so long ago, but the truth is that I haven’t had the need nor the time and the Spanish Internet providers are not helping either.\nThe vmbridge script is used to start the dnsmasq server and setup the NAT rules when the interface is brought up and remove the firewall rules and stop the dnsmasq server when it is brought down.\nThe vmbridge scriptThe vmbridge script launches an instance of dnsmasq\tthat binds to the bridge interface (vmbr0 in our case) that is used as DNS and DHCP server.\nThe DNS server reads the /etc/hosts file to publish local DNS names and forwards all the other requests to the the dnsmasq server launched by NetworkManager that is listening on the loopback interface.\nAs this server already does catching we disable it for our server, with the added advantage that, if we change networks, new requests go to the new resolvers because the DNS server handled by NetworkManager gets restarted and flushes its cache (this is useful if we connect to a new network that has internal DNS servers that are configured to do split DNS for internal services; if we use this model all requests get the internal address as soon as the DNS server is queried again).\nThe DHCP server is configured to provide IPs to unknown hosts for a sub range of the addresses on the bridge network and use fixed IPs if the /etc/ethers file has a MAC with a matching hostname on the /etc/hosts file.\nTo make things work with old DHCP clients the script also adds checksums to the DHCP packets using iptables (when the interface is not linked to a physical device the kernel does not add checksums, but we can fix it adding a rule on the mangle table).\nIf we want external connectivity we can pass the nat argument and then the script creates a MASQUERADE rule for the bridge network and enables IP forwarding.\nThe script source code is the following:\n/usr/local/sbin/vmbridge #!/bin/sh set -e # --------- # VARIABLES # --------- LOCAL_DOMAIN=\u0026#34;vmnet\u0026#34; MIN_IP_LEASE=\u0026#34;192\u0026#34; MAX_IP_LEASE=\u0026#34;223\u0026#34; # --------- # FUNCTIONS # --------- get_net() { NET=\u0026#34;$( ip a ls \u0026#34;${BRIDGE}\u0026#34; 2\u0026gt;/dev/null | sed -ne \u0026#39;s/^.*inet \\(.*\\) brd.*$/\\1/p\u0026#39; )\u0026#34; [ \u0026#34;$NET\u0026#34; ] || return 1 } checksum_fix_start() { iptables -t mangle -A POSTROUTING -o \u0026#34;${BRIDGE}\u0026#34; -p udp --dport 68 \\ -j CHECKSUM --checksum-fill 2\u0026gt;/dev/null || true } checksum_fix_stop() { iptables -t mangle -D POSTROUTING -o \u0026#34;${BRIDGE}\u0026#34; -p udp --dport 68 \\ -j CHECKSUM --checksum-fill 2\u0026gt;/dev/null || true } nat_start() { [ \u0026#34;$NAT\u0026#34; = \u0026#34;yes\u0026#34; ] || return 0 # Configure NAT iptables -t nat -A POSTROUTING -s \u0026#34;${NET}\u0026#34; ! -d \u0026#34;${NET}\u0026#34; -j MASQUERADE # Enable forwarding (just in case) echo 1 \u0026gt;/proc/sys/net/ipv4/ip_forward } nat_stop() { [ \u0026#34;$NAT\u0026#34; = \u0026#34;yes\u0026#34; ] || return 0 iptables -t nat -D POSTROUTING -s \u0026#34;${NET}\u0026#34; ! -d \u0026#34;${NET}\u0026#34; \\ -j MASQUERADE 2\u0026gt;/dev/null || true } do_start() { # Bridge address _addr=\u0026#34;${NET%%/*}\u0026#34; # DNS leases (between .MIN_IP_LEASE and .MAX_IP_LEASE) _dhcp_range=\u0026#34;${_addr%.*}.${MIN_IP_LEASE},${_addr%.*}.${MAX_IP_LEASE}\u0026#34; # Bridge mtu _mtu=\u0026#34;$( ip link show dev \u0026#34;${BRIDGE}\u0026#34; | sed -n -e \u0026#39;/mtu/ { s/^.*mtu \\([0-9]\\+\\).*$/\\1/p }\u0026#39; )\u0026#34; # Compute extra dnsmasq options dnsmasq_extra_opts=\u0026#34;\u0026#34; # Disable gateway when not using NAT if [ \u0026#34;$NAT\u0026#34; != \u0026#34;yes\u0026#34; ]; then dnsmasq_extra_opts=\u0026#34;$dnsmasq_extra_opts --dhcp-option=3\u0026#34; fi # Adjust MTU size if needed if [ -n \u0026#34;$_mtu\u0026#34; ] \u0026amp;\u0026amp; [ \u0026#34;$_mtu\u0026#34; -ne \u0026#34;1500\u0026#34; ]; then dnsmasq_extra_opts=\u0026#34;$dnsmasq_extra_opts --dhcp-option=26,$_mtu\u0026#34; fi # shellcheck disable=SC2086 dnsmasq --bind-interfaces \\ --cache-size=\u0026#34;0\u0026#34; \\ --conf-file=\u0026#34;/dev/null\u0026#34; \\ --dhcp-authoritative \\ --dhcp-leasefile=\u0026#34;/var/lib/misc/dnsmasq.${BRIDGE}.leases\u0026#34; \\ --dhcp-no-override \\ --dhcp-range \u0026#34;${_dhcp_range}\u0026#34; \\ --domain=\u0026#34;${LOCAL_DOMAIN}\u0026#34; \\ --except-interface=\u0026#34;lo\u0026#34; \\ --expand-hosts \\ --interface=\u0026#34;${BRIDGE}\u0026#34; \\ --listen-address \u0026#34;${_addr}\u0026#34; \\ --no-resolv \\ --pid-file=\u0026#34;${PIDF}\u0026#34; \\ --read-ethers \\ --server=\u0026#34;127.0.0.1\u0026#34; \\ $dnsmasq_extra_opts checksum_fix_start nat_start } do_stop() { nat_stop checksum_fix_stop if [ -f \u0026#34;${PIDF}\u0026#34; ]; then kill \u0026#34;$(cat \u0026#34;${PIDF}\u0026#34;)\u0026#34; || true rm -f \u0026#34;${PIDF}\u0026#34; fi } do_status() { if [ -f \u0026#34;${PIDF}\u0026#34; ] \u0026amp;\u0026amp; kill -HUP \u0026#34;$(cat \u0026#34;${PIDF}\u0026#34;)\u0026#34;; then echo \u0026#34;dnsmasq RUNNING\u0026#34; else echo \u0026#34;dnsmasq NOT running\u0026#34; fi } do_reload() { [ -f \u0026#34;${PIDF}\u0026#34; ] \u0026amp;\u0026amp; kill -HUP \u0026#34;$(cat \u0026#34;${PIDF}\u0026#34;)\u0026#34; } usage() { echo \u0026#34;Uso: $0 BRIDGE (start|stop [nat])|status|reload\u0026#34; exit 1 } # ---- # MAIN # ---- [ \u0026#34;$#\u0026#34; -ge \u0026#34;2\u0026#34; ] || usage BRIDGE=\u0026#34;$1\u0026#34; OPTION=\u0026#34;$2\u0026#34; shift 2 NAT=\u0026#34;no\u0026#34; for arg in \u0026#34;$@\u0026#34;; do case \u0026#34;$arg\u0026#34; in nat) NAT=\u0026#34;yes\u0026#34; ;; *) echo \u0026#34;Unknown arg \u0026#39;$arg\u0026#39;\u0026#34; \u0026amp;\u0026amp; exit 1 ;; esac done PIDF=\u0026#34;/var/run/vmbridge-${BRIDGE}-dnsmasq.pid\u0026#34; case \u0026#34;$OPTION\u0026#34; in start) get_net \u0026amp;\u0026amp; do_start ;; stop) get_net \u0026amp;\u0026amp; do_stop ;; status) do_status ;; reload) get_net \u0026amp;\u0026amp; do_reload ;; *) echo \u0026#34;Unknown command \u0026#39;$OPTION\u0026#39;\u0026#34; \u0026amp;\u0026amp; exit 1 ;; esac # vim: ts=2:sw=2:et:ai:sts=2 NetworkManager ConfigurationThe default /etc/NetworkManager/NetworkManager.conf file has the following contents:\n[main] plugins=ifupdown,keyfile [ifupdown] managed=false Which means that it will leave interfaces managed by ifupdown alone and, by default, will send the connection DNS configuration to systemd-resolved if it is installed.\nAs we want to use dnsmasq for DNS resolution, but we don’t want NetworkManager to modify our /etc/resolv.conf we are going to add the following file (/etc/NetworkManager/conf.d/dnsmasq.conf) to our system:\n/etc/NetworkManager/conf.d/dnsmasq.conf [main] dns=dnsmasq rc-manager=unmanaged and restart the NetworkManager service:\n$ sudo systemctl restart NetworkManager.service From now on the NetworkManager will start a dnsmasq service that queries the servers provided by the DHCP servers we connect to on 127.0.0.1:53 but will not touch our /etc/resolv.conf file.\nNote: We are going to use the dnsmasq service managed by NetworkManager because it is updated automatically, consumes very little memory and avoids the need of extra tricks to make the server that we will be using for DNS \u0026amp; DHCP notice external DNS configuration changes.\nConfiguring systemd-resolvedIf we start using our own name server but our system has systemd-resolved installed we will no longer need or use the DNS stub; programs using it will use our dnsmasq server directly now, but we keep running systemd-resolved for the host programs that use its native api or access it through /etc/nsswitch.conf (when libnss-resolve is installed).\nTo disable the stub we add a /etc/systemd/resolved.conf.d/disable-stub.conf file to our machine with the following content:\n# Disable the DNS Stub Listener, we use our own dnsmasq [Resolve] DNSStubListener=no and restart the systemd-resolved to make sure that the stub is stopped:\n$ sudo systemctl restart systemd-resolved.service Adjusting /etc/resolv.confFirst we remove the existing /etc/resolv.conf file (it does not matter if it is a link or a regular file) and then create a new one that contains at least the following line (we can add a search line if is useful for us):\nnameserver 10.0.4.1 From now on we will be using the dnsmasq server launched when we bring up the vmbr0 for multiple systems:\nas our main DNS server from the host (if we use the standard /etc/nsswitch.conf and libnss-resolve is installed it is queried first, but the systemd-resolved uses it as forwarder by default if needed),as the DNS server of the Virtual Machines or containers that use DHCP for network configuration and attach their virtual interfaces to our bridge,as the DNS server of docker containers that get the DNS information from /etc/resolv.conf (if we have entries that use loopback addresses the containers that don’t use the host network tend to fail, as those addresses inside the running containers are not linked to the loopback device of the host). TestingAfter all the configuration files and scripts are in place we just need to bring up the bridge interface and check that everything works:\n$ # Bring interface up $ sudo ifup vmbr0 $ # Check that it is available $ ip a ls dev vmbr0 4: vmbr0: \u0026lt;NO-CARRIER,BROADCAST,MULTICAST,UP\u0026gt; mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 0a:b8:ef:b8:07:6c brd ff:ff:ff:ff:ff:ff inet 10.0.4.1/24 brd 10.0.4.255 scope global vmbr0 valid_lft forever preferred_lft forever $ # View the listening ports used by our dnsmasq servers $ sudo ss -tulpan | grep dnsmasq udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:((\u0026#34;dnsmasq\u0026#34;,pid=1733930,fd=4)) udp UNCONN 0 0 10.0.4.1:53 0.0.0.0:* users:((\u0026#34;dnsmasq\u0026#34;,pid=1705267,fd=6)) udp UNCONN 0 0 0.0.0.0%vmbr0:67 0.0.0.0:* users:((\u0026#34;dnsmasq\u0026#34;,pid=1705267,fd=4)) tcp LISTEN 0 32 10.0.4.1:53 0.0.0.0:* users:((\u0026#34;dnsmasq\u0026#34;,pid=1705267,fd=7)) tcp LISTEN 0 32 127.0.0.1:53 0.0.0.0:* users:((\u0026#34;dnsmasq\u0026#34;,pid=1733930,fd=5)) $ # Verify that the DNS server works on the vmbr0 address $ host www.debian.org 10.0.4.1 Name: 10.0.4.1 Address: 10.0.4.1#53 Aliases: www.debian.org has address 130.89.148.77 www.debian.org has IPv6 address 2001:67c:2564:a119::77 Managing running systemsIf we want to update DNS entries and/or MAC addresses we can edit the /etc/hosts and /etc/ethers files and reload the dnsmasq configuration using the vmbridge script:\n$ sudo /usr/local/sbin/vmbridge vmbr0 reload That call sends a signal to the running dnsmasq server and it reloads the files; after that we can refresh the DHCP addresses from the client machines or start using the new DNS names immediately.\n","permalink":"https://blogops.mixinet.net/posts/shared_networking_for_vms_and_containers/","summary":"\u003cp\u003eThis entry explains how I have configured a linux bridge, \u003ccode\u003ednsmasq\u003c/code\u003e and\n\u003ccode\u003eiptables\u003c/code\u003e to be able to run and communicate different virtualization systems\nand containers on laptops running \u003ca href=\"https://www.debian.org\" target=\"_blank\" rel=\"noopener\"\u003eDebian GNU/Linux\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eI’ve used different variations of this setup for a long time with\n\u003ca href=\"https://www.virtualbox.org/\" target=\"_blank\" rel=\"noopener\"\u003eVirtualBox\u003c/a\u003e and \u003ca href=\"https://www.linux-kvm.org/\" target=\"_blank\" rel=\"noopener\"\u003eKVM\u003c/a\u003e for\nthe \u003cstrong\u003eVirtual Machines\u003c/strong\u003e and \u003ca href=\"http://linux-vserver.org/\" target=\"_blank\" rel=\"noopener\"\u003eLinux-VServer\u003c/a\u003e,\n\u003ca href=\"https://openvz.org/\" target=\"_blank\" rel=\"noopener\"\u003eOpenVZ\u003c/a\u003e, \u003ca href=\"https://linuxcontainers.org/\" target=\"_blank\" rel=\"noopener\"\u003eLXC\u003c/a\u003e and lately\n\u003ca href=\"https://www.docker.com/\" target=\"_blank\" rel=\"noopener\"\u003eDocker\u003c/a\u003e or \u003ca href=\"https://podman.io/\" target=\"_blank\" rel=\"noopener\"\u003ePodman\u003c/a\u003e for the\n\u003cstrong\u003eContainers\u003c/strong\u003e.\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_required_packages\"\u003eRequired packages\u003c/h2\u003e\u003cp\u003eI’m running Debian Sid with \u003ccode\u003esystemd\u003c/code\u003e and \u003ccode\u003enetwork-manager\u003c/code\u003e to configure the\nWiFi and Ethernet interfaces, but for the bridge I use \u003ccode\u003ebridge-utils\u003c/code\u003e with\n\u003ccode\u003eifupdown\u003c/code\u003e (as I said this setup is old, I guess \u003ccode\u003eifupdow2\u003c/code\u003e and \u003ccode\u003eifupdown-ng\u003c/code\u003e\nwill work too).\u003c/p\u003e\n\u003cp\u003eTo start and stop the \u003ccode\u003eDNS\u003c/code\u003e and \u003ccode\u003eDHCP\u003c/code\u003e services and add NAT rules when the\nbridge is brought up or down I execute a script that uses:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003e\u003ccode\u003eip\u003c/code\u003e from \u003ccode\u003eiproute2\u003c/code\u003e to get the network information,\u003c/li\u003e\u003cli\u003e\u003ccode\u003ednsmasq\u003c/code\u003e to provide the \u003ccode\u003eDNS\u003c/code\u003e and \u003ccode\u003eDHCP\u003c/code\u003e services (currently only the\n\u003ccode\u003ednsmasq-base\u003c/code\u003e package is needed and it is recommended by \u003ccode\u003enetwork-manager\u003c/code\u003e,\nso it is probably installed),\u003c/li\u003e\u003cli\u003e\u003ccode\u003eiptables\u003c/code\u003e to configure NAT (for now \u003ccode\u003edocker\u003c/code\u003e kind of forces me to keep using\n\u003ccode\u003eiptables\u003c/code\u003e, but at some point I’d like to move to \u003ccode\u003enftables\u003c/code\u003e).\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Shared networking for Virtual Machines and Containers"},{"content":"This post describes how I’ve put together a simple static content server for kubernetes clusters using a Pod with a persistent volume and multiple containers: an sftp server to manage contents, a web server to publish them with optional access control and another one to run scripts which need access to the volume filesystem.\nThe sftp server runs using MySecureShell, the web server is nginx and the script runner uses the webhook tool to publish endpoints to call them (the calls will come from other Pods that run backend servers or are executed from Jobs or CronJobs).\nNote: This service has been developed for Kyso and the version used in our current architecture includes an additional container to index documents for Elasticsearch, but as it is not relevant for the description of the service as a general solution I’ve decided to ignore it on this post.\nHistoryThe system was developed because we had a NodeJS API with endpoints to upload files and store them on S3 compatible services that were later accessed via HTTPS, but the requirements changed and we needed to be able to publish folders instead of individual files using their original names and apply access restrictions using our API.\nThinking about our requirements the use of a regular filesystem to keep the files and folders was a good option, as uploading and serving files is simple.\nFor the upload I decided to use the sftp protocol, mainly because I already had an sftp container image based on mysecureshell prepared; once we settled on that we added sftp support to the API server and configured it to upload the files to our server instead of using S3 buckets.\nTo publish the files we added a nginx container configured to work as a reverse proxy that uses the ngx_http_auth_request_module to validate access to the files (the sub request is configurable, in our deployment we have configured it to call our API to check if the user can access a given URL).\nFinally we added a third container when we needed to execute some tasks directly on the filesystem (using kubectl exec with the existing containers did not seem a good idea, as that is not supported by CronJobs objects, for example).\nThe solution we found avoiding the NIH Syndrome (i.e. write our own tool) was to use the webhook tool to provide the endpoints to call the scripts; for now we have three:\none to get the disc usage of a PATH,one to hardlink all the files that are identical on the filesystem,one to copy files and folders from S3 buckets to our filesystem. Container definitionsmysecureshellThe mysecureshell container can be used to provide an sftp service with multiple users (although the files are owned by the same UID and GID) using standalone containers (launched with docker or podman) or in an orchestration system like kubernetes, as we are going to do here.\nThe image is generated using the following Dockerfile:\nARG ALPINE_VERSION=3.16.2 FROM alpine:$ALPINE_VERSION as builder LABEL maintainer=\u0026#34;Sergio Talens-Oliag \u0026lt;sto@mixinet.net\u0026gt;\u0026#34; RUN apk update \u0026amp;\u0026amp;\\ apk add --no-cache alpine-sdk git musl-dev \u0026amp;\u0026amp;\\ git clone https://github.com/sto/mysecureshell.git \u0026amp;\u0026amp;\\ cd mysecureshell \u0026amp;\u0026amp;\\ ./configure --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man\\ --localstatedir=/var --with-shutfile=/var/lib/misc/sftp.shut --with-debug=2 \u0026amp;\u0026amp;\\ make all \u0026amp;\u0026amp; make install \u0026amp;\u0026amp;\\ rm -rf /var/cache/apk/* FROM alpine:$ALPINE_VERSION LABEL maintainer=\u0026#34;Sergio Talens-Oliag \u0026lt;sto@mixinet.net\u0026gt;\u0026#34; COPY --from=builder /usr/bin/mysecureshell /usr/bin/mysecureshell COPY --from=builder /usr/bin/sftp-* /usr/bin/ RUN apk update \u0026amp;\u0026amp;\\ apk add --no-cache openssh shadow pwgen \u0026amp;\u0026amp;\\ sed -i -e \u0026#34;s|^.*\\(AuthorizedKeysFile\\).*$|\\1 /etc/ssh/auth_keys/%u|\u0026#34;\\ /etc/ssh/sshd_config \u0026amp;\u0026amp;\\ mkdir /etc/ssh/auth_keys \u0026amp;\u0026amp;\\ cat /dev/null \u0026gt; /etc/motd \u0026amp;\u0026amp;\\ add-shell \u0026#39;/usr/bin/mysecureshell\u0026#39; \u0026amp;\u0026amp;\\ rm -rf /var/cache/apk/* COPY bin/* /usr/local/bin/ COPY etc/sftp_config /etc/ssh/ COPY entrypoint.sh / EXPOSE 22 VOLUME /sftp ENTRYPOINT [\u0026#34;/entrypoint.sh\u0026#34;] CMD [\u0026#34;server\u0026#34;] Note: Initially the container used the mysecureshell package included in alpine, but we wanted to be able to create hardlinks from the client and the support is only available on the master branch of the source repository, that is why we are compiling our own binary using a multi-stage Dockerfile.\nNote that we are cloning the source from a fork that includes this pull request because we had to fix a couple of minor issues to make the ln command work as expected.\nThe /etc/sftp_config file is used to configure the mysecureshell server to have all the user homes under /sftp/data, only allow them to see the files under their home directories as if it were at the root of the server and close idle connections after 5m of inactivity:\netc/sftp_config # Default mysecureshell configuration \u0026lt;Default\u0026gt; # All users will have access their home directory under /sftp/data Home /sftp/data/$USER # Log to a file inside /sftp/logs/ (only works when the directory exists) LogFile /sftp/logs/mysecureshell.log # Force users to stay in their home directory StayAtHome true # Hide Home PATH, it will be shown as / VirtualChroot true # Hide real file/directory owner (just change displayed permissions) DirFakeUser true # Hide real file/directory group (just change displayed permissions) DirFakeGroup true # We do not want users to keep forever their idle connection IdleTimeOut 5m \u0026lt;/Default\u0026gt; # vim: ts=2:sw=2:et The entrypoint.sh script is the one responsible to prepare the container for the users included on the /secrets/user_pass.txt file (creates the users with their HOME directories under /sftp/data and a /bin/false shell and creates the key files from /secrets/user_keys.txt if available).\nThe script expects a couple of environment variables:\nSFTP_UID: UID used to run the daemon and for all the files, it has to be different than 0 (all the files managed by this daemon are going to be owned by the same user and group, even if the remote users are different).SFTP_GID: GID used to run the daemon and for all the files, it has to be different than 0. And can use the SSH_PORT and SSH_PARAMS values if present.\nIt also requires the following files (they can be mounted as secrets in kubernetes):\n/secrets/host_keys.txt: Text file containing the ssh server keys in mime format; the file is processed using the reformime utility (the one included on busybox) and can be generated using the gen-host-keys script included on the container (it uses ssh-keygen and makemime)./secrets/user_pass.txt: Text file containing lines of the form username:password_in_clear_text (only the users included on this file are available on the sftp server, in fact in our deployment we use only the scs user for everything). And optionally can use another one:\n/secrets/user_keys.txt: Text file that contains lines of the form username:public_ssh_ed25519_or_rsa_key; the public keys are installed on the server and can be used to log into the sftp server if the username exists on the user_pass.txt file. The contents of the entrypoint.sh script are:\nentrypoint.sh #!/bin/sh set -e # --------- # VARIABLES # --------- # Expects SSH_UID \u0026amp; SSH_GID on the environment and uses the value of the # SSH_PORT \u0026amp; SSH_PARAMS variables if present # SSH_PARAMS SSH_PARAMS=\u0026#34;-D -e -p ${SSH_PORT:=22} ${SSH_PARAMS}\u0026#34; # Fixed values # DIRECTORIES HOME_DIR=\u0026#34;/sftp/data\u0026#34; CONF_FILES_DIR=\u0026#34;/secrets\u0026#34; AUTH_KEYS_PATH=\u0026#34;/etc/ssh/auth_keys\u0026#34; # FILES HOST_KEYS=\u0026#34;$CONF_FILES_DIR/host_keys.txt\u0026#34; USER_KEYS=\u0026#34;$CONF_FILES_DIR/user_keys.txt\u0026#34; USER_PASS=\u0026#34;$CONF_FILES_DIR/user_pass.txt\u0026#34; USER_SHELL_CMD=\u0026#34;/usr/bin/mysecureshell\u0026#34; # TYPES HOST_KEY_TYPES=\u0026#34;dsa ecdsa ed25519 rsa\u0026#34; # --------- # FUNCTIONS # --------- # Validate HOST_KEYS, USER_PASS, SFTP_UID and SFTP_GID _check_environment() { # Check the ssh server keys ... we don\u0026#39;t boot if we don\u0026#39;t have them if [ ! -f \u0026#34;$HOST_KEYS\u0026#34; ]; then cat \u0026lt;\u0026lt;EOF We need the host keys on the \u0026#39;$HOST_KEYS\u0026#39; file to proceed. Call the \u0026#39;gen-host-keys\u0026#39; script to create and export them on a mime file. EOF exit 1 fi # Check that we have users ... if we don\u0026#39;t we can\u0026#39;t continue if [ ! -f \u0026#34;$USER_PASS\u0026#34; ]; then cat \u0026lt;\u0026lt;EOF We need at least the \u0026#39;$USER_PASS\u0026#39; file to provision users. Call the \u0026#39;gen-users-tar\u0026#39; script to create a tar file to create an archive that contains public and private keys for users, a \u0026#39;user_keys.txt\u0026#39; with the public keys of the users and a \u0026#39;user_pass.txt\u0026#39; file with random passwords for them (pass the list of usernames to it). EOF exit 1 fi # Check SFTP_UID if [ -z \u0026#34;$SFTP_UID\u0026#34; ]; then echo \u0026#34;The \u0026#39;SFTP_UID\u0026#39; can\u0026#39;t be empty, pass a \u0026#39;GID\u0026#39;.\u0026#34; exit 1 fi if [ \u0026#34;$SFTP_UID\u0026#34; -eq \u0026#34;0\u0026#34; ]; then echo \u0026#34;The \u0026#39;SFTP_UID\u0026#39; can\u0026#39;t be 0, use a different \u0026#39;UID\u0026#39;\u0026#34; exit 1 fi # Check SFTP_GID if [ -z \u0026#34;$SFTP_GID\u0026#34; ]; then echo \u0026#34;The \u0026#39;SFTP_GID\u0026#39; can\u0026#39;t be empty, pass a \u0026#39;GID\u0026#39;.\u0026#34; exit 1 fi if [ \u0026#34;$SFTP_GID\u0026#34; -eq \u0026#34;0\u0026#34; ]; then echo \u0026#34;The \u0026#39;SFTP_GID\u0026#39; can\u0026#39;t be 0, use a different \u0026#39;GID\u0026#39;\u0026#34; exit 1 fi } # Adjust ssh host keys _setup_host_keys() { opwd=\u0026#34;$(pwd)\u0026#34; tmpdir=\u0026#34;$(mktemp -d)\u0026#34; cd \u0026#34;$tmpdir\u0026#34; ret=\u0026#34;0\u0026#34; reformime \u0026lt;\u0026#34;$HOST_KEYS\u0026#34; || ret=\u0026#34;1\u0026#34; for kt in $HOST_KEY_TYPES; do key=\u0026#34;ssh_host_${kt}_key\u0026#34; pub=\u0026#34;ssh_host_${kt}_key.pub\u0026#34; if [ ! -f \u0026#34;$key\u0026#34; ]; then echo \u0026#34;Missing \u0026#39;$key\u0026#39; file\u0026#34; ret=\u0026#34;1\u0026#34; fi if [ ! -f \u0026#34;$pub\u0026#34; ]; then echo \u0026#34;Missing \u0026#39;$pub\u0026#39; file\u0026#34; ret=\u0026#34;1\u0026#34; fi if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then continue fi cat \u0026#34;$key\u0026#34; \u0026gt;\u0026#34;/etc/ssh/$key\u0026#34; chmod 0600 \u0026#34;/etc/ssh/$key\u0026#34; chown root:root \u0026#34;/etc/ssh/$key\u0026#34; cat \u0026#34;$pub\u0026#34; \u0026gt;\u0026#34;/etc/ssh/$pub\u0026#34; chmod 0600 \u0026#34;/etc/ssh/$pub\u0026#34; chown root:root \u0026#34;/etc/ssh/$pub\u0026#34; done cd \u0026#34;$opwd\u0026#34; rm -rf \u0026#34;$tmpdir\u0026#34; return \u0026#34;$ret\u0026#34; } # Create users _setup_user_pass() { opwd=\u0026#34;$(pwd)\u0026#34; tmpdir=\u0026#34;$(mktemp -d)\u0026#34; cd \u0026#34;$tmpdir\u0026#34; ret=\u0026#34;0\u0026#34; [ -d \u0026#34;$HOME_DIR\u0026#34; ] || mkdir \u0026#34;$HOME_DIR\u0026#34; # Make sure the data dir can be managed by the sftp user chown \u0026#34;$SFTP_UID:$SFTP_GID\u0026#34; \u0026#34;$HOME_DIR\u0026#34; # Allow the user (and root) to create directories inside the $HOME_DIR, if # we don\u0026#39;t allow it the directory creation fails on EFS (AWS) chmod 0755 \u0026#34;$HOME_DIR\u0026#34; # Create users echo \u0026#34;sftp:sftp:$SFTP_UID:$SFTP_GID:::/bin/false\u0026#34; \u0026gt;\u0026#34;newusers.txt\u0026#34; sed -n \u0026#34;/^[^#]/ { s/:/ /p }\u0026#34; \u0026#34;$USER_PASS\u0026#34; | while read -r _u _p; do echo \u0026#34;$_u:$_p:$SFTP_UID:$SFTP_GID::$HOME_DIR/$_u:$USER_SHELL_CMD\u0026#34; done \u0026gt;\u0026gt;\u0026#34;newusers.txt\u0026#34; newusers --badnames newusers.txt # Disable write permission on the directory to forbid remote sftp users to # remove their own root dir (they have already done it); we adjust that # here to avoid issues with EFS (see before) chmod 0555 \u0026#34;$HOME_DIR\u0026#34; # Clean up the tmpdir cd \u0026#34;$opwd\u0026#34; rm -rf \u0026#34;$tmpdir\u0026#34; return \u0026#34;$ret\u0026#34; } # Adjust user keys _setup_user_keys() { if [ -f \u0026#34;$USER_KEYS\u0026#34; ]; then sed -n \u0026#34;/^[^#]/ { s/:/ /p }\u0026#34; \u0026#34;$USER_KEYS\u0026#34; | while read -r _u _k; do echo \u0026#34;$_k\u0026#34; \u0026gt;\u0026gt;\u0026#34;$AUTH_KEYS_PATH/$_u\u0026#34; done fi } # Main function exec_sshd() { _check_environment _setup_host_keys _setup_user_pass _setup_user_keys echo \u0026#34;Running: /usr/sbin/sshd $SSH_PARAMS\u0026#34; # shellcheck disable=SC2086 exec /usr/sbin/sshd -D $SSH_PARAMS } # ---- # MAIN # ---- case \u0026#34;$1\u0026#34; in \u0026#34;server\u0026#34;) exec_sshd ;; *) exec \u0026#34;$@\u0026#34; ;; esac # vim: ts=2:sw=2:et The container also includes a couple of auxiliary scripts, the first one can be used to generate the host_keys.txt file as follows:\n$ docker run --rm stodh/mysecureshell gen-host-keys \u0026gt; host_keys.txt Where the script is as simple as:\nbin/gen-host-keys #!/bin/sh set -e # Generate new host keys ssh-keygen -A \u0026gt;/dev/null # Replace hostname sed -i -e \u0026#39;s/@.*$/@mysecureshell/\u0026#39; /etc/ssh/ssh_host_*_key.pub # Print in mime format (stdout) makemime /etc/ssh/ssh_host_* # vim: ts=2:sw=2:et And there is another script to generate a .tar file that contains auth data for the list of usernames passed to it (the file contains a user_pass.txt file with random passwords for the users, public and private ssh keys for them and the user_keys.txt file that matches the generated keys).\nTo generate a tar file for the user scs we can execute the following:\n$ docker run --rm stodh/mysecureshell gen-users-tar scs \u0026gt; /tmp/scs-users.tar To see the contents and the text inside the user_pass.txt file we can do:\n$ tar tvf /tmp/scs-users.tar -rw-r--r-- root/root 21 2022-09-11 15:55 user_pass.txt -rw-r--r-- root/root 822 2022-09-11 15:55 user_keys.txt -rw------- root/root 387 2022-09-11 15:55 id_ed25519-scs -rw-r--r-- root/root 85 2022-09-11 15:55 id_ed25519-scs.pub -rw------- root/root 3357 2022-09-11 15:55 id_rsa-scs -rw------- root/root 3243 2022-09-11 15:55 id_rsa-scs.pem -rw-r--r-- root/root 729 2022-09-11 15:55 id_rsa-scs.pub $ tar xfO /tmp/scs-users.tar user_pass.txt scs:20JertRSX2Eaar4x The source of the script is:\nbin/gen-users-tar #!/bin/sh set -e # --------- # VARIABLES # --------- USER_KEYS_FILE=\u0026#34;user_keys.txt\u0026#34; USER_PASS_FILE=\u0026#34;user_pass.txt\u0026#34; # --------- # MAIN CODE # --------- # Generate user passwords and keys, return 1 if no username is received if [ \u0026#34;$#\u0026#34; -eq \u0026#34;0\u0026#34; ]; then return 1 fi opwd=\u0026#34;$(pwd)\u0026#34; tmpdir=\u0026#34;$(mktemp -d)\u0026#34; cd \u0026#34;$tmpdir\u0026#34; for u in \u0026#34;$@\u0026#34;; do ssh-keygen -q -a 100 -t ed25519 -f \u0026#34;id_ed25519-$u\u0026#34; -C \u0026#34;$u\u0026#34; -N \u0026#34;\u0026#34; ssh-keygen -q -a 100 -b 4096 -t rsa -f \u0026#34;id_rsa-$u\u0026#34; -C \u0026#34;$u\u0026#34; -N \u0026#34;\u0026#34; # Legacy RSA private key format cp -a \u0026#34;id_rsa-$u\u0026#34; \u0026#34;id_rsa-$u.pem\u0026#34; ssh-keygen -q -p -m pem -f \u0026#34;id_rsa-$u.pem\u0026#34; -N \u0026#34;\u0026#34; -P \u0026#34;\u0026#34; \u0026gt;/dev/null chmod 0600 \u0026#34;id_rsa-$u.pem\u0026#34; echo \u0026#34;$u:$(pwgen -s 16 1)\u0026#34; \u0026gt;\u0026gt;\u0026#34;$USER_PASS_FILE\u0026#34; echo \u0026#34;$u:$(cat \u0026#34;id_ed25519-$u.pub\u0026#34;)\u0026#34; \u0026gt;\u0026gt;\u0026#34;$USER_KEYS_FILE\u0026#34; echo \u0026#34;$u:$(cat \u0026#34;id_rsa-$u.pub\u0026#34;)\u0026#34; \u0026gt;\u0026gt;\u0026#34;$USER_KEYS_FILE\u0026#34; done tar cf - \u0026#34;$USER_PASS_FILE\u0026#34; \u0026#34;$USER_KEYS_FILE\u0026#34; id_* 2\u0026gt;/dev/null cd \u0026#34;$opwd\u0026#34; rm -rf \u0026#34;$tmpdir\u0026#34; # vim: ts=2:sw=2:et nginx-scsThe nginx-scs container is generated using the following Dockerfile:\nARG NGINX_VERSION=1.23.1 FROM nginx:$NGINX_VERSION LABEL maintainer=\u0026#34;Sergio Talens-Oliag \u0026lt;sto@mixinet.net\u0026gt;\u0026#34; RUN rm -f /docker-entrypoint.d/* COPY docker-entrypoint.d/* /docker-entrypoint.d/ Basically we are removing the existing docker-entrypoint.d scripts from the standard image and adding a new one that configures the web server as we want using a couple of environment variables:\nAUTH_REQUEST_URI: URL to use for the auth_request, if the variable is not found on the environment auth_request is not used.HTML_ROOT: Base directory of the web server, if not passed the default /usr/share/nginx/html is used. Note that if we don’t pass the variables everything works as if we were using the original nginx image.\nThe contents of the configuration script are:\ndocker-entrypoint.d/10-update-default-conf.sh #!/bin/sh # Replace the default.conf nginx file by our own version. set -e if [ -z \u0026#34;$HTML_ROOT\u0026#34; ]; then HTML_ROOT=\u0026#34;/usr/share/nginx/html\u0026#34; fi if [ \u0026#34;$AUTH_REQUEST_URI\u0026#34; ]; then cat \u0026gt;/etc/nginx/conf.d/default.conf \u0026lt;\u0026lt;EOF server { listen 80; server_name localhost; location / { auth_request /.auth; root $HTML_ROOT; index index.html index.htm; } location /.auth { internal; proxy_pass $AUTH_REQUEST_URI; proxy_pass_request_body off; proxy_set_header Content-Length \u0026#34;\u0026#34;; proxy_set_header X-Original-URI \\$request_uri; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } } EOF else cat \u0026gt;/etc/nginx/conf.d/default.conf \u0026lt;\u0026lt;EOF server { listen 80; server_name localhost; location / { root $HTML_ROOT; index index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } } EOF fi # vim: ts=2:sw=2:et As we will see later the idea is to use the /sftp/data or /sftp/data/scs folder as the root of the web published by this container and create an Ingress object to provide access to it outside of our kubernetes cluster.\nwebhook-scsThe webhook-scs container is generated using the following Dockerfile:\nARG ALPINE_VERSION=3.16.2 ARG GOLANG_VERSION=alpine3.16 FROM golang:$GOLANG_VERSION AS builder LABEL maintainer=\u0026#34;Sergio Talens-Oliag \u0026lt;sto@mixinet.net\u0026gt;\u0026#34; ENV WEBHOOK_VERSION 2.8.0 ENV WEBHOOK_PR 549 ENV S3FS_VERSION v1.91 WORKDIR /go/src/github.com/adnanh/webhook RUN apk update \u0026amp;\u0026amp;\\ apk add --no-cache -t build-deps curl libc-dev gcc libgcc patch RUN curl -L --silent -o webhook.tar.gz\\ https://github.com/adnanh/webhook/archive/${WEBHOOK_VERSION}.tar.gz \u0026amp;\u0026amp;\\ tar xzf webhook.tar.gz --strip 1 \u0026amp;\u0026amp;\\ curl -L --silent -o ${WEBHOOK_PR}.patch\\ https://patch-diff.githubusercontent.com/raw/adnanh/webhook/pull/${WEBHOOK_PR}.patch \u0026amp;\u0026amp;\\ patch -p1 \u0026lt; ${WEBHOOK_PR}.patch \u0026amp;\u0026amp;\\ go get -d \u0026amp;\u0026amp; \\ go build -o /usr/local/bin/webhook WORKDIR /src/s3fs-fuse RUN apk update \u0026amp;\u0026amp;\\ apk add ca-certificates build-base alpine-sdk libcurl automake autoconf\\ libxml2-dev libressl-dev mailcap fuse-dev curl-dev RUN curl -L --silent -o s3fs.tar.gz\\ https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/$S3FS_VERSION.tar.gz \u0026amp;\u0026amp;\\ tar xzf s3fs.tar.gz --strip 1 \u0026amp;\u0026amp;\\ ./autogen.sh \u0026amp;\u0026amp;\\ ./configure --prefix=/usr/local \u0026amp;\u0026amp;\\ make -j \u0026amp;\u0026amp; \\ make install FROM alpine:$ALPINE_VERSION LABEL maintainer=\u0026#34;Sergio Talens-Oliag \u0026lt;sto@mixinet.net\u0026gt;\u0026#34; WORKDIR /webhook RUN apk update \u0026amp;\u0026amp;\\ apk add --no-cache ca-certificates mailcap fuse libxml2 libcurl libgcc\\ libstdc++ rsync util-linux-misc \u0026amp;\u0026amp;\\ rm -rf /var/cache/apk/* COPY --from=builder /usr/local/bin/webhook /usr/local/bin/webhook COPY --from=builder /usr/local/bin/s3fs /usr/local/bin/s3fs COPY entrypoint.sh / COPY hooks/* ./hooks/ EXPOSE 9000 ENTRYPOINT [\u0026#34;/entrypoint.sh\u0026#34;] CMD [\u0026#34;server\u0026#34;] Again, we use a multi-stage build because in production we wanted to support a functionality that is not already on the official versions (streaming the command output as a response instead of waiting until the execution ends); this time we build the image applying the PATCH included on this pull request against a released version of the source instead of creating a fork.\nThe entrypoint.sh script is used to generate the webhook configuration file for the existing hooks using environment variables (basically the WEBHOOK_WORKDIR and the *_TOKEN variables) and launch the webhook service:\nentrypoint.sh #!/bin/sh set -e # --------- # VARIABLES # --------- WEBHOOK_BIN=\u0026#34;${WEBHOOK_BIN:-/webhook/hooks}\u0026#34; WEBHOOK_YML=\u0026#34;${WEBHOOK_YML:-/webhook/scs.yml}\u0026#34; WEBHOOK_OPTS=\u0026#34;${WEBHOOK_OPTS:--verbose}\u0026#34; # --------- # FUNCTIONS # --------- print_du_yml() { cat \u0026lt;\u0026lt;EOF - id: du execute-command: \u0026#39;$WEBHOOK_BIN/du.sh\u0026#39; command-working-directory: \u0026#39;$WORKDIR\u0026#39; response-headers: - name: \u0026#39;Content-Type\u0026#39; value: \u0026#39;application/json\u0026#39; http-methods: [\u0026#39;GET\u0026#39;] include-command-output-in-response: true include-command-output-in-response-on-error: true pass-arguments-to-command: - source: \u0026#39;url\u0026#39; name: \u0026#39;path\u0026#39; pass-environment-to-command: - source: \u0026#39;string\u0026#39; envname: \u0026#39;OUTPUT_FORMAT\u0026#39; name: \u0026#39;json\u0026#39; EOF } print_hardlink_yml() { cat \u0026lt;\u0026lt;EOF - id: hardlink execute-command: \u0026#39;$WEBHOOK_BIN/hardlink.sh\u0026#39; command-working-directory: \u0026#39;$WORKDIR\u0026#39; http-methods: [\u0026#39;GET\u0026#39;] include-command-output-in-response: true include-command-output-in-response-on-error: true EOF } print_s3sync_yml() { cat \u0026lt;\u0026lt;EOF - id: s3sync execute-command: \u0026#39;$WEBHOOK_BIN/s3sync.sh\u0026#39; command-working-directory: \u0026#39;$WORKDIR\u0026#39; http-methods: [\u0026#39;POST\u0026#39;] include-command-output-in-response: true include-command-output-in-response-on-error: true pass-environment-to-command: - source: \u0026#39;payload\u0026#39; envname: \u0026#39;AWS_KEY\u0026#39; name: \u0026#39;aws.key\u0026#39; - source: \u0026#39;payload\u0026#39; envname: \u0026#39;AWS_SECRET_KEY\u0026#39; name: \u0026#39;aws.secret_key\u0026#39; - source: \u0026#39;payload\u0026#39; envname: \u0026#39;S3_BUCKET\u0026#39; name: \u0026#39;s3.bucket\u0026#39; - source: \u0026#39;payload\u0026#39; envname: \u0026#39;S3_REGION\u0026#39; name: \u0026#39;s3.region\u0026#39; - source: \u0026#39;payload\u0026#39; envname: \u0026#39;S3_PATH\u0026#39; name: \u0026#39;s3.path\u0026#39; - source: \u0026#39;payload\u0026#39; envname: \u0026#39;SCS_PATH\u0026#39; name: \u0026#39;scs.path\u0026#39; stream-command-output: true EOF } print_token_yml() { if [ \u0026#34;$1\u0026#34; ]; then cat \u0026lt;\u0026lt; EOF trigger-rule: match: type: \u0026#39;value\u0026#39; value: \u0026#39;$1\u0026#39; parameter: source: \u0026#39;header\u0026#39; name: \u0026#39;X-Webhook-Token\u0026#39; EOF fi } exec_webhook() { # Validate WORKDIR if [ -z \u0026#34;$WEBHOOK_WORKDIR\u0026#34; ]; then echo \u0026#34;Must define the WEBHOOK_WORKDIR variable!\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi WORKDIR=\u0026#34;$(realpath \u0026#34;$WEBHOOK_WORKDIR\u0026#34; 2\u0026gt;/dev/null)\u0026#34; || true if [ ! -d \u0026#34;$WORKDIR\u0026#34; ]; then echo \u0026#34;The WEBHOOK_WORKDIR \u0026#39;$WEBHOOK_WORKDIR\u0026#39; is not a directory!\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi # Get TOKENS, if the DU_TOKEN or HARDLINK_TOKEN is defined that is used, if # not if the COMMON_TOKEN that is used and in other case no token is checked # (that is the default) DU_TOKEN=\u0026#34;${DU_TOKEN:-$COMMON_TOKEN}\u0026#34; HARDLINK_TOKEN=\u0026#34;${HARDLINK_TOKEN:-$COMMON_TOKEN}\u0026#34; S3_TOKEN=\u0026#34;${S3_TOKEN:-$COMMON_TOKEN}\u0026#34; # Create webhook configuration { print_du_yml print_token_yml \u0026#34;$DU_TOKEN\u0026#34; echo \u0026#34;\u0026#34; print_hardlink_yml print_token_yml \u0026#34;$HARDLINK_TOKEN\u0026#34; echo \u0026#34;\u0026#34; print_s3sync_yml print_token_yml \u0026#34;$S3_TOKEN\u0026#34; }\u0026gt;\u0026#34;$WEBHOOK_YML\u0026#34; # Run the webhook command # shellcheck disable=SC2086 exec webhook -hooks \u0026#34;$WEBHOOK_YML\u0026#34; $WEBHOOK_OPTS } # ---- # MAIN # ---- case \u0026#34;$1\u0026#34; in \u0026#34;server\u0026#34;) exec_webhook ;; *) exec \u0026#34;$@\u0026#34; ;; esac The entrypoint.sh script generates the configuration file for the webhook server calling functions that print a yaml section for each hook and optionally adds rules to validate access to them comparing the value of a X-Webhook-Token header against predefined values.\nThe expected token values are taken from environment variables, we can define a token variable for each hook (DU_TOKEN, HARDLINK_TOKEN or S3_TOKEN) and a fallback value (COMMON_TOKEN); if no token variable is defined for a hook no check is done and everybody can call it.\nThe Hook Definition documentation explains the options you can use for each hook, the ones we have right now do the following:\ndu: runs on the $WORKDIR directory, passes as first argument to the script the value of the path query parameter and sets the variable OUTPUT_FORMAT to the fixed value json (we use that to print the output of the script in JSON format instead of text).hardlink: runs on the $WORKDIR directory and takes no parameters.s3sync: runs on the $WORKDIR directory and sets a lot of environment variables from values read from the JSON encoded payload sent by the caller (all the values must be sent by the caller even if they are assigned an empty value, if they are missing the hook fails without calling the script); we also set the stream-command-output value to true to make the script show its output as it is working (we patched the webhook source to be able to use this option). The du hook scriptThe du hook script code checks if the argument passed is a directory, computes its size using the du command and prints the results in text format or as a JSON dictionary:\nhooks/du.sh #!/bin/sh set -e # Script to print disk usage for a PATH inside the scs folder # --------- # FUNCTIONS # --------- print_error() { if [ \u0026#34;$OUTPUT_FORMAT\u0026#34; = \u0026#34;json\u0026#34; ]; then echo \u0026#34;{\\\u0026#34;error\\\u0026#34;:\\\u0026#34;$*\\\u0026#34;}\u0026#34; else echo \u0026#34;$*\u0026#34; \u0026gt;\u0026amp;2 fi exit 1 } usage() { if [ \u0026#34;$OUTPUT_FORMAT\u0026#34; = \u0026#34;json\u0026#34; ]; then echo \u0026#34;{\\\u0026#34;error\\\u0026#34;:\\\u0026#34;Pass arguments as \u0026#39;?path=XXX\\\u0026#34;}\u0026#34; else echo \u0026#34;Usage: $(basename \u0026#34;$0\u0026#34;) PATH\u0026#34; \u0026gt;\u0026amp;2 fi exit 1 } # ---- # MAIN # ---- if [ \u0026#34;$#\u0026#34; -eq \u0026#34;0\u0026#34; ] || [ -z \u0026#34;$1\u0026#34; ]; then usage fi if [ \u0026#34;$1\u0026#34; = \u0026#34;.\u0026#34; ]; then DU_PATH=\u0026#34;./\u0026#34; else DU_PATH=\u0026#34;$(find . -name \u0026#34;$1\u0026#34; -mindepth 1 -maxdepth 1)\u0026#34; || true fi if [ -z \u0026#34;$DU_PATH\u0026#34; ] || [ ! -d \u0026#34;$DU_PATH/.\u0026#34; ]; then print_error \u0026#34;The provided PATH (\u0026#39;$1\u0026#39;) is not a directory\u0026#34; fi # Print disk usage in bytes for the given PATH OUTPUT=\u0026#34;$(du -b -s \u0026#34;$DU_PATH\u0026#34;)\u0026#34; if [ \u0026#34;$OUTPUT_FORMAT\u0026#34; = \u0026#34;json\u0026#34; ]; then # Format output as {\u0026#34;path\u0026#34;:\u0026#34;PATH\u0026#34;,\u0026#34;bytes\u0026#34;:\u0026#34;BYTES\u0026#34;} echo \u0026#34;$OUTPUT\u0026#34; | sed -e \u0026#34;s%^\\(.*\\)\\t.*/\\(.*\\)$%{\\\u0026#34;path\\\u0026#34;:\\\u0026#34;\\2\\\u0026#34;,\\\u0026#34;bytes\\\u0026#34;:\\\u0026#34;\\1\\\u0026#34;}%\u0026#34; | tr -d \u0026#39;\\n\u0026#39; else # Print du output as is echo \u0026#34;$OUTPUT\u0026#34; fi # vim: ts=2:sw=2:et:ai:sts=2 The hardlink hook scriptThe hardlink hook script is really simple, it just runs the util-linux version of the hardlink command on its working directory:\nhooks/hardlink.sh #!/bin/sh hardlink --ignore-time --maximize . We use that to reduce the size of the stored content; to manage versions of files and folders we keep each version on a separate directory and when one or more files are not changed this script makes them hardlinks to the same file on disc, reducing the space used on disk.\nThe s3sync hook scriptThe s3sync hook script uses the s3fs tool to mount a bucket and synchronise data between a folder inside the bucket and a directory on the filesystem using rsync; all values needed to execute the task are taken from environment variables:\nhooks/s3sync.sh #!/bin/ash set -euo pipefail set -o errexit set -o errtrace # Functions finish() { ret=\u0026#34;$1\u0026#34; echo \u0026#34;\u0026#34; echo \u0026#34;Script exit code: $ret\u0026#34; exit \u0026#34;$ret\u0026#34; } # Check variables if [ -z \u0026#34;$AWS_KEY\u0026#34; ] || [ -z \u0026#34;$AWS_SECRET_KEY\u0026#34; ] || [ -z \u0026#34;$S3_BUCKET\u0026#34; ] || [ -z \u0026#34;$S3_PATH\u0026#34; ] || [ -z \u0026#34;$SCS_PATH\u0026#34; ]; then [ \u0026#34;$AWS_KEY\u0026#34; ] || echo \u0026#34;Set the AWS_KEY environment variable\u0026#34; [ \u0026#34;$AWS_SECRET_KEY\u0026#34; ] || echo \u0026#34;Set the AWS_SECRET_KEY environment variable\u0026#34; [ \u0026#34;$S3_BUCKET\u0026#34; ] || echo \u0026#34;Set the S3_BUCKET environment variable\u0026#34; [ \u0026#34;$S3_PATH\u0026#34; ] || echo \u0026#34;Set the S3_PATH environment variable\u0026#34; [ \u0026#34;$SCS_PATH\u0026#34; ] || echo \u0026#34;Set the SCS_PATH environment variable\u0026#34; finish 1 fi if [ \u0026#34;$S3_REGION\u0026#34; ] \u0026amp;\u0026amp; [ \u0026#34;$S3_REGION\u0026#34; != \u0026#34;us-east-1\u0026#34; ]; then EP_URL=\u0026#34;endpoint=$S3_REGION,url=https://s3.$S3_REGION.amazonaws.com\u0026#34; else EP_URL=\u0026#34;endpoint=us-east-1\u0026#34; fi # Prepare working directory WORK_DIR=\u0026#34;$(mktemp -p \u0026#34;$HOME\u0026#34; -d)\u0026#34; MNT_POINT=\u0026#34;$WORK_DIR/s3data\u0026#34; PASSWD_S3FS=\u0026#34;$WORK_DIR/.passwd-s3fs\u0026#34; # Check the moutpoint if [ ! -d \u0026#34;$MNT_POINT\u0026#34; ]; then mkdir -p \u0026#34;$MNT_POINT\u0026#34; elif mountpoint \u0026#34;$MNT_POINT\u0026#34;; then echo \u0026#34;There is already something mounted on \u0026#39;$MNT_POINT\u0026#39;, aborting!\u0026#34; finish 1 fi # Create password file touch \u0026#34;$PASSWD_S3FS\u0026#34; chmod 0400 \u0026#34;$PASSWD_S3FS\u0026#34; echo \u0026#34;$AWS_KEY:$AWS_SECRET_KEY\u0026#34; \u0026gt;\u0026#34;$PASSWD_S3FS\u0026#34; # Mount s3 bucket as a filesystem s3fs -o dbglevel=info,retries=5 -o \u0026#34;$EP_URL\u0026#34; -o \u0026#34;passwd_file=$PASSWD_S3FS\u0026#34; \\ \u0026#34;$S3_BUCKET\u0026#34; \u0026#34;$MNT_POINT\u0026#34; echo \u0026#34;Mounted bucket \u0026#39;$S3_BUCKET\u0026#39; on \u0026#39;$MNT_POINT\u0026#39;\u0026#34; # Remove the password file, just in case rm -f \u0026#34;$PASSWD_S3FS\u0026#34; # Check source PATH ret=\u0026#34;0\u0026#34; SRC_PATH=\u0026#34;$MNT_POINT/$S3_PATH\u0026#34; if [ ! -d \u0026#34;$SRC_PATH\u0026#34; ]; then echo \u0026#34;The S3_PATH \u0026#39;$S3_PATH\u0026#39; can\u0026#39;t be found!\u0026#34; ret=1 fi # Compute SCS_UID \u0026amp; SCS_GID (by default based on the working directory owner) SCS_UID=\u0026#34;${SCS_UID:=$(stat -c \u0026#34;%u\u0026#34; \u0026#34;.\u0026#34; 2\u0026gt;/dev/null)}\u0026#34; || true SCS_GID=\u0026#34;${SCS_GID:=$(stat -c \u0026#34;%g\u0026#34; \u0026#34;.\u0026#34; 2\u0026gt;/dev/null)}\u0026#34; || true # Check destination PATH DST_PATH=\u0026#34;./$SCS_PATH\u0026#34; if [ \u0026#34;$ret\u0026#34; -eq \u0026#34;0\u0026#34; ] \u0026amp;\u0026amp; [ -d \u0026#34;$DST_PATH\u0026#34; ]; then mkdir -p \u0026#34;$DST_PATH\u0026#34; || ret=\u0026#34;$?\u0026#34; fi # Copy using rsync if [ \u0026#34;$ret\u0026#34; -eq \u0026#34;0\u0026#34; ]; then rsync -rlptv --chown=\u0026#34;$SCS_UID:$SCS_GID\u0026#34; --delete --stats \\ \u0026#34;$SRC_PATH/\u0026#34; \u0026#34;$DST_PATH/\u0026#34; || ret=\u0026#34;$?\u0026#34; fi # Unmount the S3 bucket umount -f \u0026#34;$MNT_POINT\u0026#34; echo \u0026#34;Called umount for \u0026#39;$MNT_POINT\u0026#39;\u0026#34; # Remove mount point dir rmdir \u0026#34;$MNT_POINT\u0026#34; # Remove WORK_DIR rmdir \u0026#34;$WORK_DIR\u0026#34; # We are done finish \u0026#34;$ret\u0026#34; # vim: ts=2:sw=2:et:ai:sts=2 Deployment objectsThe system is deployed as a StatefulSet with one replica.\nOur production deployment is done on AWS and to be able to scale we use EFS for our PersistenVolume; the idea is that the volume has no size limit, its AccessMode can be set to ReadWriteMany and we can mount it from multiple instances of the Pod without issues, even if they are in different availability zones.\nFor development we use k3d and we are also able to scale the StatefulSet for testing because we use a ReadWriteOnce PVC, but it points to a hostPath that is backed up by a folder that is mounted on all the compute nodes, so in reality Pods in different k3d nodes use the same folder on the host.\nsecrets.yamlThe secrets file contains the files used by the mysecureshell container that can be generated using kubernetes pods as follows (we are only creating the scs user):\n$ kubectl run \u0026#34;mysecureshell\u0026#34; --restart=\u0026#39;Never\u0026#39; --quiet --rm --stdin \\ --image \u0026#34;stodh/mysecureshell:latest\u0026#34; -- gen-host-keys \u0026gt;\u0026#34;./host_keys.txt\u0026#34; $ kubectl run \u0026#34;mysecureshell\u0026#34; --restart=\u0026#39;Never\u0026#39; --quiet --rm --stdin \\ --image \u0026#34;stodh/mysecureshell:latest\u0026#34; -- gen-users-tar scs \u0026gt;\u0026#34;./users.tar\u0026#34; Once we have the files we can generate the secrets.yaml file as follows:\n$ tar xf ./users.tar user_keys.txt user_pass.txt $ kubectl --dry-run=client -o yaml create secret generic \u0026#34;scs-secret\u0026#34; \\ --from-file=\u0026#34;host_keys.txt=host_keys.txt\u0026#34; \\ --from-file=\u0026#34;user_keys.txt=user_keys.txt\u0026#34; \\ --from-file=\u0026#34;user_pass.txt=user_pass.txt\u0026#34; \u0026gt; ./secrets.yaml The resulting secrets.yaml will look like the following file (the base64 would match the content of the files, of course):\nsecrets.yaml apiVersion: v1 data: host_keys.txt: TWlt... user_keys.txt: c2Nz... user_pass.txt: c2Nz... kind: Secret metadata: creationTimestamp: null name: scs-secret pvc.yamlThe persistent volume claim for a simple deployment (one with only one instance of the statefulSet) can be as simple as this:\npvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scs-pvc labels: app.kubernetes.io/name: scs spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi On this definition we don’t set the storageClassName to use the default one.\nVolumes in our development environment (k3d)In our development deployment we create the following PersistentVolume as required by the Local Persistence Volume Static Provisioner (note that the /volumes/scs-pv has to be created by hand, in our k3d system we mount the same host directory on the /volumes path of all the nodes and create the scs-pv directory by hand before deploying the persistent volume):\nk3d-pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: scs-pv labels: app.kubernetes.io/name: scs spec: capacity: storage: 8Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete claimRef: name: scs-pvc storageClassName: local-storage local: path: /volumes/scs-pv nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: node.kubernetes.io/instance-type operator: In values: - k3s Note: The nodeAffinity section is required but in practice the current definition selects all k3d nodes.\nAnd to make sure that everything works as expected we update the PVC definition to add the right storageClassName:\nk3d-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scs-pvc labels: app.kubernetes.io/name: scs spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: local-storage Volumes in our production environment (aws)In the production deployment we don’t create the PersistentVolume (we are using the aws-efs-csi-driver which supports Dynamic Provisioning) but we add the storageClassName (we set it to the one mapped to the EFS driver, i.e. efs-sc) and set ReadWriteMany as the accessMode:\nefs-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: scs-pvc labels: app.kubernetes.io/name: scs spec: accessModes: - ReadWriteMany resources: requests: storage: 8Gi storageClassName: efs-sc statefulset.yamlThe definition of the statefulSet is as follows:\nstatefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: scs labels: app.kubernetes.io/name: scs spec: serviceName: scs replicas: 1 selector: matchLabels: app: scs template: metadata: labels: app: scs spec: containers: - name: nginx image: stodh/nginx-scs:latest ports: - containerPort: 80 name: http env: - name: AUTH_REQUEST_URI value: \u0026#34;\u0026#34; - name: HTML_ROOT value: /sftp/data volumeMounts: - mountPath: /sftp name: scs-datadir - name: mysecureshell image: stodh/mysecureshell:latest ports: - containerPort: 22 name: ssh securityContext: capabilities: add: - IPC_OWNER env: - name: SFTP_UID value: \u0026#39;2020\u0026#39; - name: SFTP_GID value: \u0026#39;2020\u0026#39; volumeMounts: - mountPath: /secrets name: scs-file-secrets readOnly: true - mountPath: /sftp name: scs-datadir - name: webhook image: stodh/webhook-scs:latest securityContext: privileged: true ports: - containerPort: 9000 name: webhook-http env: - name: WEBHOOK_WORKDIR value: /sftp/data/scs volumeMounts: - name: devfuse mountPath: /dev/fuse - mountPath: /sftp name: scs-datadir volumes: - name: devfuse hostPath: path: /dev/fuse - name: scs-file-secrets secret: secretName: scs-secrets - name: scs-datadir persistentVolumeClaim: claimName: scs-pvc Notes about the containers:\nnginx: As this is an example the web server is not using an AUTH_REQUEST_URI and uses the /sftp/data directory as the root of the web (to get to the files uploaded for the scs user we will need to use /scs/ as a prefix on the URLs).mysecureshell: We are adding the IPC_OWNER capability to the container to be able to use some of the sftp-* commands inside it, but they are not really needed, so adding the capability is optional.webhook: We are launching this container in privileged mode to be able to use the s3fs-fuse, as it will not work otherwise for now (see this kubernetes issue); if the functionality is not needed the container can be executed with regular privileges; besides, as we are not enabling public access to this service we don’t define *_TOKEN variables (if required the values should be read from a Secret object). Notes about the volumes:\nthe devfuse volume is only needed if we plan to use the s3fs command on the webhook container, if not we can remove the volume definition and its mounts. service.yamlTo be able to access the different services on the statefulset we publish the relevant ports using the following Service object:\nservice.yaml apiVersion: v1 kind: Service metadata: name: scs-svc labels: app.kubernetes.io/name: scs spec: ports: - name: ssh port: 22 protocol: TCP targetPort: 22 - name: http port: 80 protocol: TCP targetPort: 80 - name: webhook-http port: 9000 protocol: TCP targetPort: 9000 selector: app: scs ingress.yamlTo download the scs files from the outside we can add an ingress object like the following (the definition is for testing using the localhost name):\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: scs-ingress labels: app.kubernetes.io/name: scs spec: ingressClassName: nginx rules: - host: \u0026#39;localhost\u0026#39; http: paths: - path: /scs pathType: Prefix backend: service: name: scs-svc port: number: 80 DeploymentTo deploy the statefulSet we create a namespace and apply the object definitions shown before:\n$ kubectl create namespace scs-demo namespace/scs-demo created $ kubectl -n scs-demo apply -f secrets.yaml secret/scs-secrets created $ kubectl -n scs-demo apply -f pvc.yaml persistentvolumeclaim/scs-pvc created $ kubectl -n scs-demo apply -f statefulset.yaml statefulset.apps/scs created $ kubectl -n scs-demo apply -f service.yaml service/scs-svc created $ kubectl -n scs-demo apply -f ingress.yaml ingress.networking.k8s.io/scs-ingress created Once the objects are deployed we can check that all is working using kubectl:\n$ kubectl -n scs-demo get all,secrets,ingress NAME READY STATUS RESTARTS AGE pod/scs-0 3/3 Running 0 24s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/scs-svc ClusterIP 10.43.0.47 \u0026lt;none\u0026gt; 22/TCP,80/TCP,9000/TCP 21s NAME READY AGE statefulset.apps/scs 1/1 24s NAME TYPE DATA AGE secret/default-token-mwcd7 kubernetes.io/service-account-token 3 53s secret/scs-secrets Opaque 3 39s NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/scs-ingress nginx localhost 172.21.0.5 80 17s At this point we are ready to use the system.\nUsage examplesFile uploadsAs previously mentioned in our system the idea is to use the sftp server from other Pods, but to test the system we are going to do a kubectl port-forward and connect to the server using our host client and the password we have generated (it is on the user_pass.txt file, inside the users.tar archive):\n$ kubectl -n scs-demo port-forward service/scs-svc 2020:22 \u0026amp; Forwarding from 127.0.0.1:2020 -\u0026gt; 22 Forwarding from [::1]:2020 -\u0026gt; 22 $ PF_PID=$! $ sftp -P 2020 scs@127.0.0.1 1 Handling connection for 2020 The authenticity of host \u0026#39;[127.0.0.1]:2020 ([127.0.0.1]:2020)\u0026#39; can\u0026#39;t be \\ established. ED25519 key fingerprint is SHA256:eHNwCnyLcSSuVXXiLKeGraw0FT/4Bb/yjfqTstt+088. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added \u0026#39;[127.0.0.1]:2020\u0026#39; (ED25519) to the list of known \\ hosts. scs@127.0.0.1\u0026#39;s password: ********** Connected to 127.0.0.1. sftp\u0026gt; ls -la drwxr-xr-x 2 sftp sftp 4096 Sep 25 14:47 . dr-xr-xr-x 3 sftp sftp 4096 Sep 25 14:36 .. sftp\u0026gt; !date -R \u0026gt; /tmp/date.txt 2 sftp\u0026gt; put /tmp/date.txt . Uploading /tmp/date.txt to /date.txt date.txt 100% 32 27.8KB/s 00:00 sftp\u0026gt; ls -l -rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt sftp\u0026gt; ln date.txt date.txt.1 3 sftp\u0026gt; ls -l -rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt -rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1 sftp\u0026gt; put /tmp/date.txt date.txt.2 4 Uploading /tmp/date.txt to /date.txt.2 date.txt 100% 32 27.8KB/s 00:00 sftp\u0026gt; ls -l 5 -rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt -rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1 -rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt.2 sftp\u0026gt; exit $ kill \u0026#34;$PF_PID\u0026#34; [1] + terminated kubectl -n scs-demo port-forward service/scs-svc 2020:22We connect to the sftp service on the forwarded port with the scs user.We put a file we have created on the host on the directory.We do a hard link of the uploaded file.We put a second copy of the file we created locally.On the file list we can see that the two first files have two hardlinks File retrievalsIf our ingress is configured right we can download the date.txt file from the URL http://localhost/scs/date.txt:\n$ curl -s http://localhost/scs/date.txt Sun, 25 Sep 2022 17:21:51 +0200 Use of the webhook containerTo finish this post we are going to show how we can call the hooks directly, from a CronJob and from a Job.\nDirect script call (du)In our deployment the direct calls are done from other Pods, to simulate it we are going to do a port-forward and call the script with an existing PATH (the root directory) and a bad one:\n$ kubectl -n scs-demo port-forward service/scs-svc 9000:9000 \u0026gt;/dev/null \u0026amp; $ PF_PID=$! $ JSON=\u0026#34;$(curl -s \u0026#34;http://localhost:9000/hooks/du?path=.\u0026#34;)\u0026#34; $ echo $JSON {\u0026#34;path\u0026#34;:\u0026#34;\u0026#34;,\u0026#34;bytes\u0026#34;:\u0026#34;4160\u0026#34;} $ JSON=\u0026#34;$(curl -s \u0026#34;http://localhost:9000/hooks/du?path=foo\u0026#34;)\u0026#34; $ echo $JSON {\u0026#34;error\u0026#34;:\u0026#34;The provided PATH (\u0026#39;foo\u0026#39;) is not a directory\u0026#34;} $ kill $PF_PID As we only have files on the base directory we print the disk usage of the . PATH and the output is in json format because we export OUTPUT_FORMAT with the value json on the webhook configuration.\nCronjobs (hardlink)As explained before, the webhook container can be used to run cronjobs; the following one uses an alpine container to call the hardlink script each minute (that setup is for testing, obviously):\nwebhook-cronjob.yaml apiVersion: batch/v1 kind: CronJob metadata: name: hardlink labels: cronjob: \u0026#39;hardlink\u0026#39; spec: schedule: \u0026#34;* */1 * * *\u0026#34; concurrencyPolicy: Replace jobTemplate: spec: template: metadata: labels: cronjob: \u0026#39;hardlink\u0026#39; spec: containers: - name: hardlink-cronjob image: alpine:latest command: [\u0026#34;wget\u0026#34;, \u0026#34;-q\u0026#34;, \u0026#34;-O-\u0026#34;, \u0026#34;http://scs-svc:9000/hooks/hardlink\u0026#34;] restartPolicy: Never The following console session shows how we create the object, allow a couple of executions and remove it (in production we keep it running but once a day, not each minute):\n$ kubectl -n scs-demo apply -f webhook-cronjob.yaml 1 cronjob.batch/hardlink created $ kubectl -n scs-demo get pods -l \u0026#34;cronjob=hardlink\u0026#34; -w 2 NAME READY STATUS RESTARTS AGE hardlink-27735351-zvpnb 0/1 Pending 0 0s hardlink-27735351-zvpnb 0/1 ContainerCreating 0 0s hardlink-27735351-zvpnb 0/1 Completed 0 2s ^C $ kubectl -n scs-demo logs pod/hardlink-27735351-zvpnb 3 Mode: real Method: sha256 Files: 3 Linked: 1 files Compared: 0 xattrs Compared: 1 files Saved: 32 B Duration: 0.000220 seconds $ sleep 60 $ kubectl -n scs-demo get pods -l \u0026#34;cronjob=hardlink\u0026#34; 4 NAME READY STATUS RESTARTS AGE hardlink-27735351-zvpnb 0/1 Completed 0 83s hardlink-27735352-br5rn 0/1 Completed 0 23s $ kubectl -n scs-demo logs pod/hardlink-27735352-br5rn 5 Mode: real Method: sha256 Files: 3 Linked: 0 files Compared: 0 xattrs Compared: 0 files Saved: 0 B Duration: 0.000070 seconds $ kubectl -n scs-demo delete -f webhook-cronjob.yaml 6 cronjob.batch \u0026#34;hardlink\u0026#34; deletedThis command creates the cronjob object.This checks the pods with our cronjob label, we interrupt it once we see that the first run has been completed.With this command we see the output of the execution, as this is the fist execution we see that date.txt.2 has been replaced by a hardlink (the summary does not name the file, but it is the only option knowing the contents from the original upload).After waiting a little bit we check the pods executed again to get the name of the latest one.The log now shows that nothing was done.As this is a demo, we delete the cronjob. Jobs (s3sync)The following job can be used to synchronise the contents of a directory in a S3 bucket with the SCS Filesystem:\njob.yaml apiVersion: batch/v1 kind: Job metadata: name: s3sync labels: cronjob: \u0026#39;s3sync\u0026#39; spec: template: metadata: labels: cronjob: \u0026#39;s3sync\u0026#39; spec: containers: - name: s3sync-job image: alpine:latest command: - \u0026#34;wget\u0026#34; - \u0026#34;-q\u0026#34; - \u0026#34;--header\u0026#34; - \u0026#34;Content-Type: application/json\u0026#34; - \u0026#34;--post-file\u0026#34; - \u0026#34;/secrets/s3sync.json\u0026#34; - \u0026#34;-O-\u0026#34; - \u0026#34;http://scs-svc:9000/hooks/s3sync\u0026#34; volumeMounts: - mountPath: /secrets name: job-secrets readOnly: true restartPolicy: Never volumes: - name: job-secrets secret: secretName: webhook-job-secrets The file with parameters for the script must be something like this:\ns3sync.json { \u0026#34;aws\u0026#34;: { \u0026#34;key\u0026#34;: \u0026#34;********************\u0026#34;, \u0026#34;secret_key\u0026#34;: \u0026#34;****************************************\u0026#34; }, \u0026#34;s3\u0026#34;: { \u0026#34;region\u0026#34;: \u0026#34;eu-north-1\u0026#34;, \u0026#34;bucket\u0026#34;: \u0026#34;blogops-test\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;test\u0026#34; }, \u0026#34;scs\u0026#34;: { \u0026#34;path\u0026#34;: \u0026#34;test\u0026#34; } } Once we have both files we can run the Job as follows:\n$ kubectl -n scs-demo create secret generic webhook-job-secrets \\ 1 --from-file=\u0026#34;s3sync.json=s3sync.json\u0026#34; secret/webhook-job-secrets created $ kubectl -n scs-demo apply -f webhook-job.yaml 2 job.batch/s3sync created $ kubectl -n scs-demo get pods -l \u0026#34;cronjob=s3sync\u0026#34; 3 NAME READY STATUS RESTARTS AGE s3sync-zx2cj 0/1 Completed 0 12s $ kubectl -n scs-demo logs s3sync-zx2cj 4 Mounted bucket \u0026#39;s3fs-test\u0026#39; on \u0026#39;/root/tmp.jiOjaF/s3data\u0026#39; sending incremental file list created directory ./test ./ kyso.png Number of files: 2 (reg: 1, dir: 1) Number of created files: 2 (reg: 1, dir: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 15,075 bytes Total transferred file size: 15,075 bytes Literal data: 15,075 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.147 seconds File list transfer time: 0.000 seconds Total bytes sent: 15,183 Total bytes received: 74 sent 15,183 bytes received 74 bytes 30,514.00 bytes/sec total size is 15,075 speedup is 0.99 Called umount for \u0026#39;/root/tmp.jiOjaF/s3data\u0026#39; Script exit code: 0 $ kubectl -n scs-demo delete -f webhook-job.yaml 5 job.batch \u0026#34;s3sync\u0026#34; deleted $ kubectl -n scs-demo delete secrets webhook-job-secrets 6 secret \u0026#34;webhook-job-secrets\u0026#34; deletedHere we create the webhook-job-secrets secret that contains the s3sync.json file.This command runs the job.Checking the label cronjob=s3sync we get the Pods executed by the job.Here we print the logs of the completed job.Once we are finished we remove the Job.And also the secret. Final remarksThis post has been longer than I expected, but I believe it can be useful for someone; in any case, next time I’ll try to explain something shorter or will split it into multiple entries.\n","permalink":"https://blogops.mixinet.net/posts/k8s_static_content_server/","summary":"\u003cp\u003eThis post describes how I’ve put together a simple static content server for\nkubernetes clusters using a Pod with a persistent volume and multiple\ncontainers: an \u003cstrong\u003esftp\u003c/strong\u003e server to manage contents, a \u003cstrong\u003eweb server\u003c/strong\u003e to publish them\nwith optional access control and another one to run scripts which need access\nto the volume filesystem.\u003c/p\u003e\n\u003cp\u003eThe \u003cstrong\u003esftp\u003c/strong\u003e server runs using\n\u003ca href=\"https://https://github.com/mysecureshell/mysecureshell\" target=\"_blank\" rel=\"noopener\"\u003eMySecureShell\u003c/a\u003e, the \u003cstrong\u003eweb\nserver\u003c/strong\u003e is \u003ca href=\"https://nginx.org/\" target=\"_blank\" rel=\"noopener\"\u003enginx\u003c/a\u003e and the script runner uses the\n\u003ca href=\"https://github.com/adnanh/webhook\" target=\"_blank\" rel=\"noopener\"\u003ewebhook\u003c/a\u003e tool to publish endpoints to call\nthem (the calls will come from other Pods that run backend servers or are\nexecuted from \u003ccode\u003eJobs\u003c/code\u003e or \u003ccode\u003eCronJobs\u003c/code\u003e).\u003c/p\u003e\n\u003caside class=\"admonition-block note\" role=\"note\"\u003e\u003ch6 class=\"block-title label-only\"\u003e\u003cspan class=\"title-label\"\u003eNote: \u003c/span\u003e\u003c/h6\u003e\u003cp\u003eThis service has been developed for \u003ca href=\"https://about.kyso.io/\" target=\"_blank\" rel=\"noopener\"\u003eKyso\u003c/a\u003e and the\nversion used in our current\n\u003ca href=\"https://docs.kyso.io/getting-started/architecture-and-deployment\" target=\"_blank\" rel=\"noopener\"\u003earchitecture\u003c/a\u003e\nincludes an additional container to index documents for\n\u003ca href=\"https://www.elastic.co/elasticsearch/\" target=\"_blank\" rel=\"noopener\"\u003eElasticsearch\u003c/a\u003e, but as it is not\nrelevant for the description of the service as a general solution I’ve decided\nto ignore it on this post.\u003c/p\u003e\u003c/aside\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_history\"\u003eHistory\u003c/h2\u003e\u003cp\u003eThe system was developed because we had a \u003ccode\u003eNodeJS\u003c/code\u003e API with endpoints to upload\nfiles and store them on S3 compatible services that were later accessed via\nHTTPS, but the requirements changed and we needed to be able to publish folders\ninstead of individual files using their original names and apply access\nrestrictions using our API.\u003c/p\u003e\n\u003cp\u003eThinking about our requirements the use of a regular filesystem to keep the\nfiles and folders was a good option, as uploading and serving files is simple.\u003c/p\u003e\n\u003cp\u003eFor the upload I decided to use the \u003cstrong\u003esftp\u003c/strong\u003e protocol, mainly because I already\nhad an \u003cstrong\u003esftp\u003c/strong\u003e container image based on\n\u003ca href=\"https://github.com/mysecureshell/mysecureshell\" target=\"_blank\" rel=\"noopener\"\u003emysecureshell\u003c/a\u003e prepared; once\nwe settled on that we added \u003cstrong\u003esftp\u003c/strong\u003e support to the API server and configured it\nto upload the files to our server instead of using \u003cem\u003eS3 buckets\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eTo publish the files we added a \u003ca href=\"https://nginx.org/\" target=\"_blank\" rel=\"noopener\"\u003enginx\u003c/a\u003e container configured\nto work as a reverse proxy that uses the\n\u003ca href=\"https://nginx.org/en/docs/http/ngx_http_auth_request_module.html\" target=\"_blank\" rel=\"noopener\"\u003engx_http_auth_request_module\u003c/a\u003e\nto validate access to the files (the sub request is configurable, in our\ndeployment we have configured it to call our API to check if the user can\naccess a given URL).\u003c/p\u003e\n\u003cp\u003eFinally we added a third container when we needed to execute some tasks\ndirectly on the filesystem (using \u003ccode\u003ekubectl exec\u003c/code\u003e with the existing containers\ndid not seem a good idea, as that is not supported by \u003ccode\u003eCronJobs\u003c/code\u003e objects, for\nexample).\u003c/p\u003e\n\u003cp\u003eThe solution we found avoiding the NIH Syndrome (i.e. write our own tool) was\nto use the \u003ca href=\"https://github.com/adnanh/webhook\" target=\"_blank\" rel=\"noopener\"\u003ewebhook\u003c/a\u003e tool to provide the\nendpoints to call the scripts; for now we have three:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003eone to get the disc usage of a \u003ccode\u003ePATH\u003c/code\u003e,\u003c/li\u003e\u003cli\u003eone to \u003ccode\u003ehardlink\u003c/code\u003e all the files that are identical on the filesystem,\u003c/li\u003e\u003cli\u003eone to copy files and folders from S3 buckets to our filesystem.\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"Kubernetes Static Content Server"},{"content":"Since a long time ago I’ve been a gitlab-ce user, in fact I’ve set it up on three of the last four companies I’ve worked for (initially I installed it using the omnibus packages on a debian server but on the last two places I moved to the docker based installation, as it is easy to maintain and we don’t need a big installation as the teams using it are small).\nOn the company I work for now (kyso) we are using it to host all our internal repositories and to do all the CI/CD work (the automatic deployments are triggered by web hooks in some cases, but the rest is all done using gitlab-ci).\nThe majority of projects are using nodejs as programming language and we have automated the publication of npm packages on our gitlab instance npm registry and even the publication into the npmjs registry.\nTo publish the packages we have added rules to the gitlab-ci configuration of the relevant repositories and we publish them when a tag is created.\nAs the we are lazy by definition, I configured the system to use the tag as the package version; I tested if the contents of the package.json where in sync with the expected version and if it was not I updated it and did a force push of the tag with the updated file using the following code on the script that publishes the package:\n# Update package version \u0026amp; add it to the .build-args INITIAL_PACKAGE_VERSION=\u0026#34;$(npm pkg get version|tr -d \u0026#39;\u0026#34;\u0026#39;)\u0026#34; npm version --allow-same --no-commit-hooks --no-git-tag-version \\ \u0026#34;$CI_COMMIT_TAG\u0026#34; UPDATED_PACKAGE_VERSION=\u0026#34;$(npm pkg get version|tr -d \u0026#39;\u0026#34;\u0026#39;)\u0026#34; echo \u0026#34;UPDATED_PACKAGE_VERSION=$UPDATED_PACKAGE_VERSION\u0026#34; \u0026gt;\u0026gt; .build-args # Update tag if the version was updated or abort if [ \u0026#34;$INITIAL_PACKAGE_VERSION\u0026#34; != \u0026#34;$UPDATED_PACKAGE_VERSION\u0026#34; ]; then if [ -n \u0026#34;$CI_GIT_USER\u0026#34; ] \u0026amp;\u0026amp; [ -n \u0026#34;$CI_GIT_TOKEN\u0026#34; ]; then git commit -m \u0026#34;Updated version from tag $CI_COMMIT_TAG\u0026#34; package.json git tag -f \u0026#34;$CI_COMMIT_TAG\u0026#34; -m \u0026#34;Updated version from tag\u0026#34; git push -f -o ci.skip origin \u0026#34;$CI_COMMIT_TAG\u0026#34; else echo \u0026#34;!!! ERROR !!!\u0026#34; echo \u0026#34;The updated tag could not be uploaded.\u0026#34; echo \u0026#34;Set CI_GIT_USER and CI_GIT_TOKEN or fix the \u0026#39;package.json\u0026#39; file\u0026#34; echo \u0026#34;!!! ERROR !!!\u0026#34; exit 1 fi fi This feels a little dirty (we are leaving commits on the tag but not updating the original branch); I thought about trying to find the branch using the tag and update it, but I drop the idea pretty soon as there were multiple issues to consider (i.e. we can have tags pointing to commits present in multiple branches and even if it only points to one the tag does not have to be the HEAD of the branch making the inclusion difficult).\nIn any case this system was working, so we left it until we started to publish to the NPM Registry; as we are using a token to push the packages that we don’t want all developers to have access to (right now it would not matter, but when the team grows it will) I started to use gitlab protected branches on the projects that need it and adjusting the .npmrc file using protected variables.\nThe problem then was that we can no longer do a standard force push for a branch (that is the main point of the protected branches feature) unless we use the gitlab api, so the tags with the wrong version started to fail.\nAs the way things were being done seemed dirty anyway I thought that the best way of fixing things was to forbid users to push a tag that includes a version that does not match the package.json version.\nAfter thinking about it we decided to use githooks on the gitlab server for the repositories that need it, as we are only interested in tags we are going to use the update hook; it is executed once for each ref to be updated, and takes three parameters:\nthe name of the ref being updated,the old object name stored in the ref,and the new object name to be stored in the ref. To install our hook we have found the gitaly relative path of each repo and located it on the server filesystem (as I said we are using docker and the gitlab’s data directory is on /srv/gitlab/data, so the path to the repo has the form /srv/gitlab/data/git-data/repositories/@hashed/xx/yy/hash.git).\nOnce we have the directory we need to:\ncreate a custom_hooks sub directory inside it,add the update script (as we only need one script we used that instead of creating an update.d directory, the good thing is that this will also work with a standard git server renaming the base directory to hooks instead of custom_hooks),make it executable, andchange the directory and file ownership to make sure it can be read and executed from the gitlab container On a console session:\n$ cd /srv/gitlab/data/git-data/repositories/@hashed/xx/yy/hash.git $ mkdir custom_hooks $ edit_or_copy custom_hooks/update $ chmod 0755 custom_hooks/update $ chown --reference=. -R custom_hooks The update script we are using is as follows:\n#!/bin/sh set -e # kyso update hook # # Right now it checks version.txt or package.json versions against the tag name # (it supports a \u0026#39;v\u0026#39; prefix on the tag) # Arguments ref_name=\u0026#34;$1\u0026#34; old_rev=\u0026#34;$2\u0026#34; new_rev=\u0026#34;$3\u0026#34; # Initial test if [ -z \u0026#34;$ref_name\u0026#34; ] || [ -z \u0026#34;$old_rev\u0026#34; ] || [ -z \u0026#34;$new_rev\u0026#34; ]; then echo \u0026#34;usage: $0 \u0026lt;ref\u0026gt; \u0026lt;oldrev\u0026gt; \u0026lt;newrev\u0026gt;\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi # Get the tag short name tag_name=\u0026#34;${ref_name##refs/tags/}\u0026#34; # Exit if the update is not for a tag if [ \u0026#34;$tag_name\u0026#34; = \u0026#34;$ref_name\u0026#34; ]; then exit 0 fi # Get the null rev value (string of zeros) zero=$(git hash-object --stdin \u0026lt;/dev/null | tr \u0026#39;0-9a-f\u0026#39; \u0026#39;0\u0026#39;) # Get if the tag is new or not if [ \u0026#34;$old_rev\u0026#34; = \u0026#34;$zero\u0026#34; ]; then new_tag=\u0026#34;true\u0026#34; else new_tag=\u0026#34;false\u0026#34; fi # Get the type of revision: # - delete: if the new_rev is zero # - commit: annotated tag # - tag: un-annotated tag if [ \u0026#34;$new_rev\u0026#34; = \u0026#34;$zero\u0026#34; ]; then new_rev_type=\u0026#34;delete\u0026#34; else new_rev_type=\u0026#34;$(git cat-file -t \u0026#34;$new_rev\u0026#34;)\u0026#34; fi # Exit if we are deleting a tag (nothing to check here) if [ \u0026#34;$new_rev_type\u0026#34; = \u0026#34;delete\u0026#34; ]; then exit 0 fi # Check the version against the tag (supports version.txt \u0026amp; package.json) if git cat-file -e \u0026#34;$new_rev:version.txt\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then version=\u0026#34;$(git cat-file -p \u0026#34;$new_rev:version.txt\u0026#34;)\u0026#34; if [ \u0026#34;$version\u0026#34; = \u0026#34;$tag_name\u0026#34; ] || [ \u0026#34;$version\u0026#34; = \u0026#34;${tag_name#v}\u0026#34; ]; then exit 0 else EMSG=\u0026#34;tag \u0026#39;$tag_name\u0026#39; and \u0026#39;version.txt\u0026#39; contents \u0026#39;$version\u0026#39; don\u0026#39;t match\u0026#34; echo \u0026#34;GL-HOOK-ERR: $EMSG\u0026#34; exit 1 fi elif git cat-file -e \u0026#34;$new_rev:package.json\u0026#34; \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then version=\u0026#34;$( git cat-file -p \u0026#34;$new_rev:package.json\u0026#34; | jsonpath version | tr -d \u0026#39;\\[\\]\u0026#34;\u0026#39; )\u0026#34; if [ \u0026#34;$version\u0026#34; = \u0026#34;$tag_name\u0026#34; ] || [ \u0026#34;$version\u0026#34; = \u0026#34;${tag_name#v}\u0026#34; ]; then exit 0 else EMSG=\u0026#34;tag \u0026#39;$tag_name\u0026#39; and \u0026#39;package.json\u0026#39; version \u0026#39;$version\u0026#39; don\u0026#39;t match\u0026#34; echo \u0026#34;GL-HOOK-ERR: $EMSG\u0026#34; exit 1 fi else # No version.txt or package.json file found exit 0 fi Some comments about it:\nwe are only looking for tags, if the ref_name does not have the prefix refs/tags/ the script does an exit 0,although we are checking if the tag is new or not we are not using the value (in gitlab that is handled by the protected tag feature),if we are deleting a tag the script does an exit 0, we don’t need to check anything in that case,we are ignoring if the tag is annotated or not (we set the new_rev_type to tag or commit, but we don’t use the value),we test first the version.txt file and if it does not exist we check the package.json file, if it does not exist either we do an exit 0, as there is no version to check against and we allow that on a tag,we add the GL-HOOK-ERR: prefix to the messages to show them on the gitlab web interface (can be tested creating a tag from it),to get the version on the package.json file we use the jsonpath binary (it is installed by the jsonpath ruby gem) because it is available on the gitlab container (initially I used sed to get the value, but a real JSON parser is always a better option). Once the hook is installed when a user tries to push a tag to a repository that has a version.txt file or package.json file and the tag does not match the version (if version.txt is present it takes precedence) the push fails.\nIf the tag matches or the files are not present the tag is added if the user has permission to add it in gitlab (our hook is only executed if the user is allowed to create or update the tag).\n","permalink":"https://blogops.mixinet.net/posts/using_git_server_hooks/","summary":"\u003cp\u003eSince a long time ago I’ve been a\n\u003ca href=\"https://about.gitlab.com/install/?version=ce\" target=\"_blank\" rel=\"noopener\"\u003egitlab-ce\u003c/a\u003e user, in fact I’ve set\nit up on three of the last four companies I’ve worked for (initially I installed\nit using the \u003ca href=\"https://docs.gitlab.com/omnibus/\" target=\"_blank\" rel=\"noopener\"\u003eomnibus\u003c/a\u003e packages on a\n\u003ca href=\"https://www.debian.org/\" target=\"_blank\" rel=\"noopener\"\u003edebian\u003c/a\u003e server but on the last two places I moved to\nthe \u003ca href=\"https://docs.gitlab.com/ee/install/docker.html\" target=\"_blank\" rel=\"noopener\"\u003edocker\u003c/a\u003e based installation,\nas it is easy to maintain and we don’t need a big installation as the teams\nusing it are small).\u003c/p\u003e\n\u003cp\u003eOn the company I work for now (\u003ca href=\"https://about.kyso.io/\" target=\"_blank\" rel=\"noopener\"\u003ekyso\u003c/a\u003e) we are using it\nto host all our internal repositories and to do all the CI/CD work (the\nautomatic deployments are triggered by web hooks in some cases, but the rest is\nall done using \u003ca href=\"https://about.gitlab.com/gitlab-ci/\" target=\"_blank\" rel=\"noopener\"\u003egitlab-ci\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003eThe majority of projects are using \u003ca href=\"https://nodejs.dev/\" target=\"_blank\" rel=\"noopener\"\u003enodejs\u003c/a\u003e as programming\nlanguage and we have automated the publication of \u003ccode\u003enpm\u003c/code\u003e packages on our gitlab\ninstance \u003ca href=\"https://docs.gitlab.com/ee/user/packages/npm_registry/\" target=\"_blank\" rel=\"noopener\"\u003enpm registry\u003c/a\u003e\nand even the publication into the \u003ca href=\"https://www.npmjs.com/\" target=\"_blank\" rel=\"noopener\"\u003enpmjs\u003c/a\u003e registry.\u003c/p\u003e\n\u003cp\u003eTo publish the packages we have added rules to the \u003ccode\u003egitlab-ci\u003c/code\u003e configuration of\nthe relevant repositories and we publish them when a \u003ccode\u003etag\u003c/code\u003e is created.\u003c/p\u003e\n\u003cp\u003eAs the we are \u003cem\u003elazy\u003c/em\u003e by definition, I configured the system to use the \u003ccode\u003etag\u003c/code\u003e as\nthe package version; I tested if the contents of the \u003ccode\u003epackage.json\u003c/code\u003e where in\nsync with the expected version and if it was not I updated it and did a force\npush of the \u003ccode\u003etag\u003c/code\u003e with the updated file using the following code on the script\nthat publishes the package:\u003c/p\u003e\n\u003cdiv class=\"listing-block\"\u003e\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"sh\"\u003e\u003cspan class=\"c\"\u003e# Update package version \u0026amp; add it to the .build-args\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eINITIAL_PACKAGE_VERSION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003enpm pkg get version|tr \u003cspan class=\"nt\"\u003e-d\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u0026#34;\u0026#39;\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\nnpm version \u003cspan class=\"nt\"\u003e--allow-same\u003c/span\u003e \u003cspan class=\"nt\"\u003e--no-commit-hooks\u003c/span\u003e \u003cspan class=\"nt\"\u003e--no-git-tag-version\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\u003c/span\u003e\n  \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_COMMIT_TAG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nv\"\u003eUPDATED_PACKAGE_VERSION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e$(\u003c/span\u003enpm pkg get version|tr \u003cspan class=\"nt\"\u003e-d\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u0026#34;\u0026#39;\u003c/span\u003e\u003cspan class=\"si\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;UPDATED_PACKAGE_VERSION=\u003c/span\u003e\u003cspan class=\"nv\"\u003e$UPDATED_PACKAGE_VERSION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e .build-args\n\u003cspan class=\"c\"\u003e# Update tag if the version was updated or abort\u003c/span\u003e\n\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$INITIAL_PACKAGE_VERSION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e!=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$UPDATED_PACKAGE_VERSION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n  if\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"nt\"\u003e-n\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_GIT_USER\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"nt\"\u003e-n\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_GIT_TOKEN\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\n    \u003c/span\u003egit commit \u003cspan class=\"nt\"\u003e-m\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Updated version from tag \u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_COMMIT_TAG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e package.json\n    git tag \u003cspan class=\"nt\"\u003e-f\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_COMMIT_TAG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"nt\"\u003e-m\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Updated version from tag\u0026#34;\u003c/span\u003e\n    git push \u003cspan class=\"nt\"\u003e-f\u003c/span\u003e \u003cspan class=\"nt\"\u003e-o\u003c/span\u003e ci.skip origin \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CI_COMMIT_TAG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"k\"\u003eelse\n    \u003c/span\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;!!! ERROR !!!\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;The updated tag could not be uploaded.\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Set CI_GIT_USER and CI_GIT_TOKEN or fix the \u0026#39;package.json\u0026#39; file\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;!!! ERROR !!!\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"nb\"\u003eexit \u003c/span\u003e1\n  \u003cspan class=\"k\"\u003efi\nfi\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Using Git Server Hooks on GitLab CE to Validate Tags"},{"content":"As promised, on this post I’m going to explain how I’ve configured this blog using hugo, asciidoctor and the papermod theme, how I publish it using nginx, how I’ve integrated the remark42 comment system and how I’ve automated its publication using forgejo and json2file-go.\nIt is a long post, but I hope that at least parts of it can be interesting for some, feel free to ignore it if that is not your case …​ :wink:\nHugo ConfigurationTheme settingsThe site is using the PaperMod theme and as I’m using asciidoctor to publish my content I’ve adjusted the settings to improve how things are shown with it.\nThe current config.yml file is the one shown below (probably some of the settings are not required nor being used right now, but I’m including the current file, so this post will have always the latest version of it):\nconfig.yml baseURL: https://blogops.mixinet.net/ title: Mixinet BlogOps Copyright: | \u0026amp;copy; 2022-2025 Sergio Talens-Oliag. pagination: pagerSize: 5 theme: PaperMod destination: public/ enableInlineShortcodes: true enableRobotsTXT: true buildDrafts: false buildFuture: false buildExpired: false enableEmoji: true pygmentsUseClasses: true minify: disableXML: true minifyOutput: true languages: en: languageName: \u0026#34;English\u0026#34; description: \u0026#34;Mixinet BlogOps - https://blogops.mixinet.net/\u0026#34; author: \u0026#34;Sergio Talens-Oliag\u0026#34; weight: 1 title: Mixinet BlogOps params: homeInfoParams: Title: \u0026#34;Sergio Talens-Oliag Technical Blog\u0026#34; Content: \u0026gt; ![Mixinet BlogOps](/images/mixinet-blogops.png) taxonomies: category: categories tag: tags series: series menu: main: - name: Archive url: archives weight: 5 - name: Categories url: categories/ weight: 10 - name: Tags url: tags/ weight: 10 - name: Search url: search/ weight: 15 outputs: home: - HTML - RSS - JSON params: author: Sergio Talens-Oliag env: production defaultTheme: light disableThemeToggle: false ShowShareButtons: true ShowReadingTime: true disableSpecial1stPost: true disableHLJS: true displayFullLangName: true ShowPostNavLinks: true ShowBreadCrumbs: true ShowCodeCopyButtons: true ShowRssButtonInSectionTermList: true ShowFullTextinRSS: true ShowToc: true TocOpen: false comments: true remark42SiteID: \u0026#34;blogops\u0026#34; remark42Url: \u0026#34;https://blogops.mixinet.net/remark42\u0026#34; profileMode: enabled: false title: Sergio Talens-Oliag Technical Blog imageUrl: \u0026#34;/images/mixinet-blogops.png\u0026#34; imageTitle: Mixinet BlogOps buttons: - name: Archives url: archives - name: Categories url: categories - name: Tags url: tags social: true socialIcons: - name: CV url: \u0026#34;https://www.uv.es/~sto/cv/\u0026#34; - name: Debian url: \u0026#34;https://people.debian.org/~sto/\u0026#34; - name: GitHub url: \u0026#34;https://github.com/sto/\u0026#34; - name: GitLab url: \u0026#34;https://gitlab.com/stalens/\u0026#34; - name: Linkedin url: \u0026#34;https://www.linkedin.com/in/sergio-talens-oliag/\u0026#34; - name: RSS url: \u0026#34;index.xml\u0026#34; assets: disableHLJS: true favicon: \u0026#34;/favicon.ico\u0026#34; favicon16x16: \u0026#34;/favicon-16x16.png\u0026#34; favicon32x32: \u0026#34;/favicon-32x32.png\u0026#34; apple_touch_icon: \u0026#34;/apple-touch-icon.png\u0026#34; safari_pinned_tab: \u0026#34;/safari-pinned-tab.svg\u0026#34; fuseOpts: isCaseSensitive: false shouldSort: true location: 0 distance: 1000 threshold: 0.4 minMatchCharLength: 0 keys: [\u0026#34;title\u0026#34;, \u0026#34;permalink\u0026#34;, \u0026#34;summary\u0026#34;, \u0026#34;content\u0026#34;] markup: asciidocExt: attributes: {\u0026#39;allow-uri-read\u0026#39;: true} backend: html5s extensions: [\u0026#39;asciidoctor-html5s\u0026#39;,\u0026#39;asciidoctor-diagram\u0026#39;] failureLevel: fatal noHeaderOrFooter: true preserveTOC: false safeMode: unsafe sectionNumbers: false trace: false verbose: false workingFolderCurrent: true privacy: vimeo: disabled: false simple: true x: disabled: false enableDNT: true simple: true instagram: disabled: false simple: true youtube: disabled: false privacyEnhanced: true services: instagram: disableInlineCSS: true x: disableInlineCSS: true security: exec: allow: - \u0026#39;^asciidoctor$\u0026#39; - \u0026#39;^dart-sass-embedded$\u0026#39; - \u0026#39;^go$\u0026#39; - \u0026#39;^npx$\u0026#39; - \u0026#39;^postcss$\u0026#39; Some notes about the settings:\ndisableHLJS and assets.disableHLJS are set to true; we plan to use rouge on adoc and the inclusion of the hljs assets adds styles that collide with the ones used by rouge.ShowToc is set to true and the TocOpen setting is set to false to make the ToC appear collapsed initially. My plan was to use the asciidoctor ToC, but after trying I believe that the theme one looks nice and I don’t need to adjust styles, although it has some issues with the html5s processor (the admonition titles use \u0026lt;h6\u0026gt; and they are shown on the ToC, which is weird), to fix it I’ve copied the layouts/partial/toc.html to my site repository and replaced the range of headings to end at 5 instead of 6 (in fact 5 still seems a lot, but as I don’t think I’ll use that heading level on the posts it doesn’t really matter).params.profileMode values are adjusted, but for now I’ve left it disabled setting params.profileMode.enabled to false and I’ve set the homeInfoParams to show more or less the same content with the latest posts under it (I’ve added some styles to my custom.css style sheet to center the text and image of the first post to match the look and feel of the profile).On the asciidocExt section I’ve adjusted the backend to use html5s, I’ve added the asciidoctor-html5s and asciidoctor-diagram extensions to asciidoctor and adjusted the workingFolderCurrent to true to make asciidoctor-diagram work right (haven’t tested it yet). Theme customisationsTo write in asciidoctor using the html5s processor I’ve added some files to the assets/css/extended directory:\nAs said before, I’ve added the file assets/css/extended/custom.css to make the homeInfoParams look like the profile page and I’ve also changed a little bit some theme styles to make things look better with the html5s output:\ncustom.css /* Fix first entry alignment to make it look like the profile */ .first-entry { text-align: center; } .first-entry img { display: inline; } /** * Remove margin for .post-content code and reduce padding to make it look * better with the asciidoctor html5s output. **/ .post-content code { margin: auto 0; padding: 4px; }I’ve also added the file assets/css/extended/adoc.css with some styles taken from the asciidoctor-default.css, see this blog post about the original file; mine is the same after formatting it with css-beautify and editing it to use variables for the colors to support light and dark themes:\nadoc.css /* AsciiDoctor*/ table { border-collapse: collapse; border-spacing: 0 } .admonitionblock\u0026gt;table { border-collapse: separate; border: 0; background: none; width: 100% } .admonitionblock\u0026gt;table td.icon { text-align: center; width: 80px } .admonitionblock\u0026gt;table td.icon img { max-width: none } .admonitionblock\u0026gt;table td.icon .title { font-weight: bold; font-family: \u0026#34;Open Sans\u0026#34;, \u0026#34;DejaVu Sans\u0026#34;, sans-serif; text-transform: uppercase } .admonitionblock\u0026gt;table td.content { padding-left: 1.125em; padding-right: 1.25em; border-left: 1px solid #ddddd8; color: var(--primary) } .admonitionblock\u0026gt;table td.content\u0026gt;:last-child\u0026gt;:last-child { margin-bottom: 0 } .admonitionblock td.icon [class^=\u0026#34;fa icon-\u0026#34;] { font-size: 2.5em; text-shadow: 1px 1px 2px var(--secondary); cursor: default } .admonitionblock td.icon .icon-note::before { content: \u0026#34;\\f05a\u0026#34;; color: var(--icon-note-color) } .admonitionblock td.icon .icon-tip::before { content: \u0026#34;\\f0eb\u0026#34;; color: var(--icon-tip-color) } .admonitionblock td.icon .icon-warning::before { content: \u0026#34;\\f071\u0026#34;; color: var(--icon-warning-color) } .admonitionblock td.icon .icon-caution::before { content: \u0026#34;\\f06d\u0026#34;; color: var(--icon-caution-color) } .admonitionblock td.icon .icon-important::before { content: \u0026#34;\\f06a\u0026#34;; color: var(--icon-important-color) } .conum[data-value] { display: inline-block; color: #fff !important; background-color: rgba(100, 100, 0, .8); -webkit-border-radius: 100px; border-radius: 100px; text-align: center; font-size: .75em; width: 1.67em; height: 1.67em; line-height: 1.67em; font-family: \u0026#34;Open Sans\u0026#34;, \u0026#34;DejaVu Sans\u0026#34;, sans-serif; font-style: normal; font-weight: bold } .conum[data-value] * { color: #fff !important } .conum[data-value]+b { display: none } .conum[data-value]::after { content: attr(data-value) } pre .conum[data-value] { position: relative; top: -.125em } b.conum * { color: inherit !important } .conum:not([data-value]):empty { display: none }The previous file uses variables from a partial copy of the theme-vars.css file that changes the highlighted code background color and adds the color definitions used by the admonitions:\ntheme-vars.css :root { /* Solarized base2 */ /* --hljs-bg: rgb(238, 232, 213); */ /* Solarized base3 */ /* --hljs-bg: rgb(253, 246, 227); */ /* Solarized base02 */ --hljs-bg: rgb(7, 54, 66); /* Solarized base03 */ /* --hljs-bg: rgb(0, 43, 54); */ /* Default asciidoctor theme colors */ --icon-note-color: #19407c; --icon-tip-color: var(--primary); --icon-warning-color: #bf6900; --icon-caution-color: #bf3400; --icon-important-color: #bf0000 } .dark { --hljs-bg: rgb(7, 54, 66); /* Asciidoctor theme colors with tint for dark background */ --icon-note-color: #3e7bd7; --icon-tip-color: var(--primary); --icon-warning-color: #ff8d03; --icon-caution-color: #ff7847; --icon-important-color: #ff3030 }The previous styles use font-awesome, so I’ve downloaded its resources for version 4.7.0 (the one used by asciidoctor) storing the font-awesome.css into on the assets/css/extended dir (that way it is merged with the rest of .css files) and copying the fonts to the static/assets/fonts/ dir (will be served directly):\nFA_BASE_URL=\u0026#34;https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0\u0026#34; curl \u0026#34;$FA_BASE_URL/css/font-awesome.css\u0026#34; \\ \u0026gt; assets/css/extended/font-awesome.css for f in FontAwesome.otf fontawesome-webfont.eot \\ fontawesome-webfont.svg fontawesome-webfont.ttf \\ fontawesome-webfont.woff fontawesome-webfont.woff2; do curl \u0026#34;$FA_BASE_URL/fonts/$f\u0026#34; \u0026gt; \u0026#34;static/assets/fonts/$f\u0026#34; doneAs already said the default highlighter is disabled (it provided a css compatible with rouge) so we need a css to do the highlight styling; as rouge provides a way to export them, I’ve created the assets/css/extended/rouge.css file with the thankful_eyes theme:\nrougify style thankful_eyes \u0026gt; assets/css/extended/rouge.cssTo support the use of the html5s backend with admonitions I’ve added a variation of the example found on this blog post to assets/js/adoc-admonitions.js:\nadoc-admonitions.js // replace the default admonitions block with a table that uses a format // similar to the standard asciidoctor ... as we are using fa-icons here there // is no need to add the icons: font entry on the document. window.addEventListener(\u0026#39;load\u0026#39;, function () { const admonitions = document.getElementsByClassName(\u0026#39;admonition-block\u0026#39;) for (let i = admonitions.length - 1; i \u0026gt;= 0; i--) { const elm = admonitions[i] const type = elm.classList[1] const title = elm.getElementsByClassName(\u0026#39;block-title\u0026#39;)[0]; const label = title.getElementsByClassName(\u0026#39;title-label\u0026#39;)[0] .innerHTML.slice(0, -1); elm.removeChild(elm.getElementsByClassName(\u0026#39;block-title\u0026#39;)[0]); const text = elm.innerHTML const parent = elm.parentNode const tempDiv = document.createElement(\u0026#39;div\u0026#39;) tempDiv.innerHTML = `\u0026lt;div class=\u0026#34;admonitionblock ${type}\u0026#34;\u0026gt; \u0026lt;table\u0026gt; \u0026lt;tbody\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td class=\u0026#34;icon\u0026#34;\u0026gt; \u0026lt;i class=\u0026#34;fa icon-${type}\u0026#34; title=\u0026#34;${label}\u0026#34;\u0026gt;\u0026lt;/i\u0026gt; \u0026lt;/td\u0026gt; \u0026lt;td class=\u0026#34;content\u0026#34;\u0026gt; ${text} \u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;/tbody\u0026gt; \u0026lt;/table\u0026gt; \u0026lt;/div\u0026gt;` const input = tempDiv.childNodes[0] parent.replaceChild(input, elm) } }) and enabled its minified use on the layouts/partials/extend_footer.html file adding the following lines to it:\n{{- $admonitions := slice (resources.Get \u0026#34;js/adoc-admonitions.js\u0026#34;) | resources.Concat \u0026#34;assets/js/adoc-admonitions.js\u0026#34; | minify | fingerprint }} \u0026lt;script defer crossorigin=\u0026#34;anonymous\u0026#34; src=\u0026#34;{{ $admonitions.RelPermalink }}\u0026#34; integrity=\u0026#34;{{ $admonitions.Data.Integrity }}\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; Note: As I’ve added the font-awesome resources and the standard styling for the asciidoctor admonitions, there is no need to add anything else to make things work.\nRemark42 configurationTo integrate Remark42 with the PaperMod theme I’ve created the file layouts/partials/comments.html with the following content based on the remark42 documentation, including extra code to sync the dark/light setting with the one set on the site:\ncomments.html \u0026lt;div id=\u0026#34;remark42\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;script\u0026gt; var remark_config = { host: {{ .Site.Params.remark42Url }}, site_id: {{ .Site.Params.remark42SiteID }}, url: {{ .Permalink }}, locale: {{ .Site.Language.Lang }} }; (function(c) { /* Adjust the theme using the local-storage pref-theme if set */ if (localStorage.getItem(\u0026#34;pref-theme\u0026#34;) === \u0026#34;dark\u0026#34;) { remark_config.theme = \u0026#34;dark\u0026#34;; } else if (localStorage.getItem(\u0026#34;pref-theme\u0026#34;) === \u0026#34;light\u0026#34;) { remark_config.theme = \u0026#34;light\u0026#34;; } /* Add remark42 widget */ for(var i = 0; i \u0026lt; c.length; i++){ var d = document, s = d.createElement(\u0026#39;script\u0026#39;); s.src = remark_config.host + \u0026#39;/web/\u0026#39; + c[i] +\u0026#39;.js\u0026#39;; s.defer = true; (d.head || d.body).appendChild(s); } })(remark_config.components || [\u0026#39;embed\u0026#39;]); \u0026lt;/script\u0026gt; In development I use it with anonymous comments enabled, but to avoid SPAM the production site uses social logins (for now I’ve only enabled Github \u0026amp; Google, if someone requests additional services I’ll check them, but those were the easy ones for me initially).\nTo support theme switching with remark42 I’ve also added the following inside the layouts/partials/extend_footer.html file:\n{{- if (not site.Params.disableThemeToggle) }} \u0026lt;script\u0026gt; /* Function to change theme when the toggle button is pressed */ document.getElementById(\u0026#34;theme-toggle\u0026#34;).addEventListener(\u0026#34;click\u0026#34;, () =\u0026gt; { if (typeof window.REMARK42 != \u0026#34;undefined\u0026#34;) { if (document.body.className.includes(\u0026#39;dark\u0026#39;)) { window.REMARK42.changeTheme(\u0026#39;light\u0026#39;); } else { window.REMARK42.changeTheme(\u0026#39;dark\u0026#39;); } } }); \u0026lt;/script\u0026gt; {{- end }} With this code if the theme-toggle button is pressed we change the remark42 theme before the PaperMod one (that’s needed here only, on page loads the remark42 theme is synced with the main one using the code from the layouts/partials/comments.html shown earlier).\nDevelopment setupTo preview the site on my laptop I’m using docker-compose with the following configuration:\ndocker-compose.yaml services: hugo: image: forgejo.mixinet.net/oci/hugo-adoc:latest container_name: hugo-adoc-blogops restart: always volumes: - .:/documents command: server --bind 0.0.0.0 -D -F user: ${APP_UID}:${APP_GID} nginx: image: nginx:latest container_name: nginx-blogops restart: always volumes: - ./nginx/default.conf:/etc/nginx/conf.d/default.conf ports: - 1313:1313 remark42: image: forgejo.mixinet.net/oci/remark42:latest container_name: remark42-blogops restart: always env_file: - ./.env - ./remark42/env.dev volumes: - ./remark42/var.dev:/srv/var To run it properly we have to create the .env file with the current user ID and GID on the variables APP_UID and APP_GID (if we don’t do it the files can end up being owned by a user that is not the same as the one running the services):\n$ echo \u0026#34;APP_UID=$(id -u)\\nAPP_GID=$(id -g)\u0026#34; \u0026gt; .env The Dockerfile used to generate the hugo-adoc image is:\nDockerfile FROM golang:1.24.1-alpine3.21 AS build ARG HUGO_BUILD_TAGS=extended ARG CGO=1 ENV CGO_ENABLED=${CGO} ENV GOOS=linux ENV GO111MODULE=on ENV HUGO_VERSION=v0.145.0 ENV MAGE_VERSION=v1.15.0 WORKDIR /go/src/github.com/gohugoio/hugo RUN apk update \u0026amp;\u0026amp;\\ # gcc/g++ are required to build SASS libraries for extended version apk add --no-cache curl gcc g++ musl-dev git \u0026amp;\u0026amp;\\ download_url=\u0026#34;https://api.github.com/repos/gohugoio/hugo/tarball/${HUGO_VERSION}\u0026#34; \u0026amp;\u0026amp;\\ echo \u0026#34;$download_url\u0026#34; \u0026amp;\u0026amp;\\ curl -sL \u0026#34;$download_url\u0026#34; -o /tmp/hugo.tgz \u0026amp;\u0026amp;\\ tar xf /tmp/hugo.tgz -C . --strip-components=1 \u0026amp;\u0026amp;\\ go install github.com/magefile/mage@${MAGE_VERSION} \u0026amp;\u0026amp;\\ mage hugo \u0026amp;\u0026amp;\\ mage install \u0026amp;\u0026amp;\\ cd / \u0026amp;\u0026amp;\\ rm -rf /tmp/hugo.tgz /go/src/github.com/gohugoio/hugo/* FROM asciidoctor/docker-asciidoctor:1.82 COPY --from=build /go/bin/hugo /usr/bin/hugo RUN gem install --no-document asciidoctor-html5s \u0026amp;\u0026amp;\\ apk update \u0026amp;\u0026amp;\\ apk add --no-cache ca-certificates libc6-compat libstdc++ git \u0026amp;\u0026amp;\\ /usr/bin/hugo version \u0026amp;\u0026amp;\\ rm -rf /var/cache/apk/* # Expose port for live server EXPOSE 1313 ENTRYPOINT [\u0026#34;/usr/bin/hugo\u0026#34;] CMD [\u0026#34;\u0026#34;] If you review it you will see that I’m using the docker-asciidoctor image as the base; the idea is that this image has all I need to work with asciidoctor and to use hugo I download the code and compile it on a builder container.\nNote: Originally I downloaded the binary from their latest release page on github but it stoped working with alpine even if we install the libc6-compat package, so I decided to compile it myself.\nThe image does not launch the server by default because I don’t want it to; in fact I use the same docker-compose.yml file to publish the site in production simply calling the container without the arguments passed on the docker-compose.yml file (see later).\nWhen running the containers with docker-compose up (or docker compose up if you have the docker-compose-plugin package installed) we also launch a nginx container and the remark42 service so we can test everything together.\nThe Dockerfile for the remark42 image is the original one with an updated version of the init.sh script:\nDockerfile FROM umputun/remark42:latest COPY init.sh /init.sh The updated init.sh is similar to the original, but allows us to use an APP_GID variable and updates the /etc/group file of the container so the files get the right user and group (with the original script the group is always 1001):\ninit.sh #!/sbin/dinit /bin/sh uid=\u0026#34;$(id -u)\u0026#34; if [ \u0026#34;${uid}\u0026#34; -eq \u0026#34;0\u0026#34; ]; then echo \u0026#34;init container\u0026#34; # set container\u0026#39;s time zone cp \u0026#34;/usr/share/zoneinfo/${TIME_ZONE}\u0026#34; /etc/localtime echo \u0026#34;${TIME_ZONE}\u0026#34; \u0026gt;/etc/timezone echo \u0026#34;set timezone ${TIME_ZONE} ($(date))\u0026#34; # set UID \u0026amp; GID for the app if [ \u0026#34;${APP_UID}\u0026#34; ] || [ \u0026#34;${APP_GID}\u0026#34; ]; then [ \u0026#34;${APP_UID}\u0026#34; ] || APP_UID=\u0026#34;1001\u0026#34; [ \u0026#34;${APP_GID}\u0026#34; ] || APP_GID=\u0026#34;${APP_UID}\u0026#34; echo \u0026#34;set custom APP_UID=${APP_UID} \u0026amp; APP_GID=${APP_GID}\u0026#34; sed -i \u0026#34;s/^app:x:1001:1001:/app:x:${APP_UID}:${APP_GID}:/\u0026#34; /etc/passwd sed -i \u0026#34;s/^app:x:1001:/app:x:${APP_GID}:/\u0026#34; /etc/group else echo \u0026#34;custom APP_UID and/or APP_GID not defined, using 1001:1001\u0026#34; fi chown -R app:app /srv /home/app fi echo \u0026#34;prepare environment\u0026#34; # replace {% REMARK_URL %} by content of REMARK_URL variable find /srv -regex \u0026#39;.*\\.\\(html\\|js\\|mjs\\)$\u0026#39; -print \\ -exec sed -i \u0026#34;s|{% REMARK_URL %}|${REMARK_URL}|g\u0026#34; {} \\; if [ -n \u0026#34;${SITE_ID}\u0026#34; ]; then #replace \u0026#34;site_id: \u0026#39;remark\u0026#39;\u0026#34; by SITE_ID sed -i \u0026#34;s|\u0026#39;remark\u0026#39;|\u0026#39;${SITE_ID}\u0026#39;|g\u0026#34; /srv/web/*.html fi echo \u0026#34;execute \\\u0026#34;$*\\\u0026#34;\u0026#34; if [ \u0026#34;${uid}\u0026#34; -eq \u0026#34;0\u0026#34; ]; then exec su-exec app \u0026#34;$@\u0026#34; else exec \u0026#34;$@\u0026#34; fi The environment file used with remark42 for development is quite minimal:\nenv.dev TIME_ZONE=Europe/Madrid REMARK_URL=http://localhost:1313/remark42 SITE=blogops SECRET=123456 ADMIN_SHARED_ID=sto AUTH_ANON=true EMOJI=true And the nginx/default.conf file used to publish the service locally is simple too:\ndefault.conf server { listen 1313; server_name localhost; location / { proxy_pass http://hugo:1313; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection \u0026#34;upgrade\u0026#34;; } location /remark42/ { rewrite /remark42/(.*) /$1 break; proxy_pass http://remark42:8080/; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } Production setupThe VM where I’m publishing the blog runs Debian GNU/Linux and uses binaries from local packages and applications packaged inside containers.\nTo run the containers I’m using docker-ce (I could have used podman instead, but I already had it installed on the machine, so I stayed with it).\nThe binaries used on this project are included on the following packages from the main Debian repository:\ngit to clone \u0026amp; pull the repository,jq to parse json files from shell scripts,json2file-go to save the webhook messages to files,inotify-tools to detect when new files are stored by json2file-go and launch scripts to process them,nginx to publish the site using HTTPS and work as proxy for json2file-go and remark42 (I run it using a container),task-spool to queue the scripts that update the deployment. And I’m using docker and docker compose from the debian packages on the docker repository:\ndocker-ce to run the containers,docker-compose-plugin to run docker compose (it is a plugin, so no - in the name). Note: On the following sections I’m assuming that the user doing the work belongs to the docker group, that is, has permission to run docker.\nRepository checkoutTo manage the git repository I’ve created a deploy key, added it to forgejo and cloned the project on the /srv/blogops PATH (that route is owned by a regular user that has permissions to run docker, as I said before).\nCompiling the site with hugoTo compile the site we are using the docker-compose.yml file seen before, to be able to run it first we build the container images and once we have them we launch hugo using docker compose run:\n$ cd /srv/blogops $ git pull $ docker compose build $ if [ -d \u0026#34;./public\u0026#34; ]; then rm -rf ./public; fi $ docker compose run hugo -- The compilation leaves the static HTML on /srv/blogops/public (we remove the directory first because hugo does not clean the destination folder as jekyll does).\nThe deploy script re-generates the site as described and moves the public directory to its final place for publishing.\nRunning remark42 with dockerOn the /srv/blogops/remark42 folder I have the following docker-compose.yml:\ndocker-compose.yml version: \u0026#34;2\u0026#34; services: remark42: image: forgejo.mixinet.net/oci/remark42:latest env_file: - ../.env - ./env.prod container_name: remark42 restart: always volumes: - ./var.prod:/srv/var ports: - 127.0.0.1:8042:8080 The ../.env file is loaded to get the APP_UID and APP_GID variables that are used by my version of the init.sh script to adjust file permissions and the env.prod file contains the rest of the settings for remark42, including the social network tokens (see the remark42 documentation for the available parameters, I don’t include my configuration here because some of them are secrets).\nNginx configurationThe nginx configuration for the blogops.mixinet.net site is as simple as:\nserver { listen 443 ssl http2; server_name blogops.mixinet.net; ssl_certificate /etc/letsencrypt/live/blogops.mixinet.net/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/blogops.mixinet.net/privkey.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; access_log /var/log/nginx/blogops.mixinet.net-443.access.log; error_log /var/log/nginx/blogops.mixinet.net-443.error.log; root /srv/blogops/nginx/public_html; location / { try_files $uri $uri/ =404; } include /srv/blogops/nginx/remark42.conf; } server { listen 80 ; listen [::]:80 ; server_name blogops.mixinet.net; access_log /var/log/nginx/blogops.mixinet.net-80.access.log; error_log /var/log/nginx/blogops.mixinet.net-80.error.log; if ($host = blogops.mixinet.net) { return 301 https://$host$request_uri; } return 404; } On this configuration the certificates are managed by certbot and the server root directory is on /srv/blogops/nginx/public_html and not on /srv/blogops/public; the reason for that is that I want to be able to compile without affecting the running site, the deployment script generates the site on /srv/blogops/public and if all works well we rename folders to do the switch, making the change feel almost atomic.\njson2file-go configurationAs I have a working WireGuard VPN between the machine running forgejo at my home and the VM where the blog is served, I’m going to configure the json2file-go to listen for connections on a high port using a self signed certificate and listening on IP addresses only reachable through the VPN.\nTo do it we create a systemd socket to run json2file-go and adjust its configuration to listen on a private IP (we use the FreeBind option on its definition to be able to launch the service even when the IP is not available, that is, when the VPN is down).\nThe following script can be used to set up the json2file-go configuration:\nsetup-json2file.sh #!/bin/sh set -e # --------- # VARIABLES # --------- BASE_DIR=\u0026#34;/srv/blogops/webhook\u0026#34; J2F_DIR=\u0026#34;$BASE_DIR/json2file\u0026#34; TLS_DIR=\u0026#34;$BASE_DIR/tls\u0026#34; J2F_SERVICE_NAME=\u0026#34;json2file-go\u0026#34; J2F_SERVICE_DIR=\u0026#34;/etc/systemd/system/json2file-go.service.d\u0026#34; J2F_SERVICE_OVERRIDE=\u0026#34;$J2F_SERVICE_DIR/override.conf\u0026#34; J2F_SOCKET_DIR=\u0026#34;/etc/systemd/system/json2file-go.socket.d\u0026#34; J2F_SOCKET_OVERRIDE=\u0026#34;$J2F_SOCKET_DIR/override.conf\u0026#34; J2F_BASEDIR_FILE=\u0026#34;/etc/json2file-go/basedir\u0026#34; J2F_DIRLIST_FILE=\u0026#34;/etc/json2file-go/dirlist\u0026#34; J2F_CRT_FILE=\u0026#34;/etc/json2file-go/certfile\u0026#34; J2F_KEY_FILE=\u0026#34;/etc/json2file-go/keyfile\u0026#34; J2F_CRT_PATH=\u0026#34;$TLS_DIR/crt.pem\u0026#34; J2F_KEY_PATH=\u0026#34;$TLS_DIR/key.pem\u0026#34; # ---- # MAIN # ---- # Install packages used with json2file for the blogops site sudo apt update sudo apt install -y json2file-go uuid if [ -z \u0026#34;$(type mkcert)\u0026#34; ]; then sudo apt install -y mkcert fi sudo apt clean # Configuration file values J2F_USER=\u0026#34;$(id -u)\u0026#34; J2F_GROUP=\u0026#34;$(id -g)\u0026#34; J2F_DIRLIST=\u0026#34;blogops:$(uuid)\u0026#34; J2F_LISTEN_STREAM=\u0026#34;172.31.31.1:4443\u0026#34; # Configure json2file [ -d \u0026#34;$J2F_DIR\u0026#34; ] || mkdir \u0026#34;$J2F_DIR\u0026#34; sudo sh -c \u0026#34;echo \u0026#39;$J2F_DIR\u0026#39; \u0026gt;\u0026#39;$J2F_BASEDIR_FILE\u0026#39;\u0026#34; [ -d \u0026#34;$TLS_DIR\u0026#34; ] || mkdir \u0026#34;$TLS_DIR\u0026#34; if [ ! -f \u0026#34;$J2F_CRT_PATH\u0026#34; ] || [ ! -f \u0026#34;$J2F_KEY_PATH\u0026#34; ]; then mkcert -cert-file \u0026#34;$J2F_CRT_PATH\u0026#34; -key-file \u0026#34;$J2F_KEY_PATH\u0026#34; \u0026#34;$(hostname -f)\u0026#34; fi sudo sh -c \u0026#34;echo \u0026#39;$J2F_CRT_PATH\u0026#39; \u0026gt;\u0026#39;$J2F_CRT_FILE\u0026#39;\u0026#34; sudo sh -c \u0026#34;echo \u0026#39;$J2F_KEY_PATH\u0026#39; \u0026gt;\u0026#39;$J2F_KEY_FILE\u0026#39;\u0026#34; sudo sh -c \u0026#34;cat \u0026gt;\u0026#39;$J2F_DIRLIST_FILE\u0026#39;\u0026#34; \u0026lt;\u0026lt;EOF $(echo \u0026#34;$J2F_DIRLIST\u0026#34; | tr \u0026#39;;\u0026#39; \u0026#39;\\n\u0026#39;) EOF # Service override [ -d \u0026#34;$J2F_SERVICE_DIR\u0026#34; ] || sudo mkdir \u0026#34;$J2F_SERVICE_DIR\u0026#34; sudo sh -c \u0026#34;cat \u0026gt;\u0026#39;$J2F_SERVICE_OVERRIDE\u0026#39;\u0026#34; \u0026lt;\u0026lt;EOF [Service] User=$J2F_USER Group=$J2F_GROUP EOF # Socket override [ -d \u0026#34;$J2F_SOCKET_DIR\u0026#34; ] || sudo mkdir \u0026#34;$J2F_SOCKET_DIR\u0026#34; sudo sh -c \u0026#34;cat \u0026gt;\u0026#39;$J2F_SOCKET_OVERRIDE\u0026#39;\u0026#34; \u0026lt;\u0026lt;EOF [Socket] # Set FreeBind to listen on missing addresses (the VPN can be down sometimes) FreeBind=true # Set ListenStream to nothing to clear its value and add the new value later ListenStream= ListenStream=$J2F_LISTEN_STREAM EOF # Restart and enable service sudo systemctl daemon-reload sudo systemctl stop \u0026#34;$J2F_SERVICE_NAME\u0026#34; sudo systemctl start \u0026#34;$J2F_SERVICE_NAME\u0026#34; sudo systemctl enable \u0026#34;$J2F_SERVICE_NAME\u0026#34; # ---- # vim: ts=2:sw=2:et:ai:sts=2 Warning: The script uses mkcert to create the temporary certificates, to install the package on bullseye the backports repository must be available.\nForgejo configurationTo make forgejo use our json2file-go server we go to the project and enter into the hooks/forgejo/new page, once there we create a new webhook of type forgejo and set the target URL to https://172.31.31.1:4443/blogops and on the secret field we put the token generated with uuid by the setup script:\nsed -n -e \u0026#39;s/blogops://p\u0026#39; /etc/json2file-go/dirlist The rest of the settings can be left as they are:\nTrigger on: Push eventsBranch filter: * Warning: We are using an internal IP and a self signed certificate, that means that we have to review that the webhook section of the app.ini of our forgejo server allows us to call the IP and skips the TLS verification (you can see the available options on the forgejo documentation).\nThe [webhook] section of my server looks like this:\n[webhook] ALLOWED_HOST_LIST=private SKIP_TLS_VERIFY=true Once we have the webhook configured we can try it and if it works our json2file server will store the file on the /srv/blogops/webhook/json2file/blogops/ folder.\nThe json2file spooler scriptWith the previous configuration our system is ready to receive webhook calls from forgejo and store the messages on files, but we have to do something to process those files once they are saved in our machine.\nAn option could be to use a cronjob to look for new files, but we can do better on Linux using inotify …​ we will use the inotifywait command from inotify-tools to watch the json2file output directory and execute a script each time a new file is moved inside it or closed after writing (IN_CLOSE_WRITE and IN_MOVED_TO events).\nTo avoid concurrency problems we are going to use task-spooler to launch the scripts that process the webhooks using a queue of length 1, so they are executed one by one in a FIFO queue.\nThe spooler script is this:\nblogops-spooler.sh #!/bin/sh set -e # --------- # VARIABLES # --------- BASE_DIR=\u0026#34;/srv/blogops/webhook\u0026#34; BIN_DIR=\u0026#34;$BASE_DIR/bin\u0026#34; TSP_DIR=\u0026#34;$BASE_DIR/tsp\u0026#34; WEBHOOK_COMMAND=\u0026#34;$BIN_DIR/blogops-webhook.sh\u0026#34; # --------- # FUNCTIONS # --------- queue_job() { echo \u0026#34;Queuing job to process file \u0026#39;$1\u0026#39;\u0026#34; TMPDIR=\u0026#34;$TSP_DIR\u0026#34; TS_SLOTS=\u0026#34;1\u0026#34; TS_MAXFINISHED=\u0026#34;10\u0026#34; \\ tsp -n \u0026#34;$WEBHOOK_COMMAND\u0026#34; \u0026#34;$1\u0026#34; } # ---- # MAIN # ---- INPUT_DIR=\u0026#34;$1\u0026#34; if [ ! -d \u0026#34;$INPUT_DIR\u0026#34; ]; then echo \u0026#34;Input directory \u0026#39;$INPUT_DIR\u0026#39; does not exist, aborting!\u0026#34; exit 1 fi [ -d \u0026#34;$TSP_DIR\u0026#34; ] || mkdir \u0026#34;$TSP_DIR\u0026#34; echo \u0026#34;Processing existing files under \u0026#39;$INPUT_DIR\u0026#39;\u0026#34; find \u0026#34;$INPUT_DIR\u0026#34; -type f | sort | while read -r _filename; do queue_job \u0026#34;$_filename\u0026#34; done # Use inotifywatch to process new files echo \u0026#34;Watching for new files under \u0026#39;$INPUT_DIR\u0026#39;\u0026#34; inotifywait -q -m -e close_write,moved_to --format \u0026#34;%w%f\u0026#34; -r \u0026#34;$INPUT_DIR\u0026#34; | while read -r _filename; do queue_job \u0026#34;$_filename\u0026#34; done # ---- # vim: ts=2:sw=2:et:ai:sts=2 To run it as a daemon we install it as a systemd service using the following script:\nsetup-spooler.sh #!/bin/sh set -e # --------- # VARIABLES # --------- BASE_DIR=\u0026#34;/srv/blogops/webhook\u0026#34; BIN_DIR=\u0026#34;$BASE_DIR/bin\u0026#34; J2F_DIR=\u0026#34;$BASE_DIR/json2file\u0026#34; SPOOLER_COMMAND=\u0026#34;$BIN_DIR/blogops-spooler.sh \u0026#39;$J2F_DIR\u0026#39;\u0026#34; SPOOLER_SERVICE_NAME=\u0026#34;blogops-j2f-spooler\u0026#34; SPOOLER_SERVICE_FILE=\u0026#34;/etc/systemd/system/$SPOOLER_SERVICE_NAME.service\u0026#34; # Configuration file values J2F_USER=\u0026#34;$(id -u)\u0026#34; J2F_GROUP=\u0026#34;$(id -g)\u0026#34; # ---- # MAIN # ---- # Install packages used with the webhook processor sudo apt update sudo apt install -y inotify-tools jq task-spooler sudo apt clean # Configure process service sudo sh -c \u0026#34;cat \u0026gt; $SPOOLER_SERVICE_FILE\u0026#34; \u0026lt;\u0026lt;EOF [Install] WantedBy=multi-user.target [Unit] Description=json2file processor for $J2F_USER After=docker.service [Service] Type=simple User=$J2F_USER Group=$J2F_GROUP ExecStart=$SPOOLER_COMMAND EOF # Restart and enable service sudo systemctl daemon-reload sudo systemctl stop \u0026#34;$SPOOLER_SERVICE_NAME\u0026#34; || true sudo systemctl start \u0026#34;$SPOOLER_SERVICE_NAME\u0026#34; sudo systemctl enable \u0026#34;$SPOOLER_SERVICE_NAME\u0026#34; # ---- # vim: ts=2:sw=2:et:ai:sts=2 The forgejo webhook processorFinally, the script that processes the JSON files does the following:\nFirst, it checks if the repository and branch are right,Then, it fetches and checks out the commit referenced on the JSON file,Once the files are updated, compiles the site using hugo with docker compose,If the compilation succeeds the script renames directories to swap the old version of the site by the new one. If there is a failure the script aborts but before doing it or if the swap succeeded the system sends an email to the configured address and/or the user that pushed updates to the repository with a log of what happened.\nThe current script is this one:\nblogops-webhook.sh #!/bin/sh set -e # --------- # VARIABLES # --------- # Values REPO_REF=\u0026#34;refs/heads/main\u0026#34; REPO_CLONE_URL=\u0026#34;https://forgejo.mixinet.net/mixinet/blogops.git\u0026#34; MAIL_PREFIX=\u0026#34;[BLOGOPS-WEBHOOK] \u0026#34; # Address that gets all messages, leave it empty if not wanted MAIL_TO_ADDR=\u0026#34;blogops@mixinet.net\u0026#34; # If the following variable is set to \u0026#39;true\u0026#39; the pusher gets mail on failures MAIL_ERRFILE=\u0026#34;false\u0026#34; # If the following variable is set to \u0026#39;true\u0026#39; the pusher gets mail on success MAIL_LOGFILE=\u0026#34;false\u0026#34; # forgejo\u0026#39;s conf/app.ini value of NO_REPLY_ADDRESS, it is used for email domains # when the KeepEmailPrivate option is enabled for a user NO_REPLY_ADDRESS=\u0026#34;noreply.example.org\u0026#34; # Directories BASE_DIR=\u0026#34;/srv/blogops\u0026#34; PUBLIC_DIR=\u0026#34;$BASE_DIR/public\u0026#34; NGINX_BASE_DIR=\u0026#34;$BASE_DIR/nginx\u0026#34; PUBLIC_HTML_DIR=\u0026#34;$NGINX_BASE_DIR/public_html\u0026#34; WEBHOOK_BASE_DIR=\u0026#34;$BASE_DIR/webhook\u0026#34; WEBHOOK_SPOOL_DIR=\u0026#34;$WEBHOOK_BASE_DIR/spool\u0026#34; WEBHOOK_ACCEPTED=\u0026#34;$WEBHOOK_SPOOL_DIR/accepted\u0026#34; WEBHOOK_DEPLOYED=\u0026#34;$WEBHOOK_SPOOL_DIR/deployed\u0026#34; WEBHOOK_REJECTED=\u0026#34;$WEBHOOK_SPOOL_DIR/rejected\u0026#34; WEBHOOK_TROUBLED=\u0026#34;$WEBHOOK_SPOOL_DIR/troubled\u0026#34; WEBHOOK_LOG_DIR=\u0026#34;$WEBHOOK_SPOOL_DIR/log\u0026#34; # Files TODAY=\u0026#34;$(date +%Y%m%d)\u0026#34; OUTPUT_BASENAME=\u0026#34;$(date +%Y%m%d-%H%M%S.%N)\u0026#34; WEBHOOK_LOGFILE_PATH=\u0026#34;$WEBHOOK_LOG_DIR/$OUTPUT_BASENAME.log\u0026#34; WEBHOOK_ACCEPTED_JSON=\u0026#34;$WEBHOOK_ACCEPTED/$OUTPUT_BASENAME.json\u0026#34; WEBHOOK_ACCEPTED_LOGF=\u0026#34;$WEBHOOK_ACCEPTED/$OUTPUT_BASENAME.log\u0026#34; WEBHOOK_REJECTED_TODAY=\u0026#34;$WEBHOOK_REJECTED/$TODAY\u0026#34; WEBHOOK_REJECTED_JSON=\u0026#34;$WEBHOOK_REJECTED_TODAY/$OUTPUT_BASENAME.json\u0026#34; WEBHOOK_REJECTED_LOGF=\u0026#34;$WEBHOOK_REJECTED_TODAY/$OUTPUT_BASENAME.log\u0026#34; WEBHOOK_DEPLOYED_TODAY=\u0026#34;$WEBHOOK_DEPLOYED/$TODAY\u0026#34; WEBHOOK_DEPLOYED_JSON=\u0026#34;$WEBHOOK_DEPLOYED_TODAY/$OUTPUT_BASENAME.json\u0026#34; WEBHOOK_DEPLOYED_LOGF=\u0026#34;$WEBHOOK_DEPLOYED_TODAY/$OUTPUT_BASENAME.log\u0026#34; WEBHOOK_TROUBLED_TODAY=\u0026#34;$WEBHOOK_TROUBLED/$TODAY\u0026#34; WEBHOOK_TROUBLED_JSON=\u0026#34;$WEBHOOK_TROUBLED_TODAY/$OUTPUT_BASENAME.json\u0026#34; WEBHOOK_TROUBLED_LOGF=\u0026#34;$WEBHOOK_TROUBLED_TODAY/$OUTPUT_BASENAME.log\u0026#34; # Query to get variables from a forgejo webhook json ENV_VARS_QUERY=\u0026#34;$( printf \u0026#34;%s\u0026#34; \\ \u0026#39;(. | @sh \u0026#34;gt_ref=\\(.ref);\u0026#34;),\u0026#39; \\ \u0026#39;(. | @sh \u0026#34;gt_after=\\(.after);\u0026#34;),\u0026#39; \\ \u0026#39;(.repository | @sh \u0026#34;gt_repo_clone_url=\\(.clone_url);\u0026#34;),\u0026#39; \\ \u0026#39;(.repository | @sh \u0026#34;gt_repo_name=\\(.name);\u0026#34;),\u0026#39; \\ \u0026#39;(.pusher | @sh \u0026#34;gt_pusher_full_name=\\(.full_name);\u0026#34;),\u0026#39; \\ \u0026#39;(.pusher | @sh \u0026#34;gt_pusher_email=\\(.email);\u0026#34;)\u0026#39; )\u0026#34; # --------- # Functions # --------- webhook_log() { echo \u0026#34;$(date -R) $*\u0026#34; \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; } webhook_check_directories() { for _d in \u0026#34;$WEBHOOK_SPOOL_DIR\u0026#34; \u0026#34;$WEBHOOK_ACCEPTED\u0026#34; \u0026#34;$WEBHOOK_DEPLOYED\u0026#34; \\ \u0026#34;$WEBHOOK_REJECTED\u0026#34; \u0026#34;$WEBHOOK_TROUBLED\u0026#34; \u0026#34;$WEBHOOK_LOG_DIR\u0026#34;; do [ -d \u0026#34;$_d\u0026#34; ] || mkdir \u0026#34;$_d\u0026#34; done } webhook_clean_directories() { # Try to remove empty dirs for _d in \u0026#34;$WEBHOOK_ACCEPTED\u0026#34; \u0026#34;$WEBHOOK_DEPLOYED\u0026#34; \u0026#34;$WEBHOOK_REJECTED\u0026#34; \\ \u0026#34;$WEBHOOK_TROUBLED\u0026#34; \u0026#34;$WEBHOOK_LOG_DIR\u0026#34; \u0026#34;$WEBHOOK_SPOOL_DIR\u0026#34;; do if [ -d \u0026#34;$_d\u0026#34; ]; then rmdir \u0026#34;$_d\u0026#34; 2\u0026gt;/dev/null || true fi done } webhook_accept() { webhook_log \u0026#34;Accepted: $*\u0026#34; mv \u0026#34;$WEBHOOK_JSON_INPUT_FILE\u0026#34; \u0026#34;$WEBHOOK_ACCEPTED_JSON\u0026#34; mv \u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; \u0026#34;$WEBHOOK_ACCEPTED_LOGF\u0026#34; WEBHOOK_LOGFILE_PATH=\u0026#34;$WEBHOOK_ACCEPTED_LOGF\u0026#34; } webhook_reject() { [ -d \u0026#34;$WEBHOOK_REJECTED_TODAY\u0026#34; ] || mkdir \u0026#34;$WEBHOOK_REJECTED_TODAY\u0026#34; webhook_log \u0026#34;Rejected: $*\u0026#34; if [ -f \u0026#34;$WEBHOOK_JSON_INPUT_FILE\u0026#34; ]; then mv \u0026#34;$WEBHOOK_JSON_INPUT_FILE\u0026#34; \u0026#34;$WEBHOOK_REJECTED_JSON\u0026#34; fi mv \u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; \u0026#34;$WEBHOOK_REJECTED_LOGF\u0026#34; exit 0 } webhook_deployed() { [ -d \u0026#34;$WEBHOOK_DEPLOYED_TODAY\u0026#34; ] || mkdir \u0026#34;$WEBHOOK_DEPLOYED_TODAY\u0026#34; webhook_log \u0026#34;Deployed: $*\u0026#34; mv \u0026#34;$WEBHOOK_ACCEPTED_JSON\u0026#34; \u0026#34;$WEBHOOK_DEPLOYED_JSON\u0026#34; mv \u0026#34;$WEBHOOK_ACCEPTED_LOGF\u0026#34; \u0026#34;$WEBHOOK_DEPLOYED_LOGF\u0026#34; WEBHOOK_LOGFILE_PATH=\u0026#34;$WEBHOOK_DEPLOYED_LOGF\u0026#34; } webhook_troubled() { [ -d \u0026#34;$WEBHOOK_TROUBLED_TODAY\u0026#34; ] || mkdir \u0026#34;$WEBHOOK_TROUBLED_TODAY\u0026#34; webhook_log \u0026#34;Troubled: $*\u0026#34; mv \u0026#34;$WEBHOOK_ACCEPTED_JSON\u0026#34; \u0026#34;$WEBHOOK_TROUBLED_JSON\u0026#34; mv \u0026#34;$WEBHOOK_ACCEPTED_LOGF\u0026#34; \u0026#34;$WEBHOOK_TROUBLED_LOGF\u0026#34; WEBHOOK_LOGFILE_PATH=\u0026#34;$WEBHOOK_TROUBLED_LOGF\u0026#34; } print_mailto() { _addr=\u0026#34;$1\u0026#34; _user_email=\u0026#34;\u0026#34; # Add the pusher email address unless it is from the domain NO_REPLY_ADDRESS, # which should match the value of that variable on the forgejo \u0026#39;app.ini\u0026#39; (it # is the domain used for emails when the user hides it). # shellcheck disable=SC2154 if [ -n \u0026#34;${gt_pusher_email##*@\u0026#34;${NO_REPLY_ADDRESS}\u0026#34;}\u0026#34; ] \u0026amp;\u0026amp; [ -z \u0026#34;${gt_pusher_email##*@*}\u0026#34; ]; then _user_email=\u0026#34;\\\u0026#34;$gt_pusher_full_name \u0026lt;$gt_pusher_email\u0026gt;\\\u0026#34;\u0026#34; fi if [ \u0026#34;$_addr\u0026#34; ] \u0026amp;\u0026amp; [ \u0026#34;$_user_email\u0026#34; ]; then echo \u0026#34;$_addr,$_user_email\u0026#34; elif [ \u0026#34;$_user_email\u0026#34; ]; then echo \u0026#34;$_user_email\u0026#34; elif [ \u0026#34;$_addr\u0026#34; ]; then echo \u0026#34;$_addr\u0026#34; fi } mail_success() { to_addr=\u0026#34;$MAIL_TO_ADDR\u0026#34; if [ \u0026#34;$MAIL_LOGFILE\u0026#34; = \u0026#34;true\u0026#34; ]; then to_addr=\u0026#34;$(print_mailto \u0026#34;$to_addr\u0026#34;)\u0026#34; fi if [ \u0026#34;$to_addr\u0026#34; ]; then # shellcheck disable=SC2154 subject=\u0026#34;OK - $gt_repo_name updated to commit \u0026#39;$gt_after\u0026#39;\u0026#34; mail -s \u0026#34;${MAIL_PREFIX}${subject}\u0026#34; \u0026#34;$to_addr\u0026#34; \\ \u0026lt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; fi } mail_failure() { to_addr=\u0026#34;$MAIL_TO_ADDR\u0026#34; if [ \u0026#34;$MAIL_ERRFILE\u0026#34; = true ]; then to_addr=\u0026#34;$(print_mailto \u0026#34;$to_addr\u0026#34;)\u0026#34; fi if [ \u0026#34;$to_addr\u0026#34; ]; then # shellcheck disable=SC2154 subject=\u0026#34;KO - $gt_repo_name update FAILED for commit \u0026#39;$gt_after\u0026#39;\u0026#34; mail -s \u0026#34;${MAIL_PREFIX}${subject}\u0026#34; \u0026#34;$to_addr\u0026#34; \\ \u0026lt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; fi } # ---- # MAIN # ---- # Check directories webhook_check_directories # Go to the base directory cd \u0026#34;$BASE_DIR\u0026#34; # Check if the file exists WEBHOOK_JSON_INPUT_FILE=\u0026#34;$1\u0026#34; if [ ! -f \u0026#34;$WEBHOOK_JSON_INPUT_FILE\u0026#34; ]; then webhook_reject \u0026#34;Input arg \u0026#39;$1\u0026#39; is not a file, aborting\u0026#34; fi # Parse the file webhook_log \u0026#34;Processing file \u0026#39;$WEBHOOK_JSON_INPUT_FILE\u0026#39;\u0026#34; eval \u0026#34;$(jq -r \u0026#34;$ENV_VARS_QUERY\u0026#34; \u0026#34;$WEBHOOK_JSON_INPUT_FILE\u0026#34;)\u0026#34; # Check that the repository clone url is right # shellcheck disable=SC2154 if [ \u0026#34;$gt_repo_clone_url\u0026#34; != \u0026#34;$REPO_CLONE_URL\u0026#34; ]; then webhook_reject \u0026#34;Wrong repository: \u0026#39;$gt_clone_url\u0026#39;\u0026#34; fi # Check that the branch is the right one # shellcheck disable=SC2154 if [ \u0026#34;$gt_ref\u0026#34; != \u0026#34;$REPO_REF\u0026#34; ]; then webhook_reject \u0026#34;Wrong repository ref: \u0026#39;$gt_ref\u0026#39;\u0026#34; fi # Accept the file # shellcheck disable=SC2154 webhook_accept \u0026#34;Processing \u0026#39;$gt_repo_name\u0026#39;\u0026#34; # Update the checkout ret=\u0026#34;0\u0026#34; git fetch \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then webhook_troubled \u0026#34;Repository fetch failed\u0026#34; mail_failure fi # shellcheck disable=SC2154 git checkout \u0026#34;$gt_after\u0026#34; \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then webhook_troubled \u0026#34;Repository checkout failed\u0026#34; mail_failure fi # Remove the build dir if present if [ -d \u0026#34;$PUBLIC_DIR\u0026#34; ]; then rm -rf \u0026#34;$PUBLIC_DIR\u0026#34; fi # Build site docker compose run hugo -- \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; # go back to the main branch git switch main \u0026amp;\u0026amp; git pull # Fail if public dir was missing if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ] || [ ! -d \u0026#34;$PUBLIC_DIR\u0026#34; ]; then webhook_troubled \u0026#34;Site build failed\u0026#34; mail_failure fi # Remove old public_html copies webhook_log \u0026#39;Removing old site versions, if present\u0026#39; find $NGINX_BASE_DIR -mindepth 1 -maxdepth 1 -name \u0026#39;public_html-*\u0026#39; -type d \\ -exec rm -rf {} \\; \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then webhook_troubled \u0026#34;Removal of old site versions failed\u0026#34; mail_failure fi # Switch site directory TS=\u0026#34;$(date +%Y%m%d-%H%M%S)\u0026#34; if [ -d \u0026#34;$PUBLIC_HTML_DIR\u0026#34; ]; then webhook_log \u0026#34;Moving \u0026#39;$PUBLIC_HTML_DIR\u0026#39; to \u0026#39;$PUBLIC_HTML_DIR-$TS\u0026#39;\u0026#34; mv \u0026#34;$PUBLIC_HTML_DIR\u0026#34; \u0026#34;$PUBLIC_HTML_DIR-$TS\u0026#34; \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; fi if [ \u0026#34;$ret\u0026#34; -eq \u0026#34;0\u0026#34; ]; then webhook_log \u0026#34;Moving \u0026#39;$PUBLIC_DIR\u0026#39; to \u0026#39;$PUBLIC_HTML_DIR\u0026#39;\u0026#34; mv \u0026#34;$PUBLIC_DIR\u0026#34; \u0026#34;$PUBLIC_HTML_DIR\u0026#34; \u0026gt;\u0026gt;\u0026#34;$WEBHOOK_LOGFILE_PATH\u0026#34; 2\u0026gt;\u0026amp;1 || ret=\u0026#34;$?\u0026#34; fi if [ \u0026#34;$ret\u0026#34; -ne \u0026#34;0\u0026#34; ]; then webhook_troubled \u0026#34;Site switch failed\u0026#34; mail_failure else webhook_deployed \u0026#34;Site deployed successfully\u0026#34; mail_success fi # ---- # vim: ts=2:sw=2:et:ai:sts=2 ","permalink":"https://blogops.mixinet.net/posts/new_blog_config/","summary":"\u003cp\u003eAs promised, on this post I’m going to explain how I’ve configured this blog\nusing \u003ccode\u003ehugo\u003c/code\u003e, \u003ccode\u003easciidoctor\u003c/code\u003e and the \u003ccode\u003epapermod\u003c/code\u003e theme, how I publish it using\n\u003ccode\u003enginx\u003c/code\u003e, how I’ve integrated the \u003ccode\u003eremark42\u003c/code\u003e comment system and how I’ve\nautomated its publication using \u003ccode\u003eforgejo\u003c/code\u003e and \u003ccode\u003ejson2file-go\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eIt is a long post, but I hope that at least parts of it can be interesting for\nsome, feel free to ignore it if that is not your case …​ :wink:\u003c/p\u003e\n\u003csection class=\"doc-section level-1\"\u003e\u003ch2 id=\"_hugo_configuration\"\u003eHugo Configuration\u003c/h2\u003e\u003csection class=\"doc-section level-2\"\u003e\u003ch3 id=\"_theme_settings\"\u003eTheme settings\u003c/h3\u003e\u003cp\u003eThe site is using the \u003ca href=\"https://git.io/hugopapermod\" target=\"_blank\" rel=\"noopener\"\u003ePaperMod\u003c/a\u003e theme and as I’m\nusing \u003ca href=\"https://asciidoctor.org\" target=\"_blank\" rel=\"noopener\"\u003easciidoctor\u003c/a\u003e to publish my content I’ve adjusted\nthe settings to improve how things are shown with it.\u003c/p\u003e\n\u003cp\u003eThe current \u003ccode\u003econfig.yml\u003c/code\u003e file is the one shown below (probably some of the\nsettings are not required nor being used right now, but I’m including the\ncurrent file, so this post will have always the latest version of it):\u003c/p\u003e\n\u003cfigure class=\"listing-block\"\u003e\u003cfigcaption\u003econfig.yml\u003c/figcaption\u003e\n\u003cpre class=\"rouge highlight\"\u003e\u003ccode data-lang=\"yaml\"\u003e\u003cspan class=\"na\"\u003ebaseURL\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ehttps://blogops.mixinet.net/\u003c/span\u003e\n\u003cspan class=\"na\"\u003etitle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eMixinet BlogOps\u003c/span\u003e\n\u003cspan class=\"na\"\u003eCopyright\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e|\u003c/span\u003e\n  \u003cspan class=\"s\"\u003e\u0026amp;copy; 2022-2025 Sergio Talens-Oliag.\u003c/span\u003e\n\u003cspan class=\"na\"\u003epagination\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003epagerSize\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e5\u003c/span\u003e\n\u003cspan class=\"na\"\u003etheme\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ePaperMod\u003c/span\u003e\n\u003cspan class=\"na\"\u003edestination\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003epublic/\u003c/span\u003e\n\u003cspan class=\"na\"\u003eenableInlineShortcodes\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003eenableRobotsTXT\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003ebuildDrafts\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n\u003cspan class=\"na\"\u003ebuildFuture\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n\u003cspan class=\"na\"\u003ebuildExpired\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n\u003cspan class=\"na\"\u003eenableEmoji\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003epygmentsUseClasses\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003eminify\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edisableXML\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eminifyOutput\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003elanguages\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003een\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003elanguageName\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eEnglish\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edescription\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eMixinet\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003eBlogOps\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e-\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://blogops.mixinet.net/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eauthor\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eSergio\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003eTalens-Oliag\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eweight\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n    \u003cspan class=\"na\"\u003etitle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eMixinet BlogOps\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eparams\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n      \u003cspan class=\"na\"\u003ehomeInfoParams\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n        \u003cspan class=\"na\"\u003eTitle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eSergio\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003eTalens-Oliag\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003eTechnical\u003c/span\u003e\u003cspan class=\"nv\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003eBlog\u0026#34;\u003c/span\u003e\n        \u003cspan class=\"na\"\u003eContent\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e\u0026gt;\u003c/span\u003e\n          \u003cspan class=\"s\"\u003e![Mixinet BlogOps](/images/mixinet-blogops.png)\u003c/span\u003e\n    \u003cspan class=\"na\"\u003etaxonomies\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n      \u003cspan class=\"na\"\u003ecategory\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ecategories\u003c/span\u003e\n      \u003cspan class=\"na\"\u003etag\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003etags\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eseries\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eseries\u003c/span\u003e\n    \u003cspan class=\"na\"\u003emenu\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n      \u003cspan class=\"na\"\u003emain\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n        \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eArchive\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003earchives\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eweight\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e5\u003c/span\u003e\n        \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eCategories\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ecategories/\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eweight\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e10\u003c/span\u003e\n        \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eTags\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003etags/\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eweight\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e10\u003c/span\u003e\n        \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eSearch\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003esearch/\u003c/span\u003e\n          \u003cspan class=\"na\"\u003eweight\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e15\u003c/span\u003e\n\u003cspan class=\"na\"\u003eoutputs\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003ehome\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s\"\u003eHTML\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s\"\u003eRSS\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s\"\u003eJSON\u003c/span\u003e\n\u003cspan class=\"na\"\u003eparams\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eauthor\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eSergio Talens-Oliag\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eenv\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eproduction\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edefaultTheme\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003elight\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edisableThemeToggle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowShareButtons\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowReadingTime\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edisableSpecial1stPost\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edisableHLJS\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003edisplayFullLangName\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowPostNavLinks\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowBreadCrumbs\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowCodeCopyButtons\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowRssButtonInSectionTermList\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowFullTextinRSS\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eShowToc\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eTocOpen\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n  \u003cspan class=\"na\"\u003ecomments\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eremark42SiteID\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eblogops\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eremark42Url\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://blogops.mixinet.net/remark42\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eprofileMode\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eenabled\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003etitle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eSergio Talens-Oliag Technical Blog\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eimageUrl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/images/mixinet-blogops.png\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eimageTitle\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eMixinet BlogOps\u003c/span\u003e\n    \u003cspan class=\"na\"\u003ebuttons\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eArchives\u003c/span\u003e\n        \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003earchives\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eCategories\u003c/span\u003e\n        \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ecategories\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eTags\u003c/span\u003e\n        \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003etags\u003c/span\u003e\n  \u003cspan class=\"na\"\u003esocial\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003esocialIcons\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eCV\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://www.uv.es/~sto/cv/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eDebian\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://people.debian.org/~sto/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eGitHub\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://github.com/sto/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eGitLab\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://gitlab.com/stalens/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eLinkedin\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003ehttps://www.linkedin.com/in/sergio-talens-oliag/\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"na\"\u003ename\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eRSS\u003c/span\u003e\n      \u003cspan class=\"na\"\u003eurl\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003eindex.xml\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eassets\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisableHLJS\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n    \u003cspan class=\"na\"\u003efavicon\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/favicon.ico\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003efavicon16x16\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e  \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/favicon-16x16.png\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003efavicon32x32\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e  \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/favicon-32x32.png\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eapple_touch_icon\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e  \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/apple-touch-icon.png\u0026#34;\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esafari_pinned_tab\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e  \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003e/safari-pinned-tab.svg\u0026#34;\u003c/span\u003e\n  \u003cspan class=\"na\"\u003efuseOpts\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eisCaseSensitive\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eshouldSort\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n    \u003cspan class=\"na\"\u003elocation\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e0\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edistance\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e1000\u003c/span\u003e\n    \u003cspan class=\"na\"\u003ethreshold\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e0.4\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eminMatchCharLength\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"m\"\u003e0\u003c/span\u003e\n    \u003cspan class=\"na\"\u003ekeys\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003etitle\u0026#34;\u003c/span\u003e\u003cspan class=\"pi\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003epermalink\u0026#34;\u003c/span\u003e\u003cspan class=\"pi\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003esummary\u0026#34;\u003c/span\u003e\u003cspan class=\"pi\"\u003e,\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"s\"\u003econtent\u0026#34;\u003c/span\u003e\u003cspan class=\"pi\"\u003e]\u003c/span\u003e\n\u003cspan class=\"na\"\u003emarkup\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003easciidocExt\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eattributes\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e{\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003eallow-uri-read\u0026#39;\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"nv\"\u003etrue\u003c/span\u003e\u003cspan class=\"pi\"\u003e}\u003c/span\u003e\n    \u003cspan class=\"na\"\u003ebackend\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003ehtml5s\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eextensions\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"pi\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003easciidoctor-html5s\u0026#39;\u003c/span\u003e\u003cspan class=\"pi\"\u003e,\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003easciidoctor-diagram\u0026#39;\u003c/span\u003e\u003cspan class=\"pi\"\u003e]\u003c/span\u003e\n    \u003cspan class=\"na\"\u003efailureLevel\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003efatal\u003c/span\u003e\n    \u003cspan class=\"na\"\u003enoHeaderOrFooter\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n    \u003cspan class=\"na\"\u003epreserveTOC\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esafeMode\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003eunsafe\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esectionNumbers\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003etrace\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003everbose\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eworkingFolderCurrent\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003eprivacy\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003evimeo\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisabled\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esimple\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003ex\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisabled\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eenableDNT\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esimple\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003einstagram\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisabled\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003esimple\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eyoutube\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisabled\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eprivacyEnhanced\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003eservices\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003einstagram\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisableInlineCSS\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n  \u003cspan class=\"na\"\u003ex\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003edisableInlineCSS\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003cspan class=\"na\"\u003esecurity\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n  \u003cspan class=\"na\"\u003eexec\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n    \u003cspan class=\"na\"\u003eallow\u003c/span\u003e\u003cspan class=\"pi\"\u003e:\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003e^asciidoctor$\u0026#39;\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003e^dart-sass-embedded$\u0026#39;\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003e^go$\u0026#39;\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003e^npx$\u0026#39;\u003c/span\u003e\n      \u003cspan class=\"pi\"\u003e-\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"s\"\u003e^postcss$\u0026#39;\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/figure\u003e\n\u003cp\u003eSome notes about the settings:\u003c/p\u003e\n\u003cdiv class=\"ulist\"\u003e\u003cul\u003e\u003cli\u003e\u003ccode\u003edisableHLJS\u003c/code\u003e and \u003ccode\u003eassets.disableHLJS\u003c/code\u003e are set to \u003ccode\u003etrue\u003c/code\u003e; we plan to use\n\u003ccode\u003erouge\u003c/code\u003e on \u003ccode\u003eadoc\u003c/code\u003e and the inclusion of the \u003ccode\u003ehljs\u003c/code\u003e assets adds \u003cem\u003estyles\u003c/em\u003e that\ncollide with the ones used by \u003ccode\u003erouge\u003c/code\u003e.\u003c/li\u003e\u003cli\u003e\u003ccode\u003eShowToc\u003c/code\u003e is set to \u003ccode\u003etrue\u003c/code\u003e and the \u003ccode\u003eTocOpen\u003c/code\u003e setting is set to \u003ccode\u003efalse\u003c/code\u003e to\nmake the ToC appear collapsed initially. My plan was to use the \u003ccode\u003easciidoctor\u003c/code\u003e\nToC, but after trying I believe that the theme one looks nice and I don’t\nneed to adjust styles, although it has some issues with the \u003ccode\u003ehtml5s\u003c/code\u003e\nprocessor (the admonition titles use \u003ccode\u003e\u0026lt;h6\u0026gt;\u003c/code\u003e and they are shown on the ToC,\nwhich is weird), to fix it I’ve copied the \u003ccode\u003elayouts/partial/toc.html\u003c/code\u003e to my\nsite repository and replaced the range of headings to end at \u003ccode\u003e5\u003c/code\u003e instead of\n\u003ccode\u003e6\u003c/code\u003e (in fact \u003ccode\u003e5\u003c/code\u003e still seems a lot, but as I don’t think I’ll use that heading\nlevel on the posts it doesn’t really matter).\u003c/li\u003e\u003cli\u003e\u003ccode\u003eparams.profileMode\u003c/code\u003e values are adjusted, but for now I’ve left it disabled\nsetting \u003ccode\u003eparams.profileMode.enabled\u003c/code\u003e to \u003ccode\u003efalse\u003c/code\u003e and I’ve set the\n\u003ccode\u003ehomeInfoParams\u003c/code\u003e to show more or less the same content with the latest posts\nunder it (I’ve added some styles to my \u003ccode\u003ecustom.css\u003c/code\u003e style sheet to center the\ntext and image of the first post to match the look and feel of the profile).\u003c/li\u003e\u003cli\u003eOn the \u003ccode\u003easciidocExt\u003c/code\u003e section I’ve adjusted the \u003ccode\u003ebackend\u003c/code\u003e to use \u003ccode\u003ehtml5s\u003c/code\u003e,\nI’ve added the \u003ccode\u003easciidoctor-html5s\u003c/code\u003e and \u003ccode\u003easciidoctor-diagram\u003c/code\u003e extensions to\n\u003ccode\u003easciidoctor\u003c/code\u003e and adjusted the \u003ccode\u003eworkingFolderCurrent\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e to make\n\u003ccode\u003easciidoctor-diagram\u003c/code\u003e work right (haven’t tested it yet).\u003c/li\u003e\u003c/ul\u003e\u003c/div\u003e","title":"New Blog Config"},{"content":"Welcome to my new Blog for Technical Stuff.\nFor a long time I was planning to start publishing technical articles again but to do it I wanted to replace my old blog based on ikiwiki by something more modern.\nI’ve used Jekyll with GitLab Pages to build the Intranet of the ITI and to generate internal documentation sites on Agile Content, but, as happened with ikiwiki, I felt that things were kind of slow and not as easy to maintain as I would like.\nSo on Kyso (the Company I work for right now) I switched to Hugo as the Static Site Generator (I still use GitLab Pages to automate the deployment, though), but the contents are written using the Markdown format, while my personal preference is the Asciidoc format.\nOne thing I liked about Jekyll was that it was possible to use Asciidoctor to generate the HTML simply by using the Jekyll Asciidoc plugin (I even configured my site to generate PDF documents from .adoc files using the Asciidoctor PDF converter) and, luckily for me, that is also possible with Hugo, so that is what I plan to use on this blog, in fact this post is written in .adoc.\nMy plan is to start publishing articles about things I’m working on to keep them documented for myself and maybe be useful to someone else.\nThe general intention is to write about Container Orchestration (mainly Kubernetes), CI/CD tools (currently I’m using GitLab CE for that), System Administration (with Debian GNU/Linux as my preferred OS) and that sort of things.\nMy next post will be about how I build, publish and update the Blog, but probably I will not finish it until next week, once the site is fully operational and the publishing system is tested.\nSpoiler Alert: This is a personal site, so I’m using Forgejo to host the code instead of GitLab.\nTo handle the deployment I’ve configured json2file-go to save the data sent by the hook calls and process it asynchronously using inotify-tools.\nWhen a new file is detected a script parses the JSON file using jq and builds and updates the site if appropriate.\n","permalink":"https://blogops.mixinet.net/posts/new_blog/","summary":"\u003cp\u003eWelcome to my new Blog for Technical Stuff.\u003c/p\u003e\n\u003cp\u003eFor a long time I was planning to start publishing technical articles again but\nto do it I wanted to replace my old \u003ca href=\"https://mixinet.net/~sto/blog/\" target=\"_blank\" rel=\"noopener\"\u003eblog\u003c/a\u003e\nbased on \u003ca href=\"https://ikiwiki.info/\" target=\"_blank\" rel=\"noopener\"\u003eikiwiki\u003c/a\u003e by something more modern.\u003c/p\u003e\n\u003cp\u003eI’ve used \u003ca href=\"https://jekyllrb.com/\" target=\"_blank\" rel=\"noopener\"\u003eJekyll\u003c/a\u003e with \u003ca href=\"https://about.gitlab.com\" target=\"_blank\" rel=\"noopener\"\u003eGitLab\u003c/a\u003e\n\u003ca href=\"https://docs.gitlab.com/ee/user/project/pages/\" target=\"_blank\" rel=\"noopener\"\u003ePages\u003c/a\u003e to build the Intranet of\nthe \u003ca href=\"https://iti.es/\" target=\"_blank\" rel=\"noopener\"\u003eITI\u003c/a\u003e and to generate internal documentation sites on\n\u003ca href=\"https://www.agilecontent.com/\" target=\"_blank\" rel=\"noopener\"\u003eAgile Content\u003c/a\u003e, but, as happened with \u003cstrong\u003eikiwiki\u003c/strong\u003e,\nI felt that things were kind of slow and not as easy to maintain as I would\nlike.\u003c/p\u003e\n\u003cp\u003eSo on \u003ca href=\"https://kyso.io/\" target=\"_blank\" rel=\"noopener\"\u003eKyso\u003c/a\u003e (the Company I work for right now) I switched to\n\u003ca href=\"https://gohugo.io/\" target=\"_blank\" rel=\"noopener\"\u003eHugo\u003c/a\u003e as the Static Site Generator (I still use \u003cstrong\u003eGitLab\nPages\u003c/strong\u003e to automate the deployment, though), but the contents are written using\nthe \u003ca href=\"https://daringfireball.net/projects/markdown/\" target=\"_blank\" rel=\"noopener\"\u003eMarkdown\u003c/a\u003e format, while my\npersonal preference is the \u003ca href=\"https://asciidoc.org/\" target=\"_blank\" rel=\"noopener\"\u003eAsciidoc\u003c/a\u003e format.\u003c/p\u003e\n\u003cp\u003eOne thing I liked about \u003cstrong\u003eJekyll\u003c/strong\u003e was that it was possible to use\n\u003ca href=\"https://asciidoctor.org/\" target=\"_blank\" rel=\"noopener\"\u003eAsciidoctor\u003c/a\u003e to generate the HTML simply by using the\n\u003ca href=\"https://github.com/asciidoctor/jekyll-asciidoc\" target=\"_blank\" rel=\"noopener\"\u003eJekyll Asciidoc\u003c/a\u003e plugin (I even\nconfigured my site to generate \u003cstrong\u003ePDF\u003c/strong\u003e documents from \u003cstrong\u003e.adoc\u003c/strong\u003e files using the\n\u003ca href=\"https://docs.asciidoctor.org/pdf-converter/latest/\" target=\"_blank\" rel=\"noopener\"\u003eAsciidoctor PDF converter\u003c/a\u003e)\nand, luckily for me, that is also possible with \u003cstrong\u003eHugo\u003c/strong\u003e, so that is what I plan\nto use on this blog, in fact this post is written in \u003cstrong\u003e.adoc\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eMy plan is to start publishing articles about things I’m working on to keep\nthem documented for myself and maybe be useful to someone else.\u003c/p\u003e\n\u003cp\u003eThe general intention is to write about Container Orchestration (mainly\nKubernetes), CI/CD tools (currently I’m using GitLab CE for that), System\nAdministration (with Debian GNU/Linux as my preferred OS) and that sort of\nthings.\u003c/p\u003e\n\u003cp\u003eMy next post will be about how I build, publish and update the Blog, but\nprobably I will not finish it until next week, once the site is fully\noperational and the publishing system is tested.\u003c/p\u003e\n\u003csection class=\"admonition-block caution\" role=\"doc-notice\"\u003e\u003ch6 class=\"block-title label-only\"\u003e\u003cspan class=\"title-label\"\u003eSpoiler Alert: \u003c/span\u003e\u003c/h6\u003e\u003cp\u003eThis is a personal site, so I’m using \u003ca href=\"https://forgejo.org/\" target=\"_blank\" rel=\"noopener\"\u003eForgejo\u003c/a\u003e to host the\ncode instead of \u003cstrong\u003eGitLab\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eTo handle the deployment I’ve configured\n\u003ca href=\"https://salsa.debian.org/sto/json2file-go\" target=\"_blank\" rel=\"noopener\"\u003ejson2file-go\u003c/a\u003e to save the data sent\nby the hook calls and process it asynchronously using\n\u003ca href=\"https://github.com/inotify-tools/inotify-tools\" target=\"_blank\" rel=\"noopener\"\u003einotify-tools\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWhen a new file is detected a script parses the JSON file using\n\u003ca href=\"https://stedolan.github.io/jq/\" target=\"_blank\" rel=\"noopener\"\u003ejq\u003c/a\u003e and builds and updates the site if\nappropriate.\u003c/p\u003e\u003c/section\u003e","title":"New Blog"}]