NotAShelf's Blog

Nix's Substituter List Is Not a Routing Table

Sun, 24 May 2026 00:00:00 GMT

Nix's substituter model is one of those designs that is almost right, but isn't exactly there. It is simple in itself: you list a few binary caches in nix.conf, the daemon walks them in order, and if the path you want is anywhere on the internet a build doesn't have to happen on your poor laptop from 2001. The binary cache system is often listed as a strength of NixOS , but it's actually a strength of Nix, and a bare minimum for something like NixOS to work for its users. Which is to say that it's actually perfectly fine until you have a large enough multi-cache setup for your configuration's dependencies, or in rarer cases, your projects dependencies. By which I mean multiple binary cache instances because you decided to fetch massive C++ and Rust projects from the internet that need to be built, on your system.

In the case of a rather complex setup, the first cache in your substituter list is almost always https://cache.nixos.org. It is fast, it is global, and it has the binary substitutions for your system to be built in a few minutes. It does not, however, have your overlay's packages. The second tends to be something like the nix-community cache, because you usually pull something really useful ¹ from the nix-community organization or just hope it has some overlay packages that you need in your obscure hardware setup. In less common cases you also add a third party project's Cachix (or a similar 3rd party) and occasionally, if you're technical (or dedicated) enough, your homelab's private cache. In such a setup, every narinfo lookup walks that rather large list. Every nix-shell -p hello becomes a serialized scan across four hosts on three continents because Nix has no concept of which substituter is most likely to answer for this path. It just asks them all, in the order you wrote them down, one after the other.

The Shape of the Problem

To explain why a proxy is the right answer, you---or rather, I---have to be honest about what Nix's substituter logic is:

A loop over substituters, in order.
For each: HEAD /<hash>.narinfo. If 200, fetch and use it. If 404, continue.
No concurrency. No latency tracking. No memory of which cache won last time you asked for this hash.

The substituter list is a preference, not a routing table. There is a priority field, but it is a static integer chosen at config time. It does not know that cache.nixos.org is 40 ms away on your home connection and 800 ms away from the office VPN. It does not know that your private cache is the only one that has the path. It does not know anything, because there is nothing to know. The daemon is stateless on every request. For a tool that prides itself on being a model of declarative purity, the network layer underneath is still static and request-local.

ncro, Briefly

ncro---Nix Cache Route Optimizer, pronounced Necro---is a small HTTP proxy that sits between nix-daemon and your substituters. It is about three thousand ² lines of clean, performant Rust code. It does three things:

On a narinfo lookup, it races all configured upstreams in parallel with HEAD and remembers which one won.
On a NAR fetch, it streams the body straight through to the client. No disk. No buffer. No NAR ever lives on the proxy.
It keeps a small, bounded SQLite table of route decisions so a restart doesn't force it to relearn the entire world.

That is the whole product. It is deliberately not another ncps, which mirrors caches to disk and gives you all the cache-invalidation grief that comes with mirroring caches to disk. ncro does not retain payload data once it is streamed through. ncro does, however, fully leverage its position as a proxy.

What's Actually In It

The interesting parts are the ones the architecture diagram (which you might or might not have paid attention to) doesn't show:

The race. ncro's router groups candidates by priority, then for each priority tier spawns a FuturesUnordered of HEAD requests and breaks on the first success. The tier loop is what lets you say "prefer my private cache, but only if it answers---otherwise fall through to cache.nixos.org" without writing any of that logic into Nix itself. There's a deadline pinned to the select loop so a single hung upstream can't stall the entire lookup. Failures are classified as not found, network error, and timeout because "every upstream returned 404" and "every upstream's TCP handshake died" deserve different answers to the client.
The cache, in two layers. A moka Cache sits in front of SQLite, with a 1024-entry capacity and TTL bound to the route TTL. SQLite underneath, with narinfo_bytes stored alongside the route so a hot path doesn't even need a second upstream fetch. Eviction is throttled to fire every hundred writes by abusing AtomicU64::fetch_add's pre-increment semantics. This is a detail that bit me in review because count % 100 == 0 fires on the first write, when the counter is zero. The fix was one character, but the impact was real: latency metrics were skewed until this edge case was corrected.
Health, with EMA. $L_t = \alpha R_t + (1 - \alpha) L_{t-1}$, $\alpha = 0.3$. The first sample bypasses the smoothing. Otherwise, the first probe permanently anchors to whatever junk was in ema_latency at startup. Consecutive failures bin the upstream into Active / Degraded / Down with multiplicative backoff (x1, x4, x10) so a dead cache stops costing you probe traffic. Sorting falls back to priority only when two upstreams are within 10% of each other, which is the only honest interpretation of "ties" when you're dealing with network latency that jitters by milliseconds.
Inflight dedup. If two clients ask for the same narinfo at the same time, only one race happens; the other waits on the mutex and reads the LRU on the way out. The cleanup path uses remove_if with Arc::ptr_eq because a naïve remove will happily delete someone else's goddamned Arc that landed in the slot between the time your guard read it and the time it dropped. This avoids a TOCTOU class bug in the dedup map.
Signatures. ed25519 via ed25519-dalek, public keys parsed from Nix's name:base64(key) format, the fingerprint reconstructed in the exact 1;StorePath;NarHash;NarSize;refs shape that nix store sign writes. If you configure a key for an upstream, narinfos that don't verify are rejected before they hit the cache. This is the part you cannot skip: a proxy that strips signature checks is a proxy that turns one compromised upstream into every machine pulling through you.

What I Did Not Build

There is no NAR cache. There is no mirror. There is no "warm-up" job. ncro will happily serve the same 200 MB closure ten times in a row by streaming it from upstream ten times, and that is the correct behaviour for a router. The optimization is in which upstream you ask, not in avoiding the upstream entirely. The moment you start storing NARs you are signing up for free-disk-space alerts, content-addressed-but-not-really cache poisoning, and a maintenance surface that has nothing to do with what you actually wanted, which was for nix build to stop spending 400 ms on DNS for a cache that never has the path. There is also no mesh by default. There is an optional gossip layer with signed UDP packets for the trusted-peer case, but it is off, and it should be off unless you have a clear trust and threat model for peer exchange. I plan to expand upon this in the future.

Was It Worth Writing?

Yes.

The proxy is small enough that I can keep the whole thing in my head, which I cannot say about most software I actually depend on. It solves one problem, the problem it set out to solve, and it does not try to grow into a content cache or a CI artifact store or a peer-to-peer Nix mesh---even though all three are tempting and each would complicate the design substantially. Nix's substituter logic could support this kind of behavior, but in practice it has remained a static ordered loop for years. A dedicated proxy is a practical way to add dynamic routing without patching the daemon itself.

Whether or not this interest you, I have really enjoyed working on ncro. Perhaps your needs are similar to mine, or it interests you too. In which case, why not give it a try?

Cough nh cough. It's useful, it's large, and it's annoying to build if you're building from the master branch because Rust is just like that. I can also list Hyprland in the things I fetch from its own flake and hope the cache has it working. What? I like my software fresh. ↩
A bit more than that now, but the tool itself is feature-complete and functional. Not sure if I'll stop tinkering with the tool, but the scope is well-defined by itself. Even if it does end up growing in size, it'll remain similar in codebase size and simplicity. This make ncro future-proof effortlessly. ↩

Why I think Go is a Terrible Language

Sun, 26 Apr 2026 00:00:00 GMT

I'll start with a disclaimer that I know will bother some of you. Maybe bother you a lot, or maybe make you laugh: this is not a screed from someone who doesn't understand Go. You wish it were, and frankly, I wish it was. Unfortunately for me, I have written Go in the past and I have also shipped Go to "prod" in the past. In fact, I continue to write Go when I'm too lazy or tired to care about correctness---for reasons we'll talk about shortly. It is also worth stating out right that while I am still in the process of converting to more... sane languages, some of my most critical infrastructure components have been Go for as long as I can remember. They are still Go, and they still hold.

This post has started as a Hedgedoc document I uploaded in response to questions asking me why I think Go is designed by people who understand language design very well, but refuse to follow established and beloved conventions. I am still behind this statement and throughout this post you will see exactly why I believe this. I have read much of the spec and the proposals. I have read, or at least tried to read, the team's stated reasoning and surrounding memos. The blog posts about why the things are the way they are and the such. Their defenses, arguments, and more.

What follows is a case. It is my case about why I think Go is a terrible* language. This is, put most simply, my attempt at a precise, specific, and deliberately uncharitable document where the language deserves what is coming. It is about why Go is a failure of a design philosophy dressed up and paraded around as pragmatism. The post itself is not to dictate whether you should use Go or not. I'll make it very clear that I franky could not care less about what you use. The post itself is to organize my thoughts, leave a structured and developed list of my arguments for those interested in why I don't like the language. This is also not for experts who are extremely comfortable with their language. My goal is not changing your mind, but making my case out in the open. This is also not a hate mail towards Go, but a strongly worded opinion essay. I intend to write something truly wholesome for Rust in the future, and one truly vile for Zig. For now, let's talk about Go and its design choices.

Failure by Design

The first thing I want to talk about, and perhaps the lowest hanging fruit that everyone reaches for, is the error handling patterns of Go. This is simply because it really is that bad. I think the designers fully understood exceptions, sum types, and structured error propagation. Quite sure they not only understand those patterns, but also see the individual beauty of it. They knew about Haskell's Either, about ML's type-safe exceptions and Java's checked exceptions which are flawed indeed but at least attempt correctness. Instead, Go enforces explicit error returns everywhere, even when the result is repetitive boilerplate that actively obscures control flow. Here, let me give you a better idea. A typical Go function doing three fallible operations would look something like this:

func process(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }

    data, err := io.ReadAll(f)
    if err != nil {
        return err
    }

    result, err := parse(data)
    if err != nil {
        return err
    }

    return store(result)
}

The actual logic (open, read, parse, and store) is four lines. FOUR. The error plumbing, however, is twelve. The signal-to-noise ratio is, if my math is correct, 3:1, and this is called explicit. Now compare it to Rust, where ? propagates errors with identical semantics but doesn't consume your screen free estate.

fn process(path: &str) -> Result<(), Error> {
    let data = fs::read_to_string(path)?;
    let result = parse(&data)?;
    store(result)
}

How many lines is that? 1, 2, 3... Oh yeah, FOUR.

The crucial difference isn't syntax sugar my dear reader. In Rust, Result<T, E> is a real type. E is a real type parameter. You can write generic combinators over it, map_err, and_then, unwrap_or_else. The error type is part of the function's contract in the type system, not a convention you hold in your head.

In Go, error is a single-method interface. Any type with Error() string satisfies it, which means there is no structured hierarchy, no enforcement and absolutely no static knowledge of what errors a function can actually produce. Somewhere around Go 1.12 or 1.13, the Go team added errors.Is and errors.As as a post-hoc attempt to recover structure from this mess. This is simply an admission of guilt. They rejected typed errors, and then recreated a weaker, ad-hoc version of the same concept without proper exhaustiveness, without enforced structure and without syntax support.

io.EOF is the clearest and perhaps the best illustration of where the model collapses. It is a sentinel value. A global error variable callers check with ==. Entire packages special-case it. In a language where Error is a proper enum, EOF would be just another variant the compiler forces you to handle but in Go it is a convention held together by documentation and (allegedly) discipline. It is also occasionally violated in ways that take hours to debug. The team eventually ran a formal design process. The proposal that followed introduced things like check/handle, the try builtin, a ? operator and a bit more. As you'd expect, all of those were rejected. Eventually, sometime in 2025, the Go blog has proudly declared that there will be no further syntax-level attempts. Now your boilerplate is intentional, and permanent.

The even lower hanging fruit---so low that you don't even have to lift your arm--- is nil, which is its own disaster. Go has nil, and Go has interfaces. Those two interact in a way that is genuinely (and verifiably) broken. An interface value is internally a two-word structure. That means it consists of a type pointer, and a data pointer. A nil interface means both are nil. A German walks into a bar, end of the joke.

Well not really, but a nil pointer to a concrete type type assigned to an interface variable has a non-nil type pointer and a nil data pointer, so the interface is no nil. Confused? Here, let me illustrate. This produces something like:

type MyError struct{}
func (e *MyError) Error() string { return "oops" }

func getError() error {
    var err *MyError = nil
    return err
}

func main() {
    if err := getError(); err != nil {
        fmt.Println("non-nil error") // this prints
    }
}

The function returned no error. The caller receives a non-nil error. Mind you, this is not even some obscure corner case. It regularly shows up in real, "production" codebases and the idiomatic fix is someone snarkily telling you "remember not to do this" enough times so that it never leaves your head how much you want to punch that person. The actual fix, on the other hand, is Option<T>, which represents absence in the type system rather than relying on a zero value that just happens to carry type metadata. Rust's Option<T> makes it impossible to use a value that might be absent without explicitly handling both cases. There is no typed None that looks like Some. The compiler catches it before your users do.

You'd expect nil to be at least coherent given how dominant it is, but no. nil is not even one coherent thing. You can call methods on nil pointers if the method doesn't reference the receiver. A nil slice has len 0, and is safe to range over. A nil map panics on write but not on read. A nil channel blocks forever on receive or send. A closed channel panics on send.

That was a lot of nils in one sentence. Phew. Anywho, the point is that those are not rules derived from a single principle, but a collection of special cases to memorize, and getting any one of them wrong produces a runtime panic with no compile-time indication that anything was wrong. And I think the type system's failures run much deeper than just nil. Go has no sum types, no exhaustive matching, no non-nullable types, no const generics, no immutability by default. There are basic tools for making illegal states unrepresentable, and the designers knew this. They then explicitly rejected them in favor of runtime discipline. As much as I like Discipline, that is no substitute for proper language design.

The absence of sum types is the most damaging. In Rust you typically write:

enum ParseResult {
    Integer(i64),
    Float(f64),
    Text(String),
    Unknown,
}

The compiler forces you to handle every variant. Add a new one later, and every match expression that doesn't cover it becomes a compile error. In Go you write an interface, use a type switch, and every unhandled case (silently) falls through unless you manually add a default that panics. This is a convention, not a guarantee. Nothing in the Go language actually stops you from being unkind to your future self. Nothing stops you from adding a new implementing type and never updating the thirty switch statements scattered across your codebase. You, if ever, find out at runtime. Rust, Haskell, Swift, TypeScript with discriminated unions---they all surface this as a compile error. Go surfaces it as a silent no-op, or a wrong result propagating for hours before anyone notices.

On the same cursed note, non-nullable types don't exist. Every pointer, interface, slice, map, channel or function can and will be nil. You cannot declare a variable that the compiler guarantees is never nil.

In Kotlin, String is non-nullable and String? is nullable. This is enforced at every callsite. In rust, you use Option<T> and the match handles the absent case. In go, however, the best you can do is put a comment saying "THIS PARAMETER MUST NOT BE nil" and hope that someone does not pass it within the next 3 months, leading the runtime panic. Similarly, immutability is just as absent: there is no const for struct fields, no readonly, no way to declare a value passed to a function must not be mutated. Go has const for primitive literals and that's it. Everything else is just... mutable. The alternative to immutability guarantees is "just don't mutate things!" is once again convention, or advice. There is no guarantee, and for a language catering to beginners this is too heavy of a footgun to hand loaded to the users.

The type system's gaps extend into territory the standard critique rarely reaches. encoding/json unmarshals all numbers into float64 when the target is interface{}, silently losing precision on integers exceeding the 53-bit IEEE 754 mantissa. The integer 9007199254740993 becomes 9.007199254740992e+15. The fix is Decoder.UseNumber(), but the default path discards data with no warning. A nil slice marshals to null while an empty slice marshals to []; a nil map marshals to null while an empty map marshals to {}. The distinction between "this field is absent" and "this field is present but empty" is semantically meaningful in many APIs, yet the language provides no way to declare which meaning a given value carries. The differentiation depends on a runtime property indistinguishable from emptiness in every other context.

panic(nil) is indistinguishable from no panic at all. recover() returns nil when the panic value was nil, so a deferred recovery cannot tell whether it caught a panic(nil) or nothing happened. recover() itself only works when called directly inside a deferred function---wrapping it in a helper silently breaks it, with no compiler warning and no lint to catch the error. Comparing two interface{} values with == panics at runtime if the underlying types are incomparable, such as maps or slices, because the type information is erased at the comparison site. The compiler cannot protect you because the concrete type is gone.

Until Go 1.22 in 2024, loop variables were reused across iterations:

for _, item := range items {
    go func() {
        process(item) // every goroutine sees the last item
    }()
}

This was a known problem from the Go 1.0 proposals and fixed only after twelve years. The language team initially defended the behavior as consistent with the spec. The spec was wrong. Twelve years of production incidents before a language-level fix is institutional reluctance to acknowledge a design decision---reusing a loop variable for efficiency---produced a persistent, well-understood bug surface.

Let's Talk Generics

The generics situation deserves its own section, and its own extended treatment because it is such a perfect illustration of the team's priorities. Go 1.0 shipped sometime in 2012 without generics. Which is fine. Then, for almost a decade, the team explicitly argued that code duplication is PERFECTLY FINE and was preferable to abstraction, actually. The result was interface{} abuse everywhere, reflection-based helpers, and the unfortunate go generate pipelines that turned code generation into a first-class workflow. The library itself either duplicated implementations for each concrete type, or punted to interface{} and runtime assertions. If you were a user, you were told to copy-paste sort functions.

This is supposed to be pragmatic, in case you haven't noticed.

I'm definitely not the first person to bash Go over generics, but I still want to give you its history proper. Generics finally arrived in Go 1.18 sometime in 2022, and they were deliberately constrained. You cannot define a method on a non-generic type that introduces its own parameters, which rules out a wide class of useful designs. You cannot write, for example, func (r *Repository) FindAll[T Entity]() ([]T, error).

Your workaround must be either making the entire struct generic, which may be structurally wrong, or moving to a free function, which discards the method set. Neither is satisfactory, and Rust has no such restriction. Type inference is local and frequently fails on anything involving interfaces or composite types, in cases where any reasonable inference engine would determine the parameters unambiguously. You cannot access a field through a type parameter even if every type in the constraint set shares that field. You must define a method instead, because the type system cannot reason about struct layout through constraints. There is no specialization: you cannot provide a more efficient implementation for a specific concrete type. Go 1.25 removed the core type restriction from type sets, a genuine improvement, but the rest of these limitations remain.

Structural typing compounds the type system's weaknesses in its own particular way. Any type with the right method set automatically satisfies an interface, with no declaration of intent. When a type accidentally satisfies an interface, it becomes part of an implicit contract its author never intended to make. When you change a method signature, you may silently break callers who depended on that accidental relationship, with no error at the definition site; only at the use site, possibly in a different package. The canonical workaround is:

var _ io.Writer = (*MyWriter)(nil)

A compile-time assertion that produces a type error if *MyWriter doesn't implement io.Writer.

Think about what that means: the language leaves intent so implicit that developers invented a hack: assign a nil pointer to a blank identifier with an explicit type. This is just to recover the ability to state intent. That this hack is idiomatic is more damning than any external criticism. Rust uses explicit impl Trait for Type declarations. The intent is in the code. You cannot accidentally implement a trait. If you change a trait's definition, the compiler tells you exactly which impl blocks need updating. The contract is in the source, not inferred from the method set.

Now, a common counterargument is the consumer-side interface pattern: you define a narrow interface at the call site with only the methods you need, the compiler verifies the caller provides something that fits, and changing the interface surfaces every call site that needs updating. On its own terms this is reasonable. For protocol handlers and I/O pipelines---the domains Go was designed for---accepting "anything that has a Read method" is genuinely cleaner than requiring every type to declare its lineage. The problem is that this pattern coexists with producer-side interfaces defined once and consumed everywhere: json.Marshaler, fmt.Stringer, database/sql.Scanner, encoding.TextMarshaler. These carry behavioral contracts that are not encoded in the type system. If you add a MarshalJSON() ([]byte, error) method to a struct for internal logging purposes, json.Marshal will call it instead of using reflection. Your JSON output changes silently---not a compile error, not a test failure unless you happen to have coverage on that exact path. The encoding packages are built around structural detection: json.Marshal checks for Marshaler, then TextMarshaler, then falls through to reflection. fmt.Sprintf checks for Stringer. database/sql checks for Scanner. Add the right method name, and the behaviour of your program changes at a distance without your knowledge or consent.

This problem compounds over time through interface pollution. Because types satisfy interfaces implicitly, a type that grows methods through maintenance may begin satisfying interfaces it never previously matched. A data transfer object that acquires a Write([]byte) (int, error) method because someone embedded a buffer for convenience is now an io.Writer. Code that accepted an io.Writer will accept this DTO, and the mismatch between "this is a writable buffer" and "this is a data object that happens to have a Write method" becomes a design problem the type system cannot surface.

But The Concurrency

Go's concurrency model is one of its marketed features and perhaps the first to be mentioned in response when you criticize Go. It is also where the gap between appearance and reality is widest.

Goroutines are genuinely cheap.

The "share memory by communicating" slogan gestures at CSP and the channels may look principled, however, as if to really rub salt in the wound, the language provides none of the static guarantees that would make any of it actually robust. Data races are runtime errors at best: the race detector is opt-in via go test -race, disabled in production by default, and only catches races that occur in the specific test run you happen to execute.

Not to mention, Go has no ownership model. Any goroutine can read or write any shared variable at any time. The type system has no concept of thread safety whatsoever. Rust's type system makes data races impossible by construction: Send and Sync are automatically derived marker traits, Mutex<T> requires you to acquire the lock before you can touch T, Arc<T> requires T: Send before it compiles. Rust doesn't make data races hard to write, it simply makes them impossible to compile.

Channels in Go do not enforce ownership or linearity. After you send a value on a channel, you can still use it. Nothing in the language prevents this. You cannot encode "send exactly once, then close" either and you cannot ensure a channel is closed by exactly one goroutine. Goroutines have no parent and no lifecycle tied to any scope. A function can return while the goroutines it spawned are still running, still holding references to state that should have been released, possibly blocked forever on a channel that will never receive. Goroutine leaks are trivially easy to produce. The standard mitigation is context.Context, but passing it is optional, checking the cancellation signal is optional, and nothing enforces either. Kotlin coroutines have lexically scoped lifetimes built into the runtime. Go has documentation asking you to be careful.

sync.Mutex is not parameterized over the data it protects. You lock it and then you can access anything. The relationship between a mutex and the state it guards exists in comments. Rust's Mutex<T> wraps the data it protects: the only way to access T is through the guard you get by locking it. The invariant is in the type. In Go it is in the README.

defer is useful and also function-scoped rather than block-scoped, which makes it wrong for a significant class of resource management problems. Every deferred call runs when the enclosing function returns, not when the enclosing block exits. This is fundamentally different from RAII. In C++ and Rust, a destructor runs when an object goes out of scope---the end of an if body, a loop iteration, an arbitrary block. In Go:

func processItems(items []Item) error {
    for _, item := range items {
        f, err := os.Open(item.Path)
        if err != nil {
            return err
        }
        defer f.Close()
        process(f)
    }
    return nil
}

Every file opened in the loop stays open until processItems returns. The fix is extracting the loop body into an immediately-invoked anonymous function, a workaround that exists because the primitive isn't expressive enough for what you're doing. Rust's Drop triggers at block exit without any special syntax. The resource is released when the owning variable goes out of scope and that's the end of it.

Less discussed are the channel semantics themselves, which form a matrix of runtime behaviors with no unifying principle. A send on a nil channel blocks forever. A receive from a nil channel blocks forever. Closing a nil channel panics. Sending on a closed channel panics. Receiving from a closed channel returns the zero value immediately. Closing a closed channel panics. Six distinct runtime behaviors for three states of a single construct, every one of them a runtime event with no compile-time guard. The language that prides itself on simplicity forces you to memorize this table.

time.After is another trapdoor. Used in a select loop it allocates a new timer on every iteration, and the timer is not garbage collected until it fires. Under load, the number of pending timers grows without bound. The fix is time.NewTimer and explicit Stop(), but the library offers the footgun as the shorter path and hopes the developer reads the caveat in the documentation. Likewise, sync.WaitGroup is a raw counter with Add and Done and no static enforcement that the two balance. Call Done one extra time across a goroutine boundary and the program panics with sync: negative WaitGroup counter, a message that tells you the counter went negative but not which goroutine drove it there. Rust's WaitGroup does not exist because std::sync::Arc and scoped threads make the pattern unnecessary. When you need it, Crossbeam provides one with static drop guards. Go gives you a volatile integer.

net/http ships with ServeHTTP(ResponseWriter, *Request) returning nothing. Every error path inside every handler must be handled inline instead of propagated to middleware that converts errors to responses. The community has reinvented type HandlerWithError func(http.ResponseWriter, *http.Request) error in roughly every Go web framework ever written. The standard library chose a signature that every non-trivial user must wrap or replace to achieve basic error composition.

At Least It's Simple

I think the first and most trivial element of confusion will come from how Go programs will be structured. I find it to be the exact opposite, however, as Go has little to no coherent "packaging" conventions. This is to say, package-level variables and init() functions add another class of problems.

You see, initialization order within a single file follows declaration order. Across multiple files in the same package it is not guaranteed. The compiler tries to resolve the dependency graph but the spec's response to cross-file ordering subtleties is essentially "don't rely on it." init() functions run automatically, cannot return errors, cannot accept arguments, cannot be called or tested directly, and can have arbitrary side effects.

To illustrate this issue, I'd like to give you the database/sql driver as an example. You writeimport _ "github.com/lib/pq" and the blank import registers a database driver as a side effect inside an init() call. There is no explicit initialization, no error return at the callsite, no indication from the import alone that anything has happened. If registration fails you panic. If you forget the import you get a runtime error on the first database operation. A language that claims to value explicitness ships this as the idiomatic database driver pattern.

This is the first crack in the "simple" story: the language removes visible machinery, then smuggles the machinery back in as package-level side effects. The code looks smaller because the initialization path is no longer in the code you are reading. That is not simplicity. That is hiding the causal chain.

On a similar note, the module system's history is also an embarrassment that the current state only partially redeems. While a beginners might not be interested in "hacking" the module system, they might as well be affected by its side-effects! Go shipped without dependency management. GOPATH had no versioning;go get fetched the latest commit with no lockfile and no way to specify which version you needed. The community produced Godep, Glide, dep, and govendor in succession, each incompatible and each dying in turn. Go modules, required in 1.16, fixed the core problems but introduced new ones. The major version suffix convention encodes version into the import path itself, so github.com/foo/bar/v2 is a different import path from github.com/foo/bar. Updating a direct dependency to a new major version means changing every import statement in your codebase that references it. No other major language ecosystem conflates import paths with version identity this way. The replace directive works for local development but does not propagate to consumers, so the development workflow diverges from the published module and requires manual management across multiple related modules.

This is not some unrelated ecosystem complaint stapled onto a language rant. It is the same design instinct again: avoid richer structure in the language and tooling, then push the resulting complexity into conventions, import paths, side effects, and developer memory. The beginner may not care about module mechanics on day one, but the project will care eventually, and then the bill arrives with interest.

Tooling

Of course, there's also tools not made by the community. Or, in other words, let's talk about official tooling.

As anyone who has ever used Go will tell you, and as you might have noticed throughout the post, Go has a vast first-party tooling. To its credit, I think the tooling is good in a way that doesn't get enough attention. gofmt is excellent, go test being built in is excellent. go vet caches a subset of common mistakes, but the typed-nil-as-interface bug described above is not caught by go vet. Goroutine leak detection is not built in. Correct mutex usage enforcement is not built in. Those require staticcheck, which is excellent but is third-party and requires explicit CI integration. The baseline static analysis for a Go project is significantly weaker than the language's correctness story requires. Build constraints until Go 1.17 were written as magic comments: // +build linux amd64 and not as syntax, not as a first-class feature, as comments the toolchain parses by convention. I am rather biased here, but this is not what good and reliable design looks like. I draw the line at doc comments.

This matters because tooling is the usual escape hatch offered in Go's defense. The language does not encode the invariant, but the tool will catch it. Except the first-party tools do not catch enough of the failures the language makes easy. They format the code. They run the tests. They catch some obvious mistakes. They do not turn Go into a language with non-nullable types, exhaustive matching, scoped goroutine lifetimes, typed mutex guards, or structured error variants. The missing guarantees remain missing. The deeper tooling problem is that reflect and unsafe.Pointer paper over type system gaps throughout the ecosystem. The standard library's encoding packages use reflection to serialize arbitrary types at runtime, which means structural mismatches are runtime errors. Rust's serde (which is not in stdlib) does the same work through procedural macros with full type safety at compile time. The comparison is straightforward and unflattering.

At this point the boundary between "tooling problem" and "language problem" stops being meaningful. Reflection, escape analysis output, race detection, and profiling are not external aids; they are compensatory mechanisms for properties the language does not or cannot express. Once you rely on them, you are no longer reasoning about your program purely through its types or its syntax. You are reasoning about compiler behavior, runtime behavior, and tooling diagnostics as part of the language. Those are not good cornerstones for a language aiming to be simple and powerful.

The memory model was tightened in Go 1.19 and is now more precisely specified than it was. Without an ownership model this doesn't change the practical situation much though, violations are catchable by the race detector if they happen to occur during a test run, and production is where you find out otherwise. Whether a value escapes to the heap is determined by the compiler's escape analysis, which is a heuristic, not a contract. You cannot declare that a value must stay on the stack or specify allocation strategy for a type. For most programs this is an acceptable tradeoff. For latency-sensitive systems it is not, and the debugging workflow is reading escape analysis output and profiling. There is no language-level handle on it. Go's GC has improved dramatically and achieves sub-millisecond pause times in common workloads, but stop-the-world pauses still occur, and under heavy allocation pressure or with large heaps the pause behavior becomes less predictable. For trading systems, real-time control, audio pipelines, or anything with hard latency requirements, a GC is disqualifying regardless of how good it is, because "very short pause" and "no pause" are not the same thing. Rust has no GC. Stack allocation is the default. Heap allocation is explicit through Box<T>, Arc<T>, Vec<T>. You know where every allocation is because, well, you put it there.

This is, if anything, where the section should land: Go is simple only if "simple" means fewer visible concepts on the surface. Once the program has to explain initialization, dependency identity, static analysis, reflection, unsafe escape hatches, allocation behavior, and latency, the complexity has not disappeared. It has merely moved out of the type system and into the runtime, the toolchain, the build graph, and the operational culture around the codebase.

Closing Thoughts

None of these problems are isolated. That is the actual point. A production Go service at scale is functions returning (value, error) where error is an opaque interface with several undocumented concrete implementations, callers checking errors.Is against known sentinels and falling through silently on unanticipated variants, goroutines outliving the requests that spawned them and holding references to state that should have been released, a mutex somewhere protecting the wrong data because the association was in a comment that was accurate when written and hasn't been updated since the refactor, a nil interface that isn't nil propagating up three layers before producing a panic the stack trace makes difficult to attribute.

The same pattern repeats in the supposedly simple parts of the language: package initialization hidden in init(), blank imports used for side-effect registration, module identity encoded into import paths, build constraints once parsed from comments, reflection used where the type system cannot express the shape of the program, race detection delegated to an optional runtime tool, and allocation behavior exposed through compiler diagnostics rather than language guarantees. These are not separate grievances. They are the same grievance wearing different costumes.

Every one of these failure modes is preventable in languages that encode the relevant invariants in their type systems. Every one of them requires discipline and convention to avoid in Go, and discipline and conventions fail under time pressure, team growth, and the ordinary entropy of a large codebase.

The defense of Go is that all of this is manageable. With experienced teams, with good code review, with staticcheck in CI, and with the race detector in test runs and context used consistently, Go codebases can be safe and maintainable. This is true. The question is why the language demands that entire investment of infrastructure and discipline to achieve properties that Rust, Haskell, Kotlin, and Swift provide structurally. Go was designed to solve Google's specific operational problems: fast compilation, easy onboarding for programmers of widely varying experience, and readable code review at massive scale, just to name a few. And sure, these are real goals, held with clear intent. The original team understood exactly what they were leaving out. They made deliberate tradeoffs, consistently applied.

The problem, however, is that optimizing for easy onboarding means in this case (and many others) optimizing against correctness. A language you can learn in a week is a language that does not make wrong programs hard to write. The ceiling on what you can prove at compile time is low by design, and as codebases grow and teams turn over and systems become load-bearing infrastructure, that low ceiling costs you in ways that are slow, cumulative, and difficult to attribute directly to the language rather than to the specific code that happened to fail.

Rust is by no means a perfect language. Compile times are long enough to be a real workflow problem on large projects. The borrow checker's learning curve is steep and genuinely so---not artificially steep, because the concepts it enforces are non-trivial and require building a new mental model before the errors start making sense. Async Rust is complex in ways that affect real programs: ¹ the story around async in trait methods, dyn and impl Trait, and the borrow checker in async contexts is still rough in places. The ecosystem is younger and has gaps. These are real criticisms and they should be stated plainly. Rust is harder to learn than Go. That is true and it is not nothing.

But Rust's design decisions are coherent in a way Go's are not. Sum types are embraced rather than rejected. Nil is eliminated rather than worked around. Data races are impossible rather than detectable. Mutex invariants are encoded in types rather than documented in comments. Async task lifetimes are scoped rather than floated as untracked background work. The tradeoffs Rust makes push complexity into the compiler and the learning curve, not into production incidents. Errors you make in Rust surface at compile time. Errors you make in Go surface in production. This is not a philosophical preference. The error handling is the difference between a bug your toolchain catches in two seconds and a bug that pisses you off for one long afternoon or an entire week.

Similarly, Go is not a badly implemented language either. Well, not too badly implemented. I would go so far to say that the runtime is excellent and the standard library is coherent and well-documented. The toolchain is pretty fast in ways other languages treat as aspirational. These are real achievements and I think dismissing them would be dishonest. But a language is not just its runtime. It is the guarantees it gives you, the invariants it lets you express, the class of bugs it makes structurally impossible. By that measure---the one that matters most for building software that has to be correct, not just running---Go is deliberately, knowingly, and permanently underpowered. There's no escaping it. That was a choice, made by people who understood what they were choosing and why. It was a clear set of priorities, consistently applied, that produced a language optimized for a narrow set of operational concerns at the direct expense of correctness. The gap between what Go lets you build and what you can actually prove about what you built is not a bug in Go. It is Go.

And that gap exists for a reason. Go was designed for Google's specific operational constraints circa 2009--2012: a monorepo with two billion lines of code, tens of thousands of engineers with widely varying experience, a build graph where every second of compilation time multiplied across the organization. Fast compilation, mechanical readability, and low abstraction were not aesthetic choices. They were operational requirements. The language satisfies them. The problem is that nearly every Go user is not Google. They do not have Google's SRE culture absorbing the cost of runtime failures. They do not have the code review bandwidth that makes convention-based correctness feasible at scale. For them the tradeoffs Go made for Google become liabilities. The fast compilation matters less than the bugs the language fails to prevent. The onboarding speed matters less than the invariants you cannot encode. Go was designed for an organization whose scale is unique in the industry and marketed as a general-purpose language. The mismatch between the design target and the actual user base is the root cause of most of the frustrations catalogued here.

None of this means Go is useless, or that writing Go is malpractice. The runtime is excellent. The toolchain is fast enough to be aspirational. The standard library is coherent within the constraints of the language. Generics removed the worst of the interface{} abuse. The race detector catches real bugs. Go succeeds at what it was designed for: fast compilation, rapid onboarding, and readable code at massive scale. But what it was designed for is narrower than its user base assumes, and a language optimized for one organization's operational problems at the direct expense of correctness is---for most other users---a language that makes wrong programs easy to write. The ceiling was chosen knowingly. The question is whether you want to live under it, and whether you think the tradeoff of knowing how to deal with a language's quirks is worth not picking a language with considerably less quirks.

Real Rust async has never been tried, actually. ↩

The Nihilist's Guide to Cross-Compiling Dioxus for Windows

Sun, 01 Feb 2026 00:00:00 GMT

The Rust GUI ecosystem has matured into a landscape of impressive, specialized tools. If you need immediate-mode simplicity, you reach for egui; if you want an Elm-inspired architecture, you go with Iced; and for raw, GPU-accelerated performance, GPUI is pushing the boundaries of what’s possible.

Choosing Dioxus in this environment wasn't really about a lack of options, so to speak, but about leveraging a specific, battle-tested paradigm that I have been interested in for a while now. It is a framework that allows developers to bring the declarative "hooks and components" model and the entire CSS/Tailwind ecosystem into a native context, effectively bridging the gap between high-velocity web development and the efficiency of a Rust backend.

In the case you didn't know, I come from a frontend background. As such, I expected to be right at home when I started with Dioxus. I was not wrong.

Why Dioxus

Admittedly, Dioxus occupies a very odd niche at a superficial level. It is neither a traditional, retained-mode native GUI toolkit nor a thin immediate-mode layer over a GPU abstraction. It (unapologetically) embraces the mental model of modern frontend development: declarative components, unidirectional data flow, and side effects expressed explicitly through hooks.

The choice, of course, is not about novelty. The hooks and components paradigm has already been stress-tested at a planetary scale by the web. Its strengths and weaknesses are very well understood, its ergonomics have been refined through years of iteration, and its failure modes are similar. By importing that model wholesale into Rust, Dioxus allows you to reuse not just ideas, but instincts. At least if you've written for the web before. State lives where you expect it to live. Effects run when dependencies change. UI becomes a pure function of state, with impurity carefully fenced off.

Dioxus' website will brandish itself a little differently, and talk about various advantages that may or may not convince you. One of those advantages is the one-stack-for-all you gain with Rust and being able to compile for many targets but I'd like to state out loud the quiet part, which is velocity without chaos. Compared to egui, you trade immediate-mode simplicity for structural clarity once applications grow beyond a single window. Compared to Iced, you give up strict Elm-style purity in exchange for a model that tolerates real-world side effects without contortions. And compared to GPU-first experiments like GPUI, you accept a webview boundary in return for a mature layout engine, a ubiquitous styling language, and an ecosystem of tooling that already knows how to scale.

For someone coming from a frontend background, this is less a paradigm shift than a homecoming. CSS is not an afterthought. Accessibility semantics exist by default. Layout is expressive rather than imperative. Tailwind, for better or worse, works exactly the way you expect it to. The result is a framework that lets you spend your time thinking about application behavior instead of fighting your UI toolkit.

Dioxus is not the lowest-level option, nor the most ideologically pure. It is, however, a pragmatic synthesis: Rust’s safety and performance paired with a UI model that has already survived contact with reality.

Just as importantly, it is one of the few frameworks in the Rust ecosystem that treats cross-platform delivery as a first-class concern. The same component tree can target the web, Linux, macOS, and Windows with minimal structural divergence, which makes it possible to develop primarily on Linux without relegating other platforms to second-class status.

Building with Dioxus

Enter: Webview

The decision to use a system webview carries a hidden cost that becomes painfully apparent the moment you try to ship. Because Dioxus on Windows relies on WebView2, you are no longer just compiling Rust; you are orchestrating a complex interaction with proprietary Microsoft loaders and COM interfaces. For those of us who prefer the deterministic, reproducible world of a Linux-based Nix environment, this creates a significant engineering challenge. You cannot simply cargo build your way out of a cross-compilation task that requires official Windows SDK headers and proprietary DLLs.

To bridge this gap professionally, you have to bypass the standard cross-compilation shortcuts and target the MSVC (Microsoft Visual C++) toolchain directly from Linux. While MinGW is often the "default" for cross-compiling, it frequently struggles with the specific nuances of the WebView2 SDK. By using cargo-xwin, you can fetch the actual Windows CRT and SDK headers into a local cache, allowing you to treat Windows as a first-class citizen of your build pipeline.

The Nix Way

The process begins by teaching Nix how to handle the proprietary Microsoft artifacts. Since Nix sandboxes prevent build scripts from reaching out to the internet, you must define a fixed-output derivation that fetches the official WebView2 NuGet package and extracts the necessary .lib and .dll files. This ensures that your build remains reproducible and that your linker has exactly what it needs to satisfy the Windows-specific dependencies of the wry and tao crates. My derivation looks something like:

# webview.nix
{
  stdenv,
  fetchurl,
  unzip,
}:
stdenv.mkDerivation (finalAttrs: {
  pname = "webview2-sdk";
  version = "1.0.3650.58";

  src = fetchurl {
    url = "https://www.nuget.org/api/v2/package/Microsoft.Web.WebView2/${finalAttrs.version}";
    sha256 = "sha256-kRpHISjIKsi6oMSGwjNCzJ3W59xQ11TmdnJmQsoGXGA=";
  };

  nativeBuildInputs = [unzip];

  unpackPhase = ''
    unzip $src
  '';

  installPhase = ''
    runHook preInstall
    mkdir -p $out

    # Copy the x64 libraries (we can use x86 for 32-bit targets)
    install -Dm755 build/native/x64/WebView2Loader.dll.lib $out/WebView2Loader.lib
    install -Dm755 build/native/x64/WebView2Loader.dll $out/
    runHook postInstall
  '';

})

It could be better, but I have gotten sick of trying to get the correct URL and loading the WebView runtime in Wine (which we'll talk about shortly.)

Once the SDK is available, you have to perform what is essentially environment surgery within your Nix devShell. The standard Nix clang wrapper is designed for Unix-style purity and will often reject the Windows-specific flags that cargo-xwin and MSVC-compatible crates expect. To solve this, you must explicitly direct Cargo to use lld-link as the linker and clang-cl as the C compiler. This configuration ensures that even complex C dependencies, such as SQLite, are compiled with the correct Windows headers and linked without flavor-mismatch errors.

Furthermore, you must account for the fact that a compiled Windows executable is functionally useless if it cannot find its dependencies at runtime. Unlike Linux, where we can often rely on package managers or rpath, Windows expects the WebView2Loader.dll to be present in the same directory as the .exe if it cannot be loaded from the runtime. A robust pipeline automates this by using a build.rs script that detects the Nix environment and copies the DLL from the Nix store into the target directory every time a build succeeds.

// build.rs
fn main() {
    let target = std::env::var("TARGET").unwrap_or_default();
    if target.contains("windows-msvc") {
        if let Ok(sdk_path) = std::env::var("WEBVIEW2_BIN_PATH") {
            let dll = "WebView2Loader.dll";
            let src = std::path::PathBuf::from(&sdk_path).join(dll);
            let out_dir = std::path::PathBuf::from(std::env::var("OUT_DIR").unwrap());
            let dest = out_dir.ancestors().nth(3).unwrap().join(dll);
            if src.exists() {
                std::fs::copy(&src, &dest).expect("Failed to bundle WebView2Loader.dll");
            }
        }
        println!("cargo:rerun-if-env-changed=WEBVIEW2_BIN_PATH");
    }
}

The Final Devshell

I have crafted an intricate devshell that pulls in various tools, and sets up a modern Clang-based compiler and linker pipeline to handle my builds.

{
  pkgs,
  rust-bin,
  # rust-overlay params
  extraComponents ? [],
  extraTargets ? [],
}: let
  webview2-sdk = pkgs.callPackage ./packages/webview.nix {};
in
  pkgs.mkShell {
    name = "mercant-dev";
    packages = [
      pkgs.taplo # TOML formatter
      pkgs.lldb # debugger
      pkgs.rust-analyzer-unwrapped # LSP
      pkgs.llvm
      pkgs.libiconv

      # Additional Cargo Tooling
      pkgs.cargo-nextest
      pkgs.cargo-deny

      # Build tools
      # We use the rust-overlay to get the stable Rust toolchain for various targets.
      # This is not exactly necessary, but it allows for compiling for various targets
      # with the least amount of friction.
      (rust-bin.nightly.latest.default.override {
        extensions = ["rustfmt" "rust-analyzer" "clippy"] ++ extraComponents;
        targets =
          [
            "arm-unknown-linux-gnueabihf" # Android
            "wasm32-unknown-unknown" # web
            # Windows
            "x86_64-pc-windows-msvc"
            "x86_64-pc-windows-gnu"
          ]
          ++ extraTargets;
      })

      # Cross-compiling to Windows
      pkgs.pkgsCross.mingwW64.stdenv.cc
      pkgs.pkgsCross.mingwW64.buildPackages.gcc
      pkgs.cargo-xwin

      # Link with Clang & lld
      pkgs.clang
      pkgs.lld

      # Handy CLI for packaging Dioxus apps and such
      pkgs.dioxus-cli

      # Dioxus desktop dependencies (GTK/WebKit)
      pkgs.pkg-config
      pkgs.glib
      pkgs.gtk3
      pkgs.webkitgtk_4_1
      pkgs.libsoup_3
      pkgs.cairo
      pkgs.pango
      pkgs.gdk-pixbuf
      pkgs.atk
      pkgs.xdotool # provides libxdo
      pkgs.openssl
      pkgs.kdePackages.wayland
    ];

    env = let
      mcfgthread = pkgs.pkgsCross.mingwW64.windows.mcfgthreads;
    in {
      # Allow Cargo to use lld and clang properly
      LIBCLANG_PATH = "${pkgs.libclang.lib}/lib";
      RUSTFLAGS = "-C link-arg=-fuse-ld=lld";

      # Windows cross-comp
      WEBVIEW2_BIN_PATH = "${webview2-sdk}/lib";

      CARGO_TARGET_X86_64_PC_WINDOWS_GNU_RUSTFLAGS = "-L native=${mcfgthread}/lib";
      CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_RUSTFLAGS = "-L native=${webview2-sdk}/lib";
      CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER = "lld-link";

      CC_x86_64_pc_windows_gnu = "x86_64-w64-mingw32-gcc";
      CXX_x86_64_pc_windows_gnu = "x86_64-w64-mingw32-g++";
      CC_x86_64_pc_windows_msvc = "clang-cl";
      AR_x86_64_pc_windows_msvc = "llvm-lib";

      # 'cargo llvm-cov' reads these environment variables to find these
      # binaries, which are needed to run the tests.
      LLVM_COV = "${pkgs.llvm}/bin/llvm-cov";
      LLVM_PROFDATA = "${pkgs.llvm}/bin/llvm-profdata";

      # Runtime library path for GTK/WebKit/xdotool
      LD_LIBRARY_PATH = "${pkgs.lib.makeLibraryPath [
        pkgs.xdotool
        pkgs.gtk3
        pkgs.webkitgtk_4_1
        pkgs.glib
        pkgs.cairo
        pkgs.pango
        pkgs.gdk-pixbuf
        pkgs.atk
        pkgs.libsoup_3
        pkgs.openssl
        pkgs.kdePackages.wayland
      ]}";
    };
  }

Metadata and Identity

By the time you drop into your shell and execute cargo xwin build --release --target x86_64-pc-windows-msvc, you are no longer fighting the toolchain. You have built a deterministic machine that translates your Rust source code into a native, high-fidelity Windows binary. This approach respects both the strict requirements of the Windows OS and the reproducible philosophy of Nix, resulting in a distribution workflow that is as clean as the code it compiles. The WEBVIEW2_BIN_PATH variable comes from our devShell, which sets various environment variables for the tooling.

Wine Tasting

Before arriving at this setup, it is tempting to try to paper over the problem by simply running the Windows pieces under Wine. After all, if Dioxus ultimately depends on WebView2, and WebView2 depends on the Edge runtime, why not just install the runtime into a Wine prefix and let everything discover it implicitly?

That line of reasoning leads you to commands like:

wine MicrosoftEdgeWebView2RuntimeInstallerX86.exe /install

And yes, this does install the runtime. But it does not actually solve the underlying problem.

WebView2 is split across multiple layers: a runtime installed system-wide, a loader DLL that applications link against, and a set of COM interfaces that assume a Windows loader, registry, and activation model. Wine can approximate enough of this environment to allow some binaries to start, but it cannot make those assumptions disappear. From the perspective of a cross-compiling Linux toolchain, Wine is still an opaque, mutable black box. More importantly, Wine does nothing for the build itself. Your Rust code still needs headers, import libraries, and a linker that understands MSVC semantics. Installing the runtime under Wine gives you a way to run a finished executable for ad-hoc testing, but it does not help Cargo or rustc produce that executable in the first place. At best, it delays failure until runtime. At worst, it masks missing dependencies behind Wine-specific behavior that will not exist on a real Windows system.

This is why the Wine approach ultimately gets abandoned here. It trades a hard but explicit problem for a soft, implicit one. By pulling the SDK artifacts directly into Nix and targeting MSVC properly, you surface the real constraints early, make them reproducible, and stop relying on an emulation layer as an undocumented part of your build pipeline.

If you're lucky, the correct runtime will be installed out-of-the-box in your real Windows machine. You may or may not have the same luck for Wine. Something like winetricks corefonts windowscodecs might come in handy but I've ultimately stopped trying to get Wine to work, and booted into Windows on my work machine.

Conclusion

Cross-compiling a Dioxus application for Windows from a Nix-based Linux environment is not difficult in the sense of missing tools or undocumented hacks. It is difficult because it forces you to reconcile two fundamentally different worldviews. Windows assumes mutable global state, opaque SDKs, and proprietary loaders. Nix assumes hermetic builds, explicit inputs, and reproducibility as a first principle.

The approach outlined here does not attempt to smooth over that mismatch. Instead, it accepts Windows on its own terms by using the MSVC toolchain and official SDK artifacts, while forcing those artifacts into Nix’s deterministic model through fixed-output derivations and explicit environment wiring. cargo-xwin acts as the linchpin, turning what would otherwise be an ad-hoc Wine-based mess into a repeatable, cacheable build step.

The result is not just a working executable, but a pipeline you can reason about. Every DLL is accounted for. Every header comes from a known source. Every build either succeeds deterministically or fails loudly. That matters, especially when you are shipping a GUI application whose failure modes tend to surface at runtime rather than compile time.

In the end, this is less about Dioxus specifically and more about refusing false dichotomies. You do not have to choose between modern UI paradigms and native binaries. You do not have to choose between Windows support and a sane Linux-based workflow. With enough stubbornness and a willingness to engage with the toolchains honestly, you can have all of it, and you can make it reproducible. The nihilism, such as it is, lies in accepting Windows not as a system to be admired, but as a constraint to be modeled precisely and without illusion.

Be well.

2025 Wrapped & 2026 Wishlist

Wed, 31 Dec 2025 00:00:00 GMT

Howdy, it seems creating something with "wrapped" in the name is a ritual this year around so I decided to name my 2025 recap this way... It is also fashionably late because Nix is a wonderful piece of software.

Another year almost in the books. I thought about skipping the recap thing because honestly who cares, but then I remembered I care (a little), and also the blog has been quiet lately so might as well fill the void with some self-indulgent yapping before my time runs out. If you are surprised by me posting at the last second (i.e., after it became 2026 for some people reading this), I am honestly impressed. I thought myself more predictable and you... less gullible. That said, I am sufficiently intoxicated for the New Year celebrations and jokes aside my dear reader, 2025 has been a very long year and a lot has happened. Hope you've got a drink or some snacks ready, because this is a long one.

Starting strong, NixOS 25.05 happened and I took part as a release editor. This coincided with a lot of important real-life events so I could not do as much as I hoped to but it was a good release regardless, and I'm proud of however much I was able to contribute. You can thank me for most of the grammar or typo fixes in the changelog. In the end it was the usual mix of module arguments, inconsistent defaults, and people discovering that lib.mkForce is both powerful and a crime scene. We shipped without total catastrophe, nobody died. I count that as a win.

Hyprland is also coming along nicely, for those interested. I'm a little over Hyprland at this point, but it appears that wlroots -> Aquamarine migration is continuing its slow, painful crawl toward something resembling sanity. While I'm still the resident babysitter for the community, I took off from the Nix side of thing as those are usually not deferred to me. We've got some Nix-related changes coming along, but there is some time until those are actually... tangible. Long story short, the ecosystem is still alive. That's more than most Wayland compositors can say after three years.

Project News

Projects-wise, it was a year of small-to-medium itches scratched in various languages. I think 2025 is the year I can finally call Rust my "go-to" language, and the year my honeymoon period with Go has ended. I've written about this and I intend to talk a bit more in depth about the Go language because of how confused people seem by my stance but I am quite happy with my adoption of Rust for the time being. Plenty of new Rust projects now plague my "portfolio."

Something worth noting is that I have taken over nh from my good friend ViperML (who considers the project complete and is no longer interested in maintaining it as per his original vision) to continue its maintenance as time goes on. I was a contributor to nh before this happened, so I am quite familiar with the codebase already. This gradual change in maintainers has covered the 4.0, 4.1 and 4.2 releases as I continue to publish more bug fixes and feature additions. If you are not familiar with nh, I recommend that you check it out as it is quite a handy CLI for managing NixOS systems. It will continue to improve over the time as new updates come along :)

We have also released v0.8 for nvf! It was a long time coming and was doubted by many, but after just over a year, the release is out. This update brings the Nix-for-Neovim approach of nvf to a whole new level with more robust LSP configurations, many plugin additions, new documentation and more! The reason that it took this long, of course, is that I was on a tiny side-quest writing a documentation generator for my Nix projects! You may or may not be aware that I have a lot of ongoing projects related to Nix, and documentation is most often a very high priority. Unfortunately I find the ecosystem very lacking, and I just had to create something of my own. The first half of 2025 was mostly spent working on ndg, our in-house documentation generator at feel-co. While this lives in feel-co and has been designed for feel-co (and my own) projects, it is a project for any Nix module system looking to document its options in a stylish manner. Try it, it grew on me despite the hellish CSS and Javascript misadventures.

Another noteworthy project, which ndg was actually initially designed for, is Hjem. I've been bothered by Home Manager's poor file management primitives for a long while, and all this rage ultimately bubbled up into our own in-house module system for managing one's $HOME, elegantly. Powered by SMFH, a manifest based file linker made by my good bald friend Gerg-L and designed for atomicity & correctness, Hjem has received many new features such as brand new documentation, much better module interfaces, and Darwin support---which many were excited for. It has come a very long way over the past year, and I am happy to report that it is nearing "completion." Of course, it's doubtful we'll stop tinkering on it ever but the features originally envisioned are almost fully implemented. If you are interested in fast, low-cost and correct file linking in your $HOME perhaps give it a try. It is at a stage where feedback and bugreports are very valuable to us. We eventually hope to upstream Hjem into Nixpkgs as a first-party home management system, as it is just the right amount of abstraction; almost none. Contributions are also welcome, as they are to any other project of mine!

Some other projects worth mentioning are:

MicrOS got its initial push in January --- Nixpkgs modules + runit instead of Systemd. It's still extremely niche, still requires manual prayers to boot on real hardware, and still mine. This is a long-term project that I am hoping to develop more in 2026. If you are one of those nutjobs that are against Systemd, your feedback would be appreciated. Contributions even more so.
The byproduct of a most unfortunate naming conflict, Stash has gotten a bit of attention around August and has taken its forever place in my Wayland desktop as a feature-rich clipboard "manager" with persistent history and multimedia support.
nix-bindings has appeared after my poor attempts to deal with C++, and is now a somewhat viable way of interacting with Nix's C API. I don't have a worthwhile use for this project yet, but perhaps you do. Either way, your feedback would be appreciated. It seems that Nix is ramping up the work being done for the C API, and as such the bindings will only evolve further over time. Perhaps those bindings will make their way into more projects of mine to replace instances where I shell out to Nix.
I have finally forked tuigreet to solve many of the minor issues that were bothering me. This ended up with me implementing some requested features, modernizing the codebase over a single week, and adding a new configuration system. Since tuigreet is my primary greeter and will remain as such for the foreseeable future, I plan to maintain it as much as I can. If you also use tuigreet and have had feature requests or bugs to report, then head over to the issue tracker and let me know. I promise to keep an open mind. PRs are also very welcome as you might guess. Ratatouille is an infinitely powerful TUI framework and I plan to leverage its powers further as time goes by. The result? Terribly extensible TUI greeter. You're welcome.
Microfetch has gotten much faster and has taught me a lot about Rust. It has also seen a bit of attention late 2025, so perhaps it is now adopted by more NixOS users. It will, of course, continue to get faster. Sky is the limit. In both speed and pointlessness.

Other than that, I've continued housekeeping work on watt (CPU power management), tailray (Tailscale systray in Rust), and a dozen smaller things like flint (flake input linter), and tempus. Nothing revolutionary, but they ship, people use them, and my system feels marginally less awful.

Blog Updates

My blog has remained opinionated and chaotic as it was designed to be. I think I've dropped plenty of posts within 2025, but I'd like to highlight a few that I'm particularly proud of:

"The Curse of Knowing How, or; Fixing Everything" was my most popular post this year, and it opened the door to many interesting conversations about software and the compelling urge to "fix" things. I'm told this is also called "obsession." No, not like the perfume.
"The Federation Fallacy" in June---my thoughts on federated software as I have come to adopt them in 2025. Federation is cool until you realize most people just want their little walled garden with extra steps.
"I am Not Convinced by Vibe Coding" in August---I'm quite disappointed in the state of... well, everything. Software is only one of the things AI has managed to ruin (I reckon with many more to come) but imagine how beautiful it would be if we just stopped telling glorified autocomplete to rewrite things?
"The Tradeoff Trap" shortly after---simplicity isn't a virtue if you're just sweeping edge cases under the rug and calling it minimalist.

Shorter rants and post have made their appearances, but I am not going to ruin the fun of discovering those posts for yourself, organically. I also plan to make a few more posts in early 2026 as I get to finishing them, so maybe also stay tuned? I've got an RSS feed that you can subscribe to if what I write interests you :)

Shorter rants sprinkled in about Golang deliberately dodging good patterns, mutation testing as the only test metric that matters, why Hydra is ironic hell for a declarative ecosystem, and the usual Nix-sucks-but-less-than-alternatives sermon.

Other Stuff

I think I aged like ten years over the last year. A lot of things happened, but most prominently, burnout happened. Still ongoing work on privacy/data integrity in policy contexts. Briefly relapsed to Arch in November---don't ask, it was bad, I'm back on NixOS, lesson learned (again).

Sailing, as usual, was alright when weather allowed. It seems that I am making less time for things I actually enjoy. Chess ELO still trapped in the 1100-1150 limbo but I haven't played chess in almost six months. I guess you can't climb in ELO without, you know, actually playing the damn game.

Overall? Just the slow realization that software culture is still mostly vibes, politics, and bloat, and we're all just trying to carve out tiny corners where it sucks marginally less. We make do, we always do.

2026 Wishlist: bigger than projects, smaller than miracles

This brings us to my 2026 wishlist, and I'm tired of small asks. Polish is nice, but polish on a rotting foundation is just expensive lipstick.

What I actually want this year is an industry (and our little corners of it) that starts punishing redundant complexity. Not rewarding it with stars, funding or useless engineer titles but instead make adding a dependency cost something real---social cost, maintenance cost or cognitive cost. Stop pretending that "but it's modern!" is an argument. We are in the year of 2025 and for some godforsaken reason our computer resources seem like they will be dwindling over time. Do you know what this means? OPTIMIZATION god damn you, actually think about it for once. I am tired of useless Typescript projects for desktop or Python in production. Write actual languages. Write Rust, write C, write C++ but stop overcomplicating. On this note, tooling that does not actually hate maintainers would not be so bad either. NixOS modules cleaner from the start. A CI that isn't as ugly as the nightmare that is Hydra. Less YAML worship. Less vibe "coding" in infra that people rely on daily.

Minimalism that isn't performative or cultish. Suckless is an unfunny joke sure, but it is not and has never been the endgame. We need small and correct, small and usable, small and secure. Not just "fits in my head" while leaking like a sieve. We know this is possible, because people used to do it when it was funny. Now it's important, and people have long forgotten.

I also want privacy to stop being a quirky nerd interest. This is not a niche hobby for people who like encryption diagrams. This is something normal adults should understand well enough to get angry about. Educate your wine aunt. Educate your siblings. Explain it in plain language until it clicks that "exceptional access" is just marketing speak for "someone else gets to read your shit". No more ChatControl style proposals quietly sliding through because someone mumbled "but cyberbullying uwu" and everyone nodded along to avoid looking callous. That rhetorical trick needs to stop working. I need widespread acceptance of a very boring technical truth: every backdoor is just an exploit wearing a badge. There is no such thing as a friendly vulnerability, no such thing as access that only the good guys use, no such thing as a hole that stays politely scoped to its original intent. If a system can be accessed, it will be accessed, by actors you did not anticipate and in ways you did not authorize. Governments, companies, and especially mid-tier tech bros need to internalize this as a fact of reality, not treat it as optional flavor you can trade away for optics or convenience. You wish this is paranoia, but it isn't. It is pattern recognition and anyone with more than just a goldfish brain should be able to notice the patterns as well. There is a rant coming about this, but I will spare you for today. Or rather, I will spare you the rest for today.

Last but not least, I wish for a single ecosystem somewhere, anywhere, that still chooses craft over clout. Not as branding, not as a values statement, not as a thin excuse for moral theater, but as an actual operating principle. No moral grandstanding, no trend chasing, no resume padding rewrites whose main purpose is to signal participation in whatever the current discourse cycle happens to be. Just good engineering because it is good engineering. Systems that are designed to be understood, maintained, and trusted by the people who have to live with them. We can have this. We have had this. We know exactly what it looks like when correctness, performance, and restraint are treated as first class concerns rather than optional polish.

What changed is not that nobody cares anymore. It is that we have become so fragmented that care is no longer visible. A lot of people still care, quietly, competently, and without hashtags. They just do not want to spend their limited time arguing with the loud group of critters who insist that performance can be bolted on later, that quality is premature optimization, and that maintainability is someone else's problem. "We shipped and that is all that matters" has become an excuse to stop thinking, as if shipping were the finish line rather than the point where responsibility actually begins. That mentality does not just produce bad software, it produces brittle ecosystems that slowly train everyone involved to accept decay as normal. If craft does not matter, nothing downstream does either. That idea needs to die. It needs to die for good.

Okay, I think that's it for my rant. Let's wrap it up for today.

Closing Thoughts

It has been a long and somewhat productive year. I've created a lot, contributed to a lot, and cynically criticized a lot. I do not plan to change. I will continue to create, contribute and criticize.

2025 was survivable.
2026 will probably be too.
Software will keep being disappointing.
We'll keep trying to disappoint it back a little less.

If things get better, thank the people who still care.
If they get worse... well, blame webdevs. Always blame webdevs.

I'm looking back with bittersweetness but I cherish a lot of things, and a lot of... you. I've met a many amazing people over 2025 that I am happy to call "friends" over this parasocial existence that you call the internet. Know that you are appreciated my dear reader, and see you next year, or; see you when I've got something worth saying. Have a nice one.

--- raf

The Tradeoff Trap

Mon, 11 Aug 2025 00:00:00 GMT

Not so long ago, I have spent quite a bit of time writing a new component for my site. It is not really that fancy. Though suffice it to say, it took more time than it was worth; the component is written in Rust, compiled to WASM using wasm-pack and it's shoehorned into my Astro stack through a Vite plugin that I probably regret adding to my site. It does work, technically, but it's not too clean. It's not even clever. It just... feels heavy. ¹

Truth be told, I did enjoy the experiment. I have been long fascinated by the idea behind WASM, and this was a fair excuse to finally give it a shot myself. This whole ordeal started because I wanted to fix something small: my search widget. However, I ended up dragging the rest of the codebase into it like a black hole pulling in nearby stars... as usual. Several components got written, new abstractions had to be born, and a majority of the codebase changed to accommodate this experiment. I opened a second terminal window just to keep track of my regrets.

Somewhere in the middle of this mess, I remembered when my site was a single text file. A literal .txt file dumped into /var/www/html and served via NGINX. No JavaScript, no build step. No decisions whatsoever. It just was.

So, how the hell did I end up here?

The Itch That Builds Complexity

There is this itch that I have. It has me so that simplicity and stagnation bother me a little. Hell, I can still remember the night I rewrote a single utility three times. First version: 30 minutes, dead simple, rock solid. Second version: three hours of reactive garbage and decorators that made the call stack look like performance art. Third version: a weird hybrid that used a language feature I had to re-learn every time I saw it. None of them felt right. I hated all of them. And not in the funny self-deprecating way. I mean actual disgust. Because none of them satisfied both sides of me: the builder and the tinkerer.

The itch comes when I write code that is too simple. You probably do too. It's that little voice that says: you could be more clever than this. You could abstract this. You could invent something novel. It’s not about what the code needs. It’s about what you need: a little shot of novelty, a way to prove that you still know how to do something slightly absurd. And sure, I tell myself it's for maintainers. Future contributors. Elegance.

But the truth? I do it for myself. For the thrill of wrapping a problem in something unnecessarily beautiful. To prove that I can.

The Joy Tax

Truth be told, joy matters. It drives creativity. It is also why I still spend time programming. However, it comes with a tax. Joy tells you that layering three patterns on top of each other is fine because you will totally clean it up later. Joy tells you that you are learning. Sometimes that's true. But more often, it leaves behind a breadcrumb trail of entropy. Small complications that add up over time until you're afraid to touch anything without a full test suite and a stiff drink.

Simplicity, on the other hand, has a kind of power I have a hard time describing. If I were to try and visualise it for you, I'd tell you of a puzzle piece falling into place on the first try. Just in.

Consider this. A loop. A few conditionals. No clever tricks. No helper macros that magically know what context you're in. Just boring code. And I mean that in the best way. Boring code runs fast. It fits in your head. It does not surprise you six months later.It might look uninspired but it has its own power. Simplicity means you can revisit it months later and feel like you understand it immediately. It means fewer bugs and fewer performance surprises. A plain solution usually uses less memory and runs faster without requiring a profiler.

Yet there is this intriguing feeling that accompanies simplicity. It feels almost like surrender. It feels like admitting the challenge was too difficult Maybe we fear judgement for writing something that looks pedestrian. We call it boring code instead of reliable code. We convince ourselves that plain means uninspired when in reality it often means practical and predictable.

A Fork in the Road

Projects used to come in two broad flavors. At least there are two that are relevant to us today, but the new third variant is more sinister. Traditionally there were those two flavors. Anyway, I digress.

You either get the reliable, boring version that just works and feels like a spreadsheet with a keyboard. Or you build something expressive and brilliant that collapses under its own weight the moment your attention shifts. Neither path is inherently better. What matters is context. A tiny service that processes thousands of requests per second deserves minimal overhead. Pull in fewer dependencies and write code that can be understood with a single glance. In that space, every millisecond counts and every external library is a liability. You need the plain solution.

By contrast, a research prototype meant to explore an idea might not care about raw performance. It might benefit from dynamic typing or metaprogramming even if that code will never see production. In that realm the joy of creativity and the speed of iteration trump long term maintenance. The cost of rewriting from scratch is acceptable because you will probably scrap it once you learn what works. The cost of learning something new is far less than the cost of missing new knowledge.

The real trap is believing you can have everything. High performance plus effortless maintainability plus endless enjoyment. Right. There is a narrow sweet spot in the design space but for most projects it is elusive. You end up either cutting complexity until nothing interesting remains or stacking layers until you cannot find the core logic anymore. The only skill worth mastering is knowing which sacrifice to make at what moment.

Chisel and the Sledgehammer

Sometimes you need a chisel. You want to carve something that fits perfectly, piece by piece. Other times, you need a sledgehammer. Bash it together until it stands and deal with the mess later. There is nothing noble about always choosing one tool.

Sometimes the best path forward is plain. Your code will not sparkle, but it might just hum along quietly without any unpleasant surprises. Alternatively, the best path forward might just be intricate. Your code may creak under pressure but it will teach you things you could never learn with a toy example. Neither choice diminishes you as a coder. Both are valid expressions of craft.

So the next time you stand at that crossroads, pause before you reach for the clever trick. Ask yourself what you truly need from this project. Do you need speed or delight in playing with language features? Do you need rock solid uptime or rapid feedback on a research question? Answer that and choose deliberately. Honor the tradeoff. There is no perfect solution, only the right compromise for now. When you are racing against a deadline, choose simplicity and accept the dullness. When you are prototyping a radical idea, choose sophistication and accept the technical debt. When you are building something meant to last, choose the combination that yields reliability, even if it does not inspire a standing ovation from your future self.

And remember: simplicity or sophistication, they are not sole measures of your talent. They are tools you wield to shape the experience of building and maintaining code. Master the tradeoff and you will write software that serves its purpose without betraying your own values. Know that there is no perfect answer. There is just the answer that works for this moment, and the honesty to admit what you're optimizing for.

And if the answer is 'I just want to mess around and write something cool,' then you need to own it. Just don't lie to yourself and call it a proper architecture. ²

And the performance benefits aren't even that great! What happened to my blazingly fast, memory-safe Rust? ↩
Believe me when I say that this is self-reflection more than anything. I have called my own creations "solutions" when they were just experiments. Sure they did substitute a solution for the time being, but I should have known it was not the way to go. ↩

I am Not Convinced by Vibe Coding

Sun, 03 Aug 2025 00:00:00 GMT

There has been some growing current of enthusiasm recently, around so called vibe "coding." To those of you unfamiliar with the term, if that is even possible, vibe coding refers to a software development mode where instead of writing code directly, developers interact with large language models (LLMs) using prompts, voice, or vague intent. The tools generate code, and the human nudges it back and forth by feeling---by vibe, they say---until it works.

The promise is seductive: anyone can build an app, and you don't even need to know programming! Prototypes come together in a weekend. Codebases emerge not from typing but from suggestion, improvisation, and iteration. The developer becomes a kind of conductor, gesturing at the machine to summon form from possibility.

I am, however, not entirely convinced. Not because I'm a Luddite, or because I'm resistant to change. I have used these tools (at least I tried to) and I do understand their appeal. What I reject is the framing that this is a sustainable or revolutionary new paradigm for software development. It's not. It's a generator of noise with occasional signals, and it relies entirely on your ability to tell the difference. What happens when you can't tell the difference? Have you really accounted for that?

The assumption that the technology alone solves the entire problem overlooks the complexity embedded in software development beyond mere code generation.

The Machine That Designed Cars

There's a plot point in the book The Dice Man ¹ that I keep returning to. At one point, a team invents a machine that generates car designs entirely at random. Millions of iterations, nonsensical and broken and unbuildable---but every once in a while, a design emerges that is striking. Beautiful, even. So they discard the garbage and hand-pick the good ones.

That's what I find vibe coding to be. You prompt an AI a hundred times, throw away the malformed outputs, and keep the one that works, at least on the surface. It looks right. It runs. The UI renders. The button clicks. Success, right?

I think this misconception, so to speak, comes from the fact that software programming has fewer win conditions. While in most domains a "win condition" is some universally recognized state of victory---e.g., in chess, it's checkmate; in biology, it's a Nobel-worthy discovery; in sports, it’s a gold medal---programming has no universal definition of "done right", ² no standard of elegance that everyone agrees on and no fixed endpoint. Hell, even shipping a product doesn't mean you "won." There are always bugs, tech debt, UX complaints, performance ceilings, etc. Even "perfect code" is usually replaced or rewritten within a decade.

As such the internal structure matters. Code is not just visual. It is layered with behavior, security, data access, side effects, and edge cases. You are not choosing car sketches for a brochure; you are selecting an actual vehicle that someone will drive. And if you don't understand what's under the hood, you are gambling with your users' safety. The belief that AI-generated code can be blindly trusted because it "builds and runs" dismisses these hidden complexities and the inherent unpredictability of probabilistic models. Treating generated code as a finished product rather than a draft invites risk.

What Vibe Coding Forgets

Proponents of vibe coding frame it as a democratizing force. Claiming that you do not need to know how it works, and that you can just explain what you want to the machine. Let the machine handle it. This does sound empowering to a degree, but it also breaks down under scrutiny. Truth is, someone will always have to understand the code. The difference is whether it's you now, or an unfortunate engineer, a curious security researcher or a malicious party down the road. Biggest question here is, what if someone malicious gets their hands on your vibe coded SaaS?

Blindly trusting generated code without review or comprehension is not engineering, it is nowhere near that. In fact, I think it is something to be ashamed of. The pride tied to the fact that you do not understand something is... tragic. I can only describe (most cases of) vibe coding as "aesthetic selection". The entire method hinges on your ability to recognize what is good or most of the time good looking, but good on what axis? Speed? Readability? Robustness? Security? These are not things you can judge from a glance. They take context. They take understanding.

Relying on AI as a catch-all acceleration tool ignores the essential nature of software as a system. Speed without comprehension leads to fragility and technical debt. The supposed gains are only superficial if the foundational integrity of the codebase is compromised.

The Shortcut That Becomes a Detour

I'll be honest; yes, vibe coding does accelerate prototyping. Yes, it also lowers the floor for experimentation. But we have to stop pretending that this comes without trade-offs. You are accumulating technical debt at warp speed. You are outsourcing logic to a model trained on GitHub, filled with every security mistake and anti-pattern known to man. What, you thought it was trained on good code? Hah.

Worse, you may not even notice. The very act of vibe coding discourages investigation. You didn't write this code. You don't own it. You don't feel responsible for it. If it breaks, you will just prompt again. But that's not development. That's gambling at best. Desperation at worst. Even if you are someone who understands the principles of programming, which is safer than most use cases I have witnessed, it still gives you tunnel vision to a degree unless you had a perfectly outlined plan from the start. Unless you know what exactly you were going for, then you are letting a subpar sentence generator decide what will be happening in your project. Even if you reject its proposals, it will plant the seeds of an idea in your head.

This mindset assumes that the code is only a surface-level artifact and ignores that true software development involves maintaining systems, anticipating edge cases, and managing complexity. The idea that one can simply "iterate until it works" neglects the institutional knowledge and discipline required to sustain software over time.

When Vibe Coding Makes Sense

I think there is a place for vibe coding. Or rather, AI-assisted programming. I reject the idea of prompting a sentence generator to have what I can do better while also enjoying the process. It can be a great tool for demos, quick and hacky scripts, or even internal tooling if the risk is acceptable. If you are hacking something together for one-time use or just a presentation, the speed is hard to beat. Though you must also realize that the domain is very narrow. The moment your software has users, state, uptime requirements, or a security surface, you are in a different game. At that point, the vibe must end. You need understanding. You need rigor. You need systems that are legible, testable, and maintainable.

The excitement around these tools often ignores that they require significant infrastructure: embedding the codebase, hooking into CI pipelines, and layering tests and linting to catch AI hallucinations. These are not trivial tasks, and most teams are not equipped to treat AI as a dependable collaborator rather than a toy.

You Cannot Abdicate Responsibility

No matter how arcane it looks, software is not magic. It is logic encoded for machines and maintained by humans. Tools like Copilot, Cursor or models like Claude and Gemini may be useful but it must be made perfectly clear that their use case is acting as assistants, not replacements. Using them well requires clarity, and not vibes.

What I take issue with in the usage of such tools is that I do not trust most developers to draw the line clearly. The most common and dare I say natural response to the task often is something along the lines of "Well I used it for prototyping and it worked. It can probably handle the production code as well" and to me that violates a contract of trust.

When I use software made by another person, I trust in their moral judgement and the potential consequences on a personal level. If a software programmer makes a mistake that causes my home directory to be deleted, they will feel remorse. They will also aim to never make that mistake again. What about LLMs? It'll give you an empty apology, fix its mistake for one time and move on. Will you always check the code in case it accidentally deletes your user's home directory again?

Thus I want to say that we do not need to throw out our understanding to embrace the future. We need to carry it forward. Celebrate the collective intelligence, experience and effort. Otherwise, we're not building; we are merely browsing. The cost of mistaking one for the other is always paid in production.

Closing Thoughts

It is not my intention to gatekeep anyone from doing anything. I avoid LLMs by principle, but that is a personal choice that I do not and cannot enforce on anyone. It also stems from the fact that I have only had disappointing experiences with tools like Claude. I tell it to do something, and it makes outrageous assumptions about my codebase or straight up tries to throw something away because it collides with its own shallow implementation. Maybe I'm unlucky though, who knows?

The point I really wanted to make in this post is that no matter how the app was coded (but especially if it was vibe coded) then you have a moral responsibility to understand it before you ship it. When I see something that boasts being written by LLMs, I tend to turn away and pretend that the repo does not exist. Not because I feel a moral superiority by doing so, it was never about that, but because I am not willing to take the chances of getting caught by a destructive oversight no human developer would make. After years of programming, I would like to think that human programmers inherently target or at least document production-critical bugs, whereas LLMs choose to put environment variables in production code. Sure we all know better than that, but what if you make a mistake you can't even recognize? Can AI learn from its mistakes? I don't think so.

Lastly there is the part of effort. When I see an art piece that I like, it is not just about the art itself but also the time spent on the piece. I respect the artist as much as I enjoy their art. If the piece is good, but I know the artist has spent no time on the piece, then my respect for the artwork and thus the level of my enjoyment only goes down. Sure you might have expressed an idea, but what about the journey? What have you gained out of this except for a potentially malformed artwork that is the expression of words instead of feelings? What am I even looking at? There is no beauty here, just lines. Vibe coding is very similar in principle. Yes most enterprise code is for the sake of completing a task, and doing so while making or saving the company money but what about FOSS? What happens to the individuality, to the collaborative creative wisdom of the people? Deferring inventing to AI is nothing but reductive. It can never be anything but reductive. This is not pragmatism. ³

This has been my miniature rant on vibe coding. Truth be told, I was exposed to "AI slop" more often than I'd like while in academia, and all it has done to serve me was to get on my nerves. Waste my time. Sure, don't think I guess. That said, I do not reject the idea outright. I believe there is merit in AI-assisted code reviews or project planning. Not yet though, at least not until the hype bubble pops. Hope you enjoyed this post. It was more emotional than the rest of my writing, and I'd like to hear your thoughts. If you think I've missed anything, I'm also open to further discussion.

I want to leave you with two interesting studies. One of them is on the AI-assisted academic writing and recommendations for ethical use. While ethics are whole new discussion point that I want to avoid, the practicality of AI is an interesting question. It is a relatively short read, so I recommend that you give it at least a skim. Especially if you are a student. The second study is on the impact of AI on developers in 2025. It challenges some of the baseless assumptions usually perpetuated by those that stand to gain from perpetuating the unseen (but somehow very real) productivity boost of AI in enterprise. If you would like a real statistical study (that you can pull out next time someone tells you that "AI makes developers faster, trust me") then I recommend you take a look at this one as well. If you have some other interesting articles, please feel free to contact me to let me know.

Cheers!

Or maybe its sequel, The Son of Dice Man? It's been years since I read the books... ↩
No matter what the grifters on tech Twitter will tell you, there simply isn't one. ↩
If you mention driving a car or using printing presses to replace walking and printing, I will simply defenestrate you. You know those things are not equal, and you KNOW that they solve problems of capacity. You cannot walk a thousand miles, even if you could the time spent would not be worth it. One man cannot copy a million books, even if he did it would take a lifetime. One developer, however, solve write one hundred projects. A thousand. AI only makes it less incentivized to think on those projects. ↩

The Federation Fallacy

Fri, 27 Jun 2025 00:00:00 GMT

This post is greatly inspired by the awesome article The Federation Fallacy by Alyssa Rosenzweig. Please make sure to check her out. Some of the ideas discussed here today are greatly inspired by the original post, bearing the same name.

Imagine this: the boundary between government control and corporate influence has all but disappeared. One bleeds into the other so seamlessly that you can't quite tell who's pulling the strings. And here's the kicker---you're already in the middle of it. Every time you unlock your phone, launch an app, or just exist online, you're stepping onto a battleground you didn't really sign up for.

I missed a great portion of the technological advancements that computers and the internet went through. I was blissfully offline throughout most of it. Although now that I have been diving more into OSdev and other programming concepts, my understanding of it is that technology was supposed to be the great equalizer. A tool for freedom, connection, progress. Instead, it appears to have become something else entirely---something much more insidious. Governments no longer need to strong-arm their citizens into compliance when corporations can do the dirty work for them. And the best part? Most people, perhaps even you reading this, don't even notice. The social contract didn't just get rewritten; it got outsourced.

But here is where things take a turn. The same technology used to monitor, categorize, and manipulate us also holds the potential to set us free. The internet, encryption, decentralized platforms—these aren't just tools, they are weapons in a fight most people don't even realize they're a part of. And that's the problem. Too many of us engage with technology passively, as consumers and not as participants. We think of it as a service, not a structure of power. That is a monumental mistake. ¹

Take something as basic as data ownership. Who owns your data? The reflexive answer might be, "I do, obviously." But think about it for a second. The moment you tap "Agree" on a terms-of-service agreement you didn't read, you're giving away more than you think. Your habits, your movements, maybe even your thoughts---logged, analyzed, and monetized by entities that have no obligation to you in ways that would not occur to you in your wildest dreams. And here is the brutal truth: once it's gone, it's gone. You don't get to take it back.

Now, let's talk about digital sovereignty. Sounds like something ripped from a cyberpunk novel, but it's real, and it matters. You might have not heard of it before, and frankly I have not thought much of it until my research brought me across several sources that used the term. It is the idea that individuals---and entire nations---should control their digital presence the way they control their physical borders. It is about deciding who gets access to your identity, on what terms, and with what consequences. Yet, somehow, this is is not a mainstream political discussion. We act like it is a niche concern, something for tech nerds to stress about, instead of what it really is---a fundamental issue of power in the 21st century.

Interlude: On Data Ownership

While working on this post, I was reminded several times that in some countries data ownership is, in fact, a mainstream political discussion. I am aware of this, and I really admire the level of optimism required to believe this is somehow the case for an acceptable majority. It isn't. According to Freedom House Freedom on the Net 2020 report, over 80% (the actually cited number is 86%, and I fear it has gotten worse by now) of the world's population live in in regimes where the online participation is partially or fully censored. What is the possibility that most of us own the data when most of us cannot go online without our every move being recorded?

Some European countries have been making extended efforts to ensure privacy as a right, other continents have been regressing steadily. The United States has granted unlimited access to every citizen's data to a private citizen while in Turkiye, the government has decided to form a government department to monitor and act on online activity with little to no regulation. This kind of immense censorship is also the case in China, Russia, Saudi Arabia and many more that I can count. Whatever privacy incentive you might be aware of is not the norm, nor is it setting an example for the rest of us. This is quite unfortunate, yet not at all surprising. One of the many dimensions of a state is to express power.

Federation Fallacy: Continued

Make no mistake, this is about power. Data is the raw material of modern power, the foundation upon which influence, control, and wealth are built. Those who have it shape the world. The problem? Governments and corporations have been playing this game for decades, and while you are barely of the rules, they have mastered it. The most average player of the game can barely remember their own passwords, let alone having any awareness of the game.

So what now? What do we do with this knowledge? First, we need to accept that technological literacy isn't optional anymore---it is survival. Second, resistance doesn't always mean taking to the streets, or drawing graffiti on walls. Sometimes, it is as simple as using open-source software, encrypting your messages, or just stopping for a moment to ask, "Why does it have to be this way?" The final step? That's on you. But let me be very clear: doing nothing is a choice, and it has consequences. Costly consequences.

We happen to be at a turning point. The choices we make now--about technology, governance, and personal autonomy--will shape the next century. The only question is: will we be the architects of that future? Or just subjects to it?

Federation: The False Promise of Decentralization

You might have heard of the promise of decentralization, an ideal where the power and control of the digital world are distributed evenly, away from the corporate giants and government entities that currently dominate. Many of us in the open-source and privacy communities dream of this decentralized world. We envision a future where individuals can host their own servers, own their data, and share information freely, without intermediaries like Google, Amazon, or Facebook controlling the flow.

But the reality of decentralized networks, such as federated platforms like Mastodon or the once-promising web, exposes a serious flaw in this vision. Sure, decentralization sounds like the solution to the problem of monopolistic control. But let me break it down. Decentralization isn't as simple as just opening a server and letting users do whatever they want. If you think about it, decentralization often becomes a highly technical and impractical solution for the majority. Most people aren't system administrators. They don't want to host their own email servers, ² handle constant updates, or manage security vulnerabilities. And let's be honest: it is not something most will do. So what happens? The dream of decentralization becomes a playground for the technologically "elite," and the rest of us are left either uninvolved or stuck relying on others to handle the complexity for us.

Federation attempts to solve this issue by allowing individuals or groups to host their own servers while still communicating with others across the network. Unlike a fully decentralized system where each user is entirely independent, a federated system connects multiple independent servers that can exchange data while maintaining local control. The idea behind federated platforms like Mastodon is that you can join or create independent instances and still interact with users across the wider network, much like how different email providers (Gmail, Outlook, etc.) can send messages to each other. This approach avoids the need for a single central authority while still enabling cross-platform communication.

Federation seems like a happy medium between full decentralization and reliance on centralized corporate platforms. But it comes with its own problems. Despite the ideal of decentralization, federated systems have shown a tendency to concentrate power into the hands of a few large instances. While there may be thousands of Mastodon instances, the majority of users end up clustering around a few, leaving power and control in the hands of administrators who run these larger servers. ³

Take Mastodon, for instance. It promises a decentralized, federated alternative to X/Twitter, where users are free to join different servers and interact across a network. But in practice, most people gravitate toward a handful of large instances, making the system feel centralized. The same problem plagues email, XMPP, and even the original decentralized web: while the systems themselves are designed to be federated, the reality is that scale and complexity inevitably lead to centralization. These federated systems are still subject to the whims of a few large players, who end up determining the experience for most users.

So, what does this mean for the dream of decentralization? It means that federation, despite its merits, often falls short of its ideal. It is an imperfect middle ground that tries to balance decentralization with practicality. But as the user base grows, so does the tendency for power to centralize; just in different ways.

Perhaps the solution lies not in decentralization for its own sake but in creating more democratic information spaces. Rather than solely focusing on breaking free from the power of corporations or governments, we must ask: how can we create systems where the power is distributed fairly, transparently, and with participation from all users, not just the elites who can afford to manage complex systems? Ultimately, decentralization and federation offer only partial solutions. The real goal should be creating systems that empower individuals while maintaining the democratic oversight necessary to prevent concentration of power. The dream of a decentralized future may be far from perfect, but it's not impossible. The question is, how do we design it in a way that is inclusive, scalable, and fair to all?

Closing Thoughts

Even as decentralized networks and cryptographic tools emerge as potential solutions, they strike me as far from perfect. The very technologies that promise to free us from corporate and governmental oversight are also fraught with their own limitations and vulnerabilities. Decentralization, for instance, sounds like the answer to control over one's data, yet it introduces new risks--fragmentation, inconsistent security protocols, and the lack of cohesive infrastructure. ⁴ As much as we crave autonomy, it's difficult to ignore that true digital sovereignty may require us torely on infrastructure that is, ironically, still controlled by large tech entities. Peer-to-peer networks and decentralized platforms may offer us a glimmer of freedom, but without mass adoption, scalability, and seamless user experience, they remain niche alternatives.

This creates a paradox: the tools that could allow individuals to reclaim their autonomy are often too difficult or inaccessible for the average user. The notion of "sovereignty" in the digital world, much like its political counterpart, is one of constant negotiation. Governments are learning how to integrate blockchain technologies for more efficient control, while corporations leverage surveillance capitalism to expand their grasp on user behavior. Ironically, the very same technology that could empower individuals is becoming a tool for governance and control, albeit one that is harder to trace and combat. The question we need to grapple with is whether true decentralization is even achievable in a society so deeply embedded in corporate capitalism and surveillance mechanisms. At the same time, can we ever truly achieve the autonomy we crave, or will the future of digital space always be a game of tug-of-war, where the strings are pulled by invisible powers far beyond our control?

In a way, we are all participating in this grand experiment---sometimes as innovators, sometimes as victims---and our level of awareness determines how much influence we have over our own digital future. The question isn't just about how we can protect ourselves, but about whether we can transform the system from within. We may be able to opt-out of certain platforms, but as digital life becomes more entrenched, can we afford to truly escape the architecture that shapes us? The choices we make today in how we interact with technology will set the precedent for how future generations experience their own agency and sovereignty in an increasingly connected, monitored, and data-driven world.

This is all I have for today. I think this post is already very long (and much longer than I have initially intended for it to be), but I'd love to dive deeper into something else that's been on my mind recently: the global reliance on CDNs like Cloudflare. It's wild when you think about how much of the internet today is essentially built on the backs of these centralized systems. Cloudflare and similar services are everywhere, speeding up websites, making them "more secure," and keeping them online. The catch? As we depend more on them, we're putting a lot of trust in a handful of companies that control a massive portion of the web's infrastructure. This brings up some huge questions about power and control. What happens when a company like Cloudflare decides to pull the plug on a website---or worse, if it's compromised in a way that takes down whole sections of the internet? It's one thing to have content spread across servers worldwide, but it's another when those servers are owned by a handful of corporations. It creates single points of failure and makes the internet feel more like a corporate-controlled ecosystem rather than a decentralized space for free expression. There's also the reality that these services often make the calls about what's acceptable or not, sometimes with little transparency or accountability. We've seen this with Cloudflare's past deplatforming decisions, ⁵ which leave us asking, "Who gets to decide who stays online and who doesn't?"

Sorry, I'm rambling again. I hope I was able to pique your interest today and perhaps offer a different perspective. There are many interesting subtopics I could cover, but I would like to avoid overwhelming you with a wave of unfiltered thoughts. Special thanks to @mraureliusr for the initial proofreading, and of course, thank you for reading this. Cheers!

Perhaps an inconsequential one. How many of us care about the consequences of our data being shared freely? ↩
Bad example, I admit. Self-hosting e-mail servers is notoriously annoying, but this artificial difficulty is also a testament to the problem at hand. Why must hosting something as simple as e-mail be this difficult? Why will spam-lists immediately put you on their blocklist? Can we not have better solutions? Why does most of our technology suck? ↩
Years ago, a friend of mine shared a theory with me about humanity's tendency to cluster together. He argued that shared interests, appearances, or ideas naturally bring us together. According to him, this drive leads people to form groups that reinforce these commonalities. Over time, these groups become more insular, focused on mutual validation and a narrower set of values. Looking back, his theory seems particularly evident in tech communities. These spaces form around shared goals or ideologies, often growing into large instances where members develop a sense of identity and belonging. Yet, as these groups grow, they can transform into echo chambers--spaces where the same ideas are constantly amplified. Eventually, these communities may "defederate," or be "defederate_d_", through either external pressure or internal fracture, reinforcing their separation from broader networks in an effort to preserve their core beliefs. ↩
I will avoid picking the low hanging fruit that is Matrix and its clients. Make whatever you will from this footnote. ↩
Cloudflare has made notable deplatforming decisions that have sparked debates about free speech, censorship, and the power of private companies over the internet. One significant incident occurred in 2017 when Cloudflare terminated its services to The Daily Stormer, a neo-Nazi website. This decision followed the site's claims that Cloudflare secretly supported their ideology, prompting Cloudflare's CEO, Matthew Prince, to take action. In 2019, Cloudflare ceased services to 8chan, an imageboard linked to multiple mass shootings, including the tragic event in El Paso, Texas. Prince described 8chan as a "cesspool of hate" and cited its role in inspiring tragic events as the reason for the decision. ↩

On Mutation Testing

Fri, 06 Jun 2025 00:00:00 GMT

If you have ever worked on any serious codebase, then you probably know how truly deceptive confidence can be; you push a change, tests pass, CI is all green and the voice in your head yells "MERGE ON GREEN!" Then regressions show up two days later in production. At some point, every experienced developer comes to the same realization: coverage is not correctness.

That is where mutation testing becomes a quiet revolution. It doesn't just measure code that's been run, it asks, does your test suite care if this logic is wrong?

A Practical Example

Imagine, for a moment, writing a unit test for a function that calculates discounts. It passes, because the discount is 10% and the test expects 10%. But what if the function always returns 10%, regardless of input? What if you accidentally hardcoded it and never noticed? ¹ Traditional coverage wouldn't say a word. It sees the function ran, so it's satisfied. Mutation testing wouldn't let that slide, no sir. It would change the discount from 10% to 20%, rerun the test, and when it passes anyway, it would call your bluff.

This is not a hypothetical. Mutation testing catches exactly these kinds of failures: logic that appears tested but is not meaningfully verified. Silent bugs. Assumptions that you didn't realize your tests were making. All the things that lead to 2AM incident reports and embarrassing root cause writeups, or maybe an apology to your co-maintainers...

And once you start using it, your relationship with testing changes. You stop writing tests to satisfy metrics. You stop writing tests for the CI badge. You start writing tests with intent because you know they'll be interrogated. This does something subtle, but powerful: it aligns testing with reality. Code is messy, deadlines exist, teams rotate, and not every dev has full context. Mutation testing cuts through that by asking the only question that matters: if this breaks, will anyone notice? The answer is "no" more often than it is "yes," because the noticing part usually comes when something does break.

When you start to see mutants surviving---mutants that change core logic, edge cases, or conditional flows---it stings, it becomes a scratch that you must scratch. But it also shows you exactly where your testing assumptions were too generous. That makes mutation testing a better code reviewer than most humans. It doesn't praise you for structure. It doesn't care if your mocks are elegant. It just says: this logic was wrong, and nobody noticed. It might even add a subtle "fuck you" sometimes.

Of course, you don't run this on every commit. Mutation testing is heavy. Too heavy, especially for languages like Rust. You run it during quiet phases. You target it at critical modules. You use it to audit tests after big refactors. And over time, you build up a sense for the kind of brittle logic that needs better test discipline.

In some ways, mutation testing is the opposite of test-driven development. TDD is aspirational. You write tests for the code you intend to write. Mutation testing is cynical. It assumes your code is garbage, ² and dares your tests to prove otherwise.

That tension is useful. Developers often lean too far in one direction: either overly optimistic about what their tests catch, or so cynical they abandon testing altogether. Mutation testing grounds both extremes. It doesn't require faith. It just produces data. Data that tells you, without ceremony, what would've slipped through. ³

When you are building something that matters, however... Perhaps an API others depend on, an authentication system, anything stateful or money-related. This is the kind of tool you'll wish you'd introduced earlier. You simply cannot afford the fluff, you MUST test.

Postscript

If you have never used mutation testing, don't start with the whole codebase. Instead, pick one module you think is "well tested" and run a mutation tool on it. Watch how many mutants survive. That first run will teach you more about your tests than hours of code review ever could.

And if nothing survives? Congratulations. Your tests actually work. That means something. Give yourself a pat on the back.

Not because a metric says so, but because you tried to break your code and it held its ground. Congratulations.

Mutation Testing in Rust: cargo-mutants

Rust is uniquely positioned to benefit from mutation testing, and cargo-mutants is the tool that makes it possible. It's small, sharp, and well-informed just like the language itself. It works with Rust's compilation model, not against it. It generates mutants at the source level, one mutation per run, compiles them, and executes the tests---all while preserving the surrounding structure. It doesn't really try to be clever about parsing Rust. It uses the compiler.

This is important because Rust's strict type system and ownership rules make traditional mutation tools (often written for dynamic or loosely typed languages) struggle to produce meaningful changes. Most random mutations won't even compile. cargo-mutants appears to be understanding this, so it carefully selects mutations that produce valid, buildable code. That level of respect for the language is rare in tooling. It also integrates tightly with the way Rust projects are structured. It knows about your Cargo.toml, about test targets, about workspace layouts. It even detects and reports untested public items, things that your code exposes but no test ever touches. That alone is worth the install.

The result is a tool that doesn't just churn through changes for the sake of metrics. It highlights real testing blind spots in real-world Rust code. And it does so without you having to leave the comfort of cargo. What's especially admirable about cargo-mutants is its humility. It doesn't overpromise. It doesn't pretend mutation testing is a silver bullet. It just methodically shows you where your test suite is asleep at the wheel, and in doing so, quietly encourages better code, not just better coverage.

Closing Thoughts

I discovered mutation testing today almost by accident. I was reading documentation for something else entirely. And like most good tools, it immediately reframed how I think about code quality. It's rare that a testing technique feels like a missing piece, but this one does. Not because it's complex, but because it asks the question we should've been asking all along: what if this code breaks, will my tests even notice? Mutation testing does not offer comfort. It offers clarity. And in a field full of false confidence and cargo cult coverage, that's exactly what I didn't know I needed.

As a final note, I would like to remind you that mutation testing is heavy. With cargo-mutants, it has taken me around an hour to test around 2000 lines of code. Though my hardware is not the best, I can't imagine having to spend around an hour every time I have to evaluate my codebase for the possibility of regressions. It is critical that we learn to evaluate the codebase mentally as if mutation tests will be ran on it anyway.

Cheers!

This is a hypothetical example, but it is exactly what I have done. I got too preoccupied with investigating a different part of the codebase and with writing tests, and forgot about un-hardcoding my dummy test values. The result was meaningless tests. Yikes. ↩
How often is it not, lol. ↩
And maybe that's what makes mutation testing hard to sell. It doesn't flatter you. It shows you the difference between looking tested and being tested. Between surface coverage and real confidence. ↩

The Curse of Knowing How, or; Fixing Everything

Thu, 24 Apr 2025 00:00:00 GMT

import Banner from "@components/Banner.astro";

It starts innocently.

You rename a batch of files with a ten-line Python script, or you alias a common git command to shave off two keystrokes. Maybe you build a small shell function to format JSON from the clipboard.

You're not even trying to be clever. You're just solving tiny problems. Making the machine do what it should have done in the first place. And then something happens. You cross a threshold. You look at your tools, your environment, your operating system---even your editor---and suddenly everything is fair game.

You could rebuild that (if you wanted to).
You could improve that (if you wanted to).

Then someone challenges you. As banter maybe, perhaps jokingly but also with a dash of hope. Then the air in the room changes.

It suddenly becomes something else. It becomes:

You should.

And from that moment forward, the world is broken in new and specific ways that only you can see.

Technical Capability as a Moral Weight

Before I could program, broken software was frustrating but ignorable. For years I've simply "used" a computer, as a consumer. I was what companies were concerned with tricking into buying their products, or subscribing to their services. Not the technical geek that they prefer to avoid with their software releases, or banning from their games based on an OS.

Now it has become provocative. I can see the patterns that I wish I couldn't, find oversights that I can attribute to a certain understanding (or the lack thereof) of a certain concept and I can hear what has been echoing in the head of the computer illiterate person who conjured the program I have to debug.

I notice flaws like a good surgeon notices a limp.
Why the hell does this site send ten megabytes of JavaScript for a static page?
Why is the CLI output not parseable by awk?
Why is this config hardcoded when it could be declarative?

Those things are not just questions, they are accusations. And, unfortunately, they do not stop.

Now that I've learned to notice, my perception of software has changed in its entirety.

Every piece of software becomes a TODO list.
Every system becomes a scaffolding for a better one.
Every inconvenience becomes an indictment of inaction.

One Must Imagine Sisyphus Happy

Like Camus' Sisyphus, we are condemned to push the boulder of our own systems uphill---one fix, one refactor, one script at a time. But unlike the story of Sisyphus, the curse is not placed onto you by some god. We built the boulder ourselves. And we keep polishing it on the way up.

I've lost count of how many projects I have started that began with some variation of "Yeah, I could build this but better."

A static site generator because the existing ones had too many opinions.
A note-taking tool because I didn't like the way others structured metadata.
A CLI task runner because Make is cryptic and Taskfile is YAML hell.
A personal wiki engine in Rust, then in Go, then in Nim, then back to Markdown.
A homelab dashboard because I don't like webslop.

The list continues, and trust me it does continue. My dev directory, as it stands, is nearing 30 gigabytes.

If you ask me, I was solving real, innocent problems. But in hindsight, I was also feeding something else: a compulsion to assert control. Every new tool I built was a sandbox I owned: No weird bugs. No legacy constraints. No decisions I didn't entirely agree with. Until, of course, I became the legacy.

Kafka once wrote that "a cage went in search of a bird." ¹ That is what these projects can become. Empty systems we keep building, waiting for purpose, for clarity, for... salvation? I'm not sure how else would you call this pursuit.

Entropy Is Undefeated

Now let's go back. Back to when we didn't know better.

Software doesn't stay solved. Every solution you write starts to rot the moment it exists. Not now, not later, but eventually. Libraries deprecate. APIs change. Performance regressions creep in. Your once-perfect tool breaks silently because libfoo.so is now libfoo.so.2. ²

I have had scripts silently fail because a website changed its HTML layout.
I have had configuration formats break because of upstream version bumps.
I have had Docker containers die because Alpine Linux rotated a mirror URL.

In each case, the immediate emotional response was not just inconvenience but something that moreso resembles guilt. I built this, and I do know better. How could I not have foreseen this? Time to fix it.

If you replace every part of the system over time, is it still the same tool? Does it still serve the same purpose? Do you?

The Illusion of Finality

I think we lie to ourselves.

"If I just get this setup right, I'll never have to touch it again."
"If I just write this one tool, my workflow will be seamless."
"If I automate this, I'll save time forever." ³
"Write once, run everywhere." My ass.

It is, I admit, a seductive lie. It frames programming as a conquest of sorts. A series of battles you win, or challenges you complete. But the imaginary war never ends. You don't build a castle. You dig trenches. And they flood every time it rains. The trials are never complete.

Technical Work as Emotional Regulation

On the theme of filling this post with literary references, let me quote the Stoic Marcus Aurelius.

You have power over your mind--not outside events. Realize this, and you will find strength.

But programming lures us into believing we can control the outside events. That is where the suffering begins. There is something deeper happening here. This is not just about software.

I believe sometimes building things is how we self-soothe. We write a new tool or a script because we are in a desperate need for a small victory. We write a new tool because we are overwhelmed. Refactor it, not because the code is messy, but your life is. We chase the perfect system because it gives us something to hold onto when everything else is spinning. This is the lesson I've taken from using NixOS.

I have written entire applications just to avoid thinking about why I was unhappy. Programming gives you instant feedback. You run the thing, and it works. Or it doesn't, and you fix it. Either way, you're doing something.

That kind of agency is addictive. Especially when the rest of life doesn't offer it. We program because we can, even when we shouldn't. Because at least it gives us something to rebel against.

The Burnout You Don't See Coming

Burnout does not just come from overwork. It comes from overresponsibility.

And programming, once internalized deeply enough, makes everything feel like your responsibility. The bloated website. The inefficient script. The clunky onboarding process at your job. You could fix it. So why aren't you?

The truth you are very well aware of is that you can't fix it all. You know this, you always knew it regardless of your level of skill. But try telling that to the part of your brain that sees every inefficiency as a moral failing.

Nietzsche warned of gazing too long into the abyss. But he did not warn what happens when the abyss is a Makefile or a 30k line of code Typescript project.

Learning to Let Go

So where is the exit? Is this akin to Sartre's depiction of hell, where hell is other people and how they interact with your software? Or is it some weird backwards hell where people create software that you have to interact with?

The first step is recognizing that not everything broken is yours to fix.
Not every tool needs replacing.
Not every bad experience is a call to action.

Sometimes, it's OK to just use the thing. Sometimes it's enough to know why it's broken---even if you don't fix it. Sometimes the most disciplined thing you can do is walk away from the problem you know how to solve. There's a kind of strength in that.

Not apathy, no. Nor laziness. Just... some restraint.

A New Kind of Skill

What if the real skill isn't technical mastery? Or better yet what if it's emotional clarity?

Knowing which problems are worth your energy.
Knowing which projects are worth maintaining.
Knowing when you're building to help—and when you're building to cope.
Knowing when to stop.

This is what I'm trying to learn now. After the excitement. After the obsession. After the burnout. I'm trying to let things stay a little broken. Because I've realized I don't want to fix everything. I just want to feel OK in a world that often isn't. I can fix something, but not everything.

You learn how to program. You learn how to fix things. But the hardest thing you'll ever learn is when to leave them broken.

And maybe that's the most human skill of all.

Post-Mortem

As of 6th of May, this post has blown up on Hackernews. First of all, thank you all for your support and the kind words that I have received on various platforms. This post was not written with any other intention than getting some things off my chest and off my mind, but it was humbling to see that it resonated with many different people of various origins. What surprised me the most was that it seems to resonate people outside the field of tech as well, so it seems the problem is not as exclusive to something as "arcane" as programming. Thank you also to those who were kind enough to point out small technical flaws in the site that I now feel compelled to fix, and those who pointed out small typos I've forgotten to push after storyboarding.

It was also quite valuable in the sense that I was able to see my webserver configuration is in fact able to handle lots of traffic. There was no downtime aside from a small hiccup after I messed up a configuration option while trying to relax my ratelimits to give the influx of readers a nicer experience.

Lastly, thank you all that left their insight about the post itself (my writing, etc.) as comments on Hackernews or lobste.rs. I have no doubt missed some, but know that it matters a lot to me. Of course, feel free to contact me directly via e-mail (or any other method you prefer as detailed in the about page) to let me know what could be better.

Thank you, and all the best!

--raf

From, I believe, Kafka's The Zurau Aphorisms. ↩
Nix solves this. Or does it? Nix was a can of worms of its own. ↩
Remember when you spent 2 hours automating a 30 minute task? Yeah, it's that again. ↩

Considerations

Mon, 07 Apr 2025 00:00:00 GMT

I am a sick man... I am a spiteful man. I am an unattractive man. I believe my liver is diseased. However, I know nothing at all about my disease, and do not know for certain what ails me. I don’t consult a doctor for it, and never have, though I have a respect for medicine and doctors. Besides, I am extremely superstitious, sufficiently so to respect medicine, anyway (I am well-educated enough not to be superstitious, but I am superstitious). No, I refuse to consult a doctor out of spite. That you probably will not understand. Well, I understand it, though. Of course, I can’t explain who it is precisely that I am mortifying in this case by my spite: I am perfectly well aware that by all this I am only injuring myself and no one else. But still, if I don’t consult a doctor, it is out of spite. My liver is bad; well—let it get worse!

The first page of Dostoyevsky's Notes From Underground has always been fascinating to me. Not because I fully relate to this sick, spiteful, unattractive man, but because I share a sentiment with him: spite. I have always carried it with me, as long as I have been self-aware. I do things when I'm spiteful; I do things because I am spiteful, and it usually becomes a strange mixture of emotions, actions, and sentiments that gets me results.

I am in pain. I have been for as long as I can remember. My leg hurts, my hand hurts, and now my neck hurts. Age? Sports injury? Your guess is as good as mine. Yet despite all that, in spite of the pain, I prevail. I walk, I run, I lift, I write, I draw, and many more. It hurts? Sure, let it get worse.

One of the first things I have learned was trust---or rather, mistrust---shortly after spite. It was fairly obvious. Simply figuring out ways I can do things myself is more cost effective. Simply put, I don't trust. Not anyone but myself. Sure, I will share responsibilities some of the time if I deem the cost insignificant, but when there are stakes that matter, then I am alone. Which is usually always. Surrounded by people whom I love, but alone, nevertheless. Spiteful, in pain, and alone.

Despite all of that, I prevail. I am successful, I have accomplished things that people my age have not even dreamed of, and I should probably be proud of this. I'm not. I feel disappointed, but not unhappy. Never unhappy. How I feel cannot be described as unhappy, or depressed, or anything of that sort. Neither happy nor unhappy. Disappointed, at best. Today was a normal day, nothing out of the ordinary. Yet I feel overwhelmed by disappointment more than ever. Acceptance, perhaps, of the fact that it is meaningless. Disappointed that knowing how things will end up in 5 years, 10, pushing onwards as if there is something on the horizon. Does it matter? I don't know, but I'm not particularly sure if I care either. I want to give up, with my whole being. I have never been more sure of something. There is not a doubt in my mind that I want to leave everything behind. And go where? I don't know. I don't care. Yet I know that even if I did that, even if I left everything and everyone behind, the ideas and memories would continue to haunt me as if nothing happened. Such a cost for nothing in return.

There are many things I have invested in. Not in a way that calls for sunk costs; I invested willingly, and I don't regret any of it. Not one bit. It is not like my investments were for nothing either. I have invested my time, my efforts, and have gained trust in return. Yet, despite it being my choice in the first place, I feel powerless to betray the trust people have put in me, a trust I have never shown them. A little funny, perhaps narcissistic. I should be happy, right? That I was able to gain the trust of people who value me as a friend, a mentor, or whatever I may be to them. But I don't feel happy. I feel trapped. The idea of betraying this trust becomes my worst nightmare. If I were to describe this, I would ask you to imagine gasping for air in between choking on water while you are drowning. You know what's going to happen. You know what's next. Yet every single cell in your body screams and pushes for one more moment of survival. Every nanosecond becomes minutes until it ends.

My life was even then gloomy, ill-regulated, and as solitary as that of a savage. I made friends with no one and positively avoided talking, and buried myself more and more in my hole.

I enjoy solitude, possibly more than anything else. More than the company of people I love. It is liberating---not subconsciously overanalyzing every expression or making small predictions based on clues within a conversation—not something I do willingly. In fact, for a while now, my dream has been to get my things and live in the middle of nowhere with not a single soul around. Not one. I do not care for a warm greeting, I do not care for any appreciation. If we are going to coexist, let us do so quietly and peacefully. There is not much I can ask for. Yet it is not what is going to make me not feel whatever this feeling is called. I find myself yearning for something. It is not solitude, nor company. It is not some kind of big win that will lift my spirits, and it is not salvation. I want an end. It is beginning to feel awfully like a boring movie that I have seen, and I am dying for it to end. In fact, I spite every second of it. In fact, I sometimes think I thrive under discomfort. My spite for it makes me take one more step, just to spite some imaginary enemy. Laughable, isn't it?

I would be remiss, however, if I did not point out the irony of it all. This is all of my own making. I chose to want the things I wanted, do the things I did. I built a prison, got inside, and locked myself in. I could get out whenever I wanted to, but I don’t want to be wanting to escape. I want to know that I can simply curl up somewhere and rot away with everything moving on around me. I want it so that I can simply give up without the consequences nagging me. Let me stop grasping for air, and I will be at peace. Though chances are, nothing is going to change. I am full of spite, regardless of how I feel at any given moment, and it's not long before I find something to direct it toward. There will be something to pique my interest and distract me from the fact that I have not signed up for any of this. That will be the way things are for a while, and then back to drowning. Each time with more water in my lungs than the last, each time more aware of the inevitable. This is how I have been doing things for a while now, and it is probably going to remain that way until the end of time. Why do I even bother? Let it get worse.

This has been a piece much outside of my usual pace. Truth be told, I am writing to clear the thoughts in my mind. I don't want them; maybe you do. If you don't, well, then you should have stopped reading earlier. How about that?

My New Tech Stack

Thu, 03 Apr 2025 00:00:00 GMT

I have began programming with Javascript back in 2018. It was a spontaneous decision, made mostly in the hope of solving a basic problem that a game server that I moderated was facing. Long story short, I learned Javascript (or at least, some Javascript) over night and that was it for a while.

That somehow propelled me head-first into web development. I was young, I wanted a personal website and maybe I could even make a few bucks on the side from developing websites for people. Since I knew Javascript somewhat decently, I began looking at HTML and CSS, read some web blogs and a few programming books that describe the basics of web development and went on with my journey.

Alongside other experiences, this lead me to use Linux as my main operating system and one of the most important reasons why I chose Linux was because it was a liberating, customizable experience. As I moved to use tiling window managers (i3 at the time), I also began experimenting with writing my own desktop utilities. Learn Python, Go, C, little bit of C++ and eventually Rust; which is now most of my stack.

While my programming journey has taken a winding road through multiple languages, each offering a unique set of strengths and trade-offs. Recently, I made the decision to transition from Golang to Rust while still retaining C in my toolbox. This post details the technical reasoning behind this move, the advantages of each language, and why, despite its venerable status, C's build tooling still leaves much to be desired.

The Golang Chapter

Go was... a compelling choice to say the least. Be it for its simplicity, powerful concurrency model and rapid compilation times (which I miss to this day in Rust.) Its robust standard library and clean syntax allowed me to build distributed systems with ease. However, as projects grew in complexity, several limitations became apparent.

One very large issue was memory management. I want my software to be lean and mean but Go's garbage collection was imposing unpredictable latencies---an issue in performance-sensitive applications that are the backbone of my home network. For example, in a real-time data processing application, garbage collection pauses caused noticeable delays. Benchmarks like goperf have shown that garbage collection pauses can sometimes exceed acceptable limits, making Go much less suitable for low-latency requirements. The type system is also, despite its simplicity, a hit-or-miss and it can can feel limiting when dealing with more intricate, low-level abstractions or when requiring fine-grained control over data. Lastly, the overhead introduced by some of Go's abstractions are a bottleneck that I've grown a distaste for during my time using Nix.

These aspects led me to explore alternatives that offer more control without sacrificing safety or performance.

Why Rust?

I didn't switch to Rust because of hype. If anything, the constant evangelism around "memory safety" as if it's the only thing that matters was a turnoff. But Rust kept solving real problems I hit when working in Go and C, to the point where using anything else felt like unnecessary friction. It emerged as a natural next step. It is widely used, and even more widely shilled. While I was using Go, I had experimented with Rust several times but extremely long build times---alongside the complex type system---drove me off each time. Eventually I accepted those as a cost for correctness, and began migrating to Rust. Although, it was a slow process. The Rust book was extremely unhelpful, especially in the pacing department, but eventually and through the help of many web searches I accepted Rust. It brings several improvements that address the limitations I experienced with Golang, which I would like to go over.

1. Memory Safety without Performance Penalties

First advantage I'm quite fond of is Rust's memory safety without the need for a garbage collector. Rust's ownership model ensures memory safety at compile time without relying on a garbage collector. This allows for predictable performance, a very crucial factor that affected my decision. The borrow checker, while initially imposing a learning curve, provides a safety net that prevents data races and dangling pointers. Unlike Go, where garbage collection kicks in unpredictably, Rust's compile-time borrow checker also ensures that memory is allocated and freed at the right time. Unlike C, which is another language I enjoy writing, but not so much *using, it doesn't require explicit malloc/free calls or reference counting in most cases, meaning it balances control and automation far better than either language.

Instead of garbage collection, Rust relies on RAII (Resource Acquisition Is Initialization), where memory is freed as soon as it goes out of scope. Consequently the borrow checker ensures safe aliasing of memory, use-after-free bugs, and iterator invalidation. Nice. I want to control my memory, without having to micromanange it.

2. True Zero-Cost Abstractions

Golang's philosophy is "simple is better." The problem is that simplicity in Go often means "dumbed down." No proper generics (until recently), no inlining across package boundaries, and everything is done through interfaces that introduce runtime overhead. As such, a common issue I had with Go is that writing high-level, ergonomic code often meant sacrificing performance. Interfaces, reflection, and even some seemingly simple abstractions introduce overhead that isn't always obvious (or sometimes not even available because... I don't know.) Rust does not have this problem because of its zero-cost abstraction principle: abstractions should compile down to the most efficient possible machine code.

For example, iterators in Rust provide the same expressiveness as Go's slices but compile into raw loops without runtime overhead. Generics in Rust are fully monomorphized at compile time, meaning there's no boxing or interface dispatch cost. This means Rust allows writing expressive, high-level code while maintaining (mostly) C-level performance. I have learned recently that Rust compiles down to efficient machine code thanks to monomorphization (try saying that out loud 3 times in a row)---a process where generics are fully expanded at compile time, allowing optimizations that eliminate runtime dispatch overhead. Combined with LLVM's optimizations, Rust often generates assembly that is as fast as, or faster than, handwritten C.

3. Concurrency Without Race Conditions

Goroutines and channels are nice, but they don't prevent data races. In Go, you still need to rely on locks, atomics, etc, and most importantly careful design to avoid issues like race conditions or deadlocks. Rust enforces safety at the type level. This means that Rust code, once compiled, has mathematically guaranteed thread safety. No surprises, no debugging obscure race conditions in production. The Send and Sync traits act as static checks that prevent unsafe sharing of resources, something that Go's runtime cannot enforce. Rust's async ecosystem, using async/await and tools like Tokio, allows for excellent asynchronous programming without the pitfalls of traditional thread-based concurrency models.

4. Tooling That Actually Works

C's build tooling is archaic, Go's module system is still a mess (GOPATH was awful, and modules are only marginally better), but Rust? Cargo just works. Dependencies, versioning, cross-compilation---it's all built-in, and I don't have to waste time wrestling with third-party package managers or Makefiles. Plus, tools like clippy, rustfmt, and cargo bench integrate seamlessly. Cargo workspaces make it easy to manage multi-crate projects, allowing for better organization and dependency management. Honestly, it's impeccable overall.

Considering Zig

One thing worth nothing about C is that I enjoy writing it. The design of the language, unlike C++, is simple and it's fun to write. I find Rust fun too, but sometimes type errors get on my nerves because I appear to get caught by the most basic ones that I probably wouldn't encounter on C. And while Rust has largely taken over my system-level programming needs, I have been keeping an eye on Zig. Zig's simplicity, lack of hidden control flow, and compile-time execution capabilities make it an interesting alternative to both C and Rust in certain contexts. Though I find it still a bit immature to really "switch" to as my primary language. For now Rust will do, but I still feel compelled to mention what Zig does well.

Some of Zig's strengths include:

Manual memory management without footguns – Unlike C, Zig provides safer manual memory management patterns.
Better C interoperability than Rust – While Rust has bindgen and cbindgen, Zig is designed from the ground up to interface smoothly with C code.
No hidden control flow – No automatic panics, exceptions, or surprises in the compiled output.

I'm not replacing Rust with Zig anytime soon, but for cases where Rust's safety mechanisms feel like overkill, or where I need tighter control over binary size and startup time, Zig looks promising. It's too early to say if it'll take a permanent place in my workflow, but I am keeping an eye on it.

Why C Still Has a Place

Despite moving to Rust, C continues to hold a vital place in my workflow. Its low-level access and direct mapping to hardware make it indispensable for performance-critical tasks, interfacing with legacy systems, or working on projects where every cycle counts. However, there are areas where C leaves a lot to be desired.

Build Systems Are a Mess – Make, CMake, Autotools—none of them are great, and package management is nonexistent.
Undefined Behavior Everywhere – The sheer number of ways you can shoot yourself in the foot in C is staggering. Buffer overflows, use-after-free, integer overflows—Rust eliminates these entirely.
Lack of Modern Language Features – No generics, no proper modules, and macros are still the hacky preprocessor-based mess they have always been.

That said, C isn't going anywhere. There's too much legacy code, too many embedded systems, and too many low-level performance demands for it to disappear. If you want to write kernels, firmware, or low-level graphics code, you still want C, though you don't necessarily need it. I find Rust's bootstrappability to be a drawback, but that's for another post.

Go Has Replaced Python for Web and Scripting

For quick scripting, Go has taken over Python's role in my workflow, especially for anything web-related. Python's performance is atrocious, and while Flask is easy to use, it crumbles under load. Go, on the other hand, compiles to a single binary, has a solid HTTP stack, and doesn't require setting up a virtual environment just to run a script. The net/http package provides a robust and easy-to-use HTTP server and client, while frameworks like gorilla/mux offer more advanced routing capabilities.

That's not to say Go is perfect---its standard library is missing some utilities that Python has had forever---but for quick-and-dirty web tools, it's become my go-to and I'll take Go over Python any day. Though the bar is low, and Go is barely doing anything to remain above it.

Final Thoughts

Transitioning to Rust did not mean discarding Golang or C entirely. Each language serves its purpose:

Rust now leads my efforts in building safe, efficient, and modern systems.
C continues to be my go-to for low-level programming tasks, where control over hardware is paramount, despite its antiquated tooling. I usually try to work in single files and avoid libraries when I'm working with C, which is a good learning experience on how to build stuff but still annoying.
Golang still finds its place in projects where rapid development and simplicity are necessary, though its limitations have driven my current focus away.
Zig is an emerging contender that I may use for certain low-level applications where Rust or C might otherwise be considered.

Long thing short, Rust isn't for everything, and neither is Go or C. Each language has its niche, and the trick is knowing when to use what. Rust replaced Go for me in performance-sensitive areas, but I still keep Go around for web scripts and C for low-level work. Zig? Maybe it'll earn a place too. Time will tell.

NixOS Testing Framework I: On VM Tests

Thu, 03 Apr 2025 00:00:00 GMT

One of the things that convinced me to stay on NixOS was how easy it is to write integration tests for my existing infrastructure. Unlike traditional testing setups, which often require complex tooling, manual configuration, or fragile dependencies, NixOS makes testing nearly effortless. In fact, Nixpkgs itself contains tests for even the most trivial cases---proving just how seamless the process can be. If you've ever struggled with testing your services in other environments, where dependencies shift and system configurations break unpredictably, then this post might convince you to give NixOS a second look.

Today's article walks you through setting up your own integration tests outside the context of Nixpkgs. I'll also briefly touch on how tests are executed within Nixpkgs itself. Not long ago, the testing framework underwent some internal changes, and now that testers.runNixOSTest is stable, I believe it's the perfect time to explore how you can leverage it for your own projects.

Obtaining Nixpkgs

We must first obtain the testers from somewhere. That somewhere is--- naturally---Nixpkgs.

Testers are system dependent, and they are dependant on pkgs. I don't think this is a surprise to anyone since we will be interacting with, well, packages but the caller for runNixOSTest especially sets hostPkgs.

That said, we will need to instantiate pkgs somewhere. Usually in Nixpkgs you are already in a scope that provides testers, but since we are working outside Nixpkgs for the purposes of this post, let's get pkgs rolling. While I'm at it, I'll also provide a self-contained flake.nix that fetches and locks a Nixpkgs instance for us.

# flake.nix
{
  inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=nixpkgs-unstable";
  outputs = inputs: {
    # No import required unless you want to propagate `inputs` to the module
    nixosModules.default = ./your-service.nix;
  };
}

Let's assume for the purposes of this post that you have a small NixOS module that you want to execute tests for. This can be a short and simple Python script for a webserver, a VPN mesh controller, or anything that you might have expressed in a simple NixOS module. Let's also assume that whatever this service might be, it serves a HTML page with... healthcheck information over at port 3000. Got anything in your mind yet?

Writing the Test

One of the many handy features of flakes is that you can define checks that will be built on nix flake check unless --no-build is passed. While I'm testing outside resource constrained environments, I prefer to execute my VM tests directly on check because that is two birds with one stone.

Let's extend the flake.nix from above with a check that will be built on nix flake check or manually with nix build .#checks.<system>.your-check. Since this is meant to serve as an example, I have omitted the system abstractions and we will be testing for a single system only. You may change this however as you see fit. The key point here is to define a 'package' using runNixOSTest that will be executed in a way that you see fit.

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=nixpkgs-unstable";
  outputs = inputs: {
    nixosModules.default = ./your-service.nix;

    checks = let
      system = "x86_64-linux";
      pkgs = inputs.nixpkgs.legacyPackages.${system};
    in {
      "${system}".default = pkgs.testers.runNixOSTest { /*...*/ };
    }
  };
}

Here is how our flake will look like, for now. I want to explain the potential arguments to runNixOSTest to give you an idea before I write the rest of the test to give you an idea before I write the rest of the test.

Anatomy of `runNixOSTest`

runNixOSTest is a wrapper function around runTest from Nix_OS_ (not nix_pkgs_) library, defined in nixos/lib. It imports nixos/lib/default.nix and wraps the runTest function that is defined in NixOS library, which is inherited from the testing library inside the NixOS library inside nixpkgs... Ugh.

Regardless, and in the spirit of runTest, there are a few parameters that you can pass to runNixOSTest. I will not be covering all of them, but here are those that you want to pass for a functional test.

runNixOSTest {
  # First, the 'name' parameter. This will determine the name of the package
  # that you will be building, but from what I can tell, not much else. For
  # example a test with the name "my-test" would result in "vm-test-run-my-test"
  # being built.
  name = "my-test";

  # Next up is nodes. This is where it gets interesting, because you can define
  # as many nodes as you deem necessary in this set. More importantly, those
  # nodes are allowed to interact with each other inside the texting context.
  nodes = { /* ... */ };

  # Last but not least, the test script. The test script is what I would call
  # a *flavored* Python script, where you get access to a few special machine
  # objects as a part of Nixpkgs, and the documentation describes it as a
  # "...sequence of Python statements that perform various actions, such as
  # starting VMs, executing commands in the VMs and so on."
  testScript = "";
}

There are a few other parameters that you can use, such as extraPythonPackages that you can use to add additional Python libraries in the script. Though I'm getting ahead of myself, we're here to talk about writing tests.

Writing the Test: Continued

As I was saying. Now that we know how a test looks like, lets write it for real this time.

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=nixpkgs-unstable";
  outputs = inputs: {
    nixosModules.default = ./your-service.nix;

    checks = let
      system = "x86_64-linux";
      pkgs = inputs.nixpkgs.legacyPackages.${system};
    in {
      "${system}".default = pkgs.testers.runNixOSTest {
        # This can be anything. I like giving my test special
        # codenames. Yes, I'm quirky like that.
        name = "narwals-are-awesome";

        # Let's go with a single-node test, for now. I'm going to add an example
        # at the end of this post to give you an idea of multi-node tests.
        nodes = {
          # Names of the nodes are up to you as well. They expose the same
          # objects regardless of names, though 'machine' is mostly standard, so
          # I will go with that for now. As a bonus, you can read this with the
          # voice of Gianni Matragrano, who is said to voice anyone for a
          # chicken nugget.
          machine = {pkgs, ...}: {
            imports = [ inputs.self.nixosModules.default ];
            environment.systemPackages = [ pkgs.curl ];

            # Enable the service. This does not exist in Nixpkgs and
            # is added by our nixosModule. You can enable anything defined in
            # nixpkgs inside a machine's configuration *and* extend them with
            # your own NixOS modules.
            services.syshc.enable = true;
          };
        };


        # Now the test script. You can add a comment like /* python */ before
        # the body of this string to provide syntax highlighting via Treesitter
        # if your editor is Neovim. Neat!
        # The script will start all available nodes, wait for the service to
        # start, and then the port. Lastly it will query the / endpoint to look
        # for a specific value that indicates success. Your test cases may be
        # more complex than this, in which case you can write a more detailed
        # Python script.
        testScript = /* python */ ''
          # Function to start *all* nodes at once. This can be used as an
          # alternative to <nodename>.start() (e.g. `machine.start()`) when
          # you have multiple nodes.
          start_all()

          # wait_for_unit is a special object that will wait for a Systemd
          # unit to get into "active" state. Throws exceptions on "failed"
          # and "inactive" states as well as after timing out.
          machine.wait_for_unit("python-server")

          # Also wait for an open port on the node. In my script the service
          # binds to port 3000, so we must wait for it to open.
          machine.wait_for_open_port(3000)

          # Finally lets log the output from the service. In my example the
          # server logs health information directly in /, but you may get the
          # information from anywhere.
          status = machine.succeed("curl --fail localhost:3000")

          # Check if our server returns the expected result.
          assert "Healthy" in status, f"'{status}' is not healthy! Check failed."
        '';
      };
    };
  };
}

Running Your Test

Now lets build the check with some verbosity. -L will tell the tester to print all logs from the test, and v adds some verbosity to the Nix builder.

nix flake check -Lv

And this will print a lot of logs. Let's isolate the output that we really care about.

vm-test-run-narwals-are-awesome> (finished: waiting for TCP port 3000 on localhost, in 18
vm-test-run-narwals-are-awesome> machine: must succeed: curl --fail http://localhost:3000
vm-test-run-narwals-are-awesome> machine #   % Total    % Received % Xferd  Average Speed
vm-test-run-narwals-are-awesome> machine #                                  Dload  Upload
vm-test-run-narwals-are-awesome> machine #   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[   34.854829] syshc[637]: 127.0.0.1 - - [04/Apr/2025 09:49:17] "GET
 / HTTP/1.1" 200 -
vm-test-run-narwals-are-awesome> machine # 100   115  100   115    0     0    108      0  0:00:01  0:00:01 --:--:--   108100   115  100   115    0     0    107      0  0:00:01  0:00:01 --:-
-:--   107
vm-test-run-narwals-are-awesome> (finished: must succeed: curl --fail http://localhost:3000, in 1.43 seconds)
vm-test-run-narwals-are-awesome> (finished: run the VM test script, in 36.99 seconds)
vm-test-run-narwals-are-awesome> test script finished in 37.09s
vm-test-run-narwals-are-awesome> cleanup

And there it is. This waits for port 3000, curls it, gets the result and parses it. Since the server returned "Healthy", the check has completed successfully. Nice.

Closing Thoughts

Hope this article has served to give you an idea of how you may utilize NixOS testing framework for your projects. At least a very basic idea, enough to help get you started. Though there is much more to this framework, and more conditions that you might want to be aware of.

As such, I invite you to reach out to me and let me know of any additional cases that you want me to cover. I am happy to help clarify testing with NixOS, as I think it should be done more often outside of Nixpkgs. Moreover, this is not the end of this topic. I would like to cover the aforementioned special objects, and more complex cases surrounding multi-machine test scenarios. I also want to go over interactive tests, but that will have to be for another post. Cheers

What is nixConfig, Should You Trust It?

Mon, 31 Mar 2025 00:00:00 GMT

Those of you who have been a part of the Nix ecosystem, more specifically the community that often deals with flakes,¹ have no doubt noticed or perhaps actively use the nixConfig attribute in a flake.nix. For those unfamiliar, the NixOS wiki in the article on flakes defines it as follows:

nixConfig is an attribute set of values which reflect the values given to nix.conf. This can extend the normal behavior of a user's nix experience by adding flake-specific configuration, such as a binary cache.

Sounds good, right? Wrong. It is a ticking time bomb waiting to explode. Being able to modify your nix.conf on your system is equivalent to having full control of the Nix daemon. When you check out a repository that uses flakes and run, say, nix run on a package, you will get a prompt asking whether you trust the configuration settings set in the flake.

$ nix run .#foobar
do you want to allow configuration setting 'extra-substituters' to be set to 'https://nix-community.cachix.org' (y/N)? y
do you want to permanently mark this value as trusted (y/N)? y

Accept it, and the door is wide open. More technical users may object, saying they will know the changes being made to their system. They are not wrong, but if that is their argument, then they are incredibly dense.

Security Implications

You should be fully aware that changes made through nixConfig will affect all Nix operations within the flake, possibly increasing the attack surface through the introduction of unsafe, unsigned, and malicious binary caches. It can also introduce allowUnfree, which might cause ideological or legal (see: licensing) issues depending on the context.

nixConfig is a powerful option that can modify settings you do not want changed haphazardly—from package sources to substitution and trusted keys your build process will use. Oftentimes, this option is more of a hassle than a convenience, though some might find it incredibly useful.

There exists an accept-flake-config option that you can set as nix.settings.accept-flake-config. Keep this set to false, as automatically accepting those options---without the prompt above—is more insecure than you think. There are many vulnerabilities that can come from blindly trusting a flake's nixConfig.

If you want to go a step further, I have been running the following patch by the awesome @eclairevoyant to add a reject-flake-config option to Lix, the Nix fork, to automatically reject flakes' nixConfig.

From 25f1b8e714b13d2aa6fcdc67bedf1544bd17e45a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C3=A9clairevoyant?= <848000+eclairevoyant@users.noreply.github.com>
Date: Fri, 19 Jul 2024 09:23:27 -0400
Subject: [PATCH 3/4] feat: option reject-flake-config

---
src/libexpr/flake/config.cc       | 5 +++++
src/libfetchers/fetch-settings.hh | 4 ++++
2 files changed, 9 insertions(+)

diff --git a/src/libexpr/flake/config.cc b/src/libexpr/flake/config.cc
index 558b3e9b9..bf558e5e2 100644
--- a/src/libexpr/flake/config.cc
+++ b/src/libexpr/flake/config.cc
@@ -51,6 +51,11 @@ void ConfigFile::apply()
         else
             assert(false);
+        if (nix::fetchSettings.rejectFlakeConfig) {
+            warn("ignoring untrusted flake configuration setting '%s' due to the '%s' setting.", name, "reject-flake-config");
+            continue;
+        }
+
         bool trusted = whitelist.count(baseName);
         if (!trusted) {
             switch (nix::fetchSettings.acceptFlakeConfig.get()) {

diff --git a/src/libfetchers/fetch-settings.hh b/src/libfetchers/fetch-settings.hh
index 93123463c..67f2e4d14 100644
--- a/src/libfetchers/fetch-settings.hh
+++ b/src/libfetchers/fetch-settings.hh
@@ -108,6 +108,10 @@ struct FetchSettings : public Config
        )",
        {}, true, Xp::Flakes};

+    Setting<bool> rejectFlakeConfig{this, false, "reject-flake-config",
+        "Whether to reject nix configuration (including whitelisted settings) from a flake without prompting.",
+        {}, true, Xp::Flakes};
+
    Setting<std::string> commitLockFileSummary{
        this, "", "commit-lockfile-summary",
        R"(

Give it a try if you are running Lix and don't mind the rebuilds. This will reject the flake config, skipping the prompt to accept the values set in nixConfig.

Looking into nixConfig

Now that the ultimatum is out of the way, let's take a look at what nixConfig is and what it really does. As mentioned above, it is an attribute set of values that reflect the values given to nix.conf. That in itself is a vague explanation, but it gets the point across: it is an attribute set that takes the same values you would pass to nix.settings, which are used to construct the nix.conf in a typical NixOS configuration. Or, if you are using Nix standalone, it is an attribute set of values that would be converted to their nix.conf equivalents.

It is defined and used in libflake/flake/config.cc, and according to the getDataDir() function, this retrieves the full path of Nix's data directory (one of NIX_DATA_DIRECTORY, XDG_DATA_HOME, or $HOME/.local/share/nix/ in that order) and creates trusted-settings.json.

Nice. Now we know where to look if we accidentally accept the prompt or ever want to retract our blind trust in a flake's author.

Conclusion

Other than this path, there is not much to nixConfig. Your system does not interact with it until you enter the directory of a flake and run a command that triggers evaluation, in which case ConfigFile::apply is invoked.

In short, nixConfig is a powerful attribute that may or may not come in handy at the cost of exposing a very large attack vector. Even a lazy attacker can exploit it, and the user is not properly warned about the consequences of accepting the prompt (which defaults to true for some reason).

This has been your ever-so-informative technical rant on Nix and its undocumented mess. Hope you learned something today. Cheers.

If you don't use Nix flakes or have no idea what they are, then this post does not apply to you. In fact, I'm glad you're saving yourself the headache! ↩

Building Deterministic Typst Packages with Nix

Mon, 17 Mar 2025 00:00:00 GMT

Recently, I encountered a Typst Builder PR in Nixpkgs that implements a Typst builder, similar to other language-specific builders you might be familiar with.

While I still use Pandoc and Markdown for many tasks (including this blog), I've been using Typst extensively recently, and that got me excited to explore the new builder(s).

To test them, and possibly demonstrate their functionality, I've prepared the following flake.nix:

{
  inputs.nixpkgs.url = "github:cherrypiejam/nixpkgs?ref=typst";
  outputs = {nixpkgs, ...}: let
    systems = ["x86_64-linux"];
    forEachSystem = nixpkgs.lib.genAttrs systems;

    pkgsForEach = nixpkgs.legacyPackages;
  in {
    packages = forEachSystem (system: {
      inherit (pkgsForEach.${system}) buildTypstPackage typstPackages;
    });

    devShells = forEachSystem (system: let
      pkgs = pkgsForEach.${system};
    in {
      default = pkgs.mkShellNoCC {
        # packages provided in 'nix develop'
        packages = [
          (pkgs.typst.withPackages (ps:
            with ps; [
              polylux
              cetz_0_3_0
            ]))
        ];
      };
    });
  };
}

This flake re-exports buildTypstPackage and typstPackages from nixpkgs and uses the new typst.withPackages scope to create a deterministic Typst environment. Its usage, though subject to change, appears straightforward.

buildTypstPackage produces a "manifest" of available Typst packages from Typst Universe, which you can use within typst.withPackages's scope. For example:

typst.withPackages (ps:
 with ps; [
    polylux
    cetz
 ])
);

This allows you to use, for instance, #import "@preview/cetz:0.3.0: \*" as cetz is added to Typst's search path via the TYPST_PACKAGE_CACHE_PATH variable set to, e.g., /nix/store/02pgr69nc5lczi9rrh2xd40s1mzqrvkl-typst-0.13.1-env/lib/typst/packages, populated with the packages in your scope.

Closing Thoughts

I encourage you to review the Typst Builder PR. It seems nearly ready and will likely be included in nixpkgs soon. If you want to try it out now, consider examining the example flake I've provided above.

On Editors and Things

Tue, 11 Mar 2025 00:00:00 GMT

Hi all, good news first. I am a Release Editor for NixOS 25.05. This is a bit difficult for me to talk about, since I do not know what exactly what it entails yet but know that I am excited about it, and I wanted to share it with you, dear reader. It has been one hell of a journey so far, I am excited to see what is to come. NixOS, with its flaws, has a special place in my heart and it is great to be able to finally play a part in the grand scheme of things.

There are a few articles that I am working on, one of them being the continuation of the NixOS hardening series. Unfortunately there is not much time on my hands, and my backlog is filling up again. Worry not though, they are coming along nicely. I also have some technical rants on the horizon, since I have not written one of those in a while. Until I get the time to finish those, enjoy this heartfelt rambling on editors and the heart-to-heart in an unusual pace.

On Editors

I do a lot of, well, editing on my computer. This is sometimes for academic essays that I have to write myself, or sometimes academic essays that I have to proofread but it usually involves me sitting in front of a computer screen for hours on end and simply type away. Thanks to the relative leniency of my department, I have been able to use LaTeX in the past and more recently I have been enjoying Typst. Much of this editing, regardless of the framework, involves an editor. While it would be all too simple to use something like LibreOffice, I rarely find myself opening .docx files to edit an essay, and it's even less rare of an occurrence for me to be writing in one. What I use for most of my editing is, of course, my trusty Neovim setup.

Neovim holds an interesting place in my tool belt. I have switched off VSCode--the editor I have started with-- due to its annoyingly high RAM usage and power draw on my laptop, which made it less than ideal for when I could not easily find a place to plug my charger. Around the same time I was learning about Linux (and customizing the hell out of my Arch Linux setup) so it eventually lead to me checking out Neovim. It was something reminiscent of love at first sight. It was all snappy and intuitive and simple, I could not have asked for a better editor. Neovim was what I was looking for.

Over time I began adding plugins, usually to add features that I was missing from VSCode. LSPs and debugger plugins quickly found their way into my configuration, and over time Neovim was not quite what it first was. It had become sluggish, and complex. Worse, it had become inconsistent with the number of plugins that all insist that their way of doings things is the correct one. Around this time, I switched to NixOS and began thinking about setting up Neovim declaratively on my system. programs.neovim was not quite intuitive, and linking a nvim directory to ~/.config/nvim felt less than ideal. This lead to neovim-flake. I cannot quite call it my creation, as I have soft-forked it from a project with the same name by Jordan Isaacs, but I had many different, conflicting ideas in mind. So much so, that it warranted the soft-fork to take it in a different direction. neovim-flake, the original one, was a learning experience. I learned a lot about how to write Nix, and I also learned how not to write Nix.

A not-so-recent discovery that I have made is that I learn by doing. Explicitly by doing. Little bit (a lot) of research later, I found the state of academia to be be... disappointing, to say the least. There is a need for greater adoption of evidence-based cognitive learning strategies in educational settings and despite their proven effectiveness, teacher training programs often do not appear to be covering any cognitive learning processes. This, unfortunately, leads to many students (such as myself) distancing themselves from the education system entirely. Although I do not feel equipped to do more than just to observe. If you are, however, an expert in the field perhaps reach out to me. I would love to hear about your thoughts as well.

As I was saying, neovim-flake--now nvf--was a learning experience. More importantly, it was a testing ground for my "ideal" Neovim configuration. I have learned about Neovim as much as I have learned about Nix by maintaining nvf. The most important discovery while working on nvf and Neovim, was that Neovim is a subpar editor. I mean this in an endearing way of course; I would want my editor to become better and I think it is moving in the right direction, however, with contenders such as Helix now challenging Neovim's place it might be the time to reconsider priorities and re-allocate resources. Although I am uninterested in Helix ¹, it has pushed--moreso, nudged Neovim in the correct direction by looking at Neovim's long-time mistakes and working on fixing them. For one, Helix boasts a more consistent interface for features that are only available as plugins on Neovim. There is no conflicting design choices or UI decisions. Not yet, anyway. I think consistent UI is a good thing to have in your editor, however good your UX may be.

As I have mentioned above, Neovim has started moving in the correct direction with some of the core-features of Helix stealing the hearts of many Neovim users. After a very underwhelming 0.10 release, I believe 0.11 is taking Neovim in a better direction by analyzing the core demands of the community and trying to implement them as a part of the editor, in response to the needs and demands. The new LSP interface appears very interesting, though I would be equally happy to see a consistent interface for linters and formatters. null-ls (now none-ls) has been a frustrating experience, with how slow and clunky it is. none-ls doubled down on complexity by moving feature-complete builtins to a different repo, which you have to fetch and load yourself. This is not how an editor should function. Taking from Helix, I would be delighted to see some of the plugin exclusive features as parts of the core editor. Although, this might be wishful thinking. So far, I am excited for the prospect of natively configuring LSPs and a few UI additions that I have been looking forward to. Perhaps I'll write about Neovim again when 0.11 drops. There is also the less commonly talked about snippet engine that will be built into Neovim with 0.11. Those are all very welcome additions, and it is a great time to be a Neovim user.

For the time being, I have been aggressively dropping plugins that I don't really need or rewriting them myself as more efficient autocommands as parts of my configuration. This seems to have helped with the startup times, as some Neovim plugins are very inefficient. I have also dropped nvim-cmp because how how slow it was, but blink.nvim is restoring my "faith" in completion plugins again. The frizbee matcher is quite fast, and hopefully it will be utilized better in the future. On that note, I encourage you to do the same. Learning Lua is trivial, and if you have experience with programming you might be able to find far smarter implementations to problems "solved" by small convenience plugins.

On Things

On a more emotional side, I have been very burnt out with things. If you use any of my projects, you might have noticed that they have not been maintained on their usual pace for a while now. This is not permanent, I have many plans but unfortunately not enough time to follow through for now. Expect the language modules to be improved for nvf in the near future, and an internal rewrite of Schizofox. Hjem is, fortunately, a team project and is able to proceed with and without me. I consider other public software (such as Microfetch) I've created to be stable, but perhaps they will be picked back up as well. I have been meaning to make Microfetch even faster (can't stop, won't stop) for a while now...

Regardless, there are many nice things planned for the future but I first need some free time. Working on a few projects, and it is almost the busy season at $WORK. But I digress. I wanted to give you this as a progress report, and to explain the reasoning behind my absence. Some of those projects have been getting a lot of attention, even with my absence, so I wanted to personally thank everyone who has been submitting pull requests. You may not think much of it, but it means much to me that you have taken the time to contribute to the project(s).

Closing Thoughts

If this reads off as a little pessimistic, worry not. Nothing is changing anytime soon, or at least nothing is changing for the worse anytime soon. I will continue writing on this blog, and I will continue working on nvf as Neovim appears to be my forever editor. This was a temporary break from my usual pace of detailed, technical writing that takes more time and an opportunity to get some things off my chest, or out of my mind. As I've said before, there are multiple articles in the writing, but I would like to hear what you think I should write about too. Nix ecosystem is weirdly obscure, and sometimes independent blog articles do better than official documentation for learning.

If you have read this far, thank you. I would also like to hear your thoughts on an of the things I've talked about above. Feel free to reach out to me anytime. That said, this is all I have time for today. Thank you for reading.

This is due to my particular distaste of Lisp and Lisp-likes. Helix's so called "plugin system" is, at its core, a glorified Scheme interpreter and I do not feel positively about it. I believe there is also talks about using Scheme for configuration, and not just plugins. Compared to something like TOML, Scheme is a horrible choice and frankly that alone makes me want to distance myself from Helix. This is not to say Helix is a bad editor, I quite like the concepts it introduced to the editor scene but it does not strike me as something I can bring myself to daily drive. ↩

NixOS Security I: Systemd

Mon, 03 Mar 2025 00:00:00 GMT

A Linux system is a complex ecosystem of components, each with its own set of vulnerabilities. NixOS is no exception. While its declarative nature provides some advantages, it is not--nor can it ever be--a silver bullet against the inherent insecurities that exist in any system. This status quo--a state of perpetual vulnerability--calls you to action.

For the last 6 months or so, I have been focusing on hardening each and every single component of my NixOS installation. This post, as an attempt to document my experiences for the sake of establishing a point of reference, marks the beginning of a series dedicated to hardening the different components of a NixOS system. Given that NixOS is a systemd-based distribution, we'll start by focusing on systemd. Over the course of the series, I'll also delve into kernel and network security, although these topics require further research and are beyond the scope of this post.

In this installment, we'll explore how you can harden various systemd services on your system to reduce potential attack surfaces.

The Problem

Systemd, as a central part of many modern Linux distributions, provides various utilities designed to streamline system management. However, its centralization can also mean a single point of failure, and its wide array of features offers a large attack surface if misconfigured. One of the utilities it offers, which will come in very handy today, is systemd-analyze security.

Go ahead and run sudo systemd-analyze security on your system. You will notice very quickly that it contains a lot of "UNSAFE" and "EXPOSED" services. First of all, do not worry. Your system is probably safe. The assessments done by Systemd, as I will emphasize time and time again, are entirely arbitrary. They are rule-based score calculations based on your configuration, do not correspond to or reflect upon the actual vulnerability of your system; there is more nuance to such vulnerabilities than a rule-based analysis tool can determine. Indeed, the score assigned to each service is not an indicator of how secure or insecure it is. It is, however, a start. In the case of Systemd, hardening services is a second line of defense when the executable the service is for becomes the vulnerability.

The Solution

systemd-analyze security <unit> generates a score for a given unit, showing all the used directives. Based on limited information we have, it is possible to try and harden individual services.

By default, Systemd leaves running services in the open. This is understandable, as trying to preemptively harden a service is likely to cause conflicts with individual services and their requirements.

Although NixOS makes some effort to harden services, installed services--be it through NixOS' services or your inferior distribution's package manager--will be running with little to no hardening - which you can confirm by viewing the security report. This post, as per my experience, aims to detail potential hardening options. Throughout this post, please keep those following core principles in mind:

There is no "one size fits all" kind of hardening, all services will require your undivided attention to make sure everything continues to work as intended.
No kind of hardening can catch all kinds of exploits. The key to a secure system is to remain ever-vigilant.

Some hardening options will disable access to certain paths, or make them read-only for the service. This is quite helpful, in theory, but not every program is written intelligently and as such, not all programs will fail gracefully when they are missing access to a path. Sometimes the service will fail, and you will not be able to tell why. This is exactly why you must treat each service with special care - and harden them one service at a time instead of abstracting a generalized way of modifying multiple services at once.

Hardening Services

Systemd services in NixOS are defined through the Systemd module exposed in the Nixpkgs module system. The schema is very basic, and I suggest that you consult NixOS option search if you wish to learn more. For our purposes, just serviceConfig is enough as we will be working primarily with the [Service] field of systemd services. Documentation is very scattered, but available at systemd.unit(5), systemd.service(5) and systemd.exec(5) manpages. They contain many important tidbids, and I encourage you to go through them alongside this post during your hardening journey.

Most services enabled in NixOS, such as Miniflux under services.*, are essentially abstractions over systemd.services When you enable a service, you're essentially creating a Systemd service unit via systemd.services.<name>. Unfortunately, there are limited options (usually limited only to application-specific settings) to harden services through services.* options. Thus, we often need to strip away the abstraction and modify the settings for the Systemd services directly, using serviceConfig. You should also be aware that some services create more than one service unit, in which case systemctl list-units --type=service can help you find more services associated with one module.

Here is a very basic example to demonstrate how individual options are applied to a service.

{
  systemd.services."<serviceName>".serviceConfig = {
    ProtectClock = true;
    ProtectKernelTunables = true;
    ProtectKernelModules = true;
    ProtectKernelLogs = true;
    SystemCallFilter = "~@clock @cpu-emulation @debug @obsolete @module @mount @raw-io @reboot @swap";
    ProtectControlGroups = true;
    RestrictNamespaces = true;
    LockPersonality = true;
    MemoryDenyWriteExecute = true;
    RestrictRealtime = true;
    RestrictSUIDSGID = true;
  };
}

Options set in this example usually enough to bring a service down to MEDIUM exposure level. It removes some of the "unnecessary" permissions--although they may very well be necessary-- all services are granted by default. In short this example does meet the basic requirements for hardening, but it is also not very comprehensive. For services that reach more into the system, hardening must be done more diligently: you must assess the needs of your service and tweak its capabilities accordingly. Furthermore, you will be able to bring the score down from MEDIUM by focusing on the needs of service and applying other configuration fields.

To check how effective the change was, run systemd-analyze security <unit> before and after applying service changes, and compare the results. The output of systemd-analyze security will also provide in-depth explanation of each option that is (or is not) set, so it is worth running frequently on services that you aim to harden.

Definitions

While we are at it, let's talk about definitions. There too many options for me to cover, and as such this section will not cover each and every once of them. Instead, I would like to focus on the example I have given above to establish some baseline for how you may look at hardening.

Option	Description
ProtectClock	Prevents procee from accessing the system clock
ProtectKernelTunables	Restricts modification of kernel parameters
ProtectKernelModules	Prevents loading of kernel modules
ProtectKernelLogs	Limits access to kernel log messages
ProtectHome	Mounts /home, /root and /run/user as read-only tmpfs fs
ProtectSystem	Makes `/boot`, `/etc`, and `/usr` directories read-only
SystemCallFilter	Filters out specific system calls
ProtectControlGroups	Restricts use of control groups
RestrictNamespaces	Limits process namespaces
LockPersonality	Prevents changing process personality
MemoryDenyWriteExecute	Disallows write-execute memory mappings
RestrictRealtime	Limits real-time scheduling capabilities
RestrictSUIDSGID	Restricts SUID/SGID binaries¹

These are the definitions for some common directives that you may apply with minimum headache. You may stick those into the serviceConfig of a service in your configuration, and if the service is a basic daemon that does not need intricate FS or memory access, it should perform as expected.

The draft Systemd Sandboxing Article on Archwiki provides some insight on other options and their level of impact.

There are many directives that are used in Systemd services, with various exposure scores. While the scores differ, some of them might come in very handy. Here are a few that are worth mentioning while you work on hardening your services.

InaccessiblePaths

InaccessiblePaths is a very useful directive, which I did not about until recently, that might come in handy if the service requires access to a lot of directives. In which case, you may manually add sensitive directories that you want hidden at all costs.

DynamicUsers

Systemd system services (as opposed to user services) run as root unless they are explicitly given a user to run as. DynamicUsers is a special directive that can help you with dynamically (look I said the thing!) creating separate user accounts for each instance of the service. Each of these unique users runs their own instance of the service, providing a high level of isolation between different processes. This not only enhances security by limiting the potential impact of a compromised process but also improves resource management by isolating each instance's file system access. Although the impact of DynamicUsers on a service's exposure score is low, it is a very handy directive that you might consider if root privileges are not necessary for your service.

SystemCallFilter

One noteworthy directive is SystemCallFilter. As its name indicates, this Directive allows restricting syscalls that a service can call. This is very tricky to work with, and you can get out of hand as the process gains new features over time. Systemd, to make your life a bit easier, includes so-called "groupings"-- several groups of system calls, all prepended with @.

@swap, @reboot, @memlock and @raw-io are some examples of those groups. You may find a more detailed explanation on linux-audit.com

IP Accounting

With Systemd 235, Systemd was granted ability to track network traffic statistics for individual services or units. It allows administrators to monitor bandwidth usage, packet counts, and other network-related metrics associated with specific systemd services. This feature is implemented through the IPAddressAllow and IPAddressDeny directives in the [Service]section of a service unit file.

By enabling IP accounting, systemd can provide detailed insights into network activity, facilitating better network management, troubleshooting, and performance optimization for services running under its control. Network security is a little out of my scope today, but I encourage you to read the awesome article on IP accounting

Application

From the previous section, you should have a basic understanding of directives we will be using to harden services. Now let's take a look at the application of those directives.

Systemd's error messages for when a service is misconfigured can be vague or misleading, especially if the executable fails to properly inform the user of the error. ² Setting the log level temporarily to debug via systemctl log-level debug may help getting actually relevant information. Though do not rely too much on debug information, as it is usually equally useless.

As mentioned above, you might isolate services that need hardening with systemd-analyze security <unit> and focus on hardening each and every one of them. For example, systemd-analyze security acpid returns for me something like this:

✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                                0.1
  SupplementaryGroups=                                        Service runs as root, option does not matter
  RemoveIPC=                                                  Service runs as root, option does not apply
✗ User=/DynamicUser=                                          Service runs as root user                                                    0.4
✗ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes may change the system clock                                0.2
✗ NoNewPrivileges=                                            Service processes may acquire new privileges                                 0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ PrivateDevices=                                             Service potentially has access to hardware devices                           0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                      0.2
...

I have omitted the rest of the output, but you should get a gist of the output from this example. First field is the directive, second is the impact of the currently set value and the third field is the "vulnerability score" assigned to the service. Higher the score, more vulnerable the service. Acpid, for example, as a score of 9.6.

Acpid is, of course, not the only example. On my system, there are little over 50 systemd services running and a majority of them were ranked EXPOSED, UNSAFE or MEDIUM until I went out of my way to harden each and every single one of them. ³ My advice to you is to do the same: there are some Nix-based projects that try and handle hardening for you, but remember that there is no solution that suits all systems. Not to mention that relying on a 3rd party project for your system hardening is a security vulnerability of its own.

I find that krathalan's systemd sandboxing template serves as a good base that you may apply with minimal tweaks. In addition to providing templates for common services, it explains how certain options might interact with each other or system settings while set. Based on this template, manpages and the directives I've described (or linked) above; you should be able to go over each and every service that you find to be vulnerable. Keep in mind that it is crucial that you keep at least one stable generation on your system, as hardening can mess with even your user accounts, or TTYs as they are also managed by Systemd.

Examples

There are many services that score high in Systemd's exposure analysis, but it is impossible for me to provide configuration options for each service. Instead, I'll provide some examples for you to base your work off of. Using the information and sources I have referenced, it should not be very difficult to harden each individual service.

{
  systemd.services.acpid.serviceConfig = {
    ProtectSystem = "full";
    ProtectHome = true;
    RestrictAddressFamilies = [ "AF_INET" "AF_INET6" ];
    SystemCallFilter = "~@clock @cpu-emulation @debug @module @mount @raw-io @reboot @swap";
    ProtectKernelTunables = true;
    ProtectKernelModules = true;
  };
}

{
  systemd.services.power-profiles-daemon.serviceConfig = {
    ProtectHome = true;
    ProtectClock = true;
    ProtectKernelTunables = true;
    ProtectKernelModules = true;
    ProtectKernelLogs = true;
    SystemCallFilter = "~@clock @cpu-emulation @debug @obsolete @module @mount @swap";
    ProtectControlGroups = true;
    RestrictNamespaces = true;
    LockPersonality = true;
    MemoryDenyWriteExecute = true;
    RestrictRealtime = true;
    RestrictSUIDSGID = true;
  };
}

This is based on the example I have shown above, and should serve as a good base for most services.

Journal Hardening

It is no secret that Systemd services run with very lax permissions. One thing that is less often talked about is the Systemd journal, Journald.

As the primary logging mechanism for systemd-based distributions, Journald captures vast amounts of system and application data. However, this collection of information also presents significant security risks if left unprotected. This is why we must also take a look at hardening the system journal, which I find is essential for maintaining system integrity, protecting sensitive data, and ensuring compliance with security standards.

There are various methods through which Journald can leak sensitive information. The threat model for the Journal may differ based on your distribution (i.e., different distributions ship different configurations for the system journal), but as a general rule of thumb you should consider storing the system journal on encrypted storage, with proper permissions (e.g., 640) to protect it from unauthorized inquiries. Additionally, if you have configured Journald to send logs over the network, then then proper encryption is mandatory, or the data may be intercepted during transmission. Do treat logfiles like toxic waste, and handle them with care.

As a precaution, you might consider using volatile storage for the system journal as such:

{
  services.journald = {
    storage = "volatile"; # Store logs in memory
    upload.enable = false; # Disable remote log upload (the default)
  };
}

man 5 journald.conf provides additional insight on options you may consider setting through services.journald.extraConfig.

Last but not least, a dedicated enough attacker may attempt to run your system out of resources by filling your journal (e.g., if service output is forwarded to the journal) with bogus logs. In which case SystemMaxUse is a very useful option to set.

Conclusion

Remember that no amount of service hardening constitutes a bullet-proof security layer. Also remember that those scores assigned by Systemd are arbitrary. A service can be "safe" in the oblivious eyes of Systemd, but may risk security or privacy in other ways. While hardening services, consider the needs and attack vectors of each service that you are looking at.

Resources

I am leaving here the resources I have consulted in the past. This post aims to serve as a digestible summary of those resources as well as a guide to hardening your system, but do feel free to consult them at any time. Most of them extend beyond the scope of Systemd hardening, and will come up again in future posts. Visit them at your own discretion, you are sure to learn something new.

Security is a meaningless term without a threat model.

SUID (Set User ID) and GUID (Set Group ID) binaries are special executables that run with elevated privileges. When executed, they temporarily assume the privileges of the owner or group specified in their file attributes. This allows certain programs to perform operations that would otherwise require root-level access, such as changing system settings or accessing restricted files. However, this capability can be misused if not properly implemented, therefore it is often restricted through systemd unit options like RestrictSUIDSGID. ↩
Hint hint wink wink ↩
Okay maybe not all of them. It is not feasible to try and harden each service, as the minimum required conditions to run some services will always remain unsafe by Systemd's definition. This is what I have meant when I referred to the scores as arbitrary. Services that run as root, for example, can never be fully hardened as running as root is a security vulnerability on its own. Trying to offload the service to a dedicated user, or using DynamicUser might break functionality, therefore some services are bound to remain exposed as per Systemd. ↩

Signing Git commits with SSH keys

Mon, 24 Feb 2025 00:00:00 GMT

Resources

https://blog.gitbutler.com/git-tips-and-tricks/

Setting up SSH signing for Git commits

For the longest of times, I have been running an obscure GPG setup using Yubikeys that requires me to go back to my notes every time I wish to replicate it on a new machine or a friend's machine to help them set up GPG signing on their system. Today, after roughly 6 years of pushing GPG signed commits, I have learned about SSH signing¹, and have decided to share my notes on setting it up.

A basic prerequisite is that you need your SSH key pair added to the Git service you will be using e.g., Github or Forgejo, so that you are able to push and pull via SSH. Github Documentation on SSH keys shows you how to set that up, and I imagine other platforms are not that more complicated.

Traditional Distros

If you are using a traditional distribution like Fedora or Arch² then the setup consists of two commands:

git config --global gpg.format ssh # tell Git to use SSH signing keys

Followed by the command to tell Git which key to use:

git config --global user.signingkey ~/.ssh/my-key.pub

If you wish to sign commits of a specific repository, then you may run the above command in said repository and omit --global to store the local (repository) configuration in .git/ directory of the repository you can the command in.

Remember to replace the example path I've used above with the path to your public key.

Now it is time to tell Github about your signing key. Add your public key to Github again, but this time change the key type to "Signing Key". The process should once again be similar for other public Git services.

Last, you will want to actually use the signing key that you have configured Git to use for signed commits. That's right, all commits are unsigned until you either pass -S to git commit or set git config --global commit.gpgsign true. Similarly, you can also sign selected tags with git tag -s. To sign all git tags, you must run git config --global tag.gpgsign true, or the same command without --global to sign tags in a local repository.

On NixOS

In typical NixOS fashion, such a trivial process is mind-numbingly easy. You will want to configure programs.git under either NixOS or Home-Manager module systems. As Git is rather a userspace application, I choose to use the one under home-manager but NixOS also allows you to configure git globally with /etc/gitconfig being used. Please note that the steps will change slightly if you are using the Git module under NixOS' module system.

First you would like to add programs.git.signing to your home.nix as follows:

# home.nix
programs.git = {
  signing = {
    key = "${config.home.homeDirectory}/.ssh/my-key.pub";
    signByDefault = true;
  };

  extraConfig.gpg.format = "ssh";
};

This will be enough to tell Git where to look while also setting signByDefault, meaning all of your commits will be signed by default, equivalent of passing --gpg-sign=<key id>. Great!

If you wish to take this a bit further, you may consider setting allowed signers to decide who can and cannot sign using the ssh.allowedSignersFile option under extraConfig.gpg. Do keep in mind, however, that you need to have set an allowed signers file beforehand. To do so, expand your home.nix with the new options:

# home.nix
{config, pkgs, ...}: let
  key = "<your public key>";
  email = "<your email>";

  # Write signersFile to /nix/store/<store path>
  signersFile = pkgs.writeText "git-allowed-signers" ''
    ${email} namespaces="git" ${key}
  '';
in {
  xdg.configFile."git/allowed_signers".source = signersFile;

  programs.git = {
    signing = {
      key = "${config.home.homeDirectory}/.ssh/my-key.pub";
      signByDefault = true;
    };

    extraConfig.gpg = {
      format = "ssh";
      ssh.allowedSignersFile = signersFile;
    };
  };
};

In above example, we have written a symlink to ~/.config/git/allowed_signers using xdg.configFile and pkgs.writeText. Then, programs.git.extraConfig.gpg.ssh.allowedSignersFile tells Git which users will be allowed to sign commits based on commit email.

And that is all! You may now switch Home-Manager generations with either nixos-rebuild switch (if you have Home-Manager as a NixOS module) or home-manager switch (if you are using home-manager in standalone mode). Do make sure to adapt the example to your own setup.

To be fair, I have known about SSH signing for a while now, but the sunk-cost fallacy has prevented me from ever looking into it. As a matter of fact, I am still not convinced by it, but that is because I still use a Yubikey based GPG setup which partially obsoletes SSH signing and prevents me from looking further into the matter. Regardless, this documents my experience with SSH signing, so I may return anytime. ↩
Please don't. ↩

Nix Remote Builders

Mon, 17 Feb 2025 00:00:00 GMT

Nix is a package manager--nay, a build tool¹ that is capable of scaling both vertically and horizontally through its Binary Caches and Remote Builders respectively.

You are introduced to the binary cache as soon as the moment you begin the installation process of NixOS. The ISO image you download, minimal or one of the graphical flavours, is built by Hydra ² as a result of a particular combination of module option combinations in Nixpkgs. In this sense, NixOS is a build artifact. It consists of many derivations that are put in a particular configuration, and built in a particular way to result in a NixOS ISO image. The images, or a common NixOS system, consist of many derivations.

Although Nix has a strange concept of dependencies, let us try to count everything that contributes to the building of your final system.

$ nix-store -q --requisites /run/current-system | cut -d- -f2- | sort | uniq | wc -l
2956

This is by no means 100% accurate, but it gets somewhat close. My system is the result of around 3000 separate derivations, that would need to be built if not for the binary caches Nix pulls built derivations from. Similar to the installation ISO images, Hydra builds and caches most of the derivations ³ so that you may pull them from https://cache.nixos.org. All things considered, the binary caches are distributed (and load balanced) for global reach. ⁴

Remote Builders

The topic I wanted to talk about today is Remote Builders, a less frequently employed method for scaling. Remote builders require additional setup, as they execute builds on an authorized machine rather than pull built results from a cache. While building things that might not be available in a public cache, you may offload build steps to a remote machine instead of using your current machine in the case your build process requires compilation, and higher resource usage. In this case, a machine with more resources can speed-up the process.

The structure of Nix remote builders consists of two components:

Local Machine
Remote Machine(s)

There is no limit to how many machines you may distribute your builds upon.

Remote Machine

For the sake of simplicity, let us assume that there is one remote builders that you would like to utilize for your builds. Nix utilizes remote builders by opening SSH connection to the target machine, so you must first have a user with SSH access on the target machine. Let's call this user builder. You must mark this user as a "trusted user" (users that have additional rights when connecting to the Nix daemon, such as the ability to specify additional binary caches, or to import unsigned NARs.) to utilize this user for remote builds.

On a typical NixOS setup, you would do so by adding it to trusted-users:

{
  nix.config.trusted-users = ["builder"]; # you may also use @groups instead
}

Once you make sure that the builder user on the remote machine is accessible via SSH, and is marked as a trusted user, try offloading a build to the remote machine to test your connection:

nix build nixpkgs#hello --rebuild --builders "ssh://builder@ssh.example.tld"

You can replace ssh.example.tld with a domain pointing at your server, or just the raw IP address of your machine. If SSH is running a port other than 22 on your machine, then you may also specify an alternate port to be used with the NIX_SSHOPTS environment variable. For example:

export NIX_SSHOPTS="-p 2222"
nix build nixpkgs#hello --rebuild --builders "ssh://builder@ssh.example.tld"

Would open a connection over port 2222 instead of the default 22.

Local Machine

This is a good time to warn you that sometimes, especially if your connection is shaky, adding remote builders to the mix might slow things down. Remember that everything built on the server will have to be sent back to your local machine to complete the final closure. However, if you are positive that remote builders are good addition and the network effects are negligible, then you may configure Nix to utilize certain builders on each build.

{
  # https://wiki.nixos.org/wiki/Distributed_build
  nix.buildMachines = [
    {
      hostName = "ssh.example.tld";
      sshUser = "builder";
      # 'ssh-ng' is faster if both machines are NixOS but falls flat if the
      # machine Nix will attempt a connection to is not NixOS. In such a case
      # you must use 'ssh' instead.
      protocol = "ssh-ng";

      # This can be an absolute path to a private key or it can be managed
      # with something like Agenix, or SOPS.
      sshKey = "/home/user/.ssh/builder-rsa";

      # Systems for which builds will be offloaded.
      systems = ["x86_64-linux" "i686-linux"];

      # Default is 1 but may keep the builder idle in between builds
      maxJobs = 3;
      # How fast is the builder compared to your local machine
      speedFactor = 2;

      supportedFeatures = ["big-parallel" "kvm" "nixos-test"];
    }
  ];
}

The above configuration sets up /etc/nix/machines. If you are setting up remote builders on non-NixOS, you will need to manually construct the contents of that file.

A cool trick, if you are self-hosting Hydra, is that Hydra can pick up remote builders if you tell it to. How you can do this is documented in the Hydra manual, but you might consider self-hosting Hydra on your desktop or a weaker machine like a Raspberry Pi while utilizing a stronger machine for remote builds.

nix.buildMachines also plays very nicely with your SSH configuration. For example in the case of unique ports, you may use programs.ssh.extraConfig. For example:

{
  programs.ssh.extraConfig = ''
    Host remote-builder
      User builder
      HostName ssh.builder.tld
      Port 2222
      IdentityFile /home/user/.ssh/builder-rsa # As before, Agenix will work here
  '';
}

Specifying your connection details in programs.ssh.extraConfig is a more sophisticated way of configuring the SSH connection that will be established on remote builds. As such, you may omit NIX_SSHOPTS and connection information. For the command-line example, you may now use protocol://host.

nix build nixpkgs#hello --rebuild --builders "ssh://remote-builder"

If there are more than one builder that you would like to use, lets call them remote-builder1 and remote-builder2, you can specify them in the --builders flag, separated by spaces.

nix build nixpkgs#hello --rebuild --builders "ssh://remote-builder1 ssh://remote-builder2"

Closing Thoughts

That should be everything you need to configure remote builders on your system. If you have a Tailnet, via Tailscale, you may utilize MagicDNS or something similar to simply query builders by hostname and partially skip the SSH configuration. E.g., ssh-ng://builder@my-host where my-host is available on your Tailnet, and systemd-resolved is asked to search your Tailnet domain.

In the case your local system is too weak, you may also consider passing --max-jobs 0 to the build command to execute all build tasks on your available remote machines. I pass --max-jobs to the nixos-rebuild command when I am attempting a rebuild on my weakest machine, which only has 4gb RAM available, to avoid running my system out of resources.

A very cool, but somewhat unrelated trick, is that you can use --build-host in nixos-rebuild to build on a remote host.

Nix, although most commonly referred to as a "Package Manager" is actually a build tool. Some find this definition pedantic, but package management is only a subset of what it is actually capable of. Calling it a package manager is fair, but also a bit reductive in a way that it does not communicate Nix's advantages over traditional package managers. ↩
Hydra is a Nix-based continuous build system, tasked for evaluating and building derivations from a jobset such as but not limited to Nixpkgs. In the context of Nixpkgs, Hydra is what populates the cache, logs evaluation or build failures, and makes channels available. ↩
Some derivations are not built and therefore not available in the cache. There are a few different conditions for this, but it is the case especially when the derivation has been marked specifically not to be built by Hydra. ↩
This should show you how fragile the Nix infrastructure is. As much as I am a huge fan of Nix, Nixpkgs and NixOS, without the proper funding Nix's adoption would not be possible. Nobody wants to build everything from source, all of the time. Even though content-addressed Nix would make rebuilds less troublesome, this is a problem to ponder on while considering the future of Nix and NixOS. ↩

Full Disk Encryption and Impermanence on NixOS

Sat, 15 Feb 2025 00:00:00 GMT

Impermanence is an interesting concept. Aside from its philosophical aspect, it has strange but pleasant implications on a NixOS system. Not only is it a display of sheer flexibility of NixOS as an operating system, it also provides a system that cleans itself up on each reboot. Gone are the days of manually clearing useless state some programs have left behind.

This general setup concept utilizes NixOS' ability to boot off of a disk that contains only /nix and /boot, linking appropriate devices and blocks during the boot process and deleting all state that programs may have left over my system. The end result, for me, was a fully encrypted system that uses BTRFS snapshots to restore / to its original state on each boot.

Resources

This post is based on, and inspired by several resources. I tried to cover everything you might need to set up a stateless NixOS system on your machines.

Reproduction steps

I've had to go through a few guides before I could figure out a set up that I really like. The final decision was that I would have an encrypted disk that restores itself to its former state during boot. Is it fast? Absolutely not. But it sure as hell is cool. And stateless!

To return the root (and only the root) we use a systemd service that fires shortly after the disk is encrypted but before the root is actually mounted. That way, we can unlock the disk, restore the disk to its pristine state using the snapshot we have taken during installation and mount the root to go on with our day.

Partitioning

First you want to format your disk. If you are really comfortable with bringing parted to your pre-formatted disks, by all means feel free to skip this section. I, however, choose to format a fresh disk. It is also possible to switch to an impermanent setup from your current installation, but you will be better off starting from scratch. Remember to make a backup of important state!

Start by partitioning disks into several sections. For example: sda1, sda2 and sda3. This will differ if you are using nvme disks, for example, /dev/nvme0n1p1.

# Set the disk name to make it easier
DISK=/dev/sdx # replace this with the name of the device you are using

Now set up the boot partition. This is where your bootloader will live. If you intend to persist a lot of generations, you may want to allocate more space. For example 2 GiB instead of just 1.

parted "$DISK" -- mklabel gpt
parted "$DISK" -- mkpart ESP fat32 1MiB 1GiB
parted "$DISK" -- set 1 boot on # assumes UEFI

mkfs.vfat -n BOOT "$DISK"1

Set up a swap partition You may choose to omit swap if your system has sufficient RAM available. My machine has 16GB available, so I choose to allocate 8GBs for swap for my setup.

parted "$DISK" -- mkpart Swap linux-swap 1GiB 9GiB
mkswap -L SWAP "$DISK"2
swapon "$DISK"2

I do in fact use swap in the civilized year of 2023¹. At the cost of disabling hibernation, I also choose to encrypt my swap. This guide will not cover how you may do so, but swapDevices.*.randomEncryption is a good start.

Encrypt your partition, and open it to make it available under /dev/mapper/enc. enc will be your logical volume name, you may wish to give it a different name.

cryptsetup --verify-passphrase -v luksFormat "$DISK"3 # /dev/sda3
cryptsetup open "$DISK"3 enc # the name enc is arbitrary, rename if you wish

Now partition the encrypted device block.

parted "$DISK" -- mkpart primary 9GiB 100%
mkfs.btrfs -L NIXOS /dev/mapper/enc

Mount the disk, and set up subvolumes. I use BTRFS for its flexibility and the robustness. You may do something similar using ZFS, but I will not cover that here.

mount -t btrfs /dev/mapper/enc /mnt

# First we create the subvolumes, those may differ as per your preferences
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
btrfs subvolume create /mnt/persist # some people may choose to put /persist in /mnt/nix, I am not one of those people.
btrfs subvolume create /mnt/log

Now that we have created the BTRFS subvolumes, it is time for the readonly snapshot of the root subvolume. This is what we will use to roll back our system on each boot. Compared to / on tmpfs, rolling back manually on boot has the added advantage of offering a way to restore unsaved data if your system shuts down abruptly. For example if you suddenly loose power, you may boot into a recovery system to recover anything that would be saved on /. This is an edge case of course, but I prefer having the option.

btrfs subvolume snapshot -r /mnt/root /mnt/root-blank

# Make sure to unmount, otherwise nixos-rebuild will try to remove /mnt
# and fail
umount /mnt

Mounting

After the subvolumes are created, we mount them with the options that we want. Ideally, on NixOS, you want the noatime² option and zstd compression, especially on your /nix partition.

The following is my partition layout. If you have created any other subvolumes in the step above, you will also want to mount them here. Below setup assumes that you have been following the steps as is.

# /
mount -o subvol=root,compress=zstd,noatime /dev/mapper/enc /mnt

# /home
mkdir /mnt/home
mount -o subvol=home,compress=zstd,noatime /dev/mapper/enc /mnt/home

# /nix
mkdir /mnt/nix
mount -o subvol=nix,compress=zstd,noatime /dev/mapper/enc /mnt/nix

# /persist
mkdir /mnt/persist
mount -o subvol=persist,compress=zstd,noatime /dev/mapper/enc /mnt/persist

# /var/log
mkdir -p /mnt/var/log
mount -o subvol=log,compress=zstd,noatime /dev/mapper/enc /mnt/var/log

# Do not forget to mount the boot partition!
mkdir /mnt/boot
mount "$DISK"1 /mnt/boot

Now, finally, we create Nix generate the appropriate hardware configuration for our setup.

nixos-generate-config --root /mnt

The generated configuration will be available at /mnt/etc/nixos.

Before we move on, we need to add the neededForBoot = true; to some mounted subvolumes in hardware-configuration.nix. It will look something like this:

# Do not modify this file!  It was generated by ‘nixos-generate-config'
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
  ];

  boot.initrd.availableKernelModules = ["xhci_pci" "ahci" "usb_storage" "sd_mod" "rtsx_pci_sdmmc"];
  boot.initrd.kernelModules = [];
  boot.kernelModules = ["kvm-intel"];
  boot.extraModulePackages = [];

  fileSystems."/" = {
    device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
    fsType = "btrfs";
    options = ["subvol=root"];
  };

  boot.initrd.luks.devices."enc".device = "/dev/disk/by-uuid/82144284-cf1d-4d65-9999-2e7cdc3c75d4";

  fileSystems."/home" = {
    device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
    fsType = "btrfs";
    options = ["subvol=home"];
  };

  fileSystems."/nix" = {
    device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
    fsType = "btrfs";
    options = ["subvol=nix"];
  };

  fileSystems."/persist" = {
    device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
    fsType = "btrfs";
    options = ["subvol=persist"];
    neededForBoot = true; # <- add this
  };

  fileSystems."/var/log" = {
    device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
    fsType = "btrfs";
    options = ["subvol=log"];
    neededForBoot = true; # <- add this
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/FDED-3BCF";
    fsType = "vfat";
  };

  swapDevices = [
    {device = "/dev/disk/by-uuid/0d1fc824-623b-4bb8-bf7b-63a3e657889d";}
    # if you encrypt your swap, it'll also need to be configured here
  ];

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
}

Do keep in mind that the NixOS hardware scanner cannot pick up your mount options. Which means that you should specify the options (i.e noatime) for each BTRFS subvolume that you have created in hardware-configuration.nix. You can simply add them in the options = [ ] list in quotation marks. I recommend adding at least zstd compression, and optionally noatime.

Installing

And that should be all. By this point you are pretty much ready to install with your existing config. I generally use my configuration flake to boot, so there is no need to make any revisions. If you are starting from scratch, you may consider tweaking your configuration.nix before you install the system. An editor, such as Neovim, or your preferred DE/wm make good additions to your configuration.

Once it's all done, take a deep breath and nixos-install. Once the installation is done, you'll be prompted for the root password and after that you can reboot. Now you are running NixOS on an encrypted disk. Nice!

Next up, if you are feeling really fancy today, is to configure disk erasure and impermanence.

Impermanence

To handle BTRFS snapshots and automatic rollbacks, I use a systemd service. This requires systemd to be enabled in stage1. You may enable it with boot.initrd.systemd.enable = true;. The schema for a systemd service in initrd is the same as systemd.services, except restartTriggers and reloadTriggers will not available.

{
  boot.initrd.systemd = {
    enable = true; # this enabled systemd support in stage1 - required for the below setup
    services.rollback = {
      description = "Rollback BTRFS root subvolume to a pristine state";
      wantedBy = ["initrd.target"];

      # LUKS/TPM process. If you have named your device mapper something other
      # than 'enc', then @enc will have a different name. Adjust accordingly.
      after = ["systemd-cryptsetup@enc.service"];

      # Before mounting the system root (/sysroot) during the early boot process
      before = ["sysroot.mount"];

      unitConfig.DefaultDependencies = "no";
      serviceConfig.Type = "oneshot";
      script = ''
        mkdir -p /mnt

        # We first mount the BTRFS root to /mnt
        # so we can manipulate btrfs subvolumes.
        mount -o subvol=/ /dev/mapper/enc /mnt

        # While we're tempted to just delete /root and create
        # a new snapshot from /root-blank, /root is already
        # populated at this point with a number of subvolumes,
        # which makes `btrfs subvolume delete` fail.
        # So, we remove them first.
        #
        # /root contains subvolumes:
        # - /root/var/lib/portables
        # - /root/var/lib/machines

        btrfs subvolume list -o /mnt/root |
          cut -f9 -d' ' |
          while read subvolume; do
            echo "deleting /$subvolume subvolume..."
            btrfs subvolume delete "/mnt/$subvolume"
          done &&
          echo "deleting /root subvolume..." &&
          btrfs subvolume delete /mnt/root
        echo "restoring blank /root subvolume..."
        btrfs subvolume snapshot /mnt/root-blank /mnt/root

        # Once we're done rolling back to a blank snapshot,
        # we can unmount /mnt and continue on the boot process.
        umount /mnt
      '';
    };
  };
}

You may opt in for boot.initrd.postDeviceCommands = lib.mkBefore '' as this blog post suggests. I opt-in for a Systemd service as a service is a more powerful option of handling service dependencies. With postDeviceCommands, we would be sticking some bash code haphazardly in the stage-1 script, with a systemd service we will be holding granular control over the service order.

Implications & Workarounds

An impermanent setup has certain implications, for example, some files such as saved networks for network-manager will be deleted on each reboot. While a little clunky, Impermanence is a great solution to our problem. Impermanence exposes to our system an environment.persistence."<dirName>" option with its NixOS module, which we can use to make certain directories or files permanent. My setup is as follows.

# inputs needs to be added to your 'specialArgs' in the lib.nixosSystem call.
# In some setups, individual inputs are passed to specialArgs directly and as
# such your setup may differ just a little bit. Note that if you are not using
# flakes, inputs will not be available at all. In which case, you must manually
# fetch impermanence with fetchTarball, or use the appropriate channel. I
# recommend using flakes since they offer a better UX overall.
{inputs, ...}: {
  imports = [inputs.impermanence.nixosModules.impermanence];

  environment.persistence."/persist" = {
    directories = [
      "/etc/nixos"
      "/etc/NetworkManager/system-connections"
      "/etc/secureboot"
      "/var/db/sudo"
    ];

    files = [
      "/etc/machine-id"

      # Required for SSH. If you have keys with different algorithms, then
      # you must also persist them here.
      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
      "/etc/ssh/ssh_host_rsa_key"
      "/etc/ssh/ssh_host_rsa_key.pub"
      # if you use docker or LXD, also persist their directories
    ];
  };
}

And that is pretty much it. If everything went well, you should now be telling your friends about your new system boasting full disk encryption and root rollbacks. They, in turn, should be looking at you like cavemen. Arch users could never. May they bask in their supreme glory.

Closing Notes

As interesting as Impermanence is, it is also a little risky. The constant new changes Nixpkgs receives on a daily basis might mean that a service may suddenly stop using the state directory that you have configured it to use, and move to a directory that you have not yet persisted. While running such a set up, you must make sure to pay close attention to your system and back up everything accordingly. Make sure to persist important directories, and have backups ready in case something changes without your knowledge.

Home Impermanence

Silly. $HOME is where state belongs. I will not cover this, and I do not encourage that you make your home directory impermanent. If you insist on setting it up, you may use a similar systemd service in systemd.services and Impermanence's Home-Manager module.

Why?

Honestly, why not?

Okay real answer. All imperative distributions suffer from something called configuration drift. It is when your system constantly moving forward, through updates and over time, leaving junk files that no longer have any use but still remain on your disk. This is sometimes a security vulnerability, and sometimes just general annoyance. Impermanence completely eliminates configuration drift, and makes sure that your system is sparkly clean on each boot. Cool, right?

I could be using tmpfs for / at this point in time. Unfortunately, since I share this setup on some of my low-end laptops, I've got no RAM to spare - which is exactly why I have opted out with BTRFS. It is a reliable filesystem that I am used to, and it allows for us to use a script that we'll see later on. ↩
Read about noatime here ↩

When, Why and How to Extend Nixpkgs' Standard Library

Sat, 01 Feb 2025 00:00:00 GMT

In plenty of cases, you might need to write your own custom functions and store them somewhere. While it is conceptually possible and easy to define them inside an argument you might want to stick in for example specialArgs in the context of a NixOS configuration, there are easier and more ergonomic ways of doing so.

What is `nixpkgs.lib`

In the context of Nix/OS, nixpkgs.lib refers to a module within the Nixpkgs repository that acts as a collection of helpful functions and other utilities designed around usage in Nixpkgs and by extension NixOS configurations. We often use those functions to simplify our configurations and the Nix package build processes. It is available as a top-level attribute as nixpkgs.lib, but also inside pkgs as pkgs.lib. While using lib.nixosSystem, it is also added to your system's specialArgs¹ so that you can add it to the argument set (i.e. the line that goes {pkgs, lib, ...} at the top of a file) in your NixOS configuration easily.

Why would you need to extend `nixpkgs.lib`

While the library functions provided by nixpkgs is quite extensive and usually suits my needs, I sometimes feel the need to define my own function or wrap an existing function to complete a task. Normally we can handle the process of a function inside a simple let in and be well off, but there may be times you need to re-use the existing function across your configuration file. In such times, you might want to either write your own lib and inherit it at the source of your flake.nix to then inherit them across your configuration.

Extending `nixpkgs.lib`

I find the easiest way of extending nixpkgs.lib to be using an "overlay", enabled by makeExtensible. ²

# The file path is arbitrary, let's assume that this is in
# lib/default.nix for the sake of simplicity.
{
  inputs,
  ...
}: inputs.nixpkgs.lib.extend (
    final: prev: {
      # Your functions go here
    }
  )

The above structure takes the existing lib from nixpkgs, which you'll remember is defined as nixpkgs.lib, and appends your own extensions to it. You may then import this library in your flake.nix to pass it to other imports and definitions.

# flake.nix
flake = let
  # Get the extended lib from ./lib/default.nix
  lib = import ./lib {inherit inputs;};
in {
  # Then you may pass it around, e.g. in imports by adding `lib`
  # to the argument set.
  nixosConfigurations = import ./hosts {inherit nixpkgs self lib;};
};

In this example, the extended library is imported from lib/default.nix in repository root where the overlay is defined. It is then passed around to make the extended lib available within all files called by flake.nix. For example, in hosts/default.nix it could be added to specialArgs to make the extended library the default in a NixOS configuration.

# hosts/default.nix
{lib, ...}: {
  # Since this is called by nixosConfigurations = ./import ...
  # we just add the configuration attributes here, one for each
  # new configuration.
  fooSystem = lib.nixosSystem {
    modules = [ ... ];
    specialArgs = {lib;};
  };

Now any and all new functions defined in the extended library will become available in any files called by modules, thanks to specialArgs.

Caveats

The problem with this approach is that it may be confusing for other people reviewing your configuration. With this approach, lib.customFunction looks identical to any lib function, which may lead to people thinking the function exists in nixpkgs itself while it is only provided by your configuration. This is not a problem per se, but if this is something that bothers you then the solution is simple though. Instead of extending nixpkgs.lib, you may define your own lib that does not inherit from nixpkgs.lib and only contains your functions. The process would be similar, and you would not need to define an overlay.

# flake.nix
flake = let
    # extended nixpkgs lib, contains my custom functions
    lib' = import ./lib {inherit nixpkgs lib inputs;};
in {
    # entry-point for nixos configurations
    nixosConfigurations = import ./hosts {inherit nixpkgs self lib';};
};

where your lib/default.nix looks like

# lib/default.nix
{nixpkgs, ...}: {
  # Define your functions here as you would do in an extension
}

Example Implementations

If you have defined your own custom library based on this post, feel free to add a project or your configuration as an example here.

Lazy Evaluation in Nix: Where Your Conditionals Go to Die

Thu, 30 Jan 2025 00:00:00 GMT

If you have spent time in traditional programming languages, then you have probably relied on conditionals (if-else, switch, case) at least once in In Nix, as a result of lazy evaluation, many of those constructs become less relevant less relevant---or outright unnecessary. This post dives into why this is the case, and hopes to save you from a few pitfalls that come with this.

What is Lazy Evaluation?

I first need to explain what Lazy evaluation is. Not just within the context of Nix, but in programming in general.

Lazy evaluation means that expressions are not evaluated until their values are actually needed. This is in contrast to strict (or eager) evaluation, where expressions are computed as soon as they are bound to a variable. In more practical terms, lazy evaluation helps avoid unnecessary computations by delaying the evaluation of expressions. It's a powerful technique for improving performance and managing resources efficiently.

Laziness in Haskell

lazyNumbers :: [Int]
lazyNumbers = [1..]

firstFiveNumbers :: [Int]
firstFiveNumbers = take 5 lazyNumbers

In this example lazyNumbers is an infinite list. However, the values are not computed all at once. The take 5 lazyNumbers expression only computes the first 5 elements of the list when required. That is the most typical example of lazy evaluation I can think of. Key point is that the numbers are only generated when needed, instead of being computed upfront.

Laziness in Python

With some friction, we can implement lazy iteration in Python.

def lazy_numbers():
    num = 1
    while True:
        yield num
        num += 1

lazy_gen = lazy_numbers()
first_five_numbers = [next(lazy_gen) for _ in range(5)]
print(first_five_numbers)

In this example, lazy_numbers() is a generator that lazily yields numbers starting from 1. Unlike a typical list, the numbers are not computed all at once. The next(lazy_gen) call only computes the next number when needed. So, when we request the first five numbers, only the first five numbers are generated.

In Nix, laziness manifests in several ways:

Unused branches of an if-else are never evaluated.
Function arguments are not evaluated unless explicitly used.¹
Attribute sets can include self-referential definitions, leading to infinite recursion errors only when an attribute directly refers to itself.²

The Death of Conditionals (or at Least Their Diminished Role)

In strict languages, conditionals are usually used to prevent expensive computations from running unnecessarily. But in Nix, the very nature of laziness means that an expensive computation inside an unused branch never executes.

Example:

let
  expensive = builtins.trace "This should not print!" (throw "Error");
  value =
    if false
    then expensive
    else "Safe";
in
  value

In an eagerly evaluated language, this would result in an error because expensive would be computed before if even runs. In Nix, however, the false branch is never evaluated, so the program executes safely.

Functions: Only Compute What is Needed

Since function arguments are evaluated lazily, unnecessary computations can be avoided naturally.

let
  alwaysReturnsOne = x: 1;
in
  # The throw becomes the function argument here, which is x.
  # Since x is never evaluated, nor is the throw so the program
  # will continue.
  alwaysReturnsOne (throw "This should never be evaluated")

Even though we pass an expression that would normally cause an error, it is never evaluated since x is not used inside the function body.

Self-Referencing With and Without Infinite Loops

Nix allows for self-referential structures, but only if evaluated lazily. Direct cyclic dependencies will cause infinite recursion unless structured carefully.

let
  # rec is a special form in Nix used for defining recursive
  # attribute sets, allowing circular references to resolve
  # lazily without infinite loops. You will find out very
  # quickly, however, that it is often discouraged.
  infiniteRec = rec {
    a = b + 1;
    b = let
      # This would cause an infinite recursion if it was evaluated
      # but the variable `infrec` is never referenced, so it passes.
      infrec = a - 1;
    in
      42;
  };
in
  infiniteRec.a  # Evaluates safely

This works because what would cause the infrec is never evaluated. If the evaluator even touched the variable infrec, the program would immediately face and infinite recursion. Instead, it uses the fixed value 42 from b so a can refer to b safely.

Here is an example that would fail.

let
  infiniteRec = rec {
    a = b + 1;
    b = a - 1;
  };
in
  infiniteRec.a  # Infinite recursion

Here both a and b depend on each other without a base case, which results in infinite recursion. The only way to prevent this is to explicitly anchor one of the values with a concrete number or expression that doesn't rely on the cycle.

Infinite recursion is not as straightforward as it looks here. While working with small sets and short files you can easily identify where the infinite recursion comes from. While using the module system to break your code into multiple files, or even repositories, you might have more difficulty identifying where exactly it comes from, because you will notice that error messages are as clueless as you are. We'll talk about the costs of deep abstractions in another post.

What is `fix`?

fix is a function used in some functional languages (such as Haskell) to define recursive expressions without requiring explicit naming. It applies a function to itself, allowing the expression to refer to itself.

fix f = f (fix f)
fact = fix (\rec n -> if n == 0 then 1 else n * rec (n - 1))

In strict (non-lazy) languages, recursion requires named definitions, such as explicitly defining fact and referencing itself. Without laziness, a function cannot pass itself as an argument to another function without being fully evaluated first---leading to a situation where the function reference does not exist at the time of evaluation. The fix function allows recursion to be encoded explicitly by ensuring that a function can reference itself even in strict evaluation contexts.

let
  factorials = rec {
    zero = 1;
    fact = n: if n == 0 then zero else n * fact (n - 1);
  };
in
  factorials.fact 5  # Evaluates to 120

Here, fact refers to itself without requiring an explicit fixed-point combinator like fix. This is because Nix only evaluates values when needed, meaning references can exist without being immediately resolved. In contrast, strict languages would require fix to explicitly establish recursion. ³

const factorial = (n) => (n === 0 ? 1 : n * factorial(n - 1));

Here, factorial is explicitly named, allowing it to refer to itself. However, if we wanted to define it without naming it explicitly, we would need something like fix.

// Why yes I've used Javascript as an example for a good reason.
// The reason is that if I've suffered with writing Haskell, then
// you must suffer reading Javascript. Ha.

// `fix` is a function that helps a function call itself
// It takes a function `f` as an argument and returns `f` applied to itself
const fix = (f) => f((x) => fix(f)(x));

const factorial = fix(
  (rec) => (n) =>
    // Base case: if n is 0, return 1 (because 0! = 1)
    n === 0 ? 1 : n * rec(n - 1), // recursive case: multiply n by the factorial of (n-1)
);

console.log(factorial(5)); // 120

Thus, while fix is necessary for recursion in strict evaluation, Nix's lazy evaluation makes it unnecessary, allowing for powerful self-referential structures with careful structuring.

Other Examples

To help give you an idea of what lazy evaluation can do for you, e.g. in your NixOS configuration, I'd like to demonstrate laziness in action.

Lazy Lists

In Nix, you can define infinite data structures like lazy lists without causing infinite loops:

let
  lazyList = {
    head = 1;
    tail = lazyList.tail; # this is an infinite recursion, if it evaluates...
  };
in
  lazyList.head

Delayed Computation in Attribute Sets

let
  config = rec {
    setting =
      if useAdvanced
      then builtins.throw "Too expensive!"
      else "default";
    useAdvanced = false;
  };
in
  # evaluates to "default", error is never thrown
  config.setting

Conclusion

Laziness is powerful, but it can lead to surprises:

Debugging is harder because errors might not surface until a deeply nested expression is finally evaluated. --show-trace is your friend most of the time.
Performance tuning requires careful observation of when expressions are actually computed.
Memory usage can balloon if large unevaluated thunks pile up, leading to unexpected memory pressure. Though, Nix's performance woes come from elsewhere.

A very good question would be "why did you write this post?" Truth is, I want to make it clear that Nix is not one of your traditional languages. There also seems to be a trend of new Nix/OS users failing to understand Nix is a programming language, and it is a functional one no less. I wanted to make it very clear that some of the "issues" (such as dreadfully long error messages) are an occupational hazard.

To conclude, laziness in Nix removes the need for many traditional control flow constructs, albeit with its own set of caveats. Rather than guarding expensive computations with explicit conditionals, you often don't need to worry at all---unused expressions simply never evaluate. However, laziness introduces its own challenges, especially when debugging or managing performance. I want to make it very clear that most of the time, you can overcome those challenges by simply thinking about how you structure your program. In short, Nix is very demanding from the user but it almost always returns your investment in full.

Next time you are working with Nix, remember that you are using a domain specific language that is bound to have its own quirks. Regardless of said quirks, Nix is a very powerful language that trivializes Infrastructure as Code, but only if you treat it as code.

Static analysis tools, such as Statix will warn you when the function argument is unused. For example in the alwaysReturnOne function I've used as an example above, Statix would've warned you about the unused argument x. ↩
This is possible due to lazy evaluation resolving circular dependencies without causing crashes. However, an infinitely recursing attribute can exist as long as it's not evaluated. ↩
I'm very well aware that there is a fix function in nixpkgs lib. I mainly want to focus on core Nix language, so applications of lib.fix are omitted this time around. I'd like to talk about fix and recursion specifically in another post, another time. This is already long as it is. ↩

Stop Scraping my Git Forge!

Tue, 07 Jan 2025 00:00:00 GMT

Not so long ago, while performing maintenance on my VPS and setting up Prometheus, Grafana, and related tools, I noticed something unexpected yet not entirely surprising. My Nginx instance was under constant bombardment of requests from a specific subnet. Public-facing servers are no strangers to port scans, but the intensity and specificity of these requests caught my attention and prompted me to investigate further.

Hello, Old Friend

I have parted ways with Facebook before its acquisition of Instagram. I hate both of those platforms and the constant infringement of privacy they curse us with. You understand my surprise when I looked at my Nginx webserver log and found thousands of requests coming my way; all with the user agent facebookexternalhit. More specifically, they were sending GET requests to my personal Git forge---a service where I store personal projects that I prefer not to entrust platforms such as Microsoft Github. Over 30,000 lines were reserved to nothing but unsolicited GET requests in my access log. What the hell?

Instinctively, I put together a quick Python script that follows the service logs for Forgejo, which at the time also logged router events, and logs unique IPs to a single file so that I can ban those IPs with firewall rules, using the trusty nftables.

import subprocess
import re
import time
from datetime import datetime

logfile = "unique_ips.log"
ipv4_pattern = re.compile(r"\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b")

def read_logged_ips():
    try:
        with open(logfile, "r") as file:
            return set(file.read().splitlines())
    except FileNotFoundError:
        return set()


def log_ip(ip, timestamp, logged_ips):
    with open(logfile, "a") as file:
        file.write(f"{timestamp} - {ip}\n")
    logged_ips.add(ip)


def main():
    logged_ips = read_logged_ips()
    process = subprocess.Popen(
        ["journalctl", "-xeu", "forgejo.service", "-f"],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
    )

    while True:
        line = process.stdout.readline()
        if "404 Not Found" in line:
            ip_match = ipv4_pattern.search(line)
            if ip_match:
                ip = ip_match.group()
                if ip not in logged_ips:
                    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
                    log_ip(ip, timestamp, logged_ips)
                    print(f"Logged new IP: {ip} at {timestamp}")


if __name__ == "__main__":
    main()

I left the script, aptly and unimaginatively named catch.py running for around an hour while I went to fix a nice meal for myself--- which you need to deal with Meta's bullshit. When I came back, there were hundreds of unique IPs in the logfile. Not a handful, hundreds. Not only that, but they were coming from several different subnets. This means

Meta is using several different providers (as evident from IP queries) to simply send bogus requests to people's webservers.
Meta is, in fact, not respecting robots.txt.¹
They feel not at all inclined to maybe stop and think sending thousands of requests to people's servers is malicious.
To my demise, banning each and every single one of those subnets would take too long, and block possibly legitimate traffic.

Get Out of My Lawn

57.141.0.16 - - [22/Nov/2024:00:36:29 +0300] "GET /NotAShelf/nyx/commits/commit/4c82e5ee05fabd315a6e5c656fd72e11c93c4cfd/homes/notashelf/services/wayland/hyprpaper HTTP/2.0" 403 146 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

Here is an example of a request from my webserver logs. Take a look at the body. They are scraping my NixOS configuration that is available on my Git forge and they are doing it commit by commit.

Following this unfortunate encounter, I have decided to start blocking any requests coming with meta-externalagent* as their user agent. Such requests now receive a 403 Forbidden response as per my adjusted Nginx configuration.

# Define access and error log paths.
access_log /var/log/nginx/forgejo_access.log;
error_log /var/log/nginx/forgejo_error.log;

# Tell Facebook's crawlers to fuck right off.
if ($http_user_agent ~* "^meta-externalagent(/[\d\.]+)?$") {
  return 403;
}

if ($http_user_agent ~* "^facebookexternalhit(/[\d\.]+)?$") {
  return 403;
}

Okay, surely this will work. There is no way that a webcrawler is designed badly or maliciously enough to disregard hundreds of error response 403s, indicating that they are forbidden from accessing that URL. Surely?

Here is a request from today, shortly after I started writing this post.

57.141.0.17 - - [07/Jan/2025:13:12:06 +0300] "GET /NotAShelf/nyx/raw/commit/93d16a3edbaa8177c76ed84c4c5b489649faa609/modules/common/options/default.nix HTTP/2.0" 403 146 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

At this point, I am not sure if this incompetence or plain stupidity.

Ban Them All

This has been ~~yet another~~ a technical rant and a gentle nudge to you that you should check your webserver logs to see if your content is being scraped for, well, as long as it has been online.

My Forgejo instance is now inaccessible to those who have not logged in---even for public repositories--- and my Nginx configuration responds with 403 to any requests coming from Meta's known crawlers. I have also tried sending an-email to their webmaster as listed from the documentation URL attached to their crawlers. It has now been 4 months since, but I am yet to receive a response, or to see an indication of those crawlers stopping soon. Meanwhile, I am left to deal with their recklessness on my own accord.

Instead, I will resort to banning ALL of them with nftables using IP ranges. It is going to be annoying, but not fruitless, to write another Python script to identify each and every single subnet those requests are coming from. Once I have a list, it will be trivial to draft the rule. I suggest that you do the same.

Stay safe.

Meta's own web-crawler documentation indicates that they may choose to disregard your robots.txt if they are performing "integrity or security checks" which in truth is just a vague way to say that you may sometimes decide to not play by the rules. At this point your bet is banning them by IP range, which turns out to be listed in the very same documentation as the result of the command whois -h whois.radb.net -- '-i origin AS32934' | grep ^route. Naturally, I do not trust the information granted "graciously" by Meta, so I will be investigating my logfiles carefully to find if there is any more. ↩

Enter: Beginner's guide to Writing Error Messages

Thu, 02 Jan 2025 00:00:00 GMT

Programs fail, even the best-written ones. You know it; you've experienced it. You wish you could avoid it, but you can't. Computers are far from perfect. As a beginner programmer, you quickly learn that you must account for both expected and unexpected errors. Handling these errors isn't too difficult, and at this point, you have two options:

Throw a stack trace that gives insight into where the error occurred.
Provide a human-readable error message explaining the issue (or its absence) and offering debugging steps, if applicable.

While you might feel inclined to favor only one of these approaches, the correct solution is to provide both---but under different circumstances. This is where verbosity comes into play. Too much information can be overwhelming; too little can be useless. So, what's the perfect middle ground?

Who asked?

When I visit your website and your backend is down (unbeknownst to me), I really don't care to see your unhandled error message in full detail. The end user, who likely has no idea what the error message means, should never encounter it. It's you---the one responsible for fixing the issue---who needs that detailed information.

The end user should instead see a simple, polite message explaining that something went wrong and reassuring them that steps are being taken to resolve it. Offer them hope, not despair. Lie if you must, but don't scare them away. Keep verbose error messages hidden—where they belong---in your backend.

On the other hand, you do need verbose stack traces or whatever form your program's detailed crash information takes. You can't omit these, and you shouldn't. Instead, tuck them behind a flag or log them in a way that's accessible only to those who need them. Ensure that anyone actively troubleshooting can easily find this detailed information, while those who don't need it remain blissfully unaware.

The Good, The Bad, and Nix

This is a Nix blog, so of course, I'm going to complain about it. Nix has a very unique (for the lack of a better word) way of reporting errors. If the author of whatever project you are consuming is thoughtful enough to handle each case (be it via an assertion, as part of the module system, or with an extensive conditional), then you're likely to receive a very informative explanation of what went wrong and how to fix it. The module system is excellent at this.

If the error is not handled, however, you're left with an obscure message pointing to where the error originates. Hundreds of lines at the very least. In the context of a NixOS system (i.e. the Nixpkgs module system), errors often trace back to the entry point of the module system or something generic like config in specialArgs. This, folks, is bad design. Not only are you expecting the user to know exactly what they're looking at (minus points for the infamous --show-trace), but you're also presenting them with an intimidating wall of text that they'll naturally avoid. It's almost as if you don't want the error to be resolved...

Nix 2.20 has made... some attempts to improve this situation, but the language remains fundamentally flawed, and such errors are, unfortunately, still unavoidable.

All your error are belong to us

Here is an example of how I would like to approach errors.

const express = require("express");
const fs = require("fs");
const path = require("path");

const app = express();
const PORT = 3000;

function logErrorToFile(err, req) {
  const logFilePath = path.join(__dirname, "error.log");
  const logMessage = `
[${new Date().toISOString()}]
Error: ${err.message}
Route: ${req.originalUrl}
Stack Trace: ${err.stack}\n`;

  fs.appendFile(logFilePath, logMessage, (error) => {
    if (error) {
      console.error("Failed to write to log file:", error.message);
    }
  });
}

function errorHandler(err, req, res, next) {
  logErrorToFile(err, req);

  res.status(500).json({
    message: "Something went wrong. We are working to resolve the issue.",
  });
}

// A route that intentionally throws an error
app.get("/error", (req, res, next) => {
  try {
    throw new Error("This is a test error");
  } catch (err) {
    next(err); // we forward the error to the error-handling middleware
  }
});

app.use((req, res) => {
  res.status(404).json({
    message: "The resource you are looking for does not exist.",
  });
});

// Centralized error-handling middleware
app.use(errorHandler);

app.listen(PORT, () => {
  console.log(`Server is running on http://localhost:${PORT}`);
});

In the example above, error-logging and user-response responsibilities are clearly separated. The errorHandler middleware returns a clean and polite JSON response to the client: "Something went wrong. We are working to resolve the issue." The user doesn't need to know anything technical, so they don't. Letting technical information to the frontend is simply poor design.

This, I think, is the golden spot. Easy to maintain, not awful and more importantly not a thousand lines of meaningless, unintelligible text.

404: According to all known laws of aviation...

Now lets consider a bad example. No, a horrible example.

const express = require("express");
const app = express();
const PORT = 3000;

app.get("/error", (req, res) => {
  // Do you want my credit card details as well?
  res.status(500).send(`
        <h1>Error</h1>
        <p>Something  wrong!</p>
        <pre>
            Error: Failed to fetch resource
            at /error:10:15
            at Layer.handle [as handle_request] (node_modules/express/lib/router/layer.js:95:5)
            at next (node_modules/express/lib/router/route.js:144:13)
            at Route.dispatch (node_modules/express/lib/router/route.js:114:3)
            at Layer.handle [as handle_request] (node_modules/express/lib/router/layer.js:95:5)
            at /error:10:15
        </pre>
        <pre>
            Environment: ${JSON.stringify(process.env, null, 2)}
        </pre>
    `);
});

// No error handling middleware or logging mechanisms. Vomit everything as is.
app.listen(PORT, () => {
  console.log(`Server running at http://localhost:${PORT}`);
});

Don't ask me where I got that error message.

In this godawful example, the end-user is bombarded with irrelevant technical details like stack traces and environment variables. This is not only overwhelming and confusing, but it also exposes sensitive information that should never be seen by the user. The error message is not actionable and serves only to frustrate the user. More importantly, it's a security risk—exposing environment variables could allow a malicious party to take advantage of what you might consider technical mumbo-jumbo (and therefore safe to spew.)

Moreover, this approach is just plain ugly. A wall of text is neither helpful nor appropriate for a user who just wants to know what went wrong and if anything is being done to fix it. Imagine receiving a message like this on your own, and ask yourself—would you ever come back to this website?

If you take anything away from this, let it be that error handling should always prioritize clarity, security and proper compartmentalization.

Final Thoughts: Verbosity as a Tool, Not a Burden

This has been translated from a far less technical rant I've dropped earlier today on some painful case of error handling¹. Looking back, it was not the first time I've faced bad error handling. Lots of software I use on a daily basis simply print everything. In hindsight, this status quo is terrible and yet we continue to endure.

Error handling should not just about fixing problems. It must be moreso about how you communicate them. I'm sure the programmer who wrote that piece of code wants to know what went wrong, and where, but I don't. Unnecessary verbosity is just as harmful as a complete lack of information. So, for the sake of establishing a standard let me provide the following as a baseline:

Tailor to Your Audience: End-users want, nay, need reassurance and simplicity. Developers need depth and precision. Separate these layers effectively. Verbosity should be opt-in. Flags, environment variables, configuration options and so on. Don't just vomit information.
Respect Security: Never expose sensitive or otherwise unnecessary details to the frontend. Verbose logs belong in the backend, under restricted access. Security through obscurity is not security.
Keep It Manageable: Avoid overwhelming developers² with too much data. Structure logs well and keep error messages concise but informative.

This was all from me. If you have anything you want to add, do feel free to contact me. Let me also know of any unique cases of good and bad error handling that you might've ran into. Cheers!

Now you know where I got the error message. ↩
There are cases where I am both the developer and the end user. Assume, for a moment, that I am working on your codebase with minimal exposure. Most of your codebase is completely foreign to me. When you throw me a stack trace that is all over the place, I am immediately less inclined to contribute to your software. Pterodactyl (the game server management panel) has a very special way of handling errors. It spews everything, but provides helpful steps for the user. This is a good alternative to separating layers. ↩

Hello World

Wed, 01 Jan 2025 00:00:00 GMT

After hours on end of reckless consideration and rewriting, I have finally crafted my static site generator---a basic (but extremely inelegant) Bash script---and my blog is now fully live. Not that I needed to tell you; after all, you're already here.

I will spare you the technical details, mostly because the code itself is agonizing to look at, but this struck me as a more streamlined way to manage my blog. Using Pandoc and jq for core functionality (plus sassc for styling), I have effectively put together an extremely portable and simple static site generator. Heavy emphasis on simple, however, as it lacks any customizability I didn not need personally. Sure, it is slow---relying on Bash and a Haskell program will do that---but it is also exactly how I wanted it. Not slow, mind you. Just, very simple and straightforward.

The script, unlike most of my projects, does not have one of my signature code names I associate with beloved projects. Still, it excels at being compact: roughly 240 lines of code handle all functionality, fitting neatly into just a few screenfuls of text. For what it's worth, it does its job and it does it well enough above my expectations. Quite satisfied with how it turned out.

What's Next

Onto more exciting news: I plan to write more frequently now. All of my previous writeups have been archived (though still accessible) until I have a chance to review or rewrite them. The old blog was more of a renderer for my personal notes, but this time around, I am aiming for more in-depth posts. After almost two years, the groundwork is finally complete and it is about time I focused on actual writing instead of over-engineering the technical side of things.

If you feel inclined to reach out, let me know if there are any topics you would like me to explore. My primary focus will be on Nix and Linux, but I might delve into technical writeups about the languages I work with, as I work with them.

Happy new year, and all the best!

Please Just Stop!

Sun, 02 Jun 2024 00:00:00 GMT

Just is a fast and powerful command runner, built with Rust. It is fast, simple and available almost everywhere with top-notch editor and CI integration.

So why do I have beef with Just?

Please, just stop

I actually have no beef with Just, I have briefly read the project page and tested it somewhat extensively on my own machine. The tool itself is impressive, however, it is also redundant when tools like Nix exist!

Just do better!

If you have previously read my blog, you probably know that I use the Nix package manager¹ on a daily basis, and NixOS as my primary system.

For those who have no idea what Nix is, it's a package manager and a build tool that supports declarative, reproducible and reliable builds & deployments anywhere. I invite you to read more about it here

Nix does not exactly compare with Just, as Just is a command runner but it provides its full functionality in Nix devShells. You can use devShells not only to declaratively install your development packages inside specific project directories (which is built upon further by projects like direnv) but also write fully fledged scripts and helpers in Bash, Python, Ruby or whatever interpreted or compiled language you want to make them available in your projects.

A common counter-argument to this is that they don't want to force people to install yet another tool, however, Just is that "yet another tool" you said you don't want people to install. If I am installing something, I might as well opt in for the extensible one.

In the wild

My intention is neither to gatekeep Just, nor to advertise Nix for new people. What I want to get at is that if you have Nix installed, you do not need Just. Simple as that.

While browsing GitHub, I have seen countless Nix projects that recommend using Just to run some simple commands (such as nix build in large projects, or nixos-rebuild in system configurations). Please just stop.

When given a powerful tool (which is installed on your system) you should be utilizing that tool to its fullest extent, not introduce a completely irrelevant tool that is inferior to what you have in every way.

If you are using NixOS and want to provide bootstrapping commands to rebuild your system, use devShells.

Conclusion

And this has been my rant. Please understand that my aim is not to discourage you from using Just, but to encourage you to use the correct tool on your belt.

If you use NixOS but have zero clue how to use devShells, then I implore you to reconsider your choice of distro. NixOS is a powerful tool that needs, nay, demands your attention and understanding.

If you do not intend to use the blessings provided by Nix to you, then you probably do not need to suffer the many side effects of Nix (such as but not limited to severe hairloss).

What to do instead?

For my curious but equally lazy readers, here is how you may use a devShell.

Start by creating a shell.nix that contains your shell environment.

{ pkgs ? import <nixpkgs> {} }:
  pkgs.mkShell {
    # nativeBuildInputs is usually what you want -- tools you need to run
    nativeBuildInputs = with pkgs.buildPackages; [ ];
}

This setup implies you are using channels, but similar instructions will apply on flakes. Regardless, acquire your development packages from nixpkgs and put them in your shell's nativeBuildInputs. If you are trying to put, e.g., a Python script available in nixpkgs, simply place it in nativeBuildInputs. shell, then you can write your script as follows:

{ pkgs ? import <nixpkgs> {} }:
  pkgs.mkShell {
    nativeBuildInputs = with pkgs.buildPackages; [
      (pkgs.writeShellScriptBin "my-builder-script" ''
        echo "hello world"
      '')
    ];
}

This will place my-builder-script in your PATH after you enter the shell with e.g. nix develop or automagically if you use something like direnv. Obviously the contents can be whatever you want, and you can write a script as complex as you need. Aside from being a simple runner that did what Just does, you will also have a fully reproducible script that fulfils your needs without installing an additional tool thats sole purpose is to run commands.

If you use Just to run your rebuild scripts, you can easily package your rebuild scripts with Nix using writeShellScript* from trivial builders collection and completely ditch Just. Remember that someone observing your NixOS configuration is less likely to use Just, and more likely to use Nix. Choose the appropriate tool for the job, lest you over or underprepare.

Nix is actually not a package manager. It has been called that was to make it more approachable for the casual users, but it is actually a build tool. Or better yet, it is lambda calculus on files. I feel the need to clarify, because package management is only one of Nix's features. Whereas running commands is Just's only function. ↩

On SysRq

Sat, 11 May 2024 00:00:00 GMT

Resources

What is SysRq?

SysRq is a "magical"¹ key combination, or possibly a literal key on your keyboard that is understood by the Linux kernel, which allows you to perform various low-level commands regardless of the system's current state. Some Linux-first computer manufacturers may include a literal SysRq key, or chances are your PrtSc key (located somewhat close to your Numpad) doubles as it.

What can I do with it?

SysRq is most often used to recover from and debug an unresponsive system, especially if you are trying to avoid doing a hard shutdown - which could potentially cause data corruption - and is recommended over a hard shutdown. Using the SysRq key, you can communicate with the Linux kernel directly and reboot your correctly, without potentially nuking active disk writes.

How do I use SysRq?

You should first check if your system is configured to support SysRq. Some distros and users (such as myself) decide to disable SysRq because it is a potential security flaw.

To check whether your system is allowed to use SysRq, simply run

cat /proc/sys/kernel/sysrq

and you will receive a value that describes the functionality of your SysRq key, or whether or not it is allowed.

Possible Values

The kernel documentation gives you a neat list of values and what they allow.

Value	Bitmask	Description
0	0x0	disable sysrq completely
1	0x1	enable all functions of sysrq
2	0x2	enable control of console logging level
4	0x4	enable control of keyboard (SAK, unraw)
8	0x8	enable debugging dumps of processes etc
16	0x10	enable sync command
32	0x20	enable remount read-only
64	0x40	enable signalling of processes (term, kill, oom-kill)
128	0x80	allow reboot/poweroff
256	0x100	allow nicing of all RT tasks

Setting SysRq Bitmask

You can set SysRq to 0 in order to disable it, 1 to allow ALL functionality and to a value >= 1 to set a bitmask² of allowed SysRq functions.

echo "number" >/proc/sys/kernel/sysrq

To combine different functionalities in the SysRq permission bitmap, you will need to perform bitwise OR operations on the values corresponding to the functionalities you want to enable. Each functionality corresponds to a specific bit in the bitmap.

E.g., let's say you want to enable the functionalities for controlling console logging level (0x2), enabling sync command (0x10), and allowing reboot/poweroff (0x80). To combine these functionalities:

0x2 (console logging level) | 0x10 (sync command) | 0x80 (reboot/poweroff)

Which would perform the following bitwise OR operation:

0x2 | 0x10 | 0x80 = 0x92

This hexadecimal value would then be converted to decimal (which the kernel commandline expects) as 146. Therefore you would run the following command to set your desired bitmask:

echo 146 > /proc/sys/kernel/sysrq

Persisting SysRq

You can enable SysRq with one command, as described above. Do keep in mind that running e.g. echo 1 > /proc/sys/kernel/sysrq will enable SysRq only until reboot and its state will not persist across reboots.

On traditional distros, you can make it persistent by adding "kernel.sysrq = 1" at the end of /etc/sysctl.d/99-sysctl.conf.

On NixOS, you will need to set boot.kernel.sysctl."kernel.sysrq" = 1; in your configuration.nix. In some cases, you might need to force this value in case it is being overridden:

boot.kernel.sysctl."kernel.sysrq" = lib.mkForce 1;

Useful applications of SysRq

There are various applications of SysRq based on the bitmask range it was allowed. Below are the two most common ones that I came across.

Rebooting with SysRq

Earlier I have mentioned that SysRq is often used to perform a proper reboot and to recover a frozen system. With a permissive enough³ bitmask, you can use the following idiom: "Reboot Even If System Utterly Broken" (also referred to as "REISUB").

R: Turns off keyboard raw mode, and sets it to XLATE.
E: Send a SIGTERM to all processes, except for init.
I: Send a SIGKILL to all processes, except for init.
S: Will attempt to sync all mounted filesystems.
U: Will attempt to re-mount all mounted filesystems as read-only.
B: Reboot!

Please be aware that "REISUB" itself is just a mnemonic, not any kind of general recommendation for the key press sequence to take back control of an unresponsive system. You should not blindly press these sequences each time without knowing their actual function as noted below⁴

To do this on your system, you what you need to do is to hold down the ALT key and the SysRq keys at the same time, hit the next queued letter while holding the key combination, and then release. Repeat this for each letter (command) in the idiom. To summarize, running the REISUB sequence would require you to hold down ALT + SysRq 6 separate times

Invoking OOM Killer

SysRq can also be used to invoke the OOM (out-of-memory) killer without causing a kernel panic if there is nothing to kill. If your system is frozen due to intense memory pressure, this could be an useful way to quickly get yourself out of a status that might corrupt your data. Simply run ALT + SysRq + f and hope that OOM killer picks something recoverable.

Keep in mind that the OOM killer, despite its well-meaning heuristics, can be unpredictable and lead to irreversible damage. Do not use this key combination casually.

Is SysRq worth it?

Those who have gone through my system configuration might have noticed that I force SysRq to be disabled with kernel.sysrq set to 0.

That is because I never had the need to use SysRq. After years on Linux, I never had to recover from a system freezing over a long duration. It either lives, or it dies. Recently I have configured OOM killer daemon to automatically kill useless applications such as Electron, that tend to drain my memory - especially on low-end systems because despite what devs might say, electron still sucks. As such, I do not think that SysRq is worth the potential security flaw right, but that question is for you to answer yourself.

Quite literally what it is called in the kernel documentation ↩
The permission bitmap is a 32-bit bitmap that determines which SysRq commands are allowed to be executed. Each bit in the bitmap corresponds to a specific SysRq command. When a SysRq command is issued, the kernel checks the corresponding bit in the permission bitmap to determine whether the command is allowed or not. ↩
The whole set of REISUB functions can be enabled by setting SysRq value to 244, although this will also enable additional functions which some may find undesirable. If in doubt, do the math and choose your bitmask range yourself. ↩
Archwiki on SysRq ↩

Headscale on NixOS, Tailscale anywhere

Sat, 11 Nov 2023 00:00:00 GMT

Today's main attraction is the Headscale setup on my VPS running NixOS, which I've finally came around to self-host.

There has been much talk about this new product called Tailscale recently around the web, especially in the last few years. Tailscale is a VPN service that makes the devices and applications we own accessible anywhere using the open source WireGuard protocol to establish encrypted point-to-point connections. I have been using Tailscale for a while now, but in an effort to move all of my services to self-owned hardware some of my services have been moved over to my NixOS server over time.

Many of Tailscale's components are open-source, especially its clients, but the server remains closed-source. Tailscale is a SaaS product and monetization naturally is a big concern, however, we care more about controlling our own data than their attempts of monetization.

This is where the (very appropriately named) Headscale comes in; Headscale is an open-source, self-hosted implementation of the Tailscale control server. The configuration is extremely straightforward, as Headscale will handle everything for us.

Running Headscale

Below is a simple configuration for the Headscale module of NixOS.

services = let
  domain = "example.com";
in {
  headscale = {
    enable = true;
    address = "0.0.0.0";
    port = 8085;

    settings = {
      server_url = "https://tailscale.${domain}";

      dns_config = {
        override_local_dns = true;
        base_domain = "${domain}";
        magic_dns = true;
        domains = ["tailscale.${domain}"];
        nameservers = [
          "9.9.9.9" # no cloudflare, nice
        ];
      };

      ip_prefixes = [
        "100.64.0.0/10"
        "fd7a:115c:a1e0::/48"
      ];
    };
  };
};

Using Headscale

We must first create a user, which we can do with

headscale users create myUser

Then on the machine that will be our client, we need to login.

tailscale up --login-server tailscale.example.com # replace this URL with your own as configured abovea

Followed by registering the machine.

# machine key will be obtained visiting the URL that is returned from the above command
headscale --user myUser nodes register --key <MACHINE_KEY>

And finally logging into your Tailnet using the URL and your machine key.

tailscale up --login-server https://tailscale.example.com --authkey <YOUR_AUTH_KEY>

And all done! Now try connecting to one of your machines using the hostname now to test if the connection is actually working. If anything goes wrong, make sure to check your DNS settings: remember, it's always the DNS.

Nix Remote Builders on non-standard OpenSSH ports

Fri, 14 Jul 2023 00:00:00 GMT

My VPS, which hosts some of my infrastructure, has been running NixOS for a while now. Although weak, I use it for distributed builds alongside the rest of my NixOS machines on a Tailscale network.

This server, due to it hosting my infrastructure that communicates with the rest of the internet (i.e my mailserver), is somewhat responsive to queries from the public - which includes very aggressive port scans (thanks, skiddies!)

To mitigate that, I have decided to change the ssh port from the default 22 ... this is not exactly a panacea, it helps alleviate ... the insane log spam I get from failed ssh requests.

The OpenSSH Configuration

First thing we've done is to configure SSH daemon to listen on the new port on your server configuration

services.openssh = {
  ports = [2222];
}

With this set, openssh on the server will now be listening on the port 2222 instead of the default 22. For the changes to take effect after a rebuild, you might need to run systemctl restart sshd.socket.

Then we want to configure our client to use the correct port for our server instead of the default 22.

programs.ssh.extraConfig = ''
    Host nix-builder
      HostName nix-builder-hostname # if you are using Tailscale, this can just be the hostname of a device on your Tailscale network
    Port 2222
'';

And done, that is all for the ssh side of things. Next up, we need to configure Next up, we need to configure our builder to use the correct host.

Nix Builder Configuration

Assuming you already have a remote builder configured, you will only need to patch the hostName with the one on your openssh.extraConfig.

{
  nix.buildMachines = [
    {
      hostName = "nix-builder-hostname";
      sshUser = "nix-builder";
      sshKey = "/path/to/key";
      systems = ["x86_64-linux"];
      maxJobs = 2;
      speedFactor = 2;
      supportedFeatures = ["kvm"];
    }
  ];
}

If you have added the correct hostName and sshUser, the builder will be picked up automatically on the next rebuild.

Home-Manager

If you are using Home-Manager, you might also want to configure your declarative ~/.config/ssh/config to use the new port. That can be achieved through programs.ssh.matchBlocks option under Home-Manager

programs.ssh.matchBlocks = {
  "builder" = {
    hostname = "nix-builder-hostname";
    user = "nix-builder";
    identityFile = "~/.ssh/builder-key";
    port = 2222;
  };
}

And that will be all. You are ready to use your new non-default port, mostly safe from port scanners.

Packaging NextJS Webapps with Nix

Sun, 21 May 2023 00:00:00 GMT

Recently I have had to go through the misfortune of hosting some websites written with NextJS on my VPS running NixOS, this note entry shall document my experience and the "easy" path I have chosen.

Packaging

The websites I hosted were of two variety: those statically exported, and those that cannot be statically exported.

Statically Exported Webapps

Statically exported ones are easy to package, because it is a matter of running npm build (or whatever your build script is) with the following NextJS settings

// next.config.js
module.exports = {
  distDir: "dist", // an arbitrary path for your export
  output: "export",
};

This will export a static website with a bunch of html files that you can then serve with nodePackages.serve or a webserver like nginx or apache. And that is the end of your worries for a statically exported website! No headache, just write a simple derivation, such as the one below

# default.nix
{
  buildNpmPackage,
  pkg-config,
  python3,
  ...
}:
buildNpmPackage {
  pname = "your-website";
  version = "0.1";

  src = ./.;
  # needs to be updated everytime you update npm dependencies
  npmDepsHash = "sha256-some-hash";
  # some npm packages may need to be built from source, because nodejs is a *terrible* ecosystem
  nativeBuildInputs = [pkg-config python3];

 # move exported website to $out
 postInstall = ''
    cp -rf dist/* $out
  '';
}

and serve its path with a simple tool after building the derivation, I find nginx to be awfully convenient for doing so, but you may choose caddy if you prefer.

Webapps that cannot be statically exported

If your website depends on API routes for some reasons, then Next will not allow you to do static export. Which means you need to run next start in some shape or form. While a systemd service is certainly a way of doing it (one that I do not recommend), a oci container works as well if not better.

You can write a "simple" docker image for your oci container to use, such as the one below

# dockerImage.nix
{
  pkgs,
  inputs,
  ...
}: {
  dockerImage = pkgs.dockerTools.buildImage {
    config = {
      WorkingDir = "/your-website";
      Cmd = ["npm" "run" "serve"];
    };

    name = "your-website";
    tag = "latest";

    fromImage = pkgs.dockerTools.buildImage {
      name = "node";
      tag = "18-alpine";
    };

    copyToRoot = pkgs.buildEnv {
      name = "image-root";

      paths = with pkgs; [
        # this package is called from a flake.nix alongside the derivation for the website
        inputs.self.packages.${pkgs.stdenv.system}.your-website
        nodejs
        bash
      ];

      pathsToLink = [
        "/bin"
        "/your-website"
      ];
    };
  };
}

Then, configure oci-containers module option to pick up the Docker image that you have built. This is a simplified version of my VPS' container setup. An example can be found in my server module

virtualisation.oci-containers = {
  backend = "podman";
  containers = {
    "website-container" = {
      autoStart = true;
      ports = [
        "3000:3000" # bind container's port 3000 to the outside port 3000 for NextJS
      ];

      extraOptions = ["--network=host"];

      image = "your-website";
      imageFile = inputs.website-flake.packages.${pkgs.stdenv.system}.dockerImage;
    };
  };
};

After a rebuild, your system will provision the container and start it on port 3000. You can access it with your-server-ip:3000 in your browser, and even configure nginx to set up a reverse proxy to assign your domain.

"example.com" = {
  locations."/".proxyPass = "http://127.0.0.1:3000";
};

This will assign your domain to your webserver, and allow outside visitors to view your "awesome" NextJS webapp.

System Backlight

Sun, 22 Jan 2023 00:00:00 GMT

Following a large system upgrade two days ago, my HP Pavilion laptop has stopped registering the intel_backlight device interface in /sys/class/backlight - which is most often used by tools such as brightnessctl or light to control active backlight level. Instinctively looking at the result of dmesg, I came across an insanely vague error message that the kernel was unable to load some modules. Thanks, I guess?

The solution

After some very extensive research, obviously on Google as every other confused Linux user would do, I came across this forum post that mentioned a change in backlight behaviour sometime after kernel version 6.1.4. Fortunately for me, the article was also referring to the ever so informative Archwiki that instructed passing one of the three kernel command-line options depending on our needs.

What I did not know at the time was that when I upgraded my kernel from 6.1.3 to 6.1.6 with a nix flake update, the acpi_backlight=none parameter had made it so that it would skip loading Intel backlight entirely. Simply switching this parameter to acpi_backlight=native as per the article above has fixed the issue.

Lessons learned

Linux devs are nerds.

NotAShelf's Blog

Nix's Substituter List Is Not a Routing Table

The Shape of the Problem

ncro, Briefly

What's Actually In It

What I Did Not Build

Was It Worth Writing?

Footnotes

Why I think Go is a Terrible Language

Failure by Design

Let's Talk Generics

But The Concurrency

At Least It's Simple

Tooling

Closing Thoughts

Footnotes

The Nihilist's Guide to Cross-Compiling Dioxus for Windows

Why Dioxus

Building with Dioxus

Enter: Webview

The Nix Way

The Final Devshell

Metadata and Identity

Wine Tasting

Conclusion

2025 Wrapped & 2026 Wishlist

Project News

Blog Updates

Other Stuff

2026 Wishlist: bigger than projects, smaller than miracles

Closing Thoughts

The Tradeoff Trap

The Itch That Builds Complexity

The Joy Tax

A Fork in the Road

Chisel and the Sledgehammer

Footnotes

I am Not Convinced by Vibe Coding

The Machine That Designed Cars

What Vibe Coding Forgets

The Shortcut That Becomes a Detour

When Vibe Coding Makes Sense

You Cannot Abdicate Responsibility

Closing Thoughts

Footnotes

The Federation Fallacy

Interlude: On Data Ownership

Federation Fallacy: Continued

Federation: The False Promise of Decentralization

Closing Thoughts

Footnotes

On Mutation Testing

A Practical Example

Postscript

Mutation Testing in Rust: cargo-mutants

Closing Thoughts

Footnotes

The Curse of Knowing How, or; Fixing Everything

Technical Capability as a Moral Weight

One Must Imagine Sisyphus Happy

Entropy Is Undefeated

The Illusion of Finality

Technical Work as Emotional Regulation

The Burnout You Don't See Coming

Learning to Let Go

A New Kind of Skill

Post-Mortem

Footnotes

Considerations

My New Tech Stack

The Golang Chapter

Why Rust?

1. Memory Safety without Performance Penalties

2. True Zero-Cost Abstractions

3. Concurrency Without Race Conditions

4. Tooling That Actually Works

Considering Zig

Why C Still Has a Place

Go Has Replaced Python for Web and Scripting

Final Thoughts

Anatomy of `runNixOSTest`

What is `nixpkgs.lib`

Why would you need to extend `nixpkgs.lib`

Extending `nixpkgs.lib`