Devenv v2 docker image already working with services?

[pbek]

Describe the bug

We are using a (private) GitLab CI to run database migration tests with devenv test.

Expand

# DB Migration Tests
devenv-test:
    rules:
        - if: $CI_PIPELINE_SOURCE != "schedule"
    stage: test
    image: ghcr.io/cachix/devenv/devenv:v2.0.2
    script: devenv test
    retry: 2

Our devenv.nix looks something like:

Expand

{
  pkgs,
  config,
  ...
}:

{
  # https://devenv.sh/supported-languages/php/
  languages.php = {
    enable = true;
    # We are using PHP 8.3 (and not 8.4), because of php-cs-fixer
    version = "8.3";
    ini = ''
      memory_limit = 256M
    '';
    extensions = [
      "ldap"
      "gd"
      "pdo_mysql" # For "devenv test"
      "soap"
      "ldap"
      "mbstring"
      "intl"
      "curl"
      "readline"
      "zip"
      "redis"
      "gmp"
      "sqlite3"
    ];
  };

  # https://devenv.sh/supported-services/mysql/
  services.mysql = {
    enable = true;
    initialDatabases = [
      { name = "db1"; }
      { name = "db2"; }
    ];
    ensureUsers = [
      {
        name = "db";
        password = "secret";
        ensurePermissions = {
          "db1.*" = "ALL PRIVILEGES";
          "db2.*" = "ALL PRIVILEGES";
        };
      }
    ];
  };

  # https://devenv.sh/integrations/dotenv/
  dotenv = {
    enable = true;
    filename = [
      # For the tests to run you need access to the SYMFONY_DECRYPTION_SECRET environment variable from .env.dev.local
      ".env.dev.local"
    ];
  };

  # https://devenv.sh/scripts/
  scripts = {
    db-clean.exec = ''
      echo "🗑️ Cleaning DB state"
      rm -rf .devenv/state/mysql/
    '';

    db-migrate.exec = ''
      echo "📥 Running DB migrations"
      php bin/console dbp:relay:core:migrate --no-interaction
    '';
  };

  env.BIN_DIR = "${config.devenv.root}/bin";

  enterShell = ''
    echo "🔨 Middleware API shell"
    echo "🐘 PHP version: $(php --version | head -n 1)"
    echo "DEVENV_ROOT: $DEVENV_ROOT"

    echo "🔗 Updating PHP symlinks for PhpStorm in ''${BIN_DIR}..."
    ln -fs ${pkgs.php83}/bin/php "''${BIN_DIR}/php"
    ln -fs ${pkgs.php83}/bin/phpdbg "''${BIN_DIR}/phpdbg"
  '';

  # https://devenv.sh/tests/
  enterTest = ''
    wait_for_port 3306
    echo "🔨 Loading .env.devenv_test"
    set -a
    source .env.devenv_test
    set +a
    echo "🔨 Doing composer install"
    composer install
    echo "🔨 Running DB clean"
    db-clean
    echo "🔨 Running DB migrations"
    db-migrate
  '';

  # https://devenv.sh/packages/
  packages = with pkgs; [
    symfony-cli
    sshuttle # for dispatch / Vendo
    fzf # fuzzy finder
  ];

  # https://devenv.sh/git-hooks/
  git-hooks = {
    excludes = [
      "config/secrets"
      "vendor"
      "bin"
    ];
    hooks = {
      php-cs-fixer = {
        enable = true;
        args = [
          "--config"
          "./.php-cs-fixer.dist.php"
        ];
        excludes = [ "config/bundles.php" ];
        # Don't spam so many messages
        require_serial = true;
      };

      prettier.args = [
        "--tab-width"
        "4"
      ];
    };
  };

  # See full reference at https://devenv.sh/reference/options/
}

This worked great so far with the devenv v1 docker image, but now with devenv v2 we run into a timeout after spending the rest of an hour waiting after a log message Error reading stdout: stream did not contain valid UTF-8. Before we get that error message it is successfully setting up the database, doing migrations and so on. But after a bit over 4 min we get that error message.

Last log lines:

Expand

[2026-03-09T14:16:52.746008+00:00] app.INFO: ++ migrating Dbp\Relay\AuthorizationBundle\Migrations\Version20251111142100 (create the authorization_group_resource_members table) {"version_name":"Dbp\\Relay\\AuthorizationBundle\\Migrations\\Version20251111142100 (create the authorization_group_resource_members table)"} []
[2026-03-09T14:16:52.746069+00:00] app.DEBUG: CREATE TABLE authorization_group_resource_members (identifier BINARY(16) NOT NULL COMMENT '(DC2Type:uuid_binary)', group_authorization_resource_identifier BINARY(16) DEFAULT NULL COMMENT '(DC2Type:uuid_binary)', member_authorization_resource_identifier BINARY(16) DEFAULT NULL COMMENT '(DC2Type:uuid_binary)', PRIMARY KEY (identifier)) DEFAULT CHARACTER SET utf8mb4 COLLATE `utf8mb4_unicode_ci` ENGINE = InnoDB  {"query":"CREATE TABLE authorization_group_resource_members (identifier BINARY(16) NOT NULL COMMENT '(DC2Type:uuid_binary)', group_authorization_resource_identifier BINARY(16) DEFAULT NULL COMMENT '(DC2Type:uuid_binary)', member_authorization_resource_identifier BINARY(16) DEFAULT NULL COMMENT '(DC2Type:uuid_binary)', PRIMARY KEY (identifier)) DEFAULT CHARACTER SET utf8mb4 COLLATE `utf8mb4_unicode_ci` ENGINE = InnoDB","params":""} []
[2026-03-09T14:16:52.752496+00:00] app.DEBUG: ALTER TABLE authorization_group_resource_members ADD CONSTRAINT FK_5C8C8E3E213E78E1 FOREIGN KEY (group_authorization_resource_identifier) REFERENCES authorization_resources (identifier) ON DELETE CASCADE  {"query":"ALTER TABLE authorization_group_resource_members ADD CONSTRAINT FK_5C8C8E3E213E78E1 FOREIGN KEY (group_authorization_resource_identifier) REFERENCES authorization_resources (identifier) ON DELETE CASCADE","params":""} []
[2026-03-09T14:16:52.768617+00:00] app.DEBUG: ALTER TABLE authorization_group_resource_members ADD CONSTRAINT FK_5C8C8E3E6F5DFD0B FOREIGN KEY (member_authorization_resource_identifier) REFERENCES authorization_resources (identifier) ON DELETE CASCADE  {"query":"ALTER TABLE authorization_group_resource_members ADD CONSTRAINT FK_5C8C8E3E6F5DFD0B FOREIGN KEY (member_authorization_resource_identifier) REFERENCES authorization_resources (identifier) ON DELETE CASCADE","params":""} []
[2026-03-09T14:16:52.789194+00:00] app.INFO: Migration Dbp\Relay\AuthorizationBundle\Migrations\Version20251111142100 migrated (took 43.2ms, used 40.5M memory) {"version":"Dbp\\Relay\\AuthorizationBundle\\Migrations\\Version20251111142100","time":43.19999999999999,"memory":"40.5M","direction":"migrated"} []
[2026-03-09T14:16:52.790500+00:00] app.INFO: ++ migrating Dbp\Relay\AuthorizationBundle\Migrations\Version20251112093200 (create the tables authorization_available_resource_class_actions and authorization_available_resource_class_action_names and populate them with the group actions) {"version_name":"Dbp\\Relay\\AuthorizationBundle\\Migrations\\Version20251112093200 (create the tables authorization_available_resource_class_actions and authorization_available_resource_class_action_names and populate them with the group actions)"} []
[2026-03-09T14:16:52.790689+00:00] app.DEBUG: CREATE TABLE authorization_available_resource_class_actions (identifier BINARY(16) NOT NULL, resource_class varchar(40) NOT NULL, action varchar(40) NOT NULL, action_type tinyint(1), PRIMARY KEY (identifier)) DEFAULT CHARACTER SET utf8mb4 COLLATE `utf8mb4_unicode_ci` ENGINE = InnoDB  {"query":"CREATE TABLE authorization_available_resource_class_actions (identifier BINARY(16) NOT NULL, resource_class varchar(40) NOT NULL, action varchar(40) NOT NULL, action_type tinyint(1), PRIMARY KEY (identifier)) DEFAULT CHARACTER SET utf8mb4 COLLATE `utf8mb4_unicode_ci` ENGINE = InnoDB","params":""} []
[2026-03-09T14:16:52.797542+00:00] app.DEBUG: ALTER TABLE authorization_available_resource_class_actions ADD CONSTRAINT unique_resource_class_action UNIQUE (resource_class, action, action_type)  {"query":"ALTER TABLE authorization_available_resource_class_actions ADD CONSTRAINT unique_resource_class_action UNIQUE (resource_class, action, action_type)","params":""} []
[2026-03-09T14:16:52.811078+00:00] app.DEBUG: CREATE TABLE authorization_available_resource_class_action_names (available_resource_class_action_identifier BINARY(16) NOT NULL, language_tag VARCHAR(2) NOT NULL, name VARCHAR(64) NOT NULL, PRIMARY KEY(available_resource_class_action_identifier, language_tag), FOREIGN KEY(available_resource_class_action_identifier) REFERENCES authorization_available_resource_class_actions(identifier) ON DELETE CASCADE)  {"query":"CREATE TABLE authorization_available_resource_class_action_names (available_resource_class_action_identifier BINARY(16) NOT NULL, language_tag VARCHAR(2) NOT NULL, name VARCHAR(64) NOT NULL, PRIMARY KEY(available_resource_class_action_identifier, language_tag), FOREIGN KEY(available_resource_class_action_identifier) REFERENCES authorization_available_resource_class_actions(identifier) ON DELETE CASCADE)","params":""} []
Error reading stdout: stream did not contain valid UTF-8
WARNING: step_script could not run to completion because the timeout was exceeded. For more control over job and script timeouts see: https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts
ERROR: Job failed: execution took longer than 1h0m0s seconds

Locally on my NixOS box I can run devenv test just fine with devenv v2!

To reproduce

Disclaimer: Will be hard to reproduce, since I can't share the full infrastructure. 😬 I was just hoping something rings a bell why the docker image is not working the same still.

Running docker run --rm -e DEVENV_HOME=/tmp/.devenv-home -v "$PWD":/work -w /work ghcr.io/cachix/devenv/devenv:v1.11.1 bash -lc 'devenv test' (or with ghcr.io/cachix/devenv/devenv:v1.11.1 in GitLab CI) did work.

Running docker run --rm -e DEVENV_HOME=/tmp/.devenv-home -v "$PWD":/work -w /work ghcr.io/cachix/devenv/devenv:v2.0.2 bash -lc 'devenv test' ran into a timeout after not outputting anything else after an error message Error reading stdout: stream did not contain valid UTF-8.

Version

devenv 2.0.1 (x86_64-linux)

[pbek]

@sandydoo, could https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv be pushed to 2.0.5, so I could test if the issue was fixed by chance?

[pbek]

Thank you, @domenkozar! When will that hit https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv so I can test? 😊

[domenkozar]

I'm looking into why containers don't build.

[domenkozar]

I've fixed the container job and it was updated in https://github.com/cachix/devenv/actions/runs/23391843159

[pbek]

Thank you very much! Will latest also be bumped on https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv? Currently, main is updated.

[pbek]

@domenkozar, @sandydoo, can it be that devenv 2.0.5 has a problem with stale caches when running devenv test? While I was trying to test and debug the main tag of https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv in GitLab I also tried to run devenv test locally and got different issues all the time. After a while I started to put in broken commands in devenv.nix on purpose, but devenv test still ran happily. Only when I removed the .devenv* folders in the project I can see changes.

Later even Sonnet came to the same conclusion:

devenv caches the result of evaluating devenv.nix in .devenv/nix-eval-cache.db. This cache was stale — it kept pointing to /nix/store/wc7hwyr2nmins82gkgr3bpm6wh44g31d-devenv-test, which was the original script with db-clean. All our devenv.nix edits were being ignored.

[domenkozar]

Yes, we've fixed that bug in 2.0.6!

[pbek]

Thank you for confirmation! And here I was, happy that 2.0.5 finally was in the cache servers of nixpkgs unstable today (where currently kde-vault and netbird is broken, and I had to do overlays for that), so I can remove my devenv 2.0.5 overlay. And now I have to wait for 2.0.6. 😂🤣

Is https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv/752239756?tag=main at 2.0.6? Will there be tags for 2.0.*?

[domenkozar]

I hope we have fixed the most annoying bugs since 2.0 release, things seems to be calming down!

You can track 2.0.6 hitting nixpkgs at https://nixpk.gs/pr-tracker.html?pr=502149

[pbek]

Is https://github.com/cachix/devenv/pkgs/container/devenv%2Fdevenv/752239756?tag=main at 2.0.6? Will there be tags for 2.0.*?

And tags for the docker image?