opam hangs when updating repositories

[mtelvers]

opam-repo-ci uses the Docker base images. The Docker base images build opam from the master branch each week. This week, we are seeing many job timeouts. The jobs hang at the point where they run opam init --reinit --ni.

 => [ 4/11] RUN opam init --reinit -ni                                                                                                                                                  
 => => # You may want to back it up before going further.                                                                                                                                      
 => => # Continue? [Y/n] y                                                                                                                                                                     
 => => # [NOTE] The 'jobs' option was reset, its value was 39 and its new value will vary according to the current number of cores on your machine. You can restore the fixed value using:     
 => => #            opam option jobs=39 --global                                                                                                                                               
 => => # Format upgrade done.                                                                                                                                                                  
 => => # <><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>                                                                                                      

Typically, this step takes < 2 minutes, but now takes > 2 hours.

Steps to reproduce. Paste the following into the shell. Change opam-2.3 to opam-dev to see the difference. On my machine, with opam-2.3 the build was completed in ~4 minutes. With opam-dev it has been running for nearly two hours

cd $(mktemp -d)
git clone --recursive "https://github.com/ocaml/opam-repository.git" && cd "opam-repository" && git fetch origin "refs/pull/27696/head" && git reset --hard 46b8cc5a
git fetch origin master
git merge --no-edit 4d8fa0fb8fce3b6c8b06f29ebcfa844c292d4f3e
cat > ../Dockerfile <<'END-OF-DOCKERFILE'
FROM ocaml/opam:debian-12-ocaml-4.09@sha256:13bd7f0979922adb13049eecc387d65d7846a3058f7dd6509738933e88bc8d4a
USER 1000:1000
WORKDIR /home/opam
RUN sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam
RUN opam init --reinit -ni
RUN opam option solver=builtin-0install && opam config report
ENV OPAMDOWNLOADJOBS="1"
ENV OPAMERRLOGLEN="0"
ENV OPAMPRECISETRACKING="1"
ENV CI="true"
ENV OPAM_REPO_CI="true"
RUN rm -rf opam-repository/
COPY --chown=1000:1000 . opam-repository/
RUN opam repository set-url --strict default opam-repository/
RUN opam update --depexts || true
RUN opam pin add -k version -yn chrome-trace.3.18.0~alpha0 3.18.0~alpha0
RUN opam reinstall chrome-trace.3.18.0~alpha0; \
    res=$?; \
    test "$res" != 31 && exit "$res"; \
    export OPAMCLI=2.0; \
    build_dir=$(opam var prefix)/.opam-switch/build; \
    failed=$(ls "$build_dir"); \
    partial_fails=""; \
    for pkg in $failed; do \
    if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"debian-12\""; then \
    echo "A package failed and has been disabled for CI using the 'x-ci-accept-failures' field."; \
    fi; \
    test "$pkg" != 'chrome-trace.3.18.0~alpha0' && partial_fails="$partial_fails $pkg"; \
    done; \
    test "${partial_fails}" != "" && echo "opam-repo-ci detected dependencies failing: ${partial_fails}"; \
    exit 1

END-OF-DOCKERFILE
docker build -f ../Dockerfile .

[mtelvers]

Update: I have switched opam-repo-ci to use opam-2.3 rather than opam-dev, which seems to have restored normal operation. 7174953145735a54ecf668c7387e57b3f2d2a411. @shonfeder @maiste

[kit-ty-kate]

Probably something from #5892. I'll have a look.

[kit-ty-kate]

After a little debugging it seems to come from the fact that the opam-repository used in the dockerfile is set to file:///home/opam/opam-repository, however it is in fact a git repository and thus opam will try to diff all the files in the .git subdirectory too and the new patch implementation fails on that (i'll have a further look, it definitely shouldn't).
Interestingly locally i get a reasonably fast assert false instead of an infinit loop. So maybe when ran in some environments there is also some performance issues to look at.

In the meantime though things could be fixed on your end by changing ocaml-base-images to initialize opam (2.0) using git+file:///home/opam/opam-repository (-k git) instead of the current -k local. Weirdly enough it seems to already be the case in ocaml-dockerfile (https://github.com/ocurrent/ocaml-dockerfile/blob/2e407583a707ae7382b4e5bb3ea5a772f4f6b04d/src-opam/opam.ml#L657) but not in docker-base-images (https://github.com/ocurrent/docker-base-images/blob/b9586aeaeb429cbce9ee8c02a457050e8897db72/src/pipeline.ml#L177), which while using the former, seems to duplicate some code. From memory i don't think this was the case before so i think this is a regression in docker-base-images.