A few other projects have started using https://codspeed.io/ for performance benchmarking. Setup is straightforward - enable the CodSpeed pytest plugin, add an @pytest.mark.benchmark test, and enable the repo in CodSpeed. Then we can start tracking performance benchmarks, with comparative flamegraphs showing the difference in execution time.

We may need to move from CircleCI to Github Actions to take advantage of this fully. Also, the reports go to Github PRs and email, however we do code review for moz-phab on Phabricator. We would need to confirm we can get similar reports for pushes to main, and hopefully also pushes to other branches for comparison.

Adds pytest-codspeed and a benchmarks GHA job authenticating to
codspeed.io via OIDC. Two end-to-end benchmarks drive submit and
patch through mozphab.main([...]) against a real git_repo_path,
with only the HTTP boundary mocked.

The action is configured for walltime mode (mode: walltime).
codspeed's CPU simulation mode would be deterministic but only
instruments the parent Python process -- moz-phab's actual workload
is dominated by git subprocesses and HTTPS calls, so simulation
reports < 1% of real wall time. Walltime is noisier on shared GHA
runners but captures the real cost; BENCHMARKING.md spells out the
tradeoff.

BENCHMARKING.md documents the setup, how to run locally, CI
integration, and codspeed's limitations for this tool. Inspired by
D295969, using codspeed instead of a bespoke framework and
exercising the real Repository layer against a real git repo.

Wraps ConduitAPI.call with a Counter and runs the real submit
and patch flows end-to-end, asserting exact per-method breakdowns.
Catches new uncached API trips and broken cache invariants; stays
quiet for refactors that only touch SimpleCache-served helpers.

Runs in the normal pytest job, not under codspeed. Intentional count
changes update the expected dict and the diff signals the perf
change to reviewers.

Mirrors the Conduit-side regression test for the subprocess
boundary. Wraps mozphab.gitcommand.check_call and check_output
with a Counter keyed by extracted git subcommand.

Baseline for one-commit submit is 18 git invocations across 14
distinct subcommands -- pinned here so future reductions surface as
explicit diff updates.

The canonical CI mode (and recommended local mode) is walltime, which
needs nothing beyond pytest --codspeed. dev/Dockerfile covers the
narrower case of running codspeed's CPU simulation mode locally,
which requires valgrind + codspeed-runner -- awkward to install
on macOS.

The image ships a Python 3.12-slim base with valgrind,
codspeed-runner v4.16.1, uv, and the dev deps pre-installed, plus
safe.directory '*' so bind-mounted repos from a different uid don't
fail with "dubious ownership". Uploading from the container needs a
personal CODSPEED_TOKEN.

Simulation mode isn't the right tool for this codebase (the dashboard
itself warns it measures < 1% of moz-phab's actual workload), so
this image is an opt-in for the rare Python-only-hot-path case.
BENCHMARKING.md spells that out.

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/review/commit/68b2af762ec9a94f951a42dd2283c14111ff20c6
[main] benchmarks: integrate codspeed for submit and patch DOT_GITHUB_OVERRIDE (Bug 2035900) r=sylvestre

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/review/commit/2442d74ecabef5f71c7b287050b84162f259011a
[main] testing: pin Conduit API call counts (Bug 2035900) r=sylvestre

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/review/commit/254cdb634b3b02dedf9b33e66cb02cdc6c070d3e
[main] testing: pin git subprocess call counts (Bug 2035900) r=sylvestre

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/review/commit/ab3953c113743dee0bc78500edac0bf1a453a9fd
[main] benchmarks: add Dockerfile for local simulation-mode runs (Bug 2035900) r=sylvestre

Authored by https://github.com/cgsheeh
https://github.com/mozilla-conduit/review/commit/d7f0d6f9b86eded273648d1532b890db635e8835
[main] ci: fix repo name in task guard (Bug 2035900)