Add -Zdead-fn-elimination: skip codegen of functions unreachable from the entry point

[yijunyu]

Implements the accepted MCP compiler-team#976: an unstable flag that skips codegen of functions a binary cannot reach from its entry point.

What it does

-Zdead-fn-elimination (off by default) adds, for binary crates only, a reachability analysis that runs after monomorphization collection and before partitioning. Functions unreachable from the entry point are dropped from the mono item set, so the backend never generates LLVM IR for them.

$ rustc -O -Zdead-fn-elimination main.rs
note: 107 unreachable functions excluded from codegen by -Z dead-fn-elimination

The flag is inert when unset.

How it works

In collect_and_partition_mono_items, after collect_crate_mono_items and before partition:

1. Build a call graph by walking each item's MIR (optimized_mir), recording direct calls, function references in constants/statics, address-taken functions, and inline-asm sym operands.
2. Seed a BFS with: reachable_set (covers exported/pub/lang-item/#[no_mangle]/#[used] items), the entry fn, and the methods of any dyn-constructed trait (dynamic dispatch is invisible to a call-graph walk, so vtable methods are seeded explicitly).
3. Run the BFS; any local fn-like item not in the result, and not otherwise required, is marked eliminable.
4. retain the mono item set, dropping only local MonoItem::Fns that are eliminable.

Conservatism

The analysis never eliminates:

• any item the monomorphization collector retained;
• lang items, exported symbols, the entry fn;
• drop glue, generic functions, async functions.

A #[cfg(debug_assertions)] invariant checks that reachable_set is a subset of the post-BFS reachable set.

Soundness testing

Verified on a stage1 build by compiling real binaries with and without the flag using the same compiler and diffing the output:

project	functions excluded	behavioral output
ripgrep	107	byte-identical to no-flag across 10 invocations (search, -n/-i/-c/-w/-v/-o, --json, recursive)
nushell	27 (6 crates)	identical across 8 pipelines (arithmetic, math sum, str ops, sort, record access, each closures)

Edge cases checked: flag-off and flag-on both produce reproducible binaries; library crates are inert (no entry seed); the #[test] harness still runs; panic=abort works; -C link-dead-code correctly defers (no elimination). Functions reached only through a closure (Iterator::map), a trait object (Box<dyn Fn>), or a function pointer are kept — only genuinely-unreachable functions are removed.
tests/codegen-units (46) and tests/ui/codegen (147) pass; tidy is clean.

Scope

This is the in-rustc flag only. It is self-contained and reviewable in isolation; there is no build-system or cross-crate coordination in this PR.

r? @oli-obk

[rustbot]

Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @oli-obk (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[bjorn3]

I still don't get what you are trying to do. If functions are unreachable from monomorphization roots (fn main(), #[no_mangle] or #[used] for binaries and a bunch more things for libraries), then the monomorphization collector should not collect them in the first place. Doing an analysis afterwards to throw away things that shouldn't have been collected in the first place seems to me like it would never eliminate anything.

Verified on a stage1 build by compiling real binaries with and without the flag using the same compiler and diffing the output:

Can you show a list of functions that are actually eliminated?