feat: Support --compress-debug-sections#1881
Conversation
250d9dc to
e21be4f
Compare
| match value { | ||
| "zlib" => args.debug_compression_kind = Some(CompressionKind::Zlib), | ||
| "zstd" => args.debug_compression_kind = Some(CompressionKind::Zstd), | ||
| value => { | ||
| args.warn_unsupported(&format!("--compress-debug-sections={value}"))?; | ||
| } |
There was a problem hiding this comment.
| match value { | |
| "zlib" => args.debug_compression_kind = Some(CompressionKind::Zlib), | |
| "zstd" => args.debug_compression_kind = Some(CompressionKind::Zstd), | |
| value => { | |
| args.warn_unsupported(&format!("--compress-debug-sections={value}"))?; | |
| } | |
| match value { | |
| "none" => args.debug_compression_kind = None, | |
| "zlib" => args.debug_compression_kind = Some(CompressionKind::Zlib), | |
| "zstd" => args.debug_compression_kind = Some(CompressionKind::Zstd), | |
| value => { | |
| args.warn_unsupported(&format!("--compress-debug-sections={value}"))?; | |
| } |
There was a problem hiding this comment.
I have no idea what GH did with that suggestion rendering, but it only adds none branch.
There was a problem hiding this comment.
Good idea. Done. I added a test for =none as well.
mati865
left a comment
There was a problem hiding this comment.
Impressive. Thought the code would be messier, but also less "verbose".
Some old build of Clang release+debuginfo with the default -O0:
❯ OUT=/tmp/bin powerprofilesctl launch -p performance hyperfine -w 3 './run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zlib' './run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zstd' './run-with ~/Projects/wild/target/release/wild --compress-debug-sections=none'
Benchmark 1: ./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zlib
Time (mean ± σ): 2.834 s ± 0.028 s [User: 0.001 s, System: 0.001 s]
Range (min … max): 2.803 s … 2.885 s 10 runs
Benchmark 2: ./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zstd
Time (mean ± σ): 2.555 s ± 0.028 s [User: 0.001 s, System: 0.001 s]
Range (min … max): 2.514 s … 2.611 s 10 runs
Benchmark 3: ./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=none
Time (mean ± σ): 1.936 s ± 0.028 s [User: 0.001 s, System: 0.001 s]
Range (min … max): 1.888 s … 1.963 s 10 runs
Summary
./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=none ran
1.32 ± 0.02 times faster than ./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zstd
1.46 ± 0.03 times faster than ./run-with ~/Projects/wild/target/release/wild --compress-debug-sections=zlib
And the sizes:
❯ ls /tmp/bin.*
.rwxr-xr-x 5,2G mateusz 2 maj 18:02 /tmp/bin.nocompress
.rwxr-xr-x 3,0G mateusz 2 maj 18:03 /tmp/bin.zlib
.rwxr-xr-x 2,5G mateusz 2 maj 18:03 /tmp/bin.zstd
Higher opt levels are nasty for linking time.
Just a question regarding compression levels.
With --compress-debug-sections=zstd LLD 22 is a bit faster than Wild (8.3s vs 8.6s), but produces slightly bigger binaries. Looks like there is a disagreement on compression levels?
❯ /bin/ls -l /tmp/bin.o*
-rwxr-xr-x 1 mateusz mateusz 2247017048 05-02 18:30 /tmp/bin.o3.ld
-rwxr-xr-x 1 mateusz mateusz 2299742096 05-02 18:28 /tmp/bin.o3.lld
-rwxr-xr-x 1 mateusz mateusz 2118940207 05-02 18:28 /tmp/bin.o3.wild
That's 2.093 GiB, 2,142 GiB and 1,973 GiB respectively. The difference seems to be coming mostly from .debug_info size.
e21be4f to
67d92ec
Compare
67d92ec to
4a387d1
Compare
|
I did some more experiments. It appears I might have been mistaken about -O levels affecting compression level. At least for lld. --compress-debug-sections doesn't let you specify a compression level, but --compress-sections (which wild doesn't yet support) does. So I did some experiments where I used --compress-sections to compress all the debug sections, then varied the -O level and also varied the compression level. It turned out, assuming I didn't make some mistake in my experiment methodology, that lld always uses level 1 for zlib and level 3 for zstd. So for not at least, I've removed the -O handling code and we now just unconditionally use the same values. |
|
Level 3 for zstd is the default one, so it makes sense: https://docs.rs/zstd/latest/zstd/constant.DEFAULT_COMPRESSION_LEVEL.html Level 1 for zstd seems to match "best speed" profile: https://docs.rs/zlib-rs/0.6.3/zlib_rs/struct.DeflateConfig.html#method.best_speed with the value at: https://docs.rs/zlib-rs/0.6.3/src/zlib_rs/c_api.rs.html#151 |
|
I converted the experiments I was doing before into a script: #!/bin/bash
set -e
D=$HOME/tmp/comp-level-test
OPT=1
FLAGS=-O$OPT
RUN_WITH=$HOME/save/wild-debug/run-with
LINKER=ld.lld
COMP=zlib
mkdir -p $D
rm -f $D/*
for L in $(seq 0 9); do
if [ $L -eq 0 ]; then
X=$COMP
LNAME=default
else
X=$COMP:$L
LNAME=level-$L
fi
OUT=$D/${LINKER}.OPT${OPT}.${COMP}.$LNAME \
$RUN_WITH $LINKER $FLAGS \
"--compress-sections=.debug_loc=$X" \
"--compress-sections=.debug_abbrev=$X" \
"--compress-sections=.debug_info=$X" \
"--compress-sections=.debug_aranges=$X" \
"--compress-sections=.debug_ranges=$X" \
"--compress-sections=.debug_str=$X" \
"--compress-sections=.debug_line=$X"
done
OUT=$D/${LINKER}.OPT${OPT}.${COMP}.debug \
$RUN_WITH $LINKER $FLAGS \
--compress-debug-sections=$COMP
ls -l $DIf I run this, I get the output: Note that the following three file sizes are exactly the same:
If I change COMP to zstd, I get the following: In this case it's the level-3 file that is equal to the debug and default files. Changing the opt-level (as passed to -O) affects the file size, but doesn't seem to affect which level matches. |
|
When you observed LLD being a tiny bit faster than Wild, was this on tmpfs or a different filesystem? I ask because I get quite different results: That's with my latest changes, which switched to level 3 for zstd. I'm pretty sure I was using level 1 for zstd before, so it should have been even faster (about 3.5 seconds). |
|
Nice idea with that test. I'll leave it up to you whether you want to keep the compression level constants, or use the mechanisms from the crates. It'd be nicer if zlib crate provided public constant like zstd does.
Oops, that was with |
|
I opted to stick with the constants that we define, since I felt that made it clearer exactly what level we were using. |
Fixes #493