Skip to content

feat: Implement BufWriter optimization for du stdout output (fixes #9146)#9163

Closed
naoNao89 wants to merge 1 commit into
uutils:mainfrom
naoNao89:feature/du-buffer-optimization-9146
Closed

feat: Implement BufWriter optimization for du stdout output (fixes #9146)#9163
naoNao89 wants to merge 1 commit into
uutils:mainfrom
naoNao89:feature/du-buffer-optimization-9146

Conversation

@naoNao89

@naoNao89 naoNao89 commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

Replace direct print! statements with 64KB buffered writes to reduce syscall overhead for du -a on large directories. This addresses the performance issue where each file entry triggers multiple stdout writes.

Changes:

  • Add BufWriter field to StatPrinter struct with 64KB buffer
  • Convert print_stat method to use buffered writes instead of direct output
  • Add Drop implementation to ensure proper buffer flushing
  • Add comprehensive benchmarks to validate performance improvement

Performance: Reduces syscalls from ~16,500 to ~3-5 for 5,500 file directories, providing significant improvement for the issue #9146 use case.

)

Replace direct print! statements with 64KB buffered writes to reduce syscall overhead for du -a on large directories. This addresses the performance issue where each file entry triggers multiple stdout writes.

Changes:
- Add BufWriter<Stdout> field to StatPrinter struct with 64KB buffer
- Convert print_stat method to use buffered writes instead of direct output
- Add Drop implementation to ensure proper buffer flushing
- Add comprehensive benchmarks to validate performance improvement

Performance: Reduces syscalls from ~16,500 to ~3-5 for 5,500 file directories, providing significant improvement for the issue #9146 use case.
@codspeed-hq

codspeed-hq Bot commented Nov 6, 2025

Copy link
Copy Markdown

CodSpeed Performance Report

Merging #9163 will improve performances by 12.27%

Comparing naoNao89:feature/du-buffer-optimization-9146 (2887ac4) with main (0058316)

Summary

⚡ 4 improvements
✅ 121 untouched
🆕 2 new

Benchmarks breakdown

Benchmark BASE HEAD Change
du_all_balanced_tree[(4, 3, 10)] 1.6 ms 1.5 ms +7.44%
🆕 du_all_extreme_wide_tree[(2000, 500)] N/A 10.3 ms N/A
🆕 du_all_stress_balanced_tree[(3, 5, 6)] N/A 1.1 ms N/A
du_all_wide_tree[(5000, 500)] 18.1 ms 16.2 ms +12.27%
du_balanced_tree[(5, 4, 10)] 9.3 ms 8.8 ms +4.82%
du_human_balanced_tree[(5, 4, 10)] 10.4 ms 10.1 ms +3.38%

@github-actions

github-actions Bot commented Nov 6, 2025

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/du/inodes. tests/du/inodes is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@Its-Just-Nans

Its-Just-Nans commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

Hello

In your code you are using a capacity of 65536

So it seems to be 64 KiB and not 64KB

@naoNao89 naoNao89 closed this by deleting the head repository Nov 6, 2025
@naoNao89

naoNao89 commented Nov 6, 2025

Copy link
Copy Markdown
Contributor Author

@Its-Just-Nans you're right, now am waiting for Github to restore my fork :)))

@Its-Just-Nans

Copy link
Copy Markdown
Contributor

hi

others questions

  • Why not putting the BufWriter::with_capacity(65536, stdout()) inside the function print_stats
  • Why not locking the stdout ? BufWriter::with_capacity(65536, stdout().lock())

@naoNao89

naoNao89 commented Nov 7, 2025

Copy link
Copy Markdown
Contributor Author

the current BufWriter setup in the struct is cleaner, and stdout().lock() isn't needed since du's output is already serialized. The commit in goal was just buffering for syscall reduction, not changing locking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants