distutil findall can choke with recursive symlinks (performance)

[ssbarnea]

BPO	44497
Nosy	@merwok, @ssbarnea, @dstufft, @corona10
PRs	bpo-44497: Avoid recursive loops on distutils findall #26873

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-06-24.16:43:21.849>
created_at = <Date 2021-06-23.09:20:31.757>
labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library', '3.11']
title = 'distutil findall can choke with recursive symlinks (performance)'
updated_at = <Date 2021-06-24.16:43:21.848>
user = 'https://github.com/ssbarnea'

bugs.python.org fields:

activity = <Date 2021-06-24.16:43:21.848>
actor = 'corona10'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-24.16:43:21.849>
closer = 'corona10'
components = ['Distutils']
creation = <Date 2021-06-23.09:20:31.757>
creator = 'ssbarnea'
dependencies = []
files = []
hgrepos = []
issue_num = 44497
keywords = ['patch']
message_count = 2.0
messages = ['396394', '396499']
nosy_count = 4.0
nosy_names = ['eric.araujo', 'ssbarnea', 'dstufft', 'corona10']
pr_nums = ['26873']
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44497'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[ssbarnea]

As the results of investigating a very poor performance of pip while trying to install some code I was able to identify that the root cause was the current implementation of distutils.filelist.findall or to be more precise the _find_all_simple function, which does followsymlinks but without any measures for preventing recursivity and duplicates.

To give an idea in my case it was taking 5-10minutes to run while the CPU was at 100%, for a repository with 95k files (most of them temp inside .tox folders). Removal of the symlinks did make it run in ~5s.

IMHO, _find_all_simple should normalize paths and avoid returning any duplicates.

Realted: https://bugs.launchpad.net/pbr/+bug/1933311

[corona10]

Since the distutils is deprecated at PEP-632, I would like to suggest changing the implementation to use setuptools.