Disable bytecode writing due to pytest's assertion rewrite (DISCUSSION)

[nicoddemus]

After some investigation I found what the problem related in pytest-dev/pytest#1888 is:

1. pytest installs an import hook which rewrites asserts in test modules (since 2.X series) and plugins and conftest files (new in 3.X).
2. The hook tries to cache the changed bytecode to disk, and to do that it calls os.mkdir.
3. Now, setuptools executes some tests in a sandboxed environment which disallows certain functions that modify the disk from being called, such as os.mkdir.
4. The sandboxed environment traps the call to os.mkdir, but to raise an error it tries to import SandboxViolation, which again triggers the import hook, which again tries to call os.mkdir, and so on.

Here's the relevant part of the traceback from a setuptools job from my fork which uses pytest 3.0.3:

self = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7fac57374198>
name = 'setuptools.extern'
path = ['/home/travis/build/nicoddemus/setuptools/setuptools']

<strip some code>

        # The requested module looks like a test file, so rewrite it. This is
        # the most magical part of the process: load the source, rewrite the
        # asserts, and load the rewritten source. We also cache the rewritten
        # module code in a special pyc. We must be aware of the possibility of
        # concurrent pytest processes rewriting and loading pycs. To avoid
        # tricky race conditions, we maintain the following invariant: The
        # cached pyc is always a complete, valid pyc. Operations on it must be
        # atomic. POSIX's atomic rename comes in handy.
        write = not sys.dont_write_bytecode
        cache_dir = os.path.join(fn_pypath.dirname, "__pycache__")
        if write:
            try:
>               os.mkdir(cache_dir)
../../../virtualenv/python3.5.2/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <setuptools.sandbox.DirectorySandbox object at 0x7fac5504db38>
path = '/home/travis/build/nicoddemus/setuptools/setuptools/__pycache__'
args = (), kw = {}
    def wrap(self, path, *args, **kw):
        if self._active:
>           path = self._remap_input(name, path, *args, **kw)
setuptools/sandbox.py:302: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <setuptools.sandbox.DirectorySandbox object at 0x7fac5504db38>
operation = 'mkdir'
path = '/home/travis/build/nicoddemus/setuptools/setuptools/__pycache__'
args = (), kw = {}
    def _remap_input(self, operation, path, *args, **kw):
        """Called for path inputs"""
        if operation in self.write_ops and not self._ok(path):
>           self._violation(operation, os.path.realpath(path), *args, **kw)
setuptools/sandbox.py:448: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <setuptools.sandbox.DirectorySandbox object at 0x7fac5504db38>
operation = 'mkdir'
args = ('/home/travis/build/nicoddemus/setuptools/setuptools/__pycache__',)
kw = {}
    def _violation(self, operation, *args, **kw):
>       from setuptools.sandbox import SandboxViolation
setuptools/sandbox.py:403: 

This explains why pytest-dev/pytest#1891 fixed this issue in pytest-3.0: pytest was trying to rewrite a bunch of modules that shouldn't be rewritten in the first place because they were not pytest plugins.

This also explains why this broke again in 3.0.3: pytest-dev/pytest#1934 was introduced. Now pytest will rewrite plugins installed in development mode, so it again tries to rewrite the setuptools own plugin during the sandboxed tests, triggering the sandbox violation.

As a quick workaround, I disabled bytecode writing so pytest will re-generate the bytecode but won't cache it into disk.

Another possible solution would be to use --assert=plain which disables the rewrite-hook completely, but then rich assertion errors would be lost.

I'm opening this PR in order to discuss other possible solutions to this problem.

[jaraco]

Thanks very much for the extensive troubleshooting and detailed description. I think the solution here will be to avoid the sandboxing altogether by relying on tox as the test runner or possible rwt as the dependency resolver instead of setup.py test. I'm moving in the direction of deprecating that altogether due to the unintended interactions between setuptools build constructs (e.g. sandboxing) and the test runner.

[nicoddemus]

I think the solution here will be to avoid the sandboxing altogether by relying on tox as the test runner or possible rwt as the dependency resolver instead of setup.py test.

Is sandboxing done by setup.py test? Oh I thought it was done somewhere in the test suite, but didn't actually searched for it.

I'm moving in the direction of deprecating that altogether due to the unintended interactions between setuptools build constructs (e.g. sandboxing) and the test runner.

What should we do with this PR? IMHO I think it is an acceptable workaround for now, it causes a very minimal (I suspect negligible) performance loss when collecting the test suite and only on local runs, as the CI servers use clean VMs for all executions.

[jaraco]

After switching the codebase to use tox for test preparation, the tests now pass. Thanks so much for the help on this.

[nicoddemus]

Glad I could help!