Version 38.5.0 fails to build extension

[djhoese]

I first noticed this on a travis build for one of my projects. NetCDF4-python (one of my dependencies) fails to build when setuptools 38.5.0 is installed. Forcing the version to 38.4.0 fixes it. See https://travis-ci.org/pytroll/satpy/jobs/337509200#L940 for the failed job which ends with this exception:

      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/opt/python/3.6.3/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/opt/python/3.6.3/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 133, in finalize_options
        _build_ext.finalize_options(self)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Distutils/build_ext.py", line 19, in finalize_options
        self.distribution.ext_modules)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 920, in cythonize
        aliases=aliases)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 815, in create_extension_list
        kwds = deps.distutils_info(file, aliases, base).values
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 670, in distutils_info
        return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 680, in transitive_merge
        node, extract, merge, seen, {}, self.cimported_files)[0]
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 685, in transitive_merge_helper
        deps = extract(node)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 650, in distutils_info0
        cimports, externs, incdirs = self.cimports_externs_incdirs(filename)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Utils.py", line 44, in wrapper
        res = cache[args] = f(self, *args)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 555, in cimports_externs_incdirs
        for include in self.included_files(filename):
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Utils.py", line 44, in wrapper
        res = cache[args] = f(self, *args)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Build/Dependencies.py", line 537, in included_files
        include_path = self.context.find_include_file(include, None)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Compiler/Main.py", line 274, in find_include_file
        error(pos, "'%s' not found" % filename)
      File "/tmp/pip-build-lpsrf1ne/netCDF4/.eggs/Cython-0.27.3-py3.6-linux-x86_64.egg/Cython/Compiler/Errors.py", line 178, in error
        raise InternalError(message)
    Cython.Compiler.Errors.InternalError: Internal compiler error: 'constants.pyx' not found

See here for a working build where I forced setuptools to 38.4.0: https://travis-ci.org/pytroll/satpy/jobs/337598254#L863

I'm hoping that this is enough to get someones attention that knows what's wrong, but if a smaller example is needed I can try to work that out.

[benoit-pierre]

I can reproduce. It looks like the issue is caused by 038baa1. The switch to new_build_ext is screwing with the build includes for some reasons: even though include is added to the extension's include_dirs, constants.pyx/netCDF4.pxi are not found. It works after patching netCDF4/_netCDF4.pyx like this:

@@ -1017,8 +1017,8 @@ from numpy import ma
 from libc.string cimport memcpy, memset
 from libc.stdlib cimport malloc, free
 import_array()
-include "constants.pyx"
-include "netCDF4.pxi"
+include "include/constants.pyx"
+include "include/netCDF4.pxi"
 IF HAS_NC_PAR:
     cimport mpi4py.MPI as MPI
     from mpi4py.libmpi cimport MPI_Comm, MPI_Info, MPI_Comm_dup, MPI_Info_dup, \

IMHO netCDF4' setup.py probably needs to be updated and the bug should be reported to them.

[jaraco]

That commit references #1270.

My expectation was that new_build_ext was a compatible drop in. Since it's not, I'm tempted to roll back that change until we can get a better understanding of what compatibility concerns might be. I'd like get a reading from the Cython project maintainers on what they recommend for a project such as netCDF4.

@scoder: Can you comment - is the switch from build_ext to new_build_ext something that's expected to have compatibility concerns? Is there a way that either the Cython-based projects or setuptools can provide compatibility behavior to smooth the transition?

[benoit-pierre]

It does look like it's not compatible: see Cython's changelog.

[benoit-pierre]

It looks like it also breaks the build_ext command:

running build_ext
Traceback (most recent call last):
  File "./setup.py", line 389, in <module>
    cmdclass=cmdclass,
  File ".../setuptools/setuptools/__init__.py", line 129, in setup                   
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup                                        
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands                                 
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 973, in run_command                                  
    cmd_obj.ensure_finalized()
  File "/usr/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized                              
    self.finalize_options()
  File ".../setuptools/setuptools/command/build_ext.py", line 133, in finalize_options
    _build_ext.finalize_options(self)
  File "~/.local/lib/python3.6/site-packages/Cython/Distutils/build_ext.py", line 20, in finalize_options
    super(build_ext, self).finalize_options()
TypeError: super(type, obj): obj must be an instance or subtype of type

Which in turns break the develop command and editable installs with pip (that's with a project that does not even build extensions).

[benoit-pierre]

Nevermind, that last issue is a bug in Cython 0.25.2, it works fine with 0.27.3.

[vallsv]

Thanks for the rollback. We had the same problem here.

The new_build_ext from cython was building the extensions at the end of the finalize_options, instead of at the build_extensions stage. As result we was not able to configure anything on our side before the cythonization.

I really think it is better to build the cython extensions at the build stage, but it is really a matter of point of view.

If you still want to use new_build_ext, please create a hook method, or wrap the cythonization into a method, like this we still can have some way to patch the cythonization stage.

[benoit-pierre]

@vallsv: just to be clear the change has not been (yet?) reverted.

[vallsv]

Oh sorry there is still no rollback :-)

Then for more explanation, on our projects (https://github.com/silx-kit/pyFAI) we are cythonizing files by ourselves to have a little more features (manage cython cache, and portable openmp options).

I try to move our code from build_extension to finalize_options, to match setuptools 38.5.0 changes but it is not working cause at this time we don't have information on the compiler (self.compiler). That is why, on our side, it would be very helpful if setuptools could provide a way to patch the build_ext cythonization stage in some clean way.

[djhoese]

With the revert and installing 38.5.1 travis tests are now passing again. Thanks.

[vallsv]

Same here. Thanks a lot.