setuptools: build shared library from C++ code, then build Cython wrapper linked to shared library

Question

We have a bunch of C++ files with classes that we wrap to Python using Cython. We use setuptools to build the Cython extension. This all works fine, we followed the guide here: http://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html

We are basically doing something like this

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules = cythonize(
           "rect.pyx",                 # our Cython source
           sources=["Rectangle.cpp"],  # additional source file(s)
           language="c++",             # generate C++ code
      ))

We don't like about this that we have to recompile everything, even if only the Cython part changes, rect.pyx in this example. In fact we never touch the .cpp files, but change the .pyx files often.

We would like to compile the .cpp files separately into a static or shared libary, then build .pyx files independently, which links to the library generated from the .cpp files. All this would be easy with make or cmake, but we want a pure Python solution that uses only setuptools. Mock-up code would look something like this:

from distutils.core import setup
from Cython.Build import cythonize

class CppLibary:
    # somehow get that to work

# this should only recompile cpplib when source files changed
cpplib = CppLibary('cpplib',
                   sources=["Rectangle.cpp"], # put static cpp code here
                   include_dirs=["include"])

setup(ext_modules = cythonize(
           "rect.pyx",                 # our Cython source
           libraries=[cpplib],         # link to cpplib
           language="c++",             # generate C++ code
      ))

Never used it and seems sparsely documented but there is a build_clib command that might work for you - stackoverflow.com/questions/16854066/… — chrisb
– chrisb, Commented Mar 14, 2018 at 2:03
Thank you, @chrisb. I looked into the related stackoverflow.com/questions/16854066/…, but build_clib from distutils did not work, and it was apparently removed from setuptools — HDembinski
– HDembinski, Commented Mar 15, 2018 at 12:50
Possible duplicate of Using distutils and build_clib to build C library — HDembinski
– HDembinski, Commented Aug 17, 2019 at 11:15

danny · Accepted Answer · 2018-03-21 12:17:01Z

6

There is a seemingly undocumented feature of setup that can do this, for example:

import os

from setuptools import setup
from Cython.Build import cythonize

ext_lib_path = 'rectangle'
include_dir = os.path.join(ext_lib_path, 'include')

sources = ['Rectangle.cpp']

# Use as macros = [('<DEFINITION>', '<VALUE>')]
# where value can be None
macros = None

ext_libraries = [['rectangle', {
               'sources': [os.path.join(ext_lib_path, src) for src in sources],
               'include_dirs': [include_dir],
               'macros': macros,
               }
]]

extensions = [Extension("rect",
              sources=["rect.pyx"],
              language="c++",
              include_dirs=[include_dir],
              libraries=['rectangle'],
)]

setup(ext_modules=cythonize(extensions),
      libraries=ext_libraries)

The libraries argument builds the external library found in directory rectangle, with include directory rectangle/include common between it and the extension.

Have also switched the import to setuptools from distutils which is deprecated, now part of setuptools.

Have not seen any documentation on this argument but seen it used in other projects.

This is untested, please provide sample files for testing if it does not work.

edited Mar 21, 2018 at 12:17

answered Mar 14, 2018 at 11:57

danny

5,2881 gold badge22 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

HDembinski Over a year ago

Thank you for the pointer, this is very helpful. I tried to follow along your example, but it does not work. ext_libraries is not compiled by setuptools, only the Cython extension. I get an error like this ld: library not found for -llibrectangle

danny Over a year ago

You may need to manually include the generated rectangle object file as an extra_objects argument to the extension as the shared object does not get installed anywhere for the linker to find it. Does the object get generated or nothing is built for rectangle at all? As above, will need to show complete code to test with, see MCVE. Note that ext_libraries should be of type list(list(str, dict))) - answer has been updated.

HDembinski Over a year ago

For anyone else who got confused by this: python setup.py develop does NOT build the clib automatically. You need to do python setup.py build_clib manually once.

FirefoxMetzger Over a year ago

Close, but not exactly. libraries will build archives (.lib) that will be linked statically. So strictly speaking this doesn't answer the question of how to build shared libraries (what I am looking for); however, it does solve the problem for most people so kudos for helping shed light on this area :)

Collectives™ on Stack Overflow

setuptools: build shared library from C++ code, then build Cython wrapper linked to shared library

1 Answer 1

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related