import-free setuptools integration
I have implemented some very simple setuptools integration that circumvents the requirement to `import cython` in `setup.py` entirely, doing away with the century-old problem of having to importing requirements during installation: <https://github.com/cython/cython/pull/477> This implementation is inspired by CFFI's setuptools integration: <https://cffi.readthedocs.org/en/latest/cdef.html#distutils-setuptools>. They do not require you to `import cffi` but simply define an additional keyword argument `cffi_modules` for `setup()`. This additional keyword argument _does not raise errors when cffi is not installed yet_ but can be used once cffi is there (as defined in `setup_requires`). Later setuptools will call cffi and have it do whatever it wants with the contents of the argument. Cython can do the exact same. Below you have the usual `setup.py`, with the chicken and egg `import` problem and a few `Extensions` that need to be `cythonize`d: import setuptools from Cython.Build import cythonize # this will fail if cython is not present prior to running `pip install this` from setuptools.extension import Extension extensions = [ Extension("fib", ["fib.pyx"]), Extension("fib2", ["fib2.pyx"]), ] setuptools.setup( name='example', version="1.0.0", packages=setuptools.find_packages(), setup_requires=['cython'], install_requires=['cython'], ext_modules=cythonize(extensions), ) With the changes I am proposing the usual ext_modules=cythonize(extensions), can be replaced with cython_modules=extensions, removing the need for `from Cython.Build import cythonize` and solving the import problem: import setuptools from setuptools.extension import Extension extensions = [ Extension("fib", ["fib.pyx"]), Extension("fib2", ["fib2.pyx"]), ] setuptools.setup( name='example', version="1.0.0", setup_requires=['cython'], # we must later require a Cython version that has this kind of integration install_requires=['cython'], cython_modules=extensions, ) This allows a nicer automated installation of tools that depend on cython and also allows end users to keep their `setup.py` cleaner and leaner. Additionally, I have also extended `cythonize()` a bit and thus allow for definitions like: import setuptools from setuptools.extension import Extension extensions = [ Extension("fib", ["fib.c"]), Extension("fib2", ["fib2.c"]), ] setuptools.setup( name='example', version="1.0.0", extras_require={'cython': ['cython']}, ext_modules=extensions, cython_modules=extensions, ) This will automatically compile pyx->c if cython is installed and a re-compilation is needed. Otherwise it will merely compile c->so. This allows devs to ship the generated C-code (as you suggest per your docs) but takes the heuristic to find out whether to compile pyx->C->so or only C->so completely out of `setup.py` and under control of cython. The change is written to be backwards compatible: - Using `cython_modules` is entirely optional and not using it is sideeffect-free - Having `cythonize()` internally automatically compile *.pyx files when *.c files are provided is disabled by default and must be enabled using the `replace_extension` keyword argument (`cython_modules` enables that option).
Nils Werner wrote:
- Having `cythonize()` internally automatically compile *.pyx files when *.c files are provided is disabled by default and must be enabled using the `replace_extension` keyword argument (`cython_modules` enables that option).
I'd be happier if it were still disabled by default even then. If I'm simply installing a package, I do *not* want it to try to recompile any .pyx files just because I happen to have cython installed. I know that isn't supposed to happen unless the .pyx is newer than the .c, but I wouldn't like to rely on that. There's a big difference between using a package and modifying it, and I'd rather be explicit about when I'm doing the latter. -- Greg
I understand. Maybe it is then better to remove the extension-swapping and ask the developer to specifically decide which extension is a cython_module and which is an ext_module (and potentially have duplicate definitions of the same extension): setuptools.setup( # ... ext_modules=[ Extension("pure_c", ["pure_c.c"]), Extension("mixed", ["mixed.c"]), ], cython_modules=[ Extension("mixed", ["mixed.pyx"]), Extension("pure_pyx", ["pure_pyx.pyx"]), ], ) In this case - `pure_c` will never automatically be cythonized - `mixed` may be cythonized - `pure_pyx` may be cythonized `pure_c` needs the developer to run `cythonize pure_c.pyx` before `python setup.py build_sdist`. Shipping a package without the resulting .c file will fail during installation. `mixed` may come with both .c and .pyx files - If cython is not installed setuptools will simply compile the C code. - If cython is installed it is up to `cythonize()` to decide if it wants to re-compile pyx->c first. `pure_pyx` is similar to `mixed` but it may run into non-obvious problems if cython is not installed and not defined as a `setup_requirement`: *It will silently not be built* and be missing from the final installation. I guess this won't be much of a problem though as it will most probably be found out quickly and be fixed with a simple additional line in setup.py. It might be a bit beyond the scope of what I am suggesting here but I have also been experimenting with adding another keyword argument and another command to setuptools: setuptools.setup( # ... cython_manual_modules=[ Extension("pure_c", ["pure_c.pyx"]), ], ) Combined with python setup.py build_pyx we could then automate the cythonization calls during development. This removes the overhead of remembering each of your pyx files from the developer and encourages them to *not write cython-related scripting logic* in their setup.py. In your case (you cythonize before sdist and dont want users to cythonize during install) you would define your package as setuptools.setup( # ... ext_modules=[ Extension("mixed", ["mixed.c"]), ], cython_manual_modules=[ Extension("mixed", ["mixed.pyx"]), ], ) And would run python setup.py build_pyx sdist to create a distribution file. Again, all these changes are backwards compatible as none of the current logic needs to be changed (the additional keyword argument and the command class have been appended to the GitHub PR). On Sun, Feb 7, 2016 at 9:38 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nils Werner wrote:
- Having `cythonize()` internally automatically compile *.pyx files when *.c files are provided is disabled by default and must be enabled using the `replace_extension` keyword argument (`cython_modules` enables that option).
I'd be happier if it were still disabled by default even then. If I'm simply installing a package, I do *not* want it to try to recompile any .pyx files just because I happen to have cython installed.
I know that isn't supposed to happen unless the .pyx is newer than the .c, but I wouldn't like to rely on that. There's a big difference between using a package and modifying it, and I'd rather be explicit about when I'm doing the latter.
-- Greg _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
participants (2)
-
Greg Ewing -
Nils Werner