[Distutils] How to implement ‘setup.py’ functionality that itself needs third-party distributions

[Leonardo Rochael Almeida]

I have faced an issue with setuptools (and even plain distutils) in the
past that has some resemblance to this.

There was a PyPI package that contained C extensions and headers, and
another python package that had more C extensions, but depended on the
location of the headers on the first package to be built. Was it numpy and
scipy? Anyway zope.proxy and zope.container had the same issue, as did
Zope's Acquisition and ExtensionClass.

The issue can be clearly summarized with the `setup.py` snippet below,
which I found at [1]:

from distutils.core import setup, Extension
import numpy

# define the extension module
cos_module_np = Extension('cos_module_np',
                          sources=['cos_module_np.c'],
                          include_dirs=[numpy.get_include()])
# run the setup
setup(ext_modules=[cos_module_np])


[1]
https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html

Semantically, it's clear that numpy is both a `setup_requires` and an
`install_requires` for the `setup.py` above. It's needed both at build time
for the header locations and at run time for dynamically linking / module
importing.

However, even if the snippet above were using `setuptools` there'd be no
way of passing `setup_requires` to the `setup()` call above, the script
will have failed with an `ImportError` long before that, in the
`include_dirs` part.

The way the zope packages solved this issue was horribly crude and
unsatisfying, but effective: they vendored the header files (and sometimes
whole extensions) of the dependency packages. First by svn:extensions, and
then by outright copying, when the `git` wave came around.

The simplest and most backward compatible fix for setuptools I could think
of would be for setup() keywords to have lazy evaluation:

In setuptools, some `setup()` keywords accept parameters in multiple
formats. For example `entry_points` accepts both a dictionary and a .ini
formatted string [2].

[2]
https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins

If `setup()` keywords were to accept callables as well, then we could delay
importing dependency packages until `setup_requires` where already
satisfied, like this:

from setuptools import setup, Extension

def ext_modules():
    import numpy

    # define the extension module
    cos_module_np = Extension('cos_module_np',
                              sources=['cos_module_np.c'],

                              include_dirs=[numpy.get_include()])
    return [cos_module_np]


# run the setup
setup(setup_requires=['numpy'], ext_modules=ext_modules)


Cheers,

Leo

On Friday, January 23, 2015, Ben Finney <ben+python at benfinney.id.au> wrote:

> Ben Finney <ben+python at benfinney.id.au> writes:
>
> > Marius Gedminas <marius at gedmin.as> writes:
> >
> > >   2. Implement metadata extraction using custom command classes[*] and
> > >      setup_requires.
> >
> > This seems to be the only one which might be feasible (and without,
> > needless duplication of information, which is part of the whole point of
> > this exercise). I will learn more and try it.
>
> Okay, the Setuptools ‘egg_info’ command is rather hostile to extension,
> but less hostile than Setuptools entry points. I have cobbled together a
> solution.
>
> The packaging code for version 2.0.4 of ‘python-daemon’:
>
> * Has no Setuptools entry points. These are not the right tool for the
>   job, it seems.
>
> * Adds a custom Setuptools command ‘write_version_info’ to parse the
>   Changelog and write the version info metadata file.
>
> * Minor hack: Uses a custom ‘egg_info’ command that will recognise and
>   run sub-commands. (Why doesn't every top-level command do this without
>   needing to be taught?)
>
> * Major hack: Re-binds class names in order to update their base class,
>   after dynamically importing ‘docutils’. (If there was a way to ensure
>   Docutils was installed before running any of the commands, this would
>   not be needed.)
>
> Thanks for all the assistance understanding the limitations and quirks
> of Distutils and Setuptools. I'm not very happy with this solution, but
> hopefully it is modular enough that it can be changed, in some happy
> future when Python's packaging features are more tractable.
>
> --
>  \              “Programs must be written for people to read, and only |
>   `\        incidentally for machines to execute.” —Abelson & Sussman, |
> _o__)              _Structure and Interpretation of Computer Programs_ |
> Ben Finney
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150123/acbe88ae/attachment.html>