Role of distutils.cfg
Hi, I have a question on how to best handle parameters to the distribution given that they can be shadowed by the global configuration file, distutils.cfg. Our project [1] contains C-extensions that include NumPy’s headers. To this end, our setup.py [2] sets the include_dirs parameter of setup() to NumPy’s include path. We have chosen this way, since it allows to add a common include path to all the extensions in one go. One advantage of this approach is that when the include_dirs parameters of the individual extensions are reconfigured (for example with a build configuration file), this does not interfere with the numpy include path. This has been working well for most of other users, but recently we got a bug report by someone for whom it doesn’t. It turns out that his system has a distutils.cfg that precedes over the include_dirs parameter to setup() [3]. My question is now: is there a policy on the correct use of distutils.cfg? After all, it can override any parameter to any distutils command. As such, is passing options like include_dirs to setup() a bad idea in the first place, or should rather the use of distutils.cfg be reserved to cases like choosing an alternative compiler? Thanks for any clarification, Christoph
On Fri, Nov 11, 2016 at 12:27 AM, Christoph Groth <christoph@grothesque.org> wrote:
Hi,
I have a question on how to best handle parameters to the distribution given that they can be shadowed by the global configuration file, distutils.cfg.
Our project [1]
You forgot to add all your links.
contains C-extensions that include NumPy’s headers. To this end, our setup.py [2] sets the include_dirs parameter of setup() to NumPy’s include path. We have chosen this way, since it allows to add a common include path to all the extensions in one go. One advantage of this approach is that when the include_dirs parameters of the individual extensions are reconfigured (for example with a build configuration file), this does not interfere with the numpy include path.
This has been working well for most of other users, but recently we got a bug report by someone for whom it doesn’t. It turns out that his system has a distutils.cfg that precedes over the include_dirs parameter to setup() [3].
My question is now: is there a policy on the correct use of distutils.cfg? After all, it can override any parameter to any distutils command. As such, is passing options like include_dirs to setup() a bad idea in the first place, or should rather the use of distutils.cfg be reserved to cases like choosing an alternative compiler?
I'm not aware of any policy, but in general I'd recommend to pass as little to setup() as possible. Most robust is to only pass metadata (name, maintainer, url, install_requires, etc.). In a number of cases you're forced to pass ext_modules or cmdclass, which usually works fine. Passing individual paths, compiler flags, etc. sounds unhealthy. Ralf
Ralf Gommers wrote:
You forgot to add all your links.
I accidentally deleted them when re-posting my message. The first time I sent it to this list without being subscribed, and it was unfortunately *silently* dropped. (I had assumed that postings by non-members are moderated.) Here they are: [1] https://pypi.python.org/pypi/kwant/1.2.2 [2] https://gitlab.kwant-project.org/kwant/kwant/blob/master/setup.py [3] https://gitlab.kwant-project.org/kwant/kwant/issues/48#note_2494
Most robust is to only pass metadata (name, maintainer, url, install_requires, etc.). In a number of cases you're forced to pass ext_modules or cmdclass, which usually works fine. Passing individual paths, compiler flags, etc. sounds unhealthy.
Sounds reasonable, thanks for your advice. Is there any alternative to passing ext_modules? Christoph
On Fri, Nov 11, 2016 at 9:12 PM, Christoph Groth <christoph@grothesque.org> wrote:
Ralf Gommers wrote:
You forgot to add all your links.
I accidentally deleted them when re-posting my message. The first time I sent it to this list without being subscribed, and it was unfortunately *silently* dropped. (I had assumed that postings by non-members are moderated.) Here they are:
[1] https://pypi.python.org/pypi/kwant/1.2.2 [2] https://gitlab.kwant-project.org/kwant/kwant/blob/master/setup.py [3] https://gitlab.kwant-project.org/kwant/kwant/issues/48#note_2494
Most robust is to only pass metadata (name, maintainer, url,
install_requires, etc.). In a number of cases you're forced to pass ext_modules or cmdclass, which usually works fine. Passing individual paths, compiler flags, etc. sounds unhealthy.
Sounds reasonable, thanks for your advice.
Is there any alternative to passing ext_modules?
What Numpy and Scipy do is pass a single Configuration instance, and define all extensions and libraries in nested setup.py files, one per submodule. Example: https://github.com/scipy/scipy/blob/master/setup.py#L327 If you pass ext_modules as a list of Extension instances, you'd likely also avoid the issue. Example: https://github.com/PyWavelets/pywt/blob/master/setup.py#L145 Ralf
On Fri, Nov 11, 2016 at 12:12 AM, Christoph Groth <christoph@grothesque.org> wrote:
Ralf Gommers wrote:
You forgot to add all your links.
I accidentally deleted them when re-posting my message. The first time I sent it to this list without being subscribed, and it was unfortunately *silently* dropped. (I had assumed that postings by non-members are moderated.) Here they are:
[1] https://pypi.python.org/pypi/kwant/1.2.2 [2] https://gitlab.kwant-project.org/kwant/kwant/blob/master/setup.py
It looks like you already have a ton of code that's looping over various representations of your ext_modules and autogenerating things... wouldn't it be fairly straightforward to have a bit more code to automatically and unconditionally stick numpy.get_include() into every extension's include_dirs? (More code than doing it at the setup.py level, for sure, but on the scale of distutils workarounds...) -n -- Nathaniel J. Smith -- https://vorpus.org
Nathaniel Smith wrote:
It looks like you already have a ton of code that's looping over various representations of your ext_modules and autogenerating things...
Indeed, some of it could be IMHO interesting for inclusion into distutils/setuptools or whatnot. For example if Cython is not to be run we warn if the the pyx files (and their dependencies) are newer.
wouldn't it be fairly straightforward to have a bit more code to automatically and unconditionally stick numpy.get_include() into every extension's include_dirs?
Sure, that’s what we are going to do now. I preferred the other solution since we have machinery in place that allows to redefine any parameter to any extension through a "build.conf" file. Now if someone redefines include_dirs in "build.conf", they will have to add the numpy include path as well. But that’s OK.
participants (3)
-
Christoph Groth -
Nathaniel Smith -
Ralf Gommers