Donald Stufft writes:

...

On Jan 17, 2015, at 7:47 PM, Ben Finney wrote: As it stands (‘python-daemon’ [0] version 2.0.3), the ‘version.py’ file is correctly included in the source distribution, correctly used by the ‘egg_info.writers’ entry point; but then ends up incorrectly installed to the run-time packages library. This causes problems for subsequent import of unrelated modules that happen to share the same name.

How can I specify to Setuptools that the file is needed in the source distribution, is needed by the entry points for Setuptools, but should not be installed along with the binary distribution?

setup_requires?

Did you want to say anything else about that, for instance, how it applies to the question or what specifically you suggest I do? As it stands, I can only quote the documentation: setup_requires A string or list of strings specifying what other distributions need to be present in order for the setup script to run. […] URL:https://setuptools.pypa.io/en/latest/setuptools.html The module in question is part of the same code base, and is not an “other distribution”. So I don't know why you suggest this option. -- \ “Every valuable human being must be a radical and a rebel, for | `\ what he must aim at is to make things better than they are.” | _o__) —Niels Bohr | Ben Finney

On Jan 18, 2015, at 8:01 PM, Nick Coghlan wrote:

On 19 January 2015 at 06:35, Ben Finney wrote:

...

Donald Stufft writes:

...

...

On Jan 17, 2015, at 7:47 PM, Ben Finney wrote: As it stands (‘python-daemon’ [0] version 2.0.3), the ‘version.py’ file is correctly included in the source distribution, correctly used by the ‘egg_info.writers’ entry point; but then ends up incorrectly installed to the run-time packages library. This causes problems for subsequent import of unrelated modules that happen to share the same name.

How can I specify to Setuptools that the file is needed in the source distribution, is needed by the entry points for Setuptools, but should not be installed along with the binary distribution?

setup_requires?

Did you want to say anything else about that, for instance, how it applies to the question or what specifically you suggest I do?

As it stands, I can only quote the documentation:

setup_requires

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. […]

URL:https://setuptools.pypa.io/en/latest/setuptools.html

The module in question is part of the same code base, and is not an “other distribution”. So I don't know why you suggest this option.

If you have a build/install time only dependency that you want to distribute, you *have* to separate it out into a separate component if you don't want it to also be present at runtime. We do not, and will not, support selective installation of subcomponents, as it's too hard to audit later. Instead, such components need to be separated out into distinct packages so that the segmentation of functionality and availability is clear to both the automated tools and to other humans.

"Extras" work the same way - they select whether or not to install optional *dependencies*, but deliberately can't be used to selectively install pieces of the package itself (the affected subcomponents are instead expected to do runtime checks to see if the optional dependencies are present and provide a useful error message if they're missing).

I’m confused what this actually is. If it’s just a file you don’t want installed… then don’t specify it in your setup.py’s setup() function in either the py_modules or the packages keyword. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jan 18, 2015, at 8:54 PM, Ben Finney wrote:

Donald Stufft writes:

...

I’m confused what this actually is.

Please look at the example I cited, ‘python-daemon’ version 2.0.3.

...

If it’s just a file you don’t want installed… then don’t specify it in your setup.py’s setup() function in either the py_modules or the packages keyword.

That doesn't have the desired effect; it is installed (unwanted) to the site packages along with the packages I do want.

Line 53 of the setup.py for python-daemon version 2.0.3 has you calling setuptools.find_packages(exclude=[“test”]) and passing the return value of that to the packages kwarg. I’m pretty sure that find_packages is going to discover version.py and add it as part of the return value which is then passing that to the packages kwarg. Adding it to the exclude will probably prevent it from being installed, although you might need to add it to MANIFEST.in to ensure it gets added to the sdist. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Ben Finney writes:

Donald Stufft writes:

...

I’m pretty sure that find_packages is going to discover version.py and add it as part of the return value which is then passing that to the packages kwarg.

That doesn't match the documentation for ‘find_packages’: […]

The ‘version.py’ module is not within any package, so I infer that ‘find_packages’ should ignore it. Is that wrong?

Testing with Python 2.7 and Python 3.4 demonstrates that, indeed, ‘find_packages’ ignores the file because it's not in a package. When I put in ‘setup.py’ a diagnostic line:: print(find_packages(exclude=["test"])) the output shows ‘version.py’ is not collected:: $ python2.7 ./setup.py check ['daemon'] running check $ python3.4 ./setup.py check ['daemon'] running check Which is as it should be. So should I expect that, if a module is not specified in the ‘packages’ parameter nor the ‘py_modules’ parameter, it will not be installed? -- \ “See, in my line of work you gotta keep repeating things over | `\ and over and over again, for the truth to sink in; to kinda | _o__) catapult the propaganda.” —George W. Bush, 2005-05 | Ben Finney

Nick Coghlan writes:

If you have a build/install time only dependency that you want to distribute, you *have* to separate it out into a separate component if you don't want it to also be present at runtime.

So, to be clear: if this module is needed during build-time for the distribution but does not have a stable API yet, you're saying it nevertheless needs to go to a repository for download as an independent distribution? That places a large burden on developing such components. Placing them in a repository for independent download strongly implies they have a reliable public API, which in this case is not yet true. The API for this component isn't stable, and so far isn't needed beyond the packaging for this one distribution. That's why I don't want to separate it out yet.

We do not, and will not, support selective installation of subcomponents, as it's too hard to audit later. Instead, such components need to be separated out into distinct packages so that the segmentation of functionality and availability is clear to both the automated tools and to other humans.

That's your authority as PyPA, of course. I hope it's clear why I'm not satisfied by the reasoning though. -- \ “Anyone who puts a small gloss on [a] fundamental technology, | `\ calls it proprietary, and then tries to keep others from | _o__) building on it, is a thief.” —Tim O'Reilly, 2000-01-25 | Ben Finney

If you're just after the ability to say "I want to include this file in the sdist, but not in the built wheel file or installed distribution" (as I now believe you are)

I think that you can add the file to the MANIFEST.in file to achieve the desired behavior. -Robert On Sun, Jan 18, 2015 at 11:39 PM, Nick Coghlan wrote:

On 19 January 2015 at 11:59, Ben Finney wrote:

...

Nick Coghlan writes:

...

If you have a build/install time only dependency that you want to distribute, you *have* to separate it out into a separate component if you don't want it to also be present at runtime.

So, to be clear: if this module is needed during build-time for the distribution but does not have a stable API yet, you're saying it nevertheless needs to go to a repository for download as an independent distribution?

I actually misunderstood your question. If you're just after the ability to say "I want to include this file in the sdist, but not in the built wheel file or installed distribution" (as I now believe you are), then you're in the implementation defined world of the significantly underspecified sdist format. I believe setting that up actually *is* possible already, but have no idea what incantation you'll need to pass to setuptools to make it do it (and the docs are unlikely to be a reliable guide).

If you'd like to volunteer for the task of reverse engineering and properly documenting how sdists work (with regression tests!), that would be quite awesome. Not necessarily *fun* from your point of view, but definitely awesome from my point of view :)

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 19 January 2015 at 17:45, Ben Finney wrote:

Nick Coghlan writes:

...

If you'd like to volunteer for the task of reverse engineering and properly documenting how sdists work (with regression tests!), that would be quite awesome. Not necessarily *fun* from your point of view, but definitely awesome from my point of view :)

I'll embark on that work and see how far I get. Thanks for identifying that as a desirable area to work on.

The contribution will be licensed under Apache License version 2.0 equally to all recipients (i.e. not signing a PSF-specific agreement). Is there a way that can be accepted?

Yes, the CLA is only necessary for CPython and the standard library itself, courtesy of the fun history with CNRI that currently requires relicensing of contributions to a Python specific license. For projects under the PyPA banner, including packaging.python.org, license in = license out is fine. That text is CC-BY-SA, so including Apache Licensed text shouldn't pose any problems. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

(Ping; I didn't see a response and would like to start this work soon.) Nick Coghlan writes:

For projects under the PyPA banner, including packaging.python.org, license in = license out is fine. That text is CC-BY-SA, so including Apache Licensed text shouldn't pose any problems.

Okay. So with which repository from URL:https://bitbucket.org/pypa/ should I start? Where would the documentation go, where would the regression tests go? You appear to be distinguishing the license for executable code versus non-executable documentation. What license conventions should I be aware of? If these are answered in a “hacking” document somewhere, a response pointing me to that document would be fine. -- \ “Quidquid latine dictum sit, altum viditur.” (“Whatever is | `\ said in Latin, sounds profound.”) —anonymous | _o__) | Ben Finney

Nick Coghlan writes:

I actually misunderstood your question. If you're just after the ability to say "I want to include this file in the sdist, but not in the built wheel file or installed distribution" (as I now believe you are)

Correct, that's the goal here.

then you're in the implementation defined world of the significantly underspecified sdist format. I believe setting that up actually *is* possible already, but have no idea what incantation you'll need to pass to setuptools to make it do it (and the docs are unlikely to be a reliable guide).

My understanding, based on an answer received elsewhere [0], is that omitting the file from ‘setup.py’ but adding it to ‘MANIFEST.in’ causes it to be included in the sdist but omitted from install targets. [0] URL:http://stackoverflow.com/questions/27930124/python-module-use-during-set... That has worked for me. But it appears to cause problems for some others, related to this module which should not be installed. Unfortunately this kind of problem (trouble post-install from a PyPI package) is difficult to test. How can I test the behaviour of ‘pip’ in this regard, without thrashing many releases of a package upload to PyPI? -- \ “The internet's completely over.… Anyway, all these computers | `\ and digital gadgets are no good. They just fill your head with | _o__) numbers and that can't be good for you.” —Prince, 2010-07-05 | Ben Finney

On Mon, Jan 19, 2015 at 07:37:35PM +1100, Ben Finney wrote:

Ben Finney writes:

...

That has worked for me. But it appears to cause problems for some others, related to this module which should not be installed.

This discussion was prompted, for me, by this bug report URL:https://alioth.debian.org/tracker/index.php?func=detail&aid=314948&group_id=100328&atid=413098. Can someone please inspect the log output from Pip and help me understand why an unrelated package is breaking when installing ‘python-daemon’ 2.0.3?

(It would be a bit easier if Alioth didn't require creating a guest account and logging in just to read a public bug report.)

The ‘python-daemon’ 2.0.3 distribution declares ‘docutils’ as a dependency with ‘setup_requires’ and ‘install_requires’. So it's expected that asking Pip to install ‘python-daemon’ will attempt to install ‘docutils’.

But why does the dependency install of ‘docutils’ then fail with::

===== Downloading/unpacking docutils (from python-daemon) […] Downloading from URL https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5=... (from https://pypi.python.org/simple/docutils/) Running setup.py (path:/home/pwj/.virtualenvs/venv/build/docutils/setup.py) egg_info for package docutils […] Installing collected packages: python-daemon, lockfile, docutils Running setup.py install for docutils Running command /home/pwj/.virtualenvs/venv/bin/python -c "import setuptools, tokenize;__file__='/home/pwj/.virtualenvs/venv/build/docutils/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-7iBtrf-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/pwj/.virtualenvs/venv/include/site/python2.7 […] Running docutils-0.12/setup.py -q bdist_egg --dist-dir /tmp/easy_install-XO8spm/docutils-0.12/egg-dist-tmp-0EsqLD Traceback (most recent call last): […] File "/home/pwj/.virtualenvs/venv/local/lib/python2.7/site-packages/pkg_resources.py", line 2147, in load ['__name__']) ImportError: No module named version

I've reproduced this error at a different stage: virtualenv /tmp/sandbox /tmp/sandbox/bin/pip install --no-use-wheel python-daemon==2.0.3 -I This tries to reinstall setuptools and fails while running 'python setup.py egg_info' in /tmp/pip-build-0DrVHy/setuptools, with the same ImportError: No module named version inside pkg_resources.py Very interesting. If I attempt pip install without the -I, I get a different (but also very weird error): Collecting python-daemon==2.0.3 Using cached python-daemon-2.0.3.tar.gz Traceback (most recent call last): File "<string>", line 20, in <module> File "/tmp/pip-build-ADeQ8n/python-daemon/setup.py", line 101, in <module> ... File "/tmp/sandbox/lib/python2.7/site-packages/setuptools/dist.py", line 311, in fetch_build_eggs ... distutils.errors.DistutilsError: Setup script exited with error: [Errno 2] No such file or directory: u'ChangeLog' AFAICT it also happens during the setup_requires processing. I was unable to reproduce either error if I tried pip install /path/to/extracted/python-daemon but perhaps I wasn't careful enough to keep all the other variables the same?

=====

Why on earth would the presence of ‘version.py’ in this distribution cause the install of ‘docutils’ to fail? Or is some other correlation going on? I have had other reports that subsequent attempts to install packages also fail with the same error.

My gut instinct is to say 'setup_requires is horrible, is there any way to avoid using it'? Of course that's not a reasonable long-term strategy. It'd be good to get to the bottom of this. I tried to debug the 'ChangeLog' error with virtualenv /tmp/sandbox /tmp/sandbox/bin/pip install --no-use-wheel --no-clean python-daemon=2.0.3 cd /tmp/pip-build-zRzNwu/python-daemon strace -e chdir,open,execve -f -o /tmp/crash.log /tmp/sandbox/bin/python setup.py egg_info and it seems to me that the error happens because setuptools changes the working directory to /tmp/easy_install-qcgHIl/docutils-0.12, and then again to /tmp/easy_install-qcgHIl/docutils-0.12/temp/easy_install-5RFCuY/docutils-0.12, right before it gives control back to python-daemon's code that attempts to read ChangeLog from the current working directory. Hmm, I think this bit in your setup.py entry_points={ "distutils.setup_keywords": [ "release_date = version:validate_distutils_release_date_value", ], "egg_info.writers": [ "{filename} = version:generate_egg_info_metadata".format( filename=metadata.version_info_filename), ], }, causes your code to run while setup_requires is busy installing docutils (in the same process!), and your code makes assumptions about the package it's used with ('there's a version.py and a ChangeLog in the current working directory') that are false when it's used with other packages, installed via setup_requires. HTH, Marius Gedminas -- I think one of the enduring tragedies of the 22nd century will be that during the 20th and 21st centuries we persistently treat nuclear reactors as if they're nuclear weapons, and nuclear weapons as if they're nuclear reactors. -- Charlie Stoss

Marius Gedminas writes:

On Mon, Jan 19, 2015 at 07:37:35PM +1100, Ben Finney wrote:

...

Why on earth would the presence of ‘version.py’ in this distribution cause the install of ‘docutils’ to fail? Or is some other correlation going on? I have had other reports that subsequent attempts to install packages also fail with the same error.

My gut instinct is to say 'setup_requires is horrible, is there any way to avoid using it'? Of course that's not a reasonable long-term strategy. It'd be good to get to the bottom of this.

Thank you very much for spending the time to do so.

Hmm, I think this bit in your setup.py [defining entry points] […] causes your code to run while setup_requires is busy installing docutils (in the same process!), and your code makes assumptions about the package it's used with ('there's a version.py and a ChangeLog in the current working directory') that are false when it's used with other packages, installed via setup_requires.

My current position would be: that's a bug in Setuptools, it should not be applying entry points defined for package FOO when running the setup for some other package BAR. Is that right? Should I report a bug against Setuptools for this behaviour? -- \ “Do unto others twenty-five percent better than you expect them | `\ to do unto you. (The twenty-five percent is [to correct] for | _o__) error.)” —Linus Pauling's Golden Rule | Ben Finney

Tres Seaver writes:

On 01/19/2015 04:57 PM, Ben Finney wrote:

...

My current position would be: that's a bug in Setuptools, it should not be applying entry points defined for package FOO when running the setup for some other package BAR.

Is that right? Should I report a bug against Setuptools for this behaviour?

Notabug. setuptools itself is extensible by means of entry points. Both entry points in your example (as cited by Marius) demonstrate that: one adds support for a new keyword argument to 'setup()'[1], and the other defines a new "writer" for 'egg-info'[2]. By design, both of those are supposed to be loaded / available for any invocation of 'setup()' in a Python where the are installed (not merely for packages which "mention" them).

I can see the logic when it's explained, but that doesn't make it not a bug IMO. That's a pretty astonishing behaviour: a declaration in the setup for one package has a mystifying effect on the setup of other packages. I won't press the point. Hopefully, though, someone does consider it a bug worth fixing. -- \ “To have the choice between proprietary software packages, is | `\ being able to choose your master. Freedom means not having a | _o__) master.” —Richard M. Stallman, 2007-05-16 | Ben Finney

Tres Seaver writes:

setuptools itself is extensible by means of entry points. Both entry points in your example (as cited by Marius) demonstrate that: one adds support for a new keyword argument to 'setup()'[1], and the other defines a new "writer" for 'egg-info'[2]. By design, both of those are supposed to be loaded / available for any invocation of 'setup()' in a Python where the are installed (not merely for packages which "mention" them).

What recourse do I have, then? I'm using entry points because it seems to be the only way I can declare functionality that resides in a module alongside the ‘setup.py’ which itself needs third-party packages. * During the distribution build stage (‘./setup.py build’ or earlier), I want to parse a reST document and derive some of the distribution metadata from that. * The ‘setup.py’ is the wrong place for this; it's complex and deserves its own module which can be imported for unit testing. * This also is totally unrelated to the functionality this distribution installs, so it doesn't belong in any of the packages to distribute for installation. * So I place it in a separate top-level module, ‘version’, only for use within ‘setup.py’. * That module itself needs a third-party distribution (‘docutils’) from PyPI. So I need that distribution to be installed *before* the ‘version’ module can be used. Apparently the ‘setup_requires’ option is the only way to do that. * Then, in order to declare how that functionality is to be used for commands like ‘./setup.py egg_info’, I have no other way than declaring a Setuptools entry point. * Declaring a Setuptools entry point makes Setuptools think the distribution-specific module must be imported by every other distribution in the same run. Of course it's not available there so those distributions fail to install. * It even thwarts the installation of ‘docutils’, the very distribution that is needed for ending this circular dependency. What am I missing? How can I implement complex functionality specific to packaging this distribution, without making an utter mess? -- \ “The whole area of [treating source code as intellectual | `\ property] is almost assuring a customer that you are not going | _o__) to do any innovation in the future.” —Gary Barnett | Ben Finney

Have you had a go with https://bitbucket.org/dholth/setup-requires ? I wrote it because it has always been problematic putting setup_requires in setup.py to try to get something that setup.py must import into the global namespace before it can run. I'd love to see some build helpers that take advantage of this feature. My version of setup-requires replaces setup.py with a short script that uses pip to install the setup requirements as specified in a special section in setup.cfg file, and then runs the real setup.py which does not have to mention any setup requirements, because they have already been installed by the setup-requires script. The setup requirements are installed into a subdirectory and are only available for the duration of the install. Then real-setup.py can use those requirements to accomplish whatever it has to do. On Mon, Jan 19, 2015 at 9:14 PM, Donald Stufft wrote:

...

On Jan 19, 2015, at 8:50 PM, Ben Finney wrote:

Tres Seaver writes:

...

setuptools itself is extensible by means of entry points. Both entry points in your example (as cited by Marius) demonstrate that: one adds support for a new keyword argument to 'setup()'[1], and the other defines a new "writer" for 'egg-info'[2]. By design, both of those are supposed to be loaded / available for any invocation of 'setup()' in a Python where the are installed (not merely for packages which "mention" them).

What recourse do I have, then?

I'm using entry points because it seems to be the only way I can declare functionality that resides in a module alongside the ‘setup.py’ which itself needs third-party packages.

* During the distribution build stage (‘./setup.py build’ or earlier), I want to parse a reST document and derive some of the distribution metadata from that.

* The ‘setup.py’ is the wrong place for this; it's complex and deserves its own module which can be imported for unit testing.

* This also is totally unrelated to the functionality this distribution installs, so it doesn't belong in any of the packages to distribute for installation.

* So I place it in a separate top-level module, ‘version’, only for use within ‘setup.py’.

* That module itself needs a third-party distribution (‘docutils’) from PyPI. So I need that distribution to be installed *before* the ‘version’ module can be used. Apparently the ‘setup_requires’ option is the only way to do that.

* Then, in order to declare how that functionality is to be used for commands like ‘./setup.py egg_info’, I have no other way than declaring a Setuptools entry point.

* Declaring a Setuptools entry point makes Setuptools think the distribution-specific module must be imported by every other distribution in the same run. Of course it's not available there so those distributions fail to install.

* It even thwarts the installation of ‘docutils’, the very distribution that is needed for ending this circular dependency.

What am I missing? How can I implement complex functionality specific to packaging this distribution, without making an utter mess?

-- \ “The whole area of [treating source code as intellectual | `\ property] is almost assuring a customer that you are not going | _o__) to do any innovation in the future.” —Gary Barnett | Ben Finney

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

I think the thing you’re missing here is that the entry point information is used to say “this project supplies these entry points”. setuptools does not offer the ability to have only build time entry points, it assumes the installed project supplies those entry points. The right way to handle this is to either import it in setup.py and do it there, or spin it out into it’s own thing that can be installed on it’s own.

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Donald Stufft writes:

setuptools does not offer the ability to have only build time entry points, it assumes the installed project supplies those entry points.

I'm now convinced that Setuptools entry points are not suitable to this purpose.

spin it out into it’s own thing that can be installed on it’s own.

Presumably this leads back to ‘setup_requires’ and ‘install_requires’?

The right way to handle this is to either import it in setup.py and do it there

I'd love to. How can I ‘import docutils’ in ‘setup.py’ without creating a circular dependency: * The correct way to get Docutils installed is to run ‘setup.py’ and have it install the dependencies. * Running ‘setup.py’ performs ‘import docutils’, and so will fail if Docutils is not already installed. What I need is a way to express “ensure Docutils is installed before continuing with other Setuptools actions” in ‘setup.py’. I don't know of a neat way to tell Setuptools that. -- \ “I'm beginning to think that life is just one long Yoko Ono | `\ album; no rhyme or reason, just a lot of incoherent shrieks and | _o__) then it's over.” —Ian Wolff | Ben Finney

Carl Meyer writes:

On 01/20/2015 07:28 PM, Ben Finney wrote:

...

What I need is a way to express “ensure Docutils is installed before continuing with other Setuptools actions” in ‘setup.py’. I don't know of a neat way to tell Setuptools that.

That is the precise purpose of the `setup_requires` kwarg.

That's too late though. What I need is to be able to instruct the build process to install Docutils so I can get stuff done *before ‘setup()’ is called*. Is there an API within Distutils or Setuptools that allows me to say “I haven't declared anything yet but go get and install distribution FOO right now”? I could do that before importing the module which needs Docutils. -- \ “The long-term solution to mountains of waste is not more | `\ landfill sites but fewer shopping centres.” —Clive Hamilton, | _o__) _Affluenza_, 2005 | Ben Finney

Just use my script. Once new-metadata has been standardized pip itself will know how to install build (setup) requirements and the script will be obsolete. Until then I think you will find it works. On Jan 20, 2015 10:27 PM, "Ben Finney" wrote:

Carl Meyer writes:

...

On 01/20/2015 07:28 PM, Ben Finney wrote:

...

What I need is a way to express “ensure Docutils is installed before continuing with other Setuptools actions” in ‘setup.py’. I don't know of a neat way to tell Setuptools that.

That is the precise purpose of the `setup_requires` kwarg.

That's too late though. What I need is to be able to instruct the build process to install Docutils so I can get stuff done *before ‘setup()’ is called*.

Is there an API within Distutils or Setuptools that allows me to say “I haven't declared anything yet but go get and install distribution FOO right now”? I could do that before importing the module which needs Docutils.

-- \ “The long-term solution to mountains of waste is not more | `\ landfill sites but fewer shopping centres.” —Clive Hamilton, | _o__) _Affluenza_, 2005 | Ben Finney

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Jan 20, 2015, at 10:27 PM, Ben Finney wrote:

Carl Meyer writes:

...

On 01/20/2015 07:28 PM, Ben Finney wrote:

...

What I need is a way to express “ensure Docutils is installed before continuing with other Setuptools actions” in ‘setup.py’. I don't know of a neat way to tell Setuptools that.

That is the precise purpose of the `setup_requires` kwarg.

That's too late though. What I need is to be able to instruct the build process to install Docutils so I can get stuff done *before ‘setup()’ is called*.

Is there an API within Distutils or Setuptools that allows me to say “I haven't declared anything yet but go get and install distribution FOO right now”? I could do that before importing the module which needs Docutils.

You could also add setup_requires=[“docutils”], and then pass in a Distclass (defined in your setup.py) and/or custom cmdclass entries which delay the import until the body of the methods on those classes. This should be after the setup_requires code has ran so the docutils import should work. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Tue, Jan 20, 2015 at 12:50:14PM +1100, Ben Finney wrote:

Tres Seaver writes:

...

setuptools itself is extensible by means of entry points. Both entry points in your example (as cited by Marius) demonstrate that: one adds support for a new keyword argument to 'setup()'[1], and the other defines a new "writer" for 'egg-info'[2]. By design, both of those are supposed to be loaded / available for any invocation of 'setup()' in a Python where the are installed (not merely for packages which "mention" them).

What recourse do I have, then?

I'm using entry points because it seems to be the only way I can declare functionality that resides in a module alongside the ‘setup.py’ which itself needs third-party packages.

Are you aware that those entry points will be used for every single package the user tries to install after installing python-daemon?

* During the distribution build stage (‘./setup.py build’ or earlier), I want to parse a reST document and derive some of the distribution metadata from that.

And presumably you don't want to use regexps for that. ;)

* The ‘setup.py’ is the wrong place for this; it's complex and deserves its own module which can be imported for unit testing.

* This also is totally unrelated to the functionality this distribution installs, so it doesn't belong in any of the packages to distribute for installation.

* So I place it in a separate top-level module, ‘version’, only for use within ‘setup.py’.

These three items sound very reasonable and shouldn't be causing problems.

* That module itself needs a third-party distribution (‘docutils’) from PyPI. So I need that distribution to be installed *before* the ‘version’ module can be used. Apparently the ‘setup_requires’ option is the only way to do that.

This is the point where my experience with setup_requires would tempt me to go "hey, regexes aren't _that_ bad". ;)

* Then, in order to declare how that functionality is to be used for commands like ‘./setup.py egg_info’, I have no other way than declaring a Setuptools entry point.

I think there's also the option of providing custom command classes to the setup() function. For completeness' sake there's also the option of monkey-patching distutils/setuptools internals (not recommended).

* Declaring a Setuptools entry point makes Setuptools think the distribution-specific module must be imported by every other distribution in the same run.

*And in every subsequent run.* This is important. Entry points live in the installed package metadata. You shouldn't ever define an entry point that points to a package or module that won't be installed.

Of course it's not available there so those distributions fail to install.

* It even thwarts the installation of ‘docutils’, the very distribution that is needed for ending this circular dependency.

What am I missing? How can I implement complex functionality specific to packaging this distribution, without making an utter mess?

I'll attempt some suggestions: 1. Simplify metadata extraction so it doesn't rely on docutils. 2. Implement metadata extraction using custom command classes[*] and setup_requires. (Sorry about the lack of specifics: I've never done this. I'm not even 100% certain the things you want can be done this way.) [*] https://docs.python.org/2/distutils/extending.html#integrating-new-commands, the bits about setup(..., cmdclass=...); not the bits about command_packages. 3. Duplicate the metadata in your setup.py; have a unit test that extracts it from your ReST and from your setup.py and compares the two. Since your release process should involve a "run all tests" step, this ought to prevent you from making releases with mismatched metadata. (Not a big fan of this one, but it's better than some alternatives, like having duplicated metadata that must be checked for consistency by hand.) 4. Move your entry points into an installed module that checks the name of the package it's working on and does nothing if it's not "python-daemon". (Not a fan of this one, it adds a moving part that can break any subsequent Python package installation.) 5. Use some replacement of setup_requires as suggested elsewhere in this thread, so you could import docutils at the top level of a setup.py. (I've never tried this approach and have no idea how reliable it is.) HTH, Marius Gedminas -- Since this protocol deals with Firewalls there are no real security considerations. -- RFC 3093

Ben Finney writes:

Marius Gedminas writes:

...

2. Implement metadata extraction using custom command classes[*] and setup_requires.

This seems to be the only one which might be feasible (and without needless duplication of information, which is part of the whole point of this exercise). I will learn more and try it.

Okay, the Setuptools ‘egg_info’ command is rather hostile to extension, but less hostile than Setuptools entry points. I have cobbled together a solution. The packaging code for version 2.0.4 of ‘python-daemon’: * Has no Setuptools entry points. These are not the right tool for the job, it seems. * Adds a custom Setuptools command ‘write_version_info’ to parse the Changelog and write the version info metadata file. * Minor hack: Uses a custom ‘egg_info’ command that will recognise and run sub-commands. (Why doesn't every top-level command do this without needing to be taught?) * Major hack: Re-binds class names in order to update their base class, after dynamically importing ‘docutils’. (If there was a way to ensure Docutils was installed before running any of the commands, this would not be needed.) Thanks for all the assistance understanding the limitations and quirks of Distutils and Setuptools. I'm not very happy with this solution, but hopefully it is modular enough that it can be changed, in some happy future when Python's packaging features are more tractable. -- \ “Programs must be written for people to read, and only | `\ incidentally for machines to execute.” —Abelson & Sussman, | _o__) _Structure and Interpretation of Computer Programs_ | Ben Finney

I have faced an issue with setuptools (and even plain distutils) in the past that has some resemblance to this. There was a PyPI package that contained C extensions and headers, and another python package that had more C extensions, but depended on the location of the headers on the first package to be built. Was it numpy and scipy? Anyway zope.proxy and zope.container had the same issue, as did Zope's Acquisition and ExtensionClass. The issue can be clearly summarized with the `setup.py` snippet below, which I found at [1]: from distutils.core import setup, Extension import numpy # define the extension module cos_module_np = Extension('cos_module_np', sources=['cos_module_np.c'], include_dirs=[numpy.get_include()]) # run the setup setup(ext_modules=[cos_module_np]) [1] https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_wit... Semantically, it's clear that numpy is both a `setup_requires` and an `install_requires` for the `setup.py` above. It's needed both at build time for the header locations and at run time for dynamically linking / module importing. However, even if the snippet above were using `setuptools` there'd be no way of passing `setup_requires` to the `setup()` call above, the script will have failed with an `ImportError` long before that, in the `include_dirs` part. The way the zope packages solved this issue was horribly crude and unsatisfying, but effective: they vendored the header files (and sometimes whole extensions) of the dependency packages. First by svn:extensions, and then by outright copying, when the `git` wave came around. The simplest and most backward compatible fix for setuptools I could think of would be for setup() keywords to have lazy evaluation: In setuptools, some `setup()` keywords accept parameters in multiple formats. For example `entry_points` accepts both a dictionary and a .ini formatted string [2]. [2] https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-ser... If `setup()` keywords were to accept callables as well, then we could delay importing dependency packages until `setup_requires` where already satisfied, like this: from setuptools import setup, Extension def ext_modules(): import numpy # define the extension module cos_module_np = Extension('cos_module_np', sources=['cos_module_np.c'], include_dirs=[numpy.get_include()]) return [cos_module_np] # run the setup setup(setup_requires=['numpy'], ext_modules=ext_modules) Cheers, Leo On Friday, January 23, 2015, Ben Finney wrote:

Ben Finney writes:

...

Marius Gedminas writes:

...

2. Implement metadata extraction using custom command classes[*] and setup_requires.

This seems to be the only one which might be feasible (and without, needless duplication of information, which is part of the whole point of this exercise). I will learn more and try it.

Okay, the Setuptools ‘egg_info’ command is rather hostile to extension, but less hostile than Setuptools entry points. I have cobbled together a solution.

The packaging code for version 2.0.4 of ‘python-daemon’:

* Has no Setuptools entry points. These are not the right tool for the job, it seems.

* Adds a custom Setuptools command ‘write_version_info’ to parse the Changelog and write the version info metadata file.

* Minor hack: Uses a custom ‘egg_info’ command that will recognise and run sub-commands. (Why doesn't every top-level command do this without needing to be taught?)

* Major hack: Re-binds class names in order to update their base class, after dynamically importing ‘docutils’. (If there was a way to ensure Docutils was installed before running any of the commands, this would not be needed.)

Thanks for all the assistance understanding the limitations and quirks of Distutils and Setuptools. I'm not very happy with this solution, but hopefully it is modular enough that it can be changed, in some happy future when Python's packaging features are more tractable.

-- \ “Programs must be written for people to read, and only | `\ incidentally for machines to execute.” —Abelson & Sussman, | _o__) _Structure and Interpretation of Computer Programs_ | Ben Finney

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 01/19/2015 01:23 AM, Ben Finney wrote: [snip]

My understanding, based on an answer received elsewhere [0], is that omitting the file from ‘setup.py’ but adding it to ‘MANIFEST.in’ causes it to be included in the sdist but omitted from install targets.

[0] URL:http://stackoverflow.com/questions/27930124/python-module-use-during-set...

That has worked for me. But it appears to cause problems for some others, related to this module which should not be installed.

That's the right answer.

Unfortunately this kind of problem (trouble post-install from a PyPI package) is difficult to test. How can I test the behaviour of ‘pip’ in this regard, without thrashing many releases of a package upload to PyPI?

Pip can install from the filesystem just as well as from PyPI. Run "python setup.py sdist" to generate your sdist file, and then run "pip install path/to/generated/sdist.tar.gz". Carl

On 01/19/2015 12:39 AM, Nick Coghlan wrote:

...

Nick Coghlan writes: I actually misunderstood your question. If you're just after the ability to say "I want to include this file in the sdist, but not in

On 19 January 2015 at 11:59, Ben Finney wrote: the built wheel file or installed distribution" (as I now believe you are), then you're in the implementation defined world of the significantly underspecified sdist format. I believe setting that up actually *is* possible already, but have no idea what incantation you'll need to pass to setuptools to make it do it (and the docs are unlikely to be a reliable guide).

This is actually quite easy, its a distutils feature not a setuptools one, and it works as documented. Just put the file in MANIFEST (or add a MANIFEST.in rule matching it), but don't reference it in any of the various setup.py kwargs that cause things to be installed. Carl

Ethan Furman writes:

However, I feel that requiring a dependency simply for the installation of the main package (the main package doesn't actually use this install-dependency, correct?) is heavy-handed and should be avoided.

It's lighter than that; the third-party dependency is needed only for building packages (sdist, or wheel). It parses a reST document to create an extra Setuptools metadata file, and then it's not needed any more. It is ideally a build-time dependency and not an install-time dependency, but I'm having a difficult time figuring out how to distinguish those so Setuptools will pay attention. -- \ “… one of the main causes of the fall of the Roman Empire was | `\ that, lacking zero, they had no way to indicate successful | _o__) termination of their C programs.” —Robert Firth | Ben Finney

On 1 February 2015 at 06:19, Ethan Furman wrote:

On 01/29/2015 08:58 PM, Ben Finney wrote:

...

Ethan Furman writes:

...

However, I feel that requiring a dependency simply for the installation of the main package (the main package doesn't actually use this install-dependency, correct?) is heavy-handed and should be avoided.

It is ideally a build-time dependency and not an install-time dependency, but I'm having a difficult time figuring out how to distinguish those so Setuptools will pay attention.

Ah, so it's needed with the sdist command and not the install command? Yeah, you definitely have my sympathies with that one. Too bad there's no easy way to specify behavior based on the command used... (hopefully someone will now tell how I am wrong about that ;) .

You're right, which is why PEP 426 has a more fine-grained dependency specification model (separating runtime, build, test and development dependencies). Other things are higher on the todo list right now than pushing that forward, but we'll get there eventually. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Jan 31, 2015 at 3:19 PM, Ethan Furman wrote:

On 01/29/2015 08:58 PM, Ben Finney wrote:

...

Ethan Furman writes:

...

However, I feel that requiring a dependency simply for the installation of the main package (the main package doesn't actually use this install-dependency, correct?) is heavy-handed and should be avoided.

It is ideally a build-time dependency and not an install-time dependency, but I'm having a difficult time figuring out how to distinguish those so Setuptools will pay attention.

Ah, so it's needed with the sdist command and not the install command? Yeah, you definitely have my sympathies with that one. Too bad there's no easy way to specify behavior based on the command used... (hopefully someone will now tell how I am wrong about that ;) .

We just subclass the command in question to do the relevant tasks (like generate some source files when running sdist). The subclassed command can just try to import the dependencies and raise a (useful) exception message if they are not available. For example Astropy recently grew Jinja2 is a dependency to build some source files. The built files are included in the sdist so casual users don't have to worry about it. Developers on the other hand will be nagged that they need Jinja2, and it isn't automatically installed, which sucks. But for the most part the only people who will need to worry about that can figure out how to install build dependencies. I'm less worried about it for them than I am for the casual user trying to run `python setup.py install` (or pip install for that matter). Erik