[Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

Wed Apr 22 12:13:41 CEST 2015

Hello again,

since I haven't got any replies yet I'm trying to make myself a bit more 
precise now. I consider the behaviour described in my original posting a 
bug. I posted to this list because the setuptools docs say "Please use 
the distutils-sig mailing list [3] for questions and discussion about 
setuptools, and the setuptools bug tracker [4] ONLY for issues you have 
confirmed via the list".

I have two questions, which I hope some expert here can answer:
1) How do I properly handle PEP420 namespace packages with setuptools 
and pip (Python >= 3.3) assuming the scenario described below?
2) Is the behaviour of setuptools/pip as I encountered it intended 
(including the solution I found) or is it a bug that should be filed?

So here is, with some more detail, what I am trying to do: I am using 
Python 3.4. There are two projects with modules in the namaspace 
coolpkg. These implement the sub-packages coolpkg.bar and and 
coolpkg.foo respectively. Both projects have (as allowed by PEP420) no 
coolpkg/__init__.py. Both projects have a setup.py using setuptools 
(15.0) to create a source distribution and will be installed using pip 
(6.1.1).

In addition to that, there is another submodule coolpkg.baz, which will 
not be packaged/installed using setuptools/pip. Instead the folder 
containing it, will we added to the PYTHONPATH.

Here is the complete layout of the projects and the additional module.

project-bar/
project-bar/setup.py
project-bar/src
project-bar/src/coolpkg
project-bar/src/coolpkg/bar
project-bar/src/coolpkg/bar/__init__.py
project-bar/src/coolpkg/bar/barmod.py
project-foo/
project-foo/setup.py
project-foo/src
project-foo/src/coolpkg
project-foo/src/coolpkg/foo
project-foo/src/coolpkg/foo/foomod.py
project-foo/src/coolpkg/foo/__init__.py
shady-folder/
shady-folder/coolpkg
shady-folder/coolpkg/baz
shady-folder/coolpkg/baz/__init__.py
shady-folder/coolpkg/baz/bazmod.py

My goal is to have a runtime configuration such that modules 
coolpkg.foo.foomod, coolpkg.bar.barmod, coolpkg.baz.bazmod can all be 
imported.

Test 1) (just a basic test to verify general setup)
Add project-bar/src, project-foo/src and shady-folder manually to 
sys.path
Result: works (obviously)

For further tests:
Create source distributions with setup.py sdist for project-bar and 
project-foo, install them with pip and put shady-folder on PYTHONPATH. 
Declare packages=['coolpkg', 'coopkg.foo'] and packages=['coolpkg', 
'coopkg.bar'] in respectice setup.py, since find_packages does not play 
well with PEP420.

Test 2)
Delcare namespace_packages=['coolpkg'] in setup.py of each project
Result: coolpkg.foo.foomod and coolpkg.bar.barmod can be imported, but 
importing coolpkg.baz.bazmod fails
Suspected reason of failure: pip creates *-nspkg.path files which 
prevent coolpkg.baz from being found (breaking PEP420)

Test 3)
DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each 
project
Result: all modules can be imported

To put it bluntly:

The setup of test 2) is the one I would expect to work when writing a 
setup.py as described by the documentation, but pip unneccessarily 
creates *-nspkg.path files which undo the whole point of PEP420.

The setup of test 3) seems like an ulgy hack as we fool pip into not 
creating *-nspkg.path files by hiding the existance of the namepsace 
package from setuptools.

To summarize my questions stated above:
Which is a bug, which is intended and is there a way to handle 
PEP420-compliant packages properly with the current versions of 
setuptools and pip.

Kind regards,
Christoph Schmitt

Attachment: Example with scripts, modiy setup.py files according to test 
2) and 3)

Am 17.04.2015 um 09:31 schrieb Christoph Schmitt:

> I am using the newest versions of setuptools (15.0) and pip (6.1.1) 
> with Python 3.4. I wanted to satisfy the following two requirements at 
> the same time, but had some trouble:
> A) creating and installing two source distribution tarballs which have 
> the same top level namespace package (no __init__.py for namespace 
> package) with setuptools and pip
> B) having other packages within the same namespace package outside of 
> <python>/site-packages in the PYTHONPATH (not managed by 
> pip/setuptools)
> 
> Since Python >= 3.3 supports namespace packages out of the box (PEP420) 
> this seemed to me like a straightforward task. However, it turned out 
> to be not. Either, setuptools would not find my modules, pip would not 
> install them properly or requirement B) was broken. The solution that 
> worked for me, was to omit the 
> namespace_packages=['whatever_namespace'] declaration in setup.py (to 
> prevent the creation of *-nspkg.pth files by pip) and not to use 
> find_packages, which does not comply with PEP420 (no packages found).
> 
> This solution is somewhat counter-intuitive and I am not sure, whether 
> it is an intended/valid configuration of a setup.py file. Definitely, 
> it is not clear from the documentation (which has a section about 
> namespace packages btw.). Since I read into 
> https://bitbucket.org/pypa/setuptools/issue/97 [1] I now know that 
> convenient support PEP420 is not easy to achieve for setuptools 
> (regarding find_packages). However it would have been very helpful, if 
> the documentation explained how to handle PEP420-compliant namespace 
> packages (without any __init__.py) in setup.py. At least I would have 
> expected a hint that there are caveats regarding PEP420 with a link to 
> issue 97.
> 
> I also created a minimal example to reproduce the issue, which I can 
> provide if anyone is interested.
> 
> Kind regards,
> Christoph Schmitt
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig [2]

Links:
------
[1] https://bitbucket.org/pypa/setuptools/issue/97
[2] https://mail.python.org/mailman/listinfo/distutils-sig
[3] http://mail.python.org/pipermail/distutils-sig/
[4] https://bitbucket.org/pypa/setuptools/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coolpkg-example.tar.gz
Type: application/x-gzip
Size: 1084 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150422/872eda8b/attachment.bin>