[Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[M.-A. Lemburg]

[adding list back on CC]

On 22.04.2015 16:11, Christoph Schmitt wrote:
> Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
>> On 22.04.2015 12:38, Robert Collins wrote:
>>> On 22 April 2015 at 22:13, Christoph Schmitt <dev-mailings at gongy.de> wrote:
>>>> Hello again,
>>>>
>>>> since I haven't got any replies yet I'm trying to make myself a bit more
>>>> precise now. I consider the behaviour described in my original posting a
>>>> bug. I posted to this list because the setuptools docs say "Please use the
>>>> distutils-sig mailing list [3] for questions and discussion about
>>>
>>>> Test 3)
>>>> DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
>>>> Result: all modules can be imported
>>>
>>> This is correct AFAICT.
>>>
>>> the setuptools namespace_packages thing predates PEP-420, and because
>>> PEP-420 namespaces don't interoperate with .pth file based packages
>>> (expecially when you get into interactions between system non-PEP-420
>>> + virtualenv PEP-420 packages!) changing this is super hard: you'll
>>> guarantee to break many existing installs.
>>>
>>> Perhaps there should be a new keyword, but since nothing is needed to
>>> make things work, it seems like it would be rather redundant.
>>
>> You can make use of the namespace_packages keyword argument to setup()
>> optional depending on which Python version is running it.
>>
>> I guess that's the only way forward unless you want to break
>> the package for previous Python versions.
>>
>> However, doing so may be hard for namespaces which are used
>> by a lot of packages.
>>
>> Perhaps setuptools could simply ignore the keyword for
>> Python 3.3+ and then rely on PEP 420 to get things working
>> in more or less the same way:
>>
>> https://www.python.org/dev/peps/pep-0420/
> 
> I would be fine with declaring namespace_packages conditionally. But doesn't this affect sdist in
> another way than install (or pip install)? If an sdist intended for use with Python < 3.3 is created
> with Python >= 3.3, the included metadata (egg-info) would look different (I don't know if pip
> relies on egg-info or setup.py).

The egg-info generated by setup.py at install time :-)

> That would apply also if namespace_packages would be ignored
> automatically for Python >= 3.3 as you proposed.
> 
> As a consequence, two distributions were neccessary. One with namespace_packages declared and
> containing an __init__.py (with pkg_resources.declare_namespace) and another one without those
> additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace
> packages in the latter case?

Like I mentioned above: it's probably better for setuptools to handle
this in a consistent way rather than changing all setup.pys.

One detail I'm not sure about is how compatible the two namespace
package techniques really are. The PEP 420 hand waves over possible
differences and only addresses pkgutil, not the setuptools
approach, which is by far more common. For most simply applications
not relying on any of the advanced features, they will most likely
be compatible.

I guess some experiments are necessary to figure that out.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/