[Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[M.-A. Lemburg]

On 22.04.2015 21:08, Robert Collins wrote:
> On 23 April 2015 at 03:09, M.-A. Lemburg <mal at egenix.com> wrote:
>> [adding list back on CC]
>>
>> On 22.04.2015 16:11, Christoph Schmitt wrote:
>>> Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
>>>> On 22.04.2015 12:38, Robert Collins wrote:
> 
> 
>>> That would apply also if namespace_packages would be ignored
>>> automatically for Python >= 3.3 as you proposed.
>>>
>>> As a consequence, two distributions were neccessary. One with namespace_packages declared and
>>> containing an __init__.py (with pkg_resources.declare_namespace) and another one without those
>>> additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace
>>> packages in the latter case?
>>
>> Like I mentioned above: it's probably better for setuptools to handle
>> this in a consistent way rather than changing all setup.pys.
> 
> I agree, but consider this situation - on any PEP-420 supporting python
> 
> Two packages: name.A and name.B. name.A already installed on the
> machine systemwide using old-style namespace path hacks, and then do a
> wheel install of name.B.
> 
> If the wheel for name.B was built expecting PEP-420, it won't be
> importable after install (because the path manipulation that sets up
> name as an old-style namespace happens in site.py).
> 
> If the wheel was built expecting old-style namespaces, but A was
> installed using PEP-420, then A won't be installable after B is
> installed (same reason).
> 
> The point of splitting the place the two are installed is to show that
> the user may not be able to fix the existing install.
> 
> So - pip would have to a) detect both styles of package, b)
> automatically install all installed-but-wrong-style versions to match
> the site installed ones. And if any of the packages in the namespace
> only support legacy, everything would be clamped down to legacy.

I don't think support mixed setups is really a practical option.

Either the namespace package is legacy all the way, or it
isn't and uses PEP 420.

Wouldn't it be possible for setuptools or pip to work this out
depending on the Python version ?

Python < 3.3: use legacy system for everything
Python >= 3.3: use PEP 420 for everything (and remove __init__.py
              files at install time as necessary)

>> One detail I'm not sure about is how compatible the two namespace
>> package techniques really are. The PEP 420 hand waves over possible
>> differences and only addresses pkgutil, not the setuptools
>> approach, which is by far more common. For most simply applications
>> not relying on any of the advanced features, they will most likely
>> be compatible.
> 
> They are entirely incompatible.

Oh, I meant from a functionality point of view, not the
technology side.

They both allow installing packages in different directory
trees and that's the only feature most namespace packages
use.

>> I guess some experiments are necessary to figure that out.
> 
> I spent a week last year debugging issues within openstack with
> namespace packages, local tree imports of same, and both pure venv and
> split system and venv environments.
> 
> Some key interesting things:
>  - the setuptools pth files aren't fully venv aware today -
> potentially fixable but the resulting pth file is fugly, so I decided
> not worth it.
>  - local tree imports work a heck of a lot better in tox etc with
> PEP-420 namespaces
>  - PEP-420 namespaces can work on older pythons with importlib, but
>  - PEP-420 and legacy packages being mixed for one namespace doesn't
> work at all today - in principle fixable via changes to both
> setuptools and importlib - but it was about here that the other
> openstack folk said 'ok wow, lets just stop using namespace packages
> :)

It's certainly easier to not use namespace packages and simply
install packages into the same tree. The main reason for
namespace packages in setuptools was the desire to stuff everything
into ZIP files (the eggs) or egg directories, even when installed,
to simplify uninstalls.

As soon as you drop that requirement you can have the package manager
deal with the complexities of having multiple packages share the
same directory in site-packages. That's solvable as pip, RPM,
apt, et al. show :-)

But ok, that doesn't solve the issue of support namespace
packages if the developers want to use them ;-)

> I think its a -lot- easier to reason about these things as two
> entirely separate features.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/