[Distutils] Expectations on how pip needs to change for Python 3.4

[Nick Coghlan]

On 14 July 2013 12:46, Donald Stufft <donald at stufft.io> wrote:

> I'm sure I've seen people say other things that have made me think "are
>> you expecting the pip maintainers to make that change?" in the various
>> threads, so I doubt this list is definitive.
>>
>
> The other big one is the one you noted about pip *not* offering a stable
> API, *but* exposing an apparently stable API to introspection.
> Introspection currently tells me that pip exports *at least* 32 public
> names (and this is without checking for public submodules that aren't
> implicitly imported by pip/__init__.py):
>
> >>> import pip; public = set(k for k, v in pip.__dict__.items() if not
> k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v,
> "__module__") or v.__name__.startswith("pip."))); print(len(public))
> 32
>
> If pip really has no stable public API, then it should properly indicate
> this under introspection (if it already uses relative imports correctly,
> then the easiest ways to achieve that are to just shove everything under a
> "pip._impl" subpackage or shuffle it sideways into a "_pip" package).
>
>
> Pip does not use relative imports. Is simply documenting the fact there is
> no public API enough? Pushing everything into a _impl or _pip directory
> makes me nervous because that's a lot of code churn (and I know there are
> people using those APIs, and while they aren't technically stable it feels
> like moving things around just for the sake of an _ in the name is
> unfriendly to those people.
>

Either the existing APIs are moved to a different name, or they get
declared stable and pip switches to "internally forked" APIs any time a
backwards incompatible change is needed for refactoring purposes (see
runpy._run_module_as_main for an example of needing to do this in the
standard library). I've had to directly deal with too many issues arising
from getting this wrong in the past for me to endorse bundling of a module
that doesn't follow this practice with CPython - if introspection indicates
an API is public, then it's public and subject to all standard library
backwards compatibility guarantees, or else we take the pain *once* and
explicitly mark it private by adding a leading underscore rather than
leaving it in limbo (contextlib._GeneratorContextManager is a standard
library example of the latter approach - it used to lack the leading
underscore, suggesting it was a public API when it's really just an
implementation detail of contextlib.contextmanager).

Mere documentation of public vs private generally doesn't cut it, as too
many people use dir(), help() and inspect() rather than the published docs
to explore APIs. The only general exception I'm aware of is "test"
packages, including the standard library's test package, and for those you
can make the case that having "test" or "tests" as a name segment is just
as clear an indicator of something being private as at least one name
segment starting with a leading underscore.

I really this is a fairly big ask for the pip maintainers, but I *don't*
consider "Oh, don't use our module API, it isn't stable" to be an adequate
answer for something that is bundled with the standard installers. Beyond
that, I don't mind if the answer is to declare the 1.5 API stable or to
sprinkle underscore where appropriate or moving everything to a private
package - the documentation and the naming conventions just need to be
consistent in their private vs public distinctions (although your points do
suggest heavily that the right answer is to accept the burden of backwards
compatibility for all APIs currently marked public, and move towards the
introduction of appropriate private APIs over time through refactoring).

 Instead of a milestone I added a PEP439 tag so that we can differentiate
between 1.5 milestone items for PEP439 and not. Ideally we don't need to
drop anything from 1.5 but just in case we do.


> I think we should probably target pip 1.5 to release in the beginning of
> December? Would need to see what the other team members think, that's a
> shorter release cycle then we normally have but I think it'd be good to
> have 1.5 out for a month or so to get real world use to make sure it
> doesn't need a patch release before inclusion in CPython (assuming the
> dates you mentioned are correct).
>

Just to confuse matters a little bit, Richard has suggested explicitly
creating a bundling PEP as a *competitor* to PEP 439, thus making it easier
to be explicit about our reasons for rejecting bootstrapping in favour of
bundling. I think that's a good way to move this forward, but I won't
actually reject 439 until the competing bundling PEP has been posted
(otherwise people might get the wrong impression that we're moving away
from the idea of making "pip install X" work out of the box, when we're
really just changing our tactics for achieving that goal).

I also realised what is probably a better idea than "python -m getpip" for
dealing with the "How do I get pip after doing a source build?": add a
"get-pip.py" utility to Tools/scripts in the cpython repo, rather than
adding anything to the standard library. This also puts us on a more solid
footing for getting pip bundled with 2.7.x at some point: we're not
touching the standard library, just the installers and the utility scripts.

The bundling PEP should also suggest to Linux packagers that pip be
considered an essential part of a fully functional Python installation.
Exactly how that is handled will be up to the distro packagers, but could
include noting pip as a recommended dependency for Python (Debian), or
rearranging the packaging to make "cpython" a package in its own right,
with "python" requiring both "cpython" and "python-pip" (while the latter
would just require cpython).

I'll post an explicit call for a PEP champion in a separate thread.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130714/84be516f/attachment-0001.html>