On 14 July 2013 12:46, Donald Stufft <donald@stufft.io> wrote:

I'm sure I've seen people say other things that have made me think "are you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive.

The other big one is the one you noted about pip *not* offering a stable API, *but* exposing an apparently stable API to introspection. Introspection currently tells me that pip exports *at least* 32 public names (and this is without checking for public submodules that aren't implicitly imported by pip/__init__.py):

>>> import pip; public = set(k for k, v in pip.__dict__.items() if not k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, "__module__") or v.__name__.startswith("pip."))); print(len(public))
32

If pip really has no stable public API, then it should properly indicate this under introspection (if it already uses relative imports correctly, then the easiest ways to achieve that are to just shove everything under a "pip._impl" subpackage or shuffle it sideways into a "_pip" package).

Pip does not use relative imports. Is simply documenting the fact there is no public API enough? Pushing everything into a _impl or _pip directory makes me nervous because that's a lot of code churn (and I know there are people using those APIs, and while they aren't technically stable it feels like moving things around just for the sake of an _ in the name is unfriendly to those people.

Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager).

Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is "test" packages, including the standard library's test package, and for those you can make the case that having "test" or "tests" as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore.

I really this is a fairly big ask for the pip maintainers, but I *don't* consider "Oh, don't use our module API, it isn't stable" to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring).

Instead of a milestone I added a PEP439 tag so that we can differentiate between 1.5 milestone items for PEP439 and not. Ideally we don't need to drop anything from 1.5 but just in case we do.

I think we should probably target pip 1.5 to release in the beginning of December? Would need to see what the other team members think, that's a shorter release cycle then we normally have but I think it'd be good to have 1.5 out for a month or so to get real world use to make sure it doesn't need a patch release before inclusion in CPython (assuming the dates you mentioned are correct).

Just to confuse matters a little bit, Richard has suggested explicitly creating a bundling PEP as a *competitor* to PEP 439, thus making it easier to be explicit about our reasons for rejecting bootstrapping in favour of bundling. I think that's a good way to move this forward, but I won't actually reject 439 until the competing bundling PEP has been posted (otherwise people might get the wrong impression that we're moving away from the idea of making "pip install X" work out of the box, when we're really just changing our tactics for achieving that goal).

I also realised what is probably a better idea than "python -m getpip" for dealing with the "How do I get pip after doing a source build?": add a "get-pip.py" utility to Tools/scripts in the cpython repo, rather than adding anything to the standard library. This also puts us on a more solid footing for getting pip bundled with 2.7.x at some point: we're not touching the standard library, just the installers and the utility scripts.

The bundling PEP should also suggest to Linux packagers that pip be considered an essential part of a fully functional Python installation. Exactly how that is handled will be up to the distro packagers, but could include noting pip as a recommended dependency for Python (Debian), or rearranging the packaging to make "cpython" a package in its own right, with "python" requiring both "cpython" and "python-pip" (while the latter would just require cpython).

I'll post an explicit call for a PEP champion in a separate thread.

Cheers,

Nick.

--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia