On Sat, Jul 13, 2013 at 8:07 AM, Donald Stufft <donald@stufft.io> wrote:

+1 to this as well. Ideally, if we go down this route, installing python just comes with pip preinstalled. However that takes place :)

Well, I don't want a "however that takes place" that causes more packaging problems in the future, so I'm -1 on that. :-) I think including it in binary installers in such a way that pip can upgrade itself is definitely a good idea. Including it in the source code distributions seems less of a benefit. In fact, I don't even think I want it as things stand right now. Story time: This week a lot of people had problems with zc.buildout, because of the new releases of setuptools and distribute. Essentially they tried to upgrade themselves and failed for various reasons, such as distribute or setuptools being installed in the system python and not being upgradeable, or not being upgradeable for other reasons, like permissions. Mostly, I had no problems, because the Python installs I use have only one package installed: Virtualenv. That means that for most buildouts I used, buildout had istelf installed setuptools or distribute in its own isolated way. I only got errors where an overly complex build tool insisted on making virtualenvs, installing distribute in them and then running buildout with those virtualenvs. In those cases I had to run bin/pip -U distribute, so it was also an easy fix. So by default I prefer to have no packages except virtualenv for my source-built Pythons. This isn't a big issue, and I'm not gonna throw a hissyfit or anything, but I think I'll -1 the idea of including pip in a source install. Having pyvenv install pip by default in the new environment is +lots from me though. Also, having pyvenv be able to install itself in a non-empty directory would be great. ;-) //Lennart

In article <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io>, Donald Stufft <donald@stufft.io> wrote:

On Jul 13, 2013, at 1:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release.

Off the top of my head, including a copy of pip as a pre-installed global site-package seems like a very reasonable suggestion. For the python.org OS X installer, it should be no problem to implement. It would be equally easy to implement for future 2.7 and 3.3 maintenance releases.

We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. If Linux packagers really don't want it installed by default they could simply just remove it and either install it along with Python, or continue to keep it how it is today as a separate package?

This sounds an unnecessary complication. I suspect that there is a small minority of users who actually build Python from source. And they should know what they are doing. I believe most users either use a distribution-provided Python (via their OS) or a third-party package provider (including python.org binary installers and their derivatives). The OS distributors are going to do what they currently do; the only change needed is to persuade them to include their pip package as a mandatory dependency. Trying to hack the Python source build process to include a copy of pip is just not worth the effort. -- Ned Deily, nad@acm.org

On Sat, Jul 13, 2013 at 2:29 AM, Ned Deily <nad@acm.org> wrote:

...

On Jul 13, 2013, at 1:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the

In article <55B209B3-9576-4CF0-B58C-2A1E692AFFF1@stufft.io>, Donald Stufft <donald@stufft.io> wrote: problem of

...

...

coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release.

Off the top of my head, including a copy of pip as a pre-installed global site-package seems like a very reasonable suggestion. For the python.org OS X installer, it should be no problem to implement. It would be equally easy to implement for future 2.7 and 3.3 maintenance releases.

Does Apple just install the python.org OS X installer for distribution, or do they build their own thing? My only worry is that Apple will not get the message about including pip and we will end up with an odd skew on OS X (I'm not worried about Linux distros as they all seem to follow Python development closely). And we obviously need to know if Martin is okay with doing something similar on Windows.

...

We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed. If Linux packagers really don't want it installed by default they could simply just remove it and either install it along with Python, or continue to keep it how it is today as a separate package?

This sounds an unnecessary complication. I suspect that there is a small minority of users who actually build Python from source. And they should know what they are doing. I believe most users either use a distribution-provided Python (via their OS) or a third-party package provider (including python.org binary installers and their derivatives). The OS distributors are going to do what they currently do; the only change needed is to persuade them to include their pip package as a mandatory dependency. Trying to hack the Python source build process to include a copy of pip is just not worth the effort.

And I doubt that will take much convincing, e.g. ActiveState includes their own installer -- PyPM -- in their distro.

On 13 July 2013 16:05, Donald Stufft <donald@stufft.io> wrote:

On Jul 13, 2013, at 1:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release.

We could simply check it into the site-packages inside the CPython source tree could we not? *Not* providing a bootstrap script and merely checking it into the default site-packages means it's available for everyone. No matter how python installed.

Source code that isn't maintained through bugs.python.org isn't getting checked into the CPython repo - see PEP 360. Getting the latest released version of something from PyPI is a different story, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 13 Jul 2013 19:05, "Paul Moore" <p.f.moore@gmail.com> wrote:

On 13 July 2013 06:31, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

* bundling a *full* copy of pip with the Python installers for Windows

and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way).

How robust is the process of upgrading pip using itself? Specifically on

Windows, where these things typically seem less reliable.

Personally, I have never upgraded pip using itself, because I only ever

install pip in virtualenvs, which don't have a lifespan as long as a pip release cycle :-) It would be easy to imagine a new pip release resulting in a *lot* of bugs raised against Python (rather than pip) saying that the upgrade fails. And of course if an upgrade fails, we can't just release a new version of pip that fixes the issue, because it's the *old* version that is installed and has to do the upgrade. So there's manual fiddling to do. Not a good experience for Python users.

My current workflow is to have absolutely nothing installed in the system

Python and use virtualenvs for everything. This is a bit extreme, but the issues I've hit in the past when package management has gone wrong have made me very cautious.

If the pip upgrade process is rock-solid, this isn't an issue, but I'm

not sure that it is, myself. I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead. Cheers, Nick.

Paul

On 13 July 2013 14:31, Brett Cannon <brett@python.org> wrote:

+1 on the inversion. I don't know what that will do to pip, it makes sense to have the installer self-contained and the packaging/building libraries be something that you grab using the installer. Having to grab the packaging infrastructure to get an installer is the more painful route.

TBH, I don't understand what "the inversion" implies. If it means pip taking all of the distlib/setuptools code that it currently uses, and making it part of pip and maintained within pip (essentially as a fork while the "inversion" is going on) then I'm not keen on that. Personally, I don't want to have to maintain that code myself - I guess if Vinay and Jason were pip maintainers and looked after that code, then that's an option. If it means pip vendoring distlib and setuptools, then OK (we do that for distlib already) but I don't see the benefit - no-ione should be doing "from pip.vendor.distlib.version import Version". I'd need to know better what it means for pip, I guess... Paul

On 13 July 2013 13:25, Nick Coghlan <ncoghlan@gmail.com> wrote:

I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead.

Agreed, up to a point. What I've worked on in the past is things like automated wheel/sdist installers for systems with no internet access at all (distributions copied onto the PC via portable disk) or behind broken proxies (Internet Explorer works to download files from the net, but nothing else does). For those environments, the key to me is that I *only* use stuff that is available in a stock Python build. Something like pyton -m getpip *will not work*, so I have to roll my own means of bootstrapping. Of course I could copy a pip sdist to the machine, unpack it and run python setup.py install. More likely, I write bootstrap code in my script to do that automatically. It's a very specialised use case, certainly, and there are plenty of ways around the issues, but I have seen "give up and just use VBScript./perl" used as the fallback, as well, and that makes me sad... Paul

On Jul 13, 2013, at 11:00 AM, Paul Moore <p.f.moore@gmail.com> wrote:

Of course, saying explicitly "only the python -m pip command line interface is stable and supported" may well be enough. But didn't we just say that setuptools and distlib depend on the pip API? So either they have special privileges (presumably because they are under the umbrella of the PyPA) or we can't avoid documenting/supporting some API...

I don't think we need this any more than if pip was not pre-installed. So still nice to have for things like chef but not a requirement. setuptools and distlib won't depend on pip, pip will just bundle them (like it already does for distlib). The idea should be that pip itself has no dependencies because a package manager with dependencies is kind of strange and can easily lead to issues where the package manager breaks and is unifiable (e.g. setuptools breaks and pip can't be used to fix it). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jul 13, 2013, at 11:05 AM, Tres Seaver <tseaver@palladion.com> wrote:

Signed PGP part On 07/13/2013 08:25 AM, Nick Coghlan wrote:

...

I think we need to flip the dependencies so that pip as the installer has all the essential code for installation from PyPI and then setuptools and distlib depend on that pip infrastructure. No need to add anything to the standard library prematurely when we can add it to pip instead.

- -1. That would effectively mean inlining the bulk of setuptools' code into pip (which is just a UI / policy shim over it). You might as well just have your bootstrapper install both pip and setuptools and be done

Unless distlib (or something like it) lands in the stdlib with enough features to support a setuptools-less pip, of course.

At-which-point-the-State-will-wither-away'ly,

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

I was planning on doing this to pip anyways. pip should not have any dependencies setuptools or otherwise. So regardless of what happens with this PEP I want pip to be inlining setuptools and providing the code to make that transparent during install. This would mean that for people who are _just_ installing packages they don't need setuptools installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jul 13, 2013, at 6:46 AM, Brett Cannon wrote:

On Sat, Jul 13, 2013 at 1:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: In addition to the long thread based on Richard's latest set of updates, I've also received a few off-list comments on the current state of the proposal. So, I figured I'd start a new thread summarising my current point of view and see where we want to go from there.

1. However we end up solving the bootstrapping problem, I'm *definitely* a fan of us updating pyvenv in 3.4 to ensure that pip is available by default in new virtual environments created with that tool. I also have an idea for a related import system feature that I'll be sending to import-sig this afternoon (it's a variant on *.pth and *.egg-link files that should be able to address a variety of existing problems, including the one of *selectively* making system and user packages available in a virtual environment in a cross-platform way without needing to copy them)

2. While I was originally a fan of the "implicit bootstrapping on demand" design, I no longer like that notion. While Richard's bootstrap script is a very nice piece of work, the edge cases and "neat tricks" have built up to the point where they trip my "if the implementation is hard to explain, it's a bad idea" filter.

Accordingly, I no longer think the implicit bootstrapping is a viable option.

3. That means there are two main options available to us that I still consider viable alternatives (the installer bundling idea was suggested in one of the off list comments I mentioned):

* an explicit bootstrapping script * bundling a *full* copy of pip with the Python installers for Windows and Mac OS X, but installing it to site-packages rather than to the standard library directory. That way pip can be used to upgrade itself as normal, rather than making it part of the standard library per se. This is then closer to the "bundled application" model adopted for IDLE in PEP 434 (we could, in fact, move to distributing idle the same way).

I'm currently leaning towards offering both, as we're going to need a tool for bootstrapping source builds, but the simplest way to bootstrap pip for Windows and Mac OS X users is to just *bundle a copy with the binary installers*. So long as the bundled copy looks *exactly* the way it would if installed later (so it can update itself), then we avoid the problem of coupling the pip update cycles to the standard library feature release cycle. The bundled version can be updated to the latest available versions when we do a Python maintenance release.

For Linux, if you're using the system Python on a Debian or Fedora derivative, then "sudo apt-get python-pip" and "sudo yum install python-pip" are both straightforward, and if you're using something else, then it's unlikely getting pip bootstrapped using the bootstrap script is a task that will bother you :)

The "python -m getpip" command is still something we will want to provide, as it is useful to people that build their own copy of Python from source.

But is it going to make a difference? If we shift to using included copies of pip in binary installers over a bootstrap I say leave out the bootstrap as anyone building from source should know how to get pip installed on their machine or venv.

The only reason I see it worth considering is if pyvenv starts bootstrapping pip and we want to support the case of pip not being installed. But if we are including it in the binary installer and are going to assume it's available through OS distros, then there isn't a need to as pip can then install pip for us into the venv and skip any initial pip bootstrap. If pip isn't found we can simply either point to the docs in the failure message or print out the one-liner it takes to install pip (and obviously there can be a --no-pip flag to skip this for people who want to install it manually like me who build from source).

IOW I think taking the worldview in Python 3.4 that pip will come installed with Python unless you build from source negates the need for the bootstrap script beyond just saying ``curl https://pypi.python.org/get-pip.py | python`` if pip isn't found.

This is highly unhelpful for dealing with systems automation. For the foreseeable future, the bulk of Python 3.4 installations will either be source installs, or homegrown packages based on source installs. The bundled pip doesn't need to be included with, say, an hg clone that you then build and install, but it does have to come with an install from an official release source tarball. --Noah

On 13 Jul, 2013, at 16:39, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

I smoke-tested the script on vanilla Python 3.3 installations on Windows and OS X. On OS X, the pip script was written to the appropriate Frameworks folder, but not to /usr/local/bin - I assume it would be simple enough to arrange that?

Not installing in /usr/local/bin is a feature that makes it easier to maintain several python installs without worrying about contamination (for example Python 3 and Python 2, or even 2.6 and 2.7). The installer by default adds the framework 'bin' directory to the environment for the shell of the user that installed the framework. Ronald