On Jul 13, 2013, at 11:01 AM, Paul Moore <p.f.moore@gmail.com> wrote:

5. Provide a stable documented programming interface.

As I said in the other thread I don't think this is required any more than it does normally. I do think we need to have testing infrastructure in pip that tests against the development branch of CPython though. If pip is going to be included in the releases we need to make sure it works prior to it being released. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 13 July 2013 16:03, Donald Stufft <donald@stufft.io> wrote:

...

1. Install to user-packages by default.

Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too.

I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information.

...

2. Not depend on setuptools (??? - Nick's "inversion" idea)

I wanted to do this anyways. It will still "depend" on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have.

Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path...

...

3. Possibly change the wrapper command name from pip to pip3 on Unix.

Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard?

...

4. Ensure that pip upgrading itself in-place is sufficiently robust and reliable that users don't get "stuck" on the Python-supplied version.

I've always used pip to upgrade pip. The only time i've had problems is when setuptools messes up (which would be prevented if bundled).

I've never tried myself, but I'm on Windows and I expect in-place stuff like this to fail. Maybe I'm paranoid :-) Again I need to check. Thanks for the comments. Paul

On Jul 13, 2013, at 12:59 PM, Brett Cannon <brett@python.org> wrote:

Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue.

I find the -m interface ugly as a primary cli api. It's ok for bonus functionality (ala json.tool) and debugging utilities (ala SimpleServer) but as a developer of user facing tools I don't think I'd ever want to tell them that they should use ``python -m`` to execute my tool. It's also a massive change in functionality from the existing pip interface. ``pip install`` is what everyone uses. The point is more or less moot though unless you're advocating not including an executable script at all. Because pip is already able to be executed with ``python -m pip`` however I don't believe i've seen anyone use that in practice. It also provides the "pip" and "pip-X.Y" commands which should probably be normalized to "pip", "pipX", and "pipX.Y". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jul 13, 2013, at 1:55 PM, Paul Moore <p.f.moore@gmail.com> wrote:

1. It's not *actually* the case that the command is always "pip". Maybe it's "pip3" if your system makes the default Python be python 2, but you want to use python 3. Maybe you're creating a virtualenv and you haven't activated it yet. In that case a plain "pip" will quietly do the wrong thing (at the moment, I don't install pip in my system python precisely to avoid this issue). 2. On Windows, ...\Python34\Scripts is not on PATH by default. Even if python is (as python.exe is in a different directory to the one distutils installs executables in). Again, you can change your own PATH. 3. There's a lot of clutter. On Windows, you have 3 executables (pip.exe, pip3.exe and pip3.4.exe) and 3 scripts alongside them. For one command.

Apart from the first of these, the issues are all Windows ones, and it's reasonable to say "well, fix the Windows setup, then, it's silly". I have some sympathy with that view, but backward compatibility and many, many years of history will make that extremely difficult. Also, some of it may simply not be fixable because people won't agree on the solution (that's been the case in the past).

1. There's no good way to make it so you don't have to modify your command depending on what python you want to install into. In the case of both ``pip`` and ``python -m pip`` the changes a person would need to make is equivalent. They need to add a version number. The virtualenv case I don't see how that's relevant at all because if pip is preinstalled then pip will be available in both the virtualenv and the system environment. So both ``python -m pip`` and ``pip`` will be operating on the system python if you don't have it activated. 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach. 3. I don't really get the clutter argument. Does it *hurt* to have extra files there? I don't think i've ever looked at the directories on my $PATH and gone "wow I wish there was _less_ things in here". Is this an actual problem people have? Even if it was I think user experience trumps this case. I'm not sure what Brett is exactly advocating for. If he just wants to document it as ``python -m pip`` well whatever. I have absolutely zero faith that method of invocation will ever become popular. Every single piece of documentation that i've ever seen out there for installing things with pip tells people to use ``pip install``. Every developer that I've ever seen out there is using ``pip install``. An explicit command is shorter, easier to type, and already has basically all of the mindshare behind it. People gravitate towards what's easiest and in my opinion ``-m`` is easier only for the folks implementing this, not for the end users. Even if ``python -m pip`` is documented we still need to handle the CLI case, and I think that following the convention used by most other programs on *Nix and by windows itself of making the commands, "pip", "pipX", and "pipX.Y" makes the most sense. If Brett is advocating we _remove_ the command line options and expose only the ``python -m pip`` command that I am vehemently against that and in my opinion that makes for a far worst experience than users have now. The very first thing I would do, if it did happen this way, is create a package "pip-sanity" on PyPI that did nothing but restored the commands and then we end up with a similar situation we have now. That people need to run some bullshit before they can start using pip in the way they want to. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jul 13, 2013, at 3:59 PM, Paul Moore <p.f.moore@gmail.com> wrote:

On 13 July 2013 19:24, Donald Stufft <donald@stufft.io> wrote: 2. This sounds like something that needs fixed on Windows. Even if you say ``-m`` for pip then things are still broken by default for any other package on PyPI that installs a script. So this feels like something wrong with Python on windows not wrong with the script approach.

It is, and it should be fixed. But in many years, nobody has managed to come up with an acceptable solution. The debates seem to be largely around what happens if you install multiple versions of Python and then remove some of them, and how badly your system PATH gets messed up by this. I don't know how many people actually do things like that, but nevertheless it's never been sorted out. (Not all of the arguments are trivial, either, there are some genuinely difficult issues to resolve, IIRC).

Ultimately, I guess there are a few options: * Accept that Windows is a problem in this regard, but don't worry about it - install executable wrappers/scripts and let the user deal with path issues.

Ultimately I think this is what the community is going to do regardless of what happens here unless we remove the command line tools all together.

* Promote "python -m pip" as a least common denominator approach, and mildly irritate people who don't use Windows (they can still use the commands, but the docs look odd to them).

This also has the problem where the existing documentation (project READMEs etc) are pointing to ``pip``. So it fractures the documentation about what the command "should" be.

* Only provide "python -m pip" and seriously annoy people who don't use Windows.

Also invalidate all the existing documentation :)

* Document the difference, which implies either a certain level of repetitious "pip install (or py -m pip install on Windows)" type of thing, or a high level "For Windows, the pip command is not available directly, you should use ``python -m pip`` in its place (or wrap this in the shell if you prefer)" which people may miss.

This is probably the most realistic approach, at least in my eyes. If the Scripts directory isn't available on windows people are going to need to know to either add it or execute it with python -m pip and that's going to include documentation outside our control. So given that there s a lot of existing documentation around ``pip`` and people are likely to continue that practice windows users will need to know that when random projects say to do ``pip install foo`` they need to translate that too ``python -m pip foo``.

It would be nice to get feedback from "normal users" on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch…

I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :)

Paul.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Because of the issues around compilation on Windows, we believe that most users avoid pip in favor of precompiled installers. The model of "download an executable that matches my Python version and run it" is more familiar than a command line tool, and unlikely to go away anytime soon. Those who use pip are going to be quite capable of managing their PATH variable to ensure the correct one is used (or they'll do what I do and use a full path). There are also GUI apps, but I have no idea how widely used they are. Sent from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: ‎7/‎13/‎2013 14:36 To: Donald Stufft<mailto:donald@stufft.io> Cc: Distutils<mailto:distutils-sig@python.org> Subject: Re: [Distutils] Expectations on how pip needs to change for Python 3.4 On 13 July 2013 21:14, Donald Stufft <donald@stufft.io<mailto:donald@stufft.io>> wrote: I don't know any windows users off hand except for you ;) (And you already said you use ``pip`` and not ``python -m pip`` which already works with pip :) You caught me :-) My problem is that I'm pretty sure I'm seriously atypical in never installing anything into my system Python. (And so I only use the pip command in activated virtualenvs, which *do* add the scripts directory to PATH). Paul.

On 13 July 2013 20:59, Paul Moore <p.f.moore@gmail.com> wrote:

It would be nice to get feedback from "normal users" on this. I suspect that the scientific community would make a good cross-section (AIUI there's quite a lot of Windows use, and for many people in the community Python is very much a tool, rather than a way of life :-)). Does anyone have links into the scipy groups? I lurk on the IPython lists, so I could ask there, at a pinch...

I don't know if I really count as a normal user but I can describe how Python is installed on the Windows machines in my faculty for scientific use. All our Windows machines have the Enthought Python Distribution (EPD) installed. This bundles CPython with numpy, scipy, matplotlib, wxpython, setuptools, pip and a whole load more. Ordinary users do not need to install numpy etc. since these are pre-installed. The bootstrap process is probably irrelevant since EPD installs easy_install and that can be used to install pip if desired. Ordinary users do not have write access to the EPD installation directory and so can only use pip/easy_install with --user anyway. On my own desktop machine which runs Windows all of the Python installations I use are inside my user directory so there is no meaningful difference between 'pip install' and 'pip install --user'. The real problem for us with using e.g. pip to install something like numpy is that it will not install the appropriate non-Python external libraries. For example, numpy ships with a just functional BLAS library but you really want to install and have it link against proper BLAS/LAPACK libraries. The good free libraries should be compiled on the target machine and pypi/pip/distutils do not currently help much with doing this. Debian (or at least Ubuntu) provides for example the ATLAS library as a source only package. This means that you can compile it on the target machine and get the most out of your CPU capabilities while still using the Debian tools to obtain the dependencies and manage the build process. The new wheel format will not help with this since even if there were an ATLAS wheel it would probably be a generic 686 binary without e.g. SSE. This is another advantage of using EPD which ships the non-free Intel MKL library. Python(x, y) is similar to EPD but is GPL'd and ships with OpenBLAS. Both distributions also ship MinGW which is useful since it's likely that our Windows machines will not have the appropriate MSVC version to match up with the CPython version. (They also don't suffer from Issue12641 so MinGW works). Oscar

On 15 July 2013 15:21, Paul Moore <p.f.moore@gmail.com> wrote:

On 15 July 2013 15:16, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

I don't know if I really count as a normal user but I can describe how Python is installed on the Windows machines in my faculty for scientific use.

Thanks, that's interesting.

Do people typically write command-line Python scripts?

A lot of researchers would run their scripts in an IDE such as Spyder (pre-installed by Enthought). This is the way that people are used to working if they are more familiar with Matlab. It's a bad idea in may ways but essentially rather than passing command line arguments a lot of people will just edit the variables at the top of their script and rerun it. Another method used is ipython which you can use to edit/run your code in a semi-interactive/semi-manual manner using the magic %edit command; this is similar to spyder. We also have a number of Linux clusters that are used to farm out big computational jobs and for this people do need to write proper command line scripts and submit the jobs via ssh (using putty rather than a real terminal) but they probably edit/test this code on the target machines.

If so, do they expect to be able to put them on PATH and run them?

Probably not. I think that most people make a folder full of scripts and either run them from an IDE or cd into the folder and run them there. Again this is basically how you would do it in Matlab. In scientific work the end user is someone who spends a lot of time writing quite small scripts that are often not really reusable and are tied in some sense to a wider project. I think that maybe 80% of the .py files I have written are command line scripts under 100 lines that produce a single figure with matplotlib. The majority of those scripts are tied to a LaTeX document and invoked by a Makefile with e.g. 'python scripts/fig1.py images/fig1.pdf'. Most of my colleagues would probably have a more manual/interactive approach than me though.

What command processor is typically used? Powershell or cmd?

I haven't seen anyone use Powershell but I assume that it is installed. It's not on my machine but I use Console2/git-bash which means that shebang lines already work for me. I do often see (unfortunate) people using cmd.exe though. Oscar

On 15 July 2013 17:47, Chris Barker - NOAA Federal <chris.barker@noaa.gov>wrote:

I don't _think_ this is just Windows Bashing: MS has done very very little to improve the whole command line experience on Windows over the years.

It's not MS-bashing. I agree with you, and I'm one of the least likely people around here to indulge in arbitrary MS-bashing. (With the exception of Steve Dower, I guess :-)) Powershell is a *great* step up from cmd, but there are still a lot of dodgy bits round the edges (mostly because of the whole "console vs GUI subsystems" thing).

All that is a way to say that Python can only make it so easy for Windows users, but what's in place is not bad, and it really makes sense for pip to use what's been there for ages, i.e. a command called "pip" (and pip2, pip3...) that sits in the same place that all other third-party Pyton "scripts" are installed. No matter how you slice it, a user will need to put that on their PATH one way or another.

Agreed, PATH manipulation is a fact of life for everyone, both Unix and Windows. Of course, what MS is telling us is: don't rely on the command line!

So a really nice thing to do for Windows users would be to provide a little GUI pip tool that's part of the standard install. (not that I'm volunteering to write it...has no none yet written a tkInter-bsed pip front-end?)

I don't think a GUI-based tool is the answer here - the command line is orders of magnitude more powerful. For simple cases yes, but we have bdist_wininst and bdist_msi for those, and they are clearly not enough.

the current setuptools exe-wrapper feels really kludgy, but it works -- it seems the only real problematic issue is the self-update problem

The self-update issue is the big one. There are others (completely incomprehensible errors if the #! line in the script is wrong, for example) but it's certainly pretty much the best solution we have at the moment. Most of my problems with the setuptools wrappers are not actually with the exe, but rather with the actual script (and its dependency on pkg_resources) that lies behind it - and that's not a Windows problem per se.

-- maybe there is a Windows guru somewhere that can fix that....

I think I'm that guru, unfortunately, and I'm not having a whole lot of luck :-) The real problem is not technical, actually - it's knowing what Windows users will actually be comfortable with. Unix users tend to assume Windows users are very uncomfortable on the command line (no offence meant to anyone by that) whereas the reality is that some are, some (like me...) really are not, and some are simply unfamiliar with the capabilities of the Windows command line through lack of need to use it (many of my colleagues, for example). I'm actually tempted to give up trying to please everyone, and just put together a solution that suits *me* and see how that flies. Second guessing what other people want makes my head hurt :-) Paul

On Mon, Jul 15, 2013 at 10:11 AM, Paul Moore <p.f.moore@gmail.com> wrote:

of Steve Dower, I guess :-)) Powershell is a *great* step up from cmd,

could we use a powershell script to launch python scripts? Maybe it wouldn't be any easier to update than an exe, but it might be more accessible.

...

Of course, what MS is telling us is: don't rely on the command line! So a really nice thing to do for Windows users would be to provide a little GUI pip tool that's part of the standard install. (not that I'm volunteering to write it...has no none yet written a tkInter-bsed pip front-end?)

I don't think a GUI-based tool is the answer here - the command line is orders of magnitude more powerful. For simple cases yes, but we have bdist_wininst and bdist_msi for those, and they are clearly not enough.

but they are really widely used -- maybe when binary wheels become ubiqitous, I'll stop using them, but I'm no command line phobic, and I usually go first to look for an installer on Windows. Over in the Mac world, we have similar issues (except a proper command line under there if you want it...), and eggs were a real issue because there was nothing to launch if you point and clicked on one. That being said, you're only going to get so far programmin python if you can't run a simple command on the command line. So maybe any GUI front-end should be part of a larger tool -- perhaps provided by IDE developers, for instance.

Most of my problems with the setuptools wrappers are not actually with the exe, but rather with the actual script (and its dependency on pkg_resources) that lies behind it - and that's not a Windows problem per se.

I've never liked pkg_resources..... ;-)

The real problem is not technical, actually - it's knowing what Windows users will actually be comfortable with. Unix users tend to assume Windows users are very uncomfortable on the command line (no offence meant to anyone by that) whereas the reality is that some are, some (like me...) really are not, and some are simply unfamiliar with the capabilities of the Windows command line through lack of need to use it (many of my colleagues, for example).

true -- and a simple command line solution is fine for most -- as I said, they'll need to deal with that one way or another if they are going to program..(and I say this as an instructor of intro to pyton classes...)

I'm actually tempted to give up trying to please everyone, and just put together a solution that suits *me* and see how that flies. Second guessing what other people want makes my head hurt :-)

fine plan! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Jul 15, 2013, at 6:22 PM, Paul Moore <p.f.moore@gmail.com> wrote:

On 15 July 2013 23:09, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:

...

For simple cases yes, but we have bdist_wininst and bdist_msi for those, and they are clearly not enough.

but they are really widely used -- maybe when binary wheels become ubiqitous, I'll stop using them, but I'm no command line phobic, and I usually go first to look for an installer on Windows.

The killer issue with bdist_wininst and bdist_msi is that they don't work with virtualenvs. I was a fan of them till I started using virtualenv, at which point they become totally useless :-(

I'm not against someone writing a GUI. But it won't be me :-)

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft <donald@stufft.io> wrote:

There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst.

To put numbers into perspective, there are ~180k total files uploaded to PyPI.

I don't hink this means that the installers aren't widely used, I think it mean they aren't distributed on PyPI. Installers are really useful for packages that require compiled code that depends on external libs -- and most of the major such package maintainers provide them. Also, numbers aren't as important as the handful of widely used, but hard to build, packages.... But they are useless with virtualenv, so I'm looking forward to binary wheels... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 16 July 2013 00:12, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

On 15 July 2013 23:39, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:

...

On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft <donald@stufft.io> wrote:

...

There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst.

...

To put numbers into perspective, there are ~180k total files uploaded to PyPI.

I don't hink this means that the installers aren't widely used, I think it mean they aren't distributed on PyPI.

Installers are really useful for packages that require compiled code that depends on external libs -- and most of the major such package maintainers provide them.

I second this. I use pip all the time for pure Python packages on Linux and Windows because it works very well for these. However when it comes to numpy, matplotlib, wxpython, PyQT4 et al. I wouldn't even attempt to use pip on Windows.

Most commonly I would do the standard Windows thing of going to the project website and looking for the downloads page. I've also used Christian Gohlke's index of science-related Windows binaries [1] which are supplied as .exe files. He says that "Most binaries are built from source code found on PyPI..." or in other words if it were easy to build these with pip then his index would be unnecessary.

When wheel distribution becomes common I hope that this situation will improve substantially though.

Precisely. At the moment, unless you need to compile code with external dependencies, pip install works fine (it's even fine for C code without dependencies if you have a compiler). But once the build process is even slightly complex, wininst installers are the only real answer. The fact that they don't work on virtualenvs is a pain, but there are 2 ways round this: 1. I believe that easy_install will install wininst installers. I've not tried myself. 2. You can use wheel convert to make wheels out of wininsts, and then pip install those. This is what I do, and it's really effective. I keep a local index of converted wheels to limit the download/convert overhead. I'd like to see more wheels on PyPI and sites like Christoph Gohlke's move to providing wheels, and preferably in a PyPI index style format, so people can pip install *anything*. But obviously that's the authors' (and people like Christoph's) choice. MSI is a lousy format in this context, because it's near-impossible to introspect, and hence convert to a wheel or anything similar. Paul

On Jul 13, 2013, at 9:59 AM, Brett Cannon wrote:

On Sat, Jul 13, 2013 at 11:15 AM, Paul Moore <p.f.moore@gmail.com> wrote: On 13 July 2013 16:03, Donald Stufft <donald@stufft.io> wrote:

...

1. Install to user-packages by default.

Do people really want this? I hadn't seen it (other than if pip was installed to user by default). I think it's a bad idea to switch this on people. I doubt the user-packages is going to be in people's default PATH so they'll easily get into cases where things are installed but they don't know where it was installed too.

I believe Nick wants to make user-packages the default. I know at least some of the pip maintainers (yourself included) have reservations. Personally, I've never used user-packages, so I don't know what issues might arise. But I hope to try it out sometime when I get the chance, just to get some specific information.

I would assume the executable script was installed next to the python binary but the library parts went into user-packages. That way -m would work for all binaries of the same version.

...

2. Not depend on setuptools (??? - Nick's "inversion" idea)

I wanted to do this anyways. It will still "depend" on it, but it will just bundle setuptools itself like its other dependencies. For pip dependencies are an implementation detail not an actual thing it can/should have.

Bundling is not the same as Nick's suggestion. I personally have no problem with bundling, but pip install with a bundled setuptools might not work because the setup subprocess won't see the bundled setuptools when it imports it in setup.py. But either way, it's doable, I just want to know if it's on the critical path...

...

3. Possibly change the wrapper command name from pip to pip3 on Unix.

Not sure on this. Ideally i'd want the commands to be pipX.Y, pipX, and pip all available and not install the less specific ones if they already exist but that might be too hard?

Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue.

No, this is not how any user ever will expect unix programs to work. I know that python -m is very cute, and I use it myself for some debug and helper functionality at times, but it can never replace normal scripts. This is a user experience expectation, and we will have to meet it. --Noah

On 13 July 2013 18:51, Donald Stufft <donald@stufft.io> wrote:

On Jul 13, 2013, at 12:59 PM, Brett Cannon <brett@python.org> wrote:

Could we just start to move away from an executable script and start promoting rather aggressively -m instead? It truly solves this problem and since the results are tied to the Python executable used (i.e. where something gets installed) it disambiguates what Python binary pip is going to work with (something I have trouble with thanks to Python 2 and 3 both being installed and each with their own pip installation). I realize older Python versions can't do this (I believe 2.6 and older can't for packages) but at least in the situation we are discussing here of bundling pip it's not an issue.

Also looking at what already ships with Python.

idle, idle2, idle2.7 smtpd.py, smtpd2.py, smptd2.7.py pydoc, pydoc2, pydoc2.7 2to3, 2to3-2, 2to3-2.7

This is also the convention anywhere someone does versioned scripts in a Python package in the ecosystem. PEP439 is there to streamline the process so that python dependencies are much easier to install and there's a smaller barrier to "entry" so that projects like Django can give simple instructions for dependencies instead of needing to opt not to have dependencies or have to give instructions on how to install the installer.

None of these commands work at the command line in a base Python install on Windows. They are all obscure enough that nobody cares (or they are GUI apps that have a start menu entry provided, i.e. Idle) - that won't be the case for pip. Paul.

2) replacing pkg_resources with distlib (vinay posted a PR for this)

3) if not #1, pip installing setuptools on-demand when building is needed (this was the old plan I think for PEP439 until the recent changes, and get's us closer to the "MEB"s model)

to be clearer for everyone, #3 depends on #2, so that pip could install setuptools from wheel (without needing setuptools)

It is easy to forget that pip only needs the "package database" part of setuptools (pkg_resources.py) to install things. With the small catch that the rest of setuptools is required to install anything besides wheels. MEBS is just about implementing build requirements properly and giving pip a consistent interface to build traditional sdists *or* any new (sdist 2.0, distil, bento) kinds of packages that may come along.

On 14 July 2013 12:46, Donald Stufft <donald@stufft.io> wrote:

I'm sure I've seen people say other things that have made me think "are

...

you expecting the pip maintainers to make that change?" in the various threads, so I doubt this list is definitive.

The other big one is the one you noted about pip *not* offering a stable API, *but* exposing an apparently stable API to introspection. Introspection currently tells me that pip exports *at least* 32 public names (and this is without checking for public submodules that aren't implicitly imported by pip/__init__.py):

...

...

...

import pip; public = set(k for k, v in pip.__dict__.items() if not k.startswith('_') and (not hasattr(v, "__name__") or hasattr(v, "__module__") or v.__name__.startswith("pip."))); print(len(public)) 32

If pip really has no stable public API, then it should properly indicate this under introspection (if it already uses relative imports correctly, then the easiest ways to achieve that are to just shove everything under a "pip._impl" subpackage or shuffle it sideways into a "_pip" package).

Pip does not use relative imports. Is simply documenting the fact there is no public API enough? Pushing everything into a _impl or _pip directory makes me nervous because that's a lot of code churn (and I know there are people using those APIs, and while they aren't technically stable it feels like moving things around just for the sake of an _ in the name is unfriendly to those people.

Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is "test" packages, including the standard library's test package, and for those you can make the case that having "test" or "tests" as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore. I really this is a fairly big ask for the pip maintainers, but I *don't* consider "Oh, don't use our module API, it isn't stable" to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring). Instead of a milestone I added a PEP439 tag so that we can differentiate between 1.5 milestone items for PEP439 and not. Ideally we don't need to drop anything from 1.5 but just in case we do.

I think we should probably target pip 1.5 to release in the beginning of December? Would need to see what the other team members think, that's a shorter release cycle then we normally have but I think it'd be good to have 1.5 out for a month or so to get real world use to make sure it doesn't need a patch release before inclusion in CPython (assuming the dates you mentioned are correct).

Just to confuse matters a little bit, Richard has suggested explicitly creating a bundling PEP as a *competitor* to PEP 439, thus making it easier to be explicit about our reasons for rejecting bootstrapping in favour of bundling. I think that's a good way to move this forward, but I won't actually reject 439 until the competing bundling PEP has been posted (otherwise people might get the wrong impression that we're moving away from the idea of making "pip install X" work out of the box, when we're really just changing our tactics for achieving that goal). I also realised what is probably a better idea than "python -m getpip" for dealing with the "How do I get pip after doing a source build?": add a "get-pip.py" utility to Tools/scripts in the cpython repo, rather than adding anything to the standard library. This also puts us on a more solid footing for getting pip bundled with 2.7.x at some point: we're not touching the standard library, just the installers and the utility scripts. The bundling PEP should also suggest to Linux packagers that pip be considered an essential part of a fully functional Python installation. Exactly how that is handled will be up to the distro packagers, but could include noting pip as a recommended dependency for Python (Debian), or rearranging the packaging to make "cpython" a package in its own right, with "python" requiring both "cpython" and "python-pip" (while the latter would just require cpython). I'll post an explicit call for a PEP champion in a separate thread. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 14 July 2013 16:19, Noah Kantrowitz <noah@coderanger.net> wrote:

...

On 14 July 2013 12:46, Donald Stufft <donald@stufft.io> wrote: Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module

On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager).

...

Respectfully, I disagree. Pip is not going in to the stdlib, and as such should not be subject to the same API stability policies as the stdlib. If the PyPA team wants to break the API every release, that is their call as the subject matter experts. Pip is not being included as a library at all. What should be subject to compat is the defined command line interface, because pip is a CLI tool. Independently of this discussion I've already been talking to the PyPA team about what they want to consider a stable API, but that is a discussion to be had over in pip-land, not here and not now. This new category of "bundled for your convenience but still external" applications will need new standards, and we should be clear about them for sure, but I think this is going too far and puts undue burden on the PyPA team. Remember the end goal is simply to get an installer in the hands of users easier.

I would also be fine with a solution where "import pip" issues a warning about API instability if sys.argv[0] indicates the main executable is something other than the pip CLI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 14, 2013, at 1:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Either the existing APIs are moved to a different name, or they get declared stable and pip switches to "internally forked" APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager).

Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is "test" packages, including the standard library's test package, and for those you can make the case that having "test" or "tests" as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore.

I really this is a fairly big ask for the pip maintainers, but I *don't* consider "Oh, don't use our module API, it isn't stable" to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring).

I agree with Noah here. In my eyes pip is either an external project with it's own polices on backwards compatibility and governance or it's part of the standard library and under the domain of Python core. I'm completely against moving it into the standard library for all the reasons I've given in the past. Maybe I'm reading too much into this but one of my primary fears here is that including pip with the Python distribution is going to lead to Python core dictating to pip what pip must do with itself. Now I don't mean to say that Python core should have no sway over pip either as they are officially blessing it and including it as part of the official releases. If I am reading too much into it then I apologize. I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jul 14, 2013, at 3:01 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 14 July 2013 16:43, Donald Stufft <donald@stufft.io> wrote: I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact.

Agreed, I think the boundaries need to be clear. If something installed by default is *only* support code for a bundled application, then it should either adhere to the standard library's backwards compatibility policies (by appropriately marking private APIs as private), or else it should issue a warning when imported by any other application. Either of those options sounds good to me.

However, I consider expecting people to "just know" (or to look at documentation to determine) which provided modules are public or private without adhering to standard naming conventions or providing an explicit runtime warning to be unreasonable.

(and yes, if "pip" goes down the runtime warning path, we should probably look into providing a runtime warning for at least the "test" namespace and possibly even the "idlelib" namespace, too)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yea I forget to talk about the *actual* change that prompted that email when I started feeling dictated to which touched upon one of my fears in this process :) I'm not against either renaming or emitting a warning. I was actually asking if just documenting the fact would be ok because I fear bugs from the code churn that renaming would cause :) I think we'd need to rename things because emitting a warning is an all or nothing ordeal and we've had requests to make certain parts of the API public for Chef and other tools like it. A question that certainly raises in my mind though is "standard library's backwards compatibility policies". What affect does this have on *actual* public API exposed from pip? Does it mean we cannot break compatibility for them until Python 4.x? That sounds very onerous for something that is installed in a way that allows easy upgrade and downgrading separately from Python to match the version requirements of someone using that library. Pip has it's own versions and develops at it's own speed. I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

I do think pip is pretty conservative about backwards compat other than the security related changes I've been doing. I think we can find the middle ground that lets things work smoothly here :). I was just making sure that we wernt going to have to keep things around for really long times like python 4 ;) On Jul 14, 2013, at 3:35 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 14 July 2013 17:13, Donald Stufft <donald@stufft.io> wrote:

...

I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize).

The main elements of CPython's backwards compatibility policy that I consider relevant are:

* Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change

We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely.

That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy:

* if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release

This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default.

On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode).

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote:

On 14 July 2013 17:13, Donald Stufft <donald@stufft.io> wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize).

The main elements of CPython's backwards compatibility policy that I consider relevant are:

* Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change

We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely.

That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy:

* if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release

This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default.

On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode).

If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will support that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is "reasonable" for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer. --Noah

On Sun, Jul 14, 2013 at 4:23 AM, Noah Kantrowitz <noah@coderanger.net>wrote:

On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote:

...

On 14 July 2013 17:13, Donald Stufft <donald@stufft.io> wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's "None") using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize).

The main elements of CPython's backwards compatibility policy that I consider relevant are:

* Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change

We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely.

That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy:

* if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release

This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default.

On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode).

If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against?

Yes, and to make that statement even stronger: all of that happening with a freshly installed copy of Python with no external packages installed. Nick's worry stems from experience (which I have also had) where people simply don't check docs as to whether something is public in the stdlib and so it ends up being considered such by users to the point that we feel obliged to support it.

I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve.

That's what we typically say for older modules, especially when we are cranky. =) But it doesn't stop the wingeing and bug reports. Luckily I think we have gotten better about this.

The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will suppor t that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is "reasonable" for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer.

I think it's all going to come down to messaging. It will have to be yelled form the top of every mountain that pip is being bundled with Python as a convenience to the community, but that it is **NOT** part of the (C)Python project and thus has it's own development process, issue tracker, etc. If Python bundles pip then python-dev will make promises about what versions we include in bugfix releases, how it's bundled, etc., but otherwise it's a separate project with its own rules.

On 17 July 2013 00:56, Donald Stufft <donald@stufft.io> wrote:

On Jul 16, 2013, at 1:36 PM, Matthias Klose <doko@ubuntu.com> wrote:

5. Support cross-compilation of extensions by default.

TBH I don't know how much of this has anything to do with pip? As far as compiling goes all pip does is call setup.py install so people are compiling with either setuptools or distutils.

I'm not involved in the current packaging work so someone else can correct me if I'm wrong but: What I'm really looking forward to as a result of all of this work on wheels and packaging metadata is that there will no longer be any need for people to use either setuptools or distutils to compile anything. Once this work is ready it will be possible for developers to build wheels using whatever tools they like and then upload them to pypi. pip will be able to select the appropriate wheel for the end user's OS, CPU, Python version etc. and install the binary wheel without needing any compiler support on the target system. What this means is that it will be possible to do all the compilation on developers machines so that the tools to do so don't need to be in the stdlib any more. Then someone will be able to release the "fancycompiler" package on pypi that will support features like cross-compilation, other developers can use it, and the end user doesn't need to care how the wheels were made. Oscar

On 17 July 2013 00:56, Donald Stufft <donald@stufft.io> wrote:

On Jul 16, 2013, at 1:36 PM, Matthias Klose <doko@ubuntu.com> wrote:

5. Support cross-compilation of extensions by default.

TBH I don't know how much of this has anything to do with pip? As far as compiling goes all pip does is call setup.py install so people are compiling with either setuptools or distutils.

Nothing at all. Cross-compilation is not handled by pip, nor is it relevant to the discussions about bundling pip with Python. The much longer-term goal of the packaging discussions around decoupling "builders" and "installers" might allow for the development of better 3rd party build tools with cross-compilation support (things in the space that tools like bento occupy) but it will never be something that pip needs to concern itself with. I doubt that the standard library (i.e. distutils or any successor) will be involved in this either - it's specialised enough that 3rd party tools are the correct way to handle it. Paul