On 15 July 2013 14:44, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

Can you not just rename the pip module to _pip?

That's a far more intrusive change, and I'm not sure I want to go there. It *will* break existing code that (rightly or wrongly) imports pip. Paul

On 15 July 2013 14:53, Brett Cannon <brett@python.org> wrote:

As long as you make sure that sys.path[0] is actually the script location then it will work (other things like .pth files, PYTHONSTARTUP, etc. could have changed things before your script started execution). But realize that a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and b) if I get my way all paths will be absolute for __file__, so you will have to just associate '' with os.getcwd() and then search for the proper directory on sys.path.

OK, that pretty much tells me that this is a bad idea. It's never going to be robust enough to work. I'm amazed actually that there's no way to say "don't add the script location to sys.path", even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E. Paul

On 15 July 2013 15:32, Nick Coghlan <ncoghlan@gmail.com> wrote:

Most of the stuff Brett mentioned there shouldn't be relevant for a directly executed script - doing sys.path.remove(os.path.dirname(os.path.abspath(__file__)) should be pretty robust in any scenario.

OK, well apart from the shadowing issue, my initial tests have looked relatively positive. So the questions are now more around whether this is how we want to go with pip, what backward compatibility issues it may have (the launcher isn't available in Python < 3.3) etc. So I'll work up a pull request for discussion by the pip devs.

...

I'm amazed actually that there's no way to say "don't add the script location to sys.path", even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E.

You'd think that, but then you'd look at getpath.c and run away (or write something like PEP 432, as I did) :P

You're a better man than I, Gunga Din :-) Paul

On 15 July 2013 18:26, Donald Stufft <donald@stufft.io> wrote:

Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues.

That's actually a very good idea. The only downside is the proliferation of extensions involved, and the need to register them. That puts it into the territory of things the installer needs to do if we're to be able to assume it. But I may propose it to python-dev (Daniel proposed a "zipped Python app" extension a while back, as well. I'm not sure what happened with that one...) Actually, this and many of the other ideas fall foul of backward compatibility issues - we can't assume the Python launcher is available prior to Python 3.3, so #! support in .py files isn't available either. I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy. 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. 3. Special case the heck out of pip upgrading itself to ignore errors from trying to replace the exe (as long as the exe is unchanged, based on a size/checksum check) This covers the replacing-the-exe issue and the entry point script problems of vendoring setuptools. I don't *like* this option, but at least it's not going to break big chunks of our userbase... There are some other options I'd still like to explore before settling on something, for example making pip install from sdist *always* build a temporary wheel and then always install from wheels - we can then introspect the wheel before installing and catch this type of issue before starting. That lets us easily avoid the "overwriting the exe" issue, as well as letting us cleanly roll back failed installs. One thing is clear - this is a longer term effort, not a quick fix... Paul

On 15 July 2013 23:21, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Paul Moore <p.f.moore <at> gmail.com> writes:

...

I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy.

Wouldn't that be the case with a bundled setuptools/pip anyway? The launcher executables are part of setuptools AFAIK.

...

2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources.

FYI the distlib executable launchers (not based on setuptools, but work the same way) and the distlib script generation approach does not require distlib to run. You might be able to make use of that in some way :-)

Yes, the physical executables aren't that much of an issue - grabbing them from somewhere and bundling them is easy enough and as you say we may already have them (I actually thought distlib included the exes, but the version bundled with pip at the moment doesn't have them). The fun bit is having to modify setuptools to do our own script wrapper management, because setuptools doesn't let us customise its process to remove the runtime dependency on a top-level pkg_resources. Having to make project-specific customisations to distutils feels like we're going in precisely the opposite direction from what the whole packaging process is trying to achieve, and it makes me feel vaguely unclean having to consider it :-) Two thoughts for the wider audience: 1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Note: This is just one relatively small step towards removing some of our dependencies on an external setuptools. It's not the whole story, and it still leaves the "upgrading a running exe wrapper" problem to address. (This discussion may be worth migrating to pypa-dev, I'm not sure how much the wider distutils audience cares about this level of detail - feel free to switch lists anyone who thinks it's appropriate). Paul

On 16 July 2013 12:04, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

On 16 July 2013 11:28, Paul Moore <p.f.moore@gmail.com> wrote:

...

Two thoughts for the wider audience:

1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI.

In what way would that affect anyone?

Sorry, you're right - that's really for the pip developers.

...

2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers?

Does that mean that an end user would need a C compiler in a situation where they previously didn't?

I don't believe so - distlib bundles the compiled code. On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... Paul

On 16 July 2013 12:42, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this).

Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. Grumble. Next time the label on the can says "worms" I need to leave the can opener alone :-) Paul

On Tue, Jul 16, 2013 at 8:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 16 July 2013 12:42, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this).

Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating.

That's not why they exist; the .exe's don't load the Python DLL, they just invoke python.exe. The existence of separate 32- and 64-bit .exe's is a Distribute innovation, actually; setuptools 0.6 doesn't use them. Instead, setuptools writes a manifest file to tell Windows that it doesn't need privilege escalation or to create a separate console. This meant that only one (32-bit) .exe was needed. I forget what happened with the Distribute approach or why the 64-bit version was kept after the merge; ISTM there was some other use for it, or at least Jason thought so. But DLL loading is not the reason. (Actually, after searching my email, my guess is that there actually *isn't* any need for the 64-bit .exe; ISTM that it was introduced solely as a false fix for the privilege escalation problem, that only fixes it for 64-bit Windows and doesn't help 32-bit.)

Am 15.07.2013 19:26, schrieb Donald Stufft:

Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues.

pip.bat? Thomas

Am 17.07.2013 15:33, schrieb Paul Moore:

On 17 July 2013 14:13, Thomas Heller <theller@ctypes.org <mailto:theller@ctypes.org>> wrote:

Am 15.07.2013 19:26, schrieb Donald Stufft:

Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues.

pip.bat?

That's my cue to cry :-)

If you missed my earlier comments about bat files, then no - bat files have a significant number of failings that make them unsuitable for this sort of thing.

Yes, missed them. Sorry that I did not read enough of the discussion, I just stumbled over some messages in this thread and .bat popped up in my head. I still use them for quite some things although I have been bitten by their problems myself often enough.

I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results.

Thomas

I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results.

No reason to be afraid of this, exe wrappers are totally the best option. As for updating .exes while they're running, the best approach is to rename the running one (e.g. 'pip.exe' -> 'pip.exe.deleteme') in the same folder and either: * delete any existing .deleteme files on next run, or * delete an existing pip.exe.deleteme file immediately before trying to rename to it Any other approach will also have corner cases, but this will be the most reliable in the context of multiple users/permissions/environment variables.

Paul

Steve

On Jul 17, 2013, at 3:12 PM, Paul Moore <p.f.moore@gmail.com> wrote:

The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. And it *doesn't* need to be handled specially if it's "python -m pip install -U pip". That's not a Windows issue, though, I was just using the Windows issue to put off having to think about it :-)

OTOH it doesn't *hurt* to do it so if a conditional is hard to sort out (I'm thinking explicitly about pip vs python -mpip) we can just genericize it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. And it *doesn't* need to be handled specially if it's "python -m pip install -U pip". That's not a Windows issue, though, I was just using the Windows issue to put off having to think about it :-)

Not especially. You wouldn't want to do it for all files, but you can do the rename for all .exe files and then try and delete the .deleteme immediately. (Or you can try and replace and gracefully handle the exception.) As long as failing to delete the old .exe doesn't cause installation to fail, 99% of the time the end result is identical to now (and the other 1% is better, because install succeeded where it would currently fail).

One thought, while I have a Windows expert's attention ;-) Is there a way (I'm not too bothered how complex it might be) of doing the equivalent of Unix exec in Windows? That is, start a new process and then have my initial process exit *without* the shell (or whatever started the first process) getting control back until the child completes? I'm assuming not, as otherwise solving the issue would be as easy as exec-ing "python -m pip" from the wrapper. But no harm in asking :-)

I had this discussion with a colleague earlier today, and there really isn't. You'd need to be cooperating with the process that started you initially (cmd.exe/powershell.exe/whatever) and if there is a way to do it, (a) I don't know what it is, and (b) it's unlikely to be consistent/reliable/documented. At best, you could start a new process at the end of installation that keeps trying to delete the .deleteme file until it succeeds (or waits for a specific process to exit). However, my gut says that it's safer just to leave the file around and try and delete it later, especially since pip is going to be used in such a wide variety of contexts. Being clever is more likely to get you into trouble. Steve