On Sun, Aug 11, 2013 at 12:17 PM, PJ Eby <pje@telecommunity.com> wrote:

May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable.

(Details I forgot to mention: the script would be in __main__.py inside the zipped application file, and it would need to change sys.path[0], because sys.path[0] will be the .pyz file itself; it should replace it with the directory containing the .pyz file before doing anything else. That would be the correct way to simulate the existing .exe approach.)

On Sun, Aug 11, 2013 at 1:58 PM, Jason R. Coombs <jaraco@jaraco.com> wrote:

This sounds like a suitable idea, but as you mention in a subsequent message, this format has issues with sys.path assumptions as well.

Meh. It's basically, a one-line fix in the __main__.py, i.e.: import sys,os.path; sys.path[0] = os.path.dirname(sys.path[0]) Not exactly rocket science. ;-)

In this case, I'm inclined to suggest yet another option (7) - create another extension to specifically represent executable but not importable scripts, perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and gui script).

Probably .pys/.pyws or .pws would be needed, due to issues with some Windows shells using extensions longer than three characters. (This came up in PEP 441 discussions on Python-Dev.)

It sounds as if there is a fundamental need for Python to define an extension that distinguishes a script from a module.

Yep.

I am aware of the PATHEXT factor. I personally add .py and .pyw to the PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add those or if pylauncher would add them (if installed separately). I was _hoping_ that was the case, but it sounds like it is not. I did include in the documentation notes about this requirement (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 ).

I do want to explore the possibility of setuptools facilitating this configuration such that it's easy for a Windows user to enable these settings even if Python does not.

It would definitely make sense to have an installer that sets this up, but it would need to be a Windows installer, I think, not a Python program. That is, I don't think setuptools can really do anything about it directly. Personally, I'm not sure I see the point of pushing for early elimination of .exe's - they don't depend on the registry or environment variables or anything else, which makes them great for standalone applications, and they work across all Python versions. Meanwhile, the experimental nature of your change -- and its inability to be the default on versions below 3.4 -- means you're going to be maintaining two sets of code for a very long time. OTOH, implementing a way to deploy an app as a .pyz/.pwz file is a useful feature in its own right, so it might not be doubling up as much. ;-)

-----Original Message----- From: Nick Coghlan [mailto:ncoghlan@gmail.com] Sent: Sunday, 11 August, 2013 17:14

We actually have a proposal on import-sig to allow module specific import path manipulation (including the ability to say "don't import this module from this directory, even though it looks like it is here"). I'd favour that mechanism over a new "not importable" file extension.

I don't believe this mechanism would suffice. My previous example was over-simplified to the general problem, which is that any script could potentially be imported as a module of the same name. So if I were to launch easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to import cython (which it does), it would import Scripts/cython.py as cython, unless there were some way to globally declare all installed scripts somewhere so they're excluded from import.

If that doesn't make it into 3.4, the proposed zipapp extensions would also serve a similar purpose, with some straightforward sys.path manipulation in __main__.py (as PJE pointed out).

Regardless what solution might be made available for Python 3.4, I'd prefer to work toward a solution that leverages pylauncher under older Pythons. After all, one of the huge benefits of pylauncher is that it supports multiple Pythons. If zipapp is the preferred mechanism for that, then so be it. I do agree that we should devise a best approach within the context of Python 3.4, and consider the backward-compatibility implications separately. My feeling is that zipapp is somewhat too convoluted in that it alters the sys.path, but then has to alter it back to simulate not being a zipapp. It's also a file within a file, meaning it can't be readily edited with a text editor, but requires a routine even just to inspect it. I guess that's more transparent than an executable, though. This approach also means that the script generation is not congruent with that on Unix systems. Using a zipapp means that the whole script generation needs to be special-cased for Windows. One of great benefits of using a simple script was that the code becomes largely unified (only requiring appending of an extension when on Windows). That is, unless zipapps can be made executable on Unix as well. Given these obstacles, do you still feel that zipapp is the best approach?

On 11 Aug 2013 21:37, "PJ Eby" <pje@telecommunity.com> wrote:

On Sun, Aug 11, 2013 at 7:31 PM, Jason R. Coombs <jaraco@jaraco.com>

...

...

-----Original Message----- From: Nick Coghlan [mailto:ncoghlan@gmail.com] Sent: Sunday, 11 August, 2013 17:14

We actually have a proposal on import-sig to allow module specific import path manipulation (including the ability to say "don't import this module from this directory, even though it looks like it is here"). I'd favour that mechanism over a new "not importable" file extension.

I don't believe this mechanism would suffice. My previous example was over-simplified to the general problem, which is that any script could potentially be imported as a module of the same name. So if I were to launch easy_install.py, it would set sys.path[0] to Scripts\ and if it were

wrote: then to

...

import cython (which it does), it would import Scripts/cython.py as cython, unless there were some way to globally declare all installed scripts somewhere so they're excluded from import.

Indeed. It really *does* need to be a "don't import this" extension, though it doesn't much matter what that extension is. Except on Windows, of course, where it has to be something associated with Python that also still works as a console app and is listed in PATHEXT.

(As you surmised earlier, my choice of '-script.py' was indeed chosen to prevent accidental importing, as the '-' ensures it's not a valid module name.)

Having an empty "cython.ref" file (extension TBC) would tell Python to skip that directory for "import cython" regardless of the presence of something that otherwise would be considered. The actual downside is that won't land until 3.4 at the earliest, while PyLauncher can associate additional extensions with itself (and modify PATHEXT) earlier than that. Having pys and pyz for "executable, but not importable" (source and zip archive forms) could be quite clean. In effect, the pys extension would bring windows to parity with *nix, where "no extension at all" has traditionally served the purpose of making it impossible to import a script. Cheers, Nick.

On 12 August 2013 14:01, PJ Eby <pje@telecommunity.com> wrote:

This means that you can actually write source as a .pyz or .pwz file on Windows, and it would Just Work -- *without any sys.path modification*.

Conversely, you can right now rename a zipped file as xxx.py and it will be run happily as a Python standalone zip file. It confuses the heck out of things like text editors and zip file managers, of course :-) We currently have 2 extensions on Windows - .py and .pyw. These get a default run action of the launcher (py.exe and pyw.exe versions respectively) and both are treated as marking Python modules (I don't know why anyone would create a Python module with a .pyw extension - sounds like an unnecessary attempt to apply "consistency" to me). Having two more that had the launcher as default action but did *not* mark Python modules might make sense. Having further extensions is fundamentally a documentation-only exercise - they will not be treated any differently by any code shipped by Python. The problem is that documentation (and user expectation) is important - nobody expects a .py file to be a binary zip file (and they'd get a shock if they opened it in a text editor). On the other hand, grabbing a huge host of file extensions (.py, .pyw, .pyo, .pyc, .pyz, .pwz, .pys, .pws) is not very friendly, as well as adding the burden of clearly documenting what all these various extensions *mean*. My view would be: 1. We can't touch .py/.pyw behaviour for backward compatibility reasons 2. A new suffix that is associated with the launcher but which does *not* mark importable modules would be good. (Call it .pys for now). 3. I'd rather not see further extensions added, six is plenty. As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Apologies if I've missed an obvious flaw - I'm way behind on sleep at the moment... Paul. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further.

On 12 August 2013 11:21, PJ Eby <pje@telecommunity.com> wrote:

On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

On 12 August 2013 14:01, PJ Eby <pje@telecommunity.com> wrote:

As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content.

Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue.

I think the issue of "too many extensions" vs. "source/binary confusion" is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP.

If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, "hey, run this as a console app via PyLauncher", not that it's a "script" (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile.

I like this idea.

...

PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further.

But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT.

Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it.

(The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.)

This sounds like a plausible approach, especially if we add the bootstrapping being considered for 3.4+ to PyLauncher for earlier versions. (Donald has a draft PEP for that, he's just making a few tweaks before publishing it for broader comment) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

PJ Eby <pje <at> telecommunity.com> writes:

If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, "hey, run this as a console app via PyLauncher", not that it's a "script" (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file

I know I'm bike-shedding here, but my preference for extension would be '.pye' as it indicates something to execute, but without indicating exactly how (i.e. that it's via a separate launcher executable). The standalone launcher already adds bindings for zip extensions but does not change PATHEXT. Any changes in this area would not be in the launcher itself, but in the standalone launcher installer (and hence would need to be replicated in the Python installer). Regards, Vinay Sajip

On 12 August 2013 19:35, Steve Dower <Steve.Dower@microsoft.com> wrote:

And any reason we need a separate extension for pyw.exe? Can't that be specified in the shebang?

The association specifies the exe to run, that exe then relaunches whatever the shebang specifies. But it's the *initial* executable's console/GUI flag that specifies the behaviour, so we need two launchers and hence two extensions. You could put pythonw in the shebang of a .py file, but that would open a console window when double clicked. Similarly, putting python in the shebang of a .pyw file will detach the script from the current console when run from the command line. Paul.

.pyx

that said, both .pye and .pyx are superior to .pyl  (is that last character an "ell", an "eye" or a "one"?)

I agree that .pyl is less readable/more confusable. We can't use .pyx, though, as that is already used in the Python ecosystem for Pyrex files (a forerunner of Cython). Regards, Vinay Sajip

Jason R. Coombs wrote:

My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable: [SNIP] 2. Executables that look like installers. If a launcher executable is used and Windows detects that it "looks like" an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window).

The problematic part of detection is the filename. The "Installer Detection" section of http://msdn.microsoft.com/en-us/library/bb530410.aspx has more details, but isn't 100% precise. I'm sure there's a precise description somewhere, but I don't know where it is.

3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable).

32-bit is sufficient, and the manifest can be embedded. Since all the executable is doing is looking for a matching/similarly-named file in its directory and launching it, there's no need for a 64-bit version, or for any differences to exist between the executables. They could all be copied from the same source whenever one is needed. (A 64-bit binary is only required when loading 64-bit DLLs. It is not required to launch a 64-bit process.)

[SNIP]

6. Two to three files to do the job of one. In fact, the "job" isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script.

While I can understand this from the POV of the implementer/maintainer, I've never heard a single Windows user mention it. And with an embedded manifest, it's no more than one .py and one .exe-per-Python-version.

[SNIP] 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful.

Do us Windows users get .sh and .DSStore files filtered out too? :) (More seriously, why isn't the onus on package developers to specify platform-specific files?)

...

Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it.

(The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.)

I'm warming up to this idea a bit, especially how it supports the most elegant approach but degrades gracefully.

py.exe is a great file to use as a launcher, but it would also be easy to make a more specific one.

Some questions that arise: Where would Setuptools expect to find these launchers? Would it expect them to be present on the system?

I don't think that's unreasonable. Perhaps they should always install into Python's path (C:\Python##\py.exe) so they are discoverable? Not everybody can install into C:\Windows\System32 (or SysWOW64)

Would it symlink or hardlink them or simply copy?

Copying is most reliable and the only way to handle installation onto different drive partitions. Some people will worry about the size, but I don't know that much can be done about that. A smaller executable could be made with no icons and relying on the OS to find "py.exe" wherever it's been installed. You can hardlink the version-specific executables safely.

Is py.exe subject to the 'looks like installer' behavior, such that it would need a manifest?

The one that's in Python 3.4 Alpha already has an embedded manifest, but I don't believe it would trigger the installer heuristics anyway.

I still feel like this approach would require substantial special-casing, but since it provides a transition to the simple, elegant approach, I'm not opposed.

Shouldn't require that much special casing, except that you probably wouldn't be able to use it with an adjacent "py.py" file (that is, you'd special case "py.exe" to have the default behaviour). Expecting the adjacent file to be "name-script.py" or "name.pyl" for "name.exe" seems reasonable to me, and neither of those modules will be importable. Cheers, Steve

On Tue, Aug 13, 2013 at 8:54 AM, Jason R. Coombs <jaraco@jaraco.com> wrote:

1. Renames, deletes, and other actions must be synchronized.

Why are you manually deleting or altering executables? Why are you renaming them at all? I've been using .exe wrappers since they were written, and have never had a single one of the issues you mention, because I never do any of the things you mention by hand. IMO that's what tools are for. Doesn't pip uninstall scripts? I may be slightly biased in my preference for .exe, because files with other extensions don't work with Cygwin (which doesn't support PATHEXT), but I work primarily with Windows Python rather than Cygwin Python. So, if there *has* to be a single file, I would greatly prefer an .exe with the script embedded, rather than a non-.exe file. It's a bit less discoverable, but at least it'll discourage anybody from editing the contents. (Because nobody should be editing generated scripts anyway.) (Also relevant: not every situation where wrapper scripts are used is going to be one where a PyLauncher install is possible. For example, portable deployment of an app to USB stick with a bundled Python can't assume PATHEXT and a globally-installed PyLauncher.)

4. Updates to the launcher won't apply to existing scripts. If the launcher is updated, the side-by-side versions will remain out-of-date until their scripts are re-installed.

This is kind of a bogus point; *any* update to how scripts are generated isn't automatically applied to existing scripts; the format in which they're written is of no relevance.

5. Files in use can't be replaced. Because a Windows executable that's in use is not allowed to be overwritten,

But they can be renamed, and deleted afterwards. For example, when updating, you can do the simple dance of: 1. delete scriptname.exe.deleteme if it exists 2. rename scriptname.exe to scriptname.exe.deleteme 3. replace scriptname.exe 4. try to delete the .deleteme file created in step 2, ignoring errors. And since this only needs to be done for the wrappers on installation tools themselves (pip, easy_install, etc.), it's not like a lot of people are going to have to write this code. It can also be further enhanced, by having the .exe wrapper check (as it exits) whether it was renamed, and if so, spin off a 'python -c "import os, time; time.sleep(0.1); os.unlink('path to .deleteme')"' and immediately exit. (Or use one of the other tricks from http://www.catch22.net/tuts/self-deleting-executables -- but I think this one is the simplest and best for our purposes, since the wrapper already knows at this point it can invoke Python using the path it previously found, and it's not doing anything questionable with process invocations that might raise red flags with security tools.)

On 13 August 2013 17:33, Paul Moore <p.f.moore@gmail.com> wrote:

On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way.

So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Oscar

Just $0.02 from a user... I'm primarily an OS-X user these days, but have to do Windows once in a while, and help others do Windows (including as an intro to Python instructor) Once I discovered setuptools "develop" mode, I never looked bak -- it is simpl;y THE way to develop code, particularly if you are working on a lib you want to use from other projects and/or actively develping utility scripts. I really like that I can add some scripts to setup.py, and use either develop mode or regular old install and get nice commands, and it works the same way on Windows and *nix (with the hood closed0 I did all this for a good while before I even noticed that exe launchers -- it "just worked". In fact, the only time I noticed the launchers was a couple years ago when a beta version of setuptools released a broken version -- very frustration -- it would fire up another command window that would then close when the script was done -- not very helpful for nosetests and the like... When I did discover how it all worked, I did think it was a little weird, but Windows simply hasn't been built for command line stuff, so you do what you have to do. Conclusions: 1) an extra bunch of files is a on-issue for most users -- we just need something that works. 2) the exe launcher is a bit fragile and hard to maintain (and even harder to debug) -- but there are smart people working on this. 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) I cant help thinking a more elegant solution exists, but maybe not. Thanks to everyone hashing this out! -Chris On Tue, Aug 13, 2013 at 12:58 PM, Paul Moore <p.f.moore@gmail.com> wrote:

On 13 August 2013 18:08, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

On 13 August 2013 17:33, Paul Moore <p.f.moore@gmail.com> wrote:

...

On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way.

So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files.

Ah, OK, thanks for the clarification.

In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash shell wrappers, or shell aliases...). Maybe an option to still use exe wrappers is worth it - but honestly, I'd say that in that context you probably have enough expertise to understand the issue and make your own solution relatively easily.

What about having in your .bashrc:

for prog in ls ~/bin/*.py; do alias $(basename $prog .py)=$prog done

(Excuse me if I got the precise details wrong there). OK, you need to rerun .bashrc if you add new scripts. It's not perfect. But it's not a showstopper either.

I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed.

Paul

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Aug 13, 2013 at 2:27 PM, Paul Moore <p.f.moore@gmail.com> wrote:

...

3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin)

Nobody is suggesting that end users mess with PATHEXT. The proposal is that the Python installer does this

ouch! I don't think I'd want PATHEXT set for *.py files -- I'd rather they get opened by an editor by default than run...or is point+click behavior different than command line -- shows you how well I know Windows.

(indeed, that's been done for Python 3.4, I don't know if the installer for the standalone launcher has been updated in the same way yet).

There are those of us still in the 2.7 world -- and I suspect for a good while.

I'm suggesting that we collect specifics on any "fragility" (can you provide details of what has gone wrong for you?)

well, for PATHEXT, the env variable has to be set right, and Windows kind of hides all that from you -- it's really a pain to edit them by hand, so if the installer doesn't do it, or someone re-builds their registry or profile, or what have you, then it'll break. Oh and the cygwin (and who know what other shell alternatives) issue. At least we can pretty much count on an exe running if the shell can find it... Fragiity for the exe approach -- all I know is that the setuptools binary was proken a while back -- but setuptools itself was in a bit of a void of not-quite-sure-if-its maintained, and not-sure-even-how-to-report-a-bug state. It seems that setuptools (or whatever this will be part of) has now been adopted by the core Python community, so we can expect good support -- let's hope so. Anyway, whether the exe approach was more or less fragile than anyting else, is sure was harder to debug/fix for anyone that doesn't know Windows and non-python development well.

...

I cant help thinking a more elegant solution exists, but maybe not.

Personally, I believe that executable .py scripts (or maybe a dedicated .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but equally, I concede, "maybe not"...

well, I guess that's the way Windows does things -- *nix has the executable bit, Windows uses extensions, so I suppose that is "the way" to get an executable script -- but it really SHOULDN'T be *.py!

I think it's worth trying, though.

agreed. I've lost track of what needs to be tried -- can't we just associate an extension with py and give it a go? (got to get that Windos VM working....)

Also, you mention develop mode. I don't use develop mode, most of my standalone scripts are single-file scripts, not anything that I'd bother packaging up with a setup.py, etc.

Good point -- it seem most of my scripts are part of a larger package, either a set of scripts, or a couple scripts that all rely on a particular package, so the whole setup.py thing works well. but for a single stand-alone script, the PATHEXT and py launcher approach seems really natural.

And many of the issues I've had with the exe wrappers are particular to built installers (wheels, wininsts) and *not* develop mode.

in theory, wheels and develop mode should do the same thing -- not sure about wininsts -- previously, the launcher thing was setuptools, not plain distutils, not sure how that was handled. but it would be best if they all did it the same way.

PS I still think that long-term a policy PEP on the recommended way of making "executable scripts" is worthwhile.

Probably, yes: "There should be one-- and preferably only one --obvious way to do it." I'm leaning toward the PATHEXT approach -- it seems more si milar to the *nix way, and perhaps easeir for lay folks to debug and fix. But I'm saying tha tthe exe method worked fine for many of us, too. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Paul Moore Sent: Tuesday, 13 August, 2013 17:28 In the interests of getting more concrete data, can I suggest that setuptools add an off-by-default option, which can be set globally using a config file or environment variable or something (so that it can be used transparently by tools like pip) to use scripts rather than exe wrappers? People can then try it and see whether they hit issues with it. Otherwise, I think we're going to remain forever stuck with theorising and guesswork. That's exactly what setuptools 1.0 does, but I want to get the first draft right, so I'm seeking comment on the technique (using env var to enable), the proper extension to use (.py has problems), and any other suggestions. I'm glad you mentioned the issues on powershell with >4 characters. I'm going to explore that issue to confirm that something like .pygs isn't viable (I use powershell and PATHEXT quite a bit).

On 14 August 2013 12:42, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

...

I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed.

There have been so many emails on this list that I can't immediately find it but somewhere Steve Dower of Microsoft said something like: ''' Don't worry. Exe wrappers are *definitely* the best solution for this. ''' (quote is approximate and emphasis added by me).

My experience has lead me to the same conclusion as Steve. It may be worth documenting the reasons why but make no mistake about it: .exe wrappers of some form are the way to go.

I would like it documented, with all the reasons, so that we don't keep rehashing this whole thing over and over. Also, if we do recommend exes, I'd like to see something (in the stdlib ultimately) that makes it trivially easy to "wrap" an existing .py script (a standalone one, not one with an associated setup.py) into an exe. Having two ways to write a command using Python is icky. Also, I'd like a single-file solution. More times than I care to remember I have put a program on a USB key or in my dropbox/skydrive and forgotten the "support stuff" (whether that's required DLLs, resource files, or whatever - it's not a Python-specific problem). But I do see your point regarding things like subprocess. It's a shame, but anything other than exes do seem to be second class citizens on Windows. BTW, you mention bat files - it bugs me endlessly that bat files seem to have a more privileged status than "other" script formats whether that's .py or .ps1 or whatever. I've never managed to 100% convince myself that they are special in a way that you can't replicate with suitable settings (PATHEXT, etc, etc). I think it's that .bat is hard-coded in the OS search algorithm or something, though. The docs are not easy to locate on the various aspects of matter. (If bat files didn't have their horrible nesting and ctrl-C handling behaviours, they'd be a viable solution...) Paul.

On 14 August 2013 16:49, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

To give an example of where these subprocess issues might matter. sphinx auto-generates Makefiles that call 'sphinx-build' with no extension. The sphinx-build command has a setuptools .exe wrapper so that it will be picked up. I wouldn't confidently assume that for all combinations of Windows version and 'make' implementation that 'make' would know how to find sphinx-build for anything other than an .exe.

OK, that's a pretty solid use case, and pretty clearly demonstrates that there will be issues with anything other than an exe. So we come full circle again - I'm pretty sure the last time this came up a month or so ago, someone came up with a scenario that convinced me to give up on executable script files. I definitely will at some point write up *some* sort of document on best practices for wrapping Python code (scripts, apps, whatever) as OS commands, in a cross-platform way. Paul

On 12 August 2013 19:14, Jason R. Coombs <jaraco@jaraco.com> wrote:

My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable:

Using a dedicated filetype and associated systemwide launcher is the ideal solution in many ways, but the behaviour is subtle, shell-dependent, and under-documented. I have been adding .PY to PATHEXT for ages now, and running .py scripts as commands with no issues - but people have raised objections to the proposal that we do this for exe wrappers, and I don't have convincing answers in all cases (mostly because "I've never seen that case" is often my response...). I can't recall details, but look back over recent threads on distutils-sig and python-dev that I've been involved in, for details. Also, a dedicated filetype won't be registered for users of older Pythons - unless they install the latest launcher. Whether that is an issue to be concerned about needs to be agreed. (Backward compatibility vs making progress, I guess...)

If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point.

IIRC, the shebang can be omitted, and in that case you get the launcher's configured default Python (from either py.ini, or built in). I tend to add /usr/bin/python{2,3} (or /usr/bin/env python if I want to respect PATH) for Unix compatibility even though I never expect the scripts to be used on Unix. Also, in my mind, this approach is most directly addressing the fundamental

challenge (distinguishing a (executable) script from a module) in much the way Unix has previously enjoyed.

Agreed - and I think that's a more important point than the merely technical issue of whether it manipulates sys.path the way we want.

I'm warming up to this idea a bit, especially how it supports the most elegant approach but degrades gracefully. Some questions that arise:

Where would Setuptools expect to find these launchers? Would it expect them to be present on the system? Would it symlink or hardlink them or simply copy? Is py.exe subject to the 'looks like installer' behavior, such that it would need a manifest?

I still feel like this approach would require substantial special-casing, but since it provides a transition to the simple, elegant approach, I'm not opposed.

You ask important questions (and I don't know the answers :-)). I suggested (in the issue where I implemented the wrapper functionality) that we add a stdlib module that "knows" where the wrapper is and can wrap a script for you (something like distlib's "distlib.script" module). But that felt a bit over-engineered and the idea was abandoned almost immediately. In practical terms, of course, there are many, many ways of writing wrappers, depending on what you want them to do, and how manageable you want to make the job of manipulating them. The current setuptools wrappers make manipulating them easy (it's just an exe plus a script, with related names) at the cost of needing multiple files. You could append the script to the exe, which is a one-file solution but harder to edit the script. You could append a zipfile to the exe, which has the advantage that you can prepend arbitrary content to a zipfile, so there are no "mixed content" issues to worry about, in theory. The exe would be coded slightly differently in each case, of course, so there's a maintenance cost... Also, of course, as was mentioned elsewhere in the thread, you need separate wrapper exes for every architecture/platform you intend to support. And unless the wrappers are added at install time, wrapped scripts change a platform-neutral script into a platform-specific exe. I would still like to see the standard be registered .pye (I'm happy with a bikeshed of this colour) and .pwe extensions which are added to PATHEXT and associated with the launcher. But someone would need to collect and document the issues which have been raised with this approach, confirm which ones are real and which are merely potential, offer solutions for the real ones and confirm that the potential ones don't actually happen (or they do!). I won't have the time to do this in the near future - but I could consider it in a month or two if someone reminds me. It should probably be a PEP, although I'm not 100% sure what that PEP would propose (what changes in Python and wording in a PEP would be needed for you to change setuptools to abandon wrappers, for instance? And why would you need a PEP?) Paul

On Aug 12, 2013, at 4:04 PM, Paul Moore <p.f.moore@gmail.com> wrote:

Also, of course, as was mentioned elsewhere in the thread, you need separate wrapper exes for every architecture/platform you intend to support. And unless the wrappers are added at install time, wrapped scripts change a platform-neutral script into a platform-specific exe.

Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Donald Stufft <donald <at> stufft.io> writes:

Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic.

That's not unreasonable, as long as they don't contain executables. With the current version of distil, built wheels don't contain executables for scripts. The executable launchers are determined / written at wheel installation time. While distlib currently uses bespoke launchers, I plan to update it before the next release to use the PEP 397 launcher compiled with SCRIPT_WRAPPER. Regards, Vinay Sajip

Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:

While distlib currently uses bespoke launchers, I plan to update it before the next release to use the PEP 397 launcher compiled with SCRIPT_WRAPPER.

One more data point - the launcher currently used by distlib is found at [1]. Since it doesn't have all the bells and whistles that the PEP 397 launcher has (e.g. no searching for Pythons in the registry - the shebang is assumed to point at the Python to launch; no customised command handling; no configuration options; no support for passing parameters to Python itself), it is a lot simpler and has smaller executables as a result (around 64K, as opposed to 100-150K for the PEP 397 launcher). This might be an issue for some, when there would potentially be lots of copies of these executables. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/simple_launcher/

On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs <jaraco@jaraco.com> wrote:

...

-----Original Message----- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco.com@python.org] On Behalf Of PJ Eby Sent: Monday, 12 August, 2013 11:22

On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

On 12 August 2013 14:01, PJ Eby <pje@telecommunity.com> wrote:

As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content.

Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue.

My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable:

1. Disparity with Unix. Better parity means cleaner code, easier documentation, and less confusion moving from platform to platform. 2. Executables that look like installers. If a launcher executable is used and Windows detects that it "looks like" an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 5. Platform support. We're fortunate that Windows is one of the most stable binary platforms out there. Nevertheless, Setuptools recently got support for AMD binaries in the launcher. By relying on an external launcher, the launcher becomes responsible for platform support. 6. Two to three files to do the job of one. In fact, the "job" isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. 7. Obfuscation of purpose. A single script pretty directly communicates its purpose. When there are multiple files, it's not obvious why they exist or what their purpose is. Indeed, I went years without realizing we had an open issue in Distribute due to a missing manifest (which was fixed in Setuptools), all because I used the 64-bit executable. While it may take some time for the community to learn what a '.pyl' is, it's easily documented and simple to grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a side-by-side executable. 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful.

...

I think the issue of "too many extensions" vs. "source/binary confusion" is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP.

If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, "hey, run this as a console app via PyLauncher", not that it's a "script" (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile.

If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point.

...

...

PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further.

But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT.

Also, in my mind, this approach is most directly addressing the fundamental challenge (distinguishing a (executable) script from a module) in much the way Unix has previously enjoyed.

...

Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it.

(The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.)

I'm warming up to this idea a bit, especially how it supports the most elegant approach but degrades gracefully. Some questions that arise:

Where would Setuptools expect to find these launchers? Would it expect them to be present on the system?

It could, which would incidentally would address your issue #8 (people whining about Windows in their Linux). ;-) Basically, if the launcher is globally installed, you don't need to copy unless you're trying to be compatible with shells that don't support PATHEXT properly. If the launcher isn't installed, you'll need it bundled, or download it on the fly from a binary distribution dependency.

Would it symlink or hardlink them or simply copy?

Hardlinks and symlinks are essentially useless on Windows, so copy.

Is py.exe subject to the 'looks like installer' behavior, such that it would need a manifest?

Yep, but it can be embedded, as Steve points out. The only reason I never did this is because the Force is insufficiently strong in this one. ;-)

I still feel like this approach would require substantial special-casing, but since it provides a transition to the simple, elegant approach, I'm not opposed.

Well, the transition would probably go something like: 1. Start embedding pylauncher (or having a dependency on an egg or wheel that contains the launcher binaries) to use in place of the existing script mechanism 2. Allow making executables without copying the launcher, provided that a global pylauncher is installed w/file association and proper PATHEXT 3. Switch off using launcher copies by default, leave it as a backward compatibility option Basically step 1 gets manifest files and launcher maintenance off of setuptools' plate, which IIUC is mainly what you want.

On 13 August 2013 01:01, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Just a thought -- is there any need in this day and age for extensions to be limited to 3 characters?

There's a bug affecting PowerShell, which Microsoft have pretty much confirmed that they won't fix, which means that longer extensions aren't handled properly when used in PATHEXT. Longer extensions *can* be used, but not for the purposes we want them :-( Paul

On Wed, Aug 14, 2013 at 7:34 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Jason R. Coombs <jaraco <at> jaraco.com> writes:

...

This means that instead of installing, for example:

Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest

Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried:

1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource.

2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step.

3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete.

4. If a 'foo-script.py' was written in step 2, it is deleted.

The launcher comes with an embedded manifest, so no external manifest is needed.

Better suggestion: just append a PEP 441 .pyz to the .exe, and no extraction is necessary; the .exe just reads out the #! part. For Python 2.6 and up, the .exe can simply pass itself as argv[1] to the interpreter. (For older Pythons, a little -c and PYTHONPATH munging is required, but should be doable.) For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-)

Thoughts?

Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.)

Better suggestion: just append a PEP 441 .pyz to the .exe, and no

IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)?

For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-)

This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding?

Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good

True.

idea.  Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process?

That could probably be taken care of with a bit of footwork.

No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file.

How would such an offset be used? Are you saying the -c scriptlet would use that offset to extract the script? Or do you mean something else?

(As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.)

It's needed only if you use the specific approach I used - which was to use the Windows UpdateResource API to embed the script as a pukka Windows resource. Of course, if you're just appending something to the .exe, you don't need ctypes. Regards, Vinay Sajip

From: Nick Coghlan <ncoghlan@gmail.com>

I believe PJE's suggestion is to expand the scope of PEP 441 a bit to include the ability to generate a valid Windows executable with the archive appended, instead of just the *nix/Python launcher shebang line.

I got that; I was commenting on the "embed the entire relevant project" part as applied to wrapped scripts.

The "script wrapper" case would then just be a particular use of the PEP 441 executable generation features:  prepend the console executable for console scripts and the GUI executable for GUI wrappers, with the wrapper itself being a __main__.py file in the attached archive.

Given that Vinay already gave the Python launcher the ability to do this (when built with different options), this sounds quite feasible to me.

It's certainly feasible - I'm looking at the implementation now. It's not quite there yet - I need to extract the shebang from the end of the stock executable/beginning of the appended archive to determine the Python to use, and then ensure that the executable is passed to that Python as the script argument. Regards, Vinay Sajip

On 14 August 2013 18:46, PJ Eby <pje@telecommunity.com> wrote:

Shouldn't naming the file .pyw already work today for that case? Certainly, the .pyw extension is already suitable for manually creating GUI scripts in a text editor. Unless there's something special about how the 'pythonw' executable processes the command line, it should work just as well for a zipped archive.

.pyw files can be imported as modules, just like .py, so you hit the issue of scripts named the same as modules that they import. Naming a zipped archive .pyw is no better or worse than naming it .py - both work most of the time, but are disconcerting to users who think the filetype implies "text" and both have the problems associated with being both executable and importable. Paul.

PJ Eby <pje <at> telecommunity.com> writes:

used for these things nowadays, rather than PATH... but I think it only works for .exe's, so there we go again back to the land of .exe's just plain Work Better On Windows.)

In that vein, I've updated distlib to install only a single .exe per script, where the script is appended to a stock launcher .exe as a zip with the script as a single __main__.py in it. I've not produced a new release, but the BitBucket repos for both distlib [1] and the launcher [2] are up to date. Note that the launcher is still not the PEP 397 launcher, but the simpler one which I developed when working on PEP 405 and whose .exes have been shipping with distlib. To try it out, you can download the latest distil.py from [3]. My smoke tests with both generated and pre-built scripts pass, but I'd be grateful if people could try it out and provide feedback. Wheel support should be up to date in terms of PEP 426 and the discussions about launchers: wheel builds should have no platform-specific files other than for binary extensions, and when installing from a wheel, wrapped scripts declared in the metadata should be installed. Regards, Vinay Sajip [1] https://bitbucket.org/pypa/distlib [2] https://bitbucket.org/vinay.sajip/simple_launcher [3] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py

On 20 Aug 2013 04:22, "Thomas Heller" <theller@ctypes.org> wrote:

Back from holidays, I read this very interesting discussion...

Am 14.08.2013 16:33, schrieb Nick Coghlan:

...

Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5),

...but I don't understand what you mean by this. Can you please explain?

Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C. Cheers, Nick.

Thanks, Thomas

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 20 Aug 2013 09:18, "Thomas Heller" <theller@ctypes.org> wrote:

Am 20.08.2013 15:42, schrieb Nick Coghlan:

...

Importing C extensions requires extracting them to a temp directory and loading them from there. Trivial in Python, a pain in C. zipimport is currently still written in C.

So what - zipimport is a builtin module (on Windows at least).

Huh? That's irrelevant to the fact that doing the tempdir creation, file extraction and subsequent import entirely in C code would be painfully tedious. Cheers, Nick.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller <theller@ctypes.org> wrote:

Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ...

(py2exe for Python 3, which is work in progress, uses this approach)

Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.)

Am 20.08.2013 19:39, schrieb PJ Eby:

I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out?

To give a definite answer, after thinking it over: It works, for quite some extensions. The main problem is this: If it does NOT work (process crashes) there is no way to find out why. It is nearly impossible to debug because you end up with all the machine code from the extensions/dlls mapped into the process and the debugger has no info about it. Thomas

On Aug 20, 2013, at 8:57 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Thomas Heller <theller <at> ctypes.org> writes:

...

Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'):

In distlib, I've built on top of the zipfile support to allow C extensions to be available. Ordinary zipfiles contain no metadata indicating the extensions available, but wheels built with distil do. The wheel handling code in distlib takes advantage of this. The Wheel.mount() API [1] takes care of this (adding the wheel to sys.path, extracting the extensions to a user-specific directory and using an import hook to call imp.load_dynamic when required, so that both Python modules and extensions are available for import). It seems to work, though when I introduced the Wheel.mount() API I was told that it was very dangerous, and the sky would fall, or something :-)

Regards,

Vinay Sajip

[1] http://distlib.readthedocs.org/en/latest/tutorial.html#mounting-wheels

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track. If Wheels are to be importable it should be enshrined in the PEP not an adhoc feature of one possible implementation. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 21 August 2013 07:36, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Donald Stufft <donald <at> stufft.io> writes:

...

Mounting Wheels seems like a bad idea, it was one of the things Daniel explicitly removed (since Wheels are basically cleaned up eggs). Adding it back in ex post facto seems like it's an idea that's going down the wrong track.

Like I said, the sky will fall. Isn't importing from zips what's being discussed in this part of the thread?

Unless something is expressly verboten, I can't see a reason why particular implementations of a PEP can't provide additional functionality, as long as they implement the contents of the standard. And in Python's consenting-adults world, I can't recall seeing any such express proscriptions.

I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format. Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration). Paul PS Variances like this make me want to go back to the original idea that wheel support functions should be implemented in the stdlib, rather than having competing implementations "in the wild". Or at least, that one implementation should be considered the reference implementation that all others need to be compatible with :-(

On Aug 21, 2013, at 3:32 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Paul Moore <p.f.moore <at> gmail.com> writes:

...

I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq using distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format.

If the additional metadata isn't there, then distlib just doesn't do anything additional - it just makes the Python modules importable (by adding the wheel to sys.path, which AFAIK is uncontroversial).

...

Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration).

Well, you don't really want to stifle innovation, do you? ;-)

As far as I can tell, Daniel's wheel implementation allows files that are not specifically mentioned in the PEP to be installed into a distribution's .dist-info. This is also allowed in distlib - ISTM this is one way in which different packaging tools can add features which are special to them, and hold state relevant to distributions they build and/or install. If you accept that multiple competing implementations if a PEP are a good thing, then they can't all be functionally identical, though they must all implement a common set of functions described in the PEP they're implementing.

I was one of the advocates for extension support in the new metadata, I want tools to be able to try things out and innovate. However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around. My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions. So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is strictly the PEPs to allow distlib to experiment. If it's experimentations turns out to be good and useful we can make PEPs for them and add them to the reference library.

As far as the advocacy for C-extension import support in wheel mounting goes, I see opposition to the idea on the basis that some people have shot themselves in the foot with a similar feature in pkg_resources. However, I've not seen any analysis that indicates (with specifics) why the feature is inherently bad (if there are problems with a specific implementation of the idea, then those could perhaps be resolved, but you can't really argue against "you're going to have a bad time, and you should feel bad").

Regards,

Vinay Sajip

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Aug 21, 2013, at 4:07 AM, Paul Moore <p.f.moore@gmail.com> wrote:

OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my "reference implementation". I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen.

Probably Wheel at this point. There's just the problem with the scripts which we need to actually get into the PEP and implemented. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Paul Moore <p.f.moore <at> gmail.com> writes:

My problem is that as someone who wants to implement code that uses the new features like wheels, I want a usable reference implementation that covers the (agreed) standards. I don't particularly want my application to incorporate support for extensions to the standard, nor do I want to have to implement my own support all the time.

The wheel implementation in distlib conforms to the existing PEPs, as far as I know. It covers the agreed standards and, as far as I know, in a reasonable way. Why should you care if there are additional methods in a Wheel class, if you never call on them? It seems a step too far for a spec of this type to dictate precisely what specific methods should or should not be present in a class. If you find that distlib's behaviour somehow violates the agreed standards, please tell me how and I'll fix it. I think I've been pretty responsive on issues raised, but please don't raise vague fears if they have no basis in fact.

In particular, at the present time there are two tools that can generate wheels (bdist_wheel and distlib) and 3 that can install them (wheel, distlib and pip). They have subtly different behaviours outside of the standard definitions, which means that they are not completely interoperable. I am not happy at all about that - and if that counts as "being against innovation", then I'm afraid that yes, I am... (I don't think it does, by the way, but you may differ).

As I understand it, any interoperability issues are an effect of everything being a work in progress and some implementations lagging behind others in terms of PEP compliance. I can't speak for Daniel, but I've certainly made efforts to ensure that distlib can work with wheel-produced wheels, even where it means using old METADATA and entry_points.txt formats. I find it always helps to focus on specifics - since the implementations can't be identical, what specifically are the differences that make you unhappy, and why do they make you unhappy? If some of those can be laid at distlib's door, fine - I think I'm generally quite responsive to specific issues raised. Also, note that some features in an implementation might be useful to some people, even if not enshrined in a PEP. For example, distlib's wheel installation provides an option to install only the site-packages parts - no scripts or headers - a feature which you specifically requested, and which I readily added (as I can see its usefulness) without playing a "it's not in the PEP" card. Is that a "harmless innovation", or a "dangerous deviation from the standard"?

At the moment the wheel PEP is lagging a little behind some of the ongoing discussions, in particular in terms of script generation. That's fine, it's a work in progress. I hope it will be updated soon so that the spec matches what's been agreed.

Well, let's not forget that distlib is a work in progress, too.

But I think we have a reasonable consensus on how scripts should work,

Do we? Someone might tell me tomorrow that the "cool" one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-)

had a standard. I'm pretty unhappy that now we do have a standard, we still have situations where a wheel generated by one tool can have problems when installed with another - try "pip install wheel --use-wheel" on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one).

If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues.

OK, so here's a concrete question for distutils-sig. If I want to use wheels in my app (built them, install them, whatever) what should I use as my "reference implementation". I don't want to implement the code myself, I just want to produce lowest-common-denominator wheels that can be used anywhere, and consume wheels that conform to the spec correctly. This is not a hypothetical question - in the first instance I'm looking to add support for loading setuptools/pip from wheels in virtualenv, and I need to know what code to bundle to make that happen.

Some will say "wheel", even though it doesn't fully implement the spec, apparently because the wheel mount code (which,naturally, doesn't do anything unless it's called) is offensive to the point that some sort of fork is warranted ;-) Seriously - I already install setuptools and pip from wheels into venvs using distil (which of course uses distlib) - have you tried it? You're welcome to try with wheels built using any project, and if you run into any issues I will do my best to fix them. Have you had a better offer? :-) Regards, Vinay Sajip

On 21 August 2013 11:56, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

...

But I think we have a reasonable consensus on how scripts should work,

Do we?

To the level of "wheels builders should write metadata that defines the scripts and wheel installers should generate the necessary wrappers" then yes. We may or may not (depending on your viewpoint) have agreement on the format of that metadata. Personally, I think that Metadata 2.0 where the wheel uses that format, and entry-points.txt for older versions, is the only realistic option here. I would like to see some clarity on the status of the wheel 1.0 spec. Where there are areas like this where the spec is either silent or missing sufficient detail to allow us to implement a common approach, should we be updating the 1.0 spec or should we be creating a 1.1 spec (I'd prefer the former)? Who does those updates? Must they go through Daniel as the author (who's been quite quiet, possibly he's hiding somewhere rather than being drawn into the fray :-))?

Someone might tell me tomorrow that the "cool" one-executable solution I've recently implemented as an option on Windows (append shebang / archive to a stock executable) is an offence against all that's holy ;-)

That won't be me. I'm more interested in discussions about interoperability issues than about how the functionality gets implemented.

had a standard. I'm pretty unhappy that now we do have a standard, we

...

still have situations where a wheel generated by one tool can have problems when installed with another - try "pip install wheel --use-wheel" on Windows to see what I mean, the exe wrappers are missing (this uses the wheel from PyPI, not a home-built one).

If distlib is proving to be a problem either as a producer or consumer of wheels, please raise issues.

I don't believe that distlib creates scripts based on entry-points.txt for pre-2.0 metadata. I have raised an issue for that. That's the only significant issue I have with distlib from the POV of what it can solve. There may be other interoperability issues where other tools can't consume things that distlib produces - I haven't tested anything like all of the combinations. We currently have 2 producers and 3 consumers. That's already 6 interactions to test. And if there *is* an issue, it's not always at all clear who to report the problem to. I'm not complaining about *any* of the implementations here. And I apologise if you in particular feel "picked on" (there's probably a distutils-sig "victim of the week" rota somewhere for this ;-)) My real concern is that we're drifting away from standards-based design towards implementation-based. And I think that updating the wheel PEP to be tighter on some of the details is what's really needed (hey, look, it's Daniel's turn to be picked on! :-)) I'd be willing to author an updated version of the wheel spec if all of the interested parties are OK with that (in particular, Nick and Daniel). If nothing else, that means that everyone can pick on me so I wouldn't feel quite so much like a troublemaker :-) Paul PS I really, really don't want anyone here to feel like anything they do is not valued. This is about tidying up the documentation, not about stifling anyone's enthusiasm or blocking any experimentation.

On Aug 21, 2013, at 4:23 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Donald Stufft <donald <at> stufft.io> writes:

...

However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around.

"Someone's personal test-bed"? How do you think all open source software starts out? What innovation isn't based on a feature someone thinks "is cool"?

From our interactions, I don't get a feeling that you're particularly interested in building tooling around distlib anyway, for reasons best known to you. This specific feature could be easily pulled, and distlib is still only at version 0.1.2. Nick has indicated he doesn't think it's appropriate to consider distlib ready for endorsement in a 3.4 time-frame, so there's plenty of time to make changes which are deemed necessary. How on earth do you expect people to try things out, see where the problems are and fix them, if you don't release such features? It would certainly help if you describe specific problems with this feature, rather than just "I don't like it".

I don't particularly care about this feature. I care about having a simple reference library. That is what I thought distlib was supposed to be (and I'm not alone). However if you want to innovate and experiment inside it then great, that's fine. I just won't tell people that distlib is the reference library.

...

My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's confusing what is standard and what are just distlib's extensions.

There are already features in distlib which aren't mentioned in any PEP - e.g. the ability to calculate dependency graphs without downloading any archives, or the ability to install scripts as single executables on Windows, or the ability to give better feedback when uninstalling.

As mentioned above, I'm not trying to argue against this particular feature I'm just trying to make sure we have a simple reference library for people to be pointed at.

...

So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is

ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall.

That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation.

There's plenty of time to try things out and make changes, so I find your approach a bit of an over-reaction, and feel that it doesn't square with what you're saying about innovation. You know that feeling you mentioned that you get, where you think everyone is just out to block you and stop you getting stuff done? Are you trying to spread that feeling around? ;-)

I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards. I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around.

Regards,

Vinay Sajip

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Aug 21, 2013, at 5:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 21 August 2013 09:56, Donald Stufft <donald@stufft.io> wrote:

...

ISTM distlib is not yet that reference library - it's just another library for most people, judging from the low level of feedback I've had overall.

That's totally fine. We just need to be clear that it's not the reference library and is instead one implementation.

Yep, there's certainly been a perception that distlib is the reference implementation. Apologies if I perpetuated that.

We do have a slightly different issue then, in that there *isn't* a reference implementation for a lot of this stuff... (I guess wheel counts as the reference implementation for wheel, doh, so that part's covered). People are starting to write code to use these new facilities, so having an actual reference implementation is important (IMO, that's one area where packaging/distutils2 got in a mess, so I'm concerned we don't fall into the same trap). We need people using the new stuff to help us identify potential issues.

I might see about making a stripped down library which only implements the PEPs and nothing else.

Paul

PS Apologies if I appear to be a little irritable on this subject. I have a series of scripts that maintain a local cache of wheels for various projects. I'm starting to hit cases where the wheels aren't usable because of subtle differences in how the spec's being implemented, and it feels like I'm trying to hit a moving target, which is what I thought having the wheel 1.0 PEP accepted was designed to avoid.

Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Aug 21, 2013, at 5:46 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 21 August 2013 10:29, Donald Stufft <donald@stufft.io> wrote: Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal).

The scripts one is the key one (and yes, that needs to be fixed by updating the spec to confirm the consensus and then updating the tools to match).

+1

Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here).

Ah yes, This one is a bit harder because it's a non standard file added by setuptools, but yea it probably should be adding that file.

I'll try to dig out the others, but they weren't anywhere near as major (some were also script-related such as inconsistent rewriting of shebang lines, which are superseded by the current state of play on scripts anyway). Generally, though, they were all areas where the 1.0 spec is silent (or just a bit vague) on details, and I was hitting implementation-defined differences. (For example, I was surprised to find that nowhere in the Wheel 1.0 spec does it actually state the metadata version that wheels should contain internally, just that it should be stored in a file called METADATA and that it should follow PKG-INFO which sort of implies that it's a 1.X format). Without an implementation that matches the spec precisely, though, it's often hard to know where to report bugs.

Paul.

I think Wheel files are (and should be) independent of the particular metadata version used. That file should contain the required information in order to know what version of the metadata is included with the Wheel. This means that as metadata evolves Wheels can just start using the new meta data version without requiring an update to the spec. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Aug 21, 2013, at 6:17 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 21 August 2013 10:48, Donald Stufft <donald@stufft.io> wrote: I think Wheel files are (and should be) independent of the particular metadata version used. That file should contain the required information in order to know what version of the metadata is included with the Wheel. This means that as metadata evolves Wheels can just start using the new meta data version without requiring an update to the spec.

That implies that any wheel reference implementation needs to expose APIs for reading and writing the metadata to/from the wheel. I don't have a problem with that, but I don't think the existing implementations do[1]... (And it could be a bit of a beast to design such an interface in a sufficiently future-proof manner, unless we also standardise a metadata object type...) Specifically, if I have a wheel and want to introspect it to find out the author email address, how do I do this? ("Doing so is not supported" is a valid answer, of course, but should also be documented...)

Paul

[1] Just checked. Distlib doesn't. Wheel doesn't (and wheel doesn't even have a fully documented API). Pip's wheel support is purely internal so doesn't count.

Yes I believe there should be an API for reading/writing to the dist-info directory and that's how a Wheel API should handle exposing that. It means you can compose api's so you have 1 API for reading/writing metadata which can be used for Wheels, Sdist 2.0, The on disk installed database format etc. Quick off the cuff design of an API for introspecting the author email address (Please realize this is totally off the top of my head and has not been thought through or played with or explored or anything that would make it reasonable to actually expect it to be sane for more than this simple example).

...

...

from ref import Metadata from ref import Wheel, Sdist2

...

...

whl = Wheel.from_file("something-something.whl") print(whl.dist_info) {"METADATA": "…", "entry_points.txt": "…"} meta = Metadata.from_mapping(whl.dist_info) print(meta["summary"]) "A Something or Other Library" sdist2 = Sdist2.from_file("something-something.sdist2") print(sdist2.dist_info) {"pydist.json": "…", "README": "…"} meta = Metadata.from_mapping(sdist2.dist_info) print(meta["summary"]) "A something or Other Library"

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 21 August 2013 11:30, Donald Stufft <donald@stufft.io> wrote:

On Aug 21, 2013, at 6:29 AM, Donald Stufft <donald@stufft.io> wrote:

introspecting the author email address

Of course I wrote that and then did summary because the location of the author email address changed between Metadata 1.x and 2.x and I didn't feel like looking up the exact difference.

:-) So would you see the API for author being if meta.version < (2, 0): author = meta["whatever_it_is_in_1.x"] else: author = meta["whatever_it_is_in_2.x"] (i.e., the metadata version would be available)? Sounds like a sensible approach. Paul.

Paul Moore <p.f.moore <at> gmail.com> writes:

That implies that any wheel reference implementation needs to expose APIs for reading and writing the metadata to/from the wheel.

Not necessarily. For example, distlib's approach side-steps the need for such a write API: you tell Wheel.build which directories contain purelib, platlib, scripts, headers etc. and it puts all the stuff that it finds in those directories into appropriate locations in the wheel, then updates RECORD, WHEEL etc. This way, it doesn't need to worry about what custom files installers put in .dist-info, and reduces the coupling between the wheel code and users of it. There's no need for a special read API either, since wheels are just zip files and you can use the zipfile API to look at anything inside a wheel. As a convenience, distlib's Wheel instances have a metadata property, which returns as a dict the pydist.json from the wheel's .dist-info. Why make it more complicated than that? Regards, Vinay Sajip

On 21 August 2013 12:22, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Paul Moore <p.f.moore <at> gmail.com> writes:

...

Another one IIRC was that distlib didn't put entry-points.txt in the .dist-info directory in the wheel (which breaks entry points). I think that's fixed now (and again, the Wheel spec is silent on what is correct behaviour here).

Right. The recent PEP 426 updates with the "commands" key supersedes the entry points stuff, so I'm not sure what exactly should be done here. The distlib code uses the latest PEP information. Since wheel is moving to pydist.json, I assume that (when it gets around to it) it will have the relevant scripts info in pydist.json, so I haven't implemented using scripts declared in entry_points.txt in distlib for that reason.

OK, I see what you're saying here. But the Wheel 1.0 spec says that metadata is in the METADATA file (and comes from PKG-INFO). So my reading of that means that Metadata 1.x will remain valid for the foreseeable future (sure, Metadata 2.0 may become acceptable *as well* but the point of having a Wheel 1.0 spec is that we won't stop supporting it for some time yet). So you need to have a pre-2.0 solution in place, and while entry-points.txt isn't explicitly stated in the wheel PEP, it's the obvious equivalent. If you want to say distlib won't support pre-Metadata 2.0 specifications of script metadata, then that's your choice - it's not contrary to the standards but I'd view it as a quality of implementation choice. I view the underspecification in the Wheel 1.0 spec as similarly a quality of detail issue, and I'd expect to fix it in either an update to Wheel 1.0, or a Wheel 1.1 which does not make the jump to pure Metadata 2.0. Paul

On 08/21/2013 03:29 AM, Donald Stufft wrote:

Can you send me a list (or post them here) of what issues you've hit? The biggest one i'm aware of is the scripts problem which is a fundamental problem with the 1.0 Wheel (or rather that any library with console entry points cannot be universal).

Since you asked, I'll mention the two that I've hit (though I think you're also aware of these already): 1) Wheel's conversion of - to _ in version strings embedded in filenames, which breaks with setuptools precedent; see https://github.com/pypa/pip/issues/1150 and https://bitbucket.org/dholth/wheel/issue/78/wheel-rewrites-versions-preventi... 2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_file... Carl

On 08/21/2013 10:32 AM, Daniel Holth wrote:

...

2) Wheel's decision to follow distutils' documentation rather than distutils' behavior when it comes to the location for installing data_files with relative paths; see https://bitbucket.org/dholth/wheel/issue/80/wheel-does-not-install-data_file...

Django has fixed it by using package_data appropriately: https://code.djangoproject.com/ticket/19252 . The problem isn't unique to wheel, the same data_files mishap happens with bdist_wininst.

"Regardless, comment 5 is correct that we jump through way too many hoops in our setup.py in order to try to trick distutils into handling data_files as if they were package_data, and that is the root cause of this bug. Instead we should just use package_data and solve the problem properly."

Yup, that's my comment you're quoting :-) I do think from the packager end using package_data is the right solution. But given the existence of distributions using data_files this way (and the likelihood that not all of them will be fixed), is there a good argument for wheel to not maintain compatibility with sdists and python setup.py install? Are there distributions out there relying on the bdist_wininst behavior? Carl

Donald Stufft <donald <at> stufft.io> writes:

I think the way you view distlib and the way other are viewing distlib are different (and that's ok). We just need to know what distlib is so we can have reasonable expectations of it. What i'm getting from you is that, at least right now, distlib isn't what I thought it was and I (and others) should stop treating it as *the* reference implementation of the new standards.

I'm not trying to stop you from innovating, I'm just trying to make sure everyone has reasonable expectations all around.

I don't see how anyone is treating it as a reference implementation, other than just talking about it being such. I have had very good feedback from one or two people who are using it, but for the most part I see no evidence that people on this list are using it to the extent they would if they really thought "this might be a good candidate for a reference implementation - I see it's early days, and there's work to do, but it seems usable, so let me check it out and see that my use cases are covered, and if anything's been overlooked". In my view, nothing deserves to be a considered a "reference implementation" other than through merit, or perhaps by "being number one in a field of one". Merit isn't earned unless the software is used and refined based on real-world experience. The time to try distlib is now (not in production environments, obviously), to allow any problems with it to be identified early in its life. That would be more helpful than any sniping from the sidelines about whether it does more than what it needs to for PEP conformance. Regards, Vinay Sajip

Donald Stufft <donald <at> stufft.io> writes:

I think you're using a completely different definition of "reference implementation" than I've ever seen used. A reference implementation

Quite possibly, but I feel justified in this case ... I'll say why below.

by definition cannot contain customizations or additions or extensions from the spec. The entire *point* of a reference implementation is to act as programatic reference to the spec. Something being the reference implementation does not speak to the quality of the implementation and as such it may not be the *best* implementation.

The packaging PEPs don't do down to programmatic detail in their specification. In the stricter definition of "reference implementation" (RI) you're talking about, a spec comes with a reference implementation but also a test suite (provided by the spec author) which conforming implementations must pass. While this is de rigueur in the Java world, it's not common in the Python world. Without such a test conformance suite, I think it's reasonable to use the looser definition of RI that I did.

An example is the wsgiref from the standard library. Very few projects actively use wsgiref for much at all if they use it at all. However it's existence means that web servers like gunicorn, mod_wsgi etc can simply test against it instead of needing to test against every implementation of WSGI.

While you can certainly use wsgiref for interoperability testing (much as I use wheel's wheel implementation for interoperability testing with distlib) I don't think WSGI implementations run a suite of tests provided in wsgiref before claiming conformance. I'm happy to be corrected if I'm wrong about that :-) In my experience of implementing PEPs (282, 391, 397, 405) all implementations have had test suites, but none have had the tests implemented by any central authority. In discussions the relevant implementations have been referred to as RIs and that's the way I'm using it now.

Which implementation is used (and ultimately possibly enshrined in the standard library) is decided through merit. Which implementation is used as the reference implementation is typically decided by the standards body (in this case, distutils-sig or Nick or whoever).

If Nick or whoever is planning to write a test suite which all implementations must pass, great. But that would (I suppose) mean tying things down to a specific Python interface at the module, class and function level - not something I've seen imposed externally before. But in the absence of such, I don't see any problem with my interpretation of RI, since the stricter interpretation only makes sense in the presence of accompanying tests generated by the spec originators. Regards, Vinay Sajip

On Wed, Aug 21, 2013 at 9:24 AM, Donald Stufft <donald@stufft.io> wrote:

An example is the wsgiref from the standard library.

It's an example, alright, but not for your side. ;-) The wsgiref library doesn't just implement the spec, it implements a ton of utility classes for use with the spec. The validator was almost an afterthought grafted on later, borrowed from another project. It implements a framework with all sorts of features that are not technically part of the spec, but are just useful if you want to implement the spec. Very few of the classes, methods, etc. in the entire package are specified by the spec, except in the sense that many of them match a calling signature defined in the PEP. (The PEP doesn't specify any method names, except for things like read() on file-like objects.) IOW, wsgiref is a collection of generally useful tools for anybody doing things with the spec, as an combination of "examples of how to do this" and "ready-to-use code for working with the spec". Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world.

On 21 Aug 2013 17:46, "Donald Stufft" <donald@stufft.io> wrote:

On Aug 21, 2013, at 3:32 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

...

Paul Moore <p.f.moore <at> gmail.com> writes:

...

I'm concerned that you need extra metadata (not described in the wheel spec) to do this. It means that there are in effect two subtly different types of wheel. To be specific, if I create a wheel for (say) pyzmq

...

distil, and mount it, everything works. But if I create the same wheel with bdist_wheel or pip, it doesn't. That, to my mind, is very bad as it damages the credibility of wheel as a standardised format.

If the additional metadata isn't there, then distlib just doesn't do anything additional - it just makes the Python modules importable (by adding the wheel to sys.path, which AFAIK is uncontroversial).

...

Can I suggest that if you need to add features like this, you need to get the wheel spec updated to mandate them, so that *all* wheels will follow the same spec. Essentially, I am -1 on any feature that uses information that is not documented in the wheel spec. Pip in particular resisted adding support for wheels until they were standardised in a PEP. It's frustrating if that PEP *still* doesn't mean that the wheel format is the same for all tools. (Note that another area where this is an issue is script wrappers, as the spec is silent about the fact that they are specified using entry-points.txt in metadata 1.x/setuptools. I've sent a proposed update to the spec to Daniel for his consideration).

Well, you don't really want to stifle innovation, do you? ;-)

As far as I can tell, Daniel's wheel implementation allows files that are not specifically mentioned in the PEP to be installed into a distribution's .dist-info. This is also allowed in distlib - ISTM this is one way in which different packaging tools can add features which are special to them, and hold state relevant to distributions they build and/or install. If you accept that multiple competing implementations if a PEP are a good

using thing,

...

then they can't all be functionally identical, though they must all implement a common set of functions described in the PEP they're implementing.

I was one of the advocates for extension support in the new metadata, I want tools to be able to try things out and innovate.

However what I don't really want is to be using someones personal testbed for features they think is cool. There's nothing *wrong* with you trying new ideas out in distlib, it just means that distib isn't the library I want to build tooling around.

Right. I wasn't really aware Vinay was adding experimental ideas to distlib, I thought it was just the proven stable core from distutils2, plus support for the draft PEPs, with the experimental stuff entirely in distil rather than in distlib. If Vinay wants to do experimental extensions, they either need to happen somewhere other than distlib, or else we need a new library which is just the candidate for standard library inclusion with *nothing* that hasn't been discussed and agreed to through the PEP process. As I said, I *thought* distlib was that library, but it appears Vinay doesn't currently see it that way :(

My basic problem is if the library we're pointing at to be the reference implementation of all of these things is adding new features it's

confusing what

is standard and what are just distlib's extensions.

So basically I want people to innovate, that's something I feel very strongly is a good thing, I just don't want innovations to happen in the reference library. Maybe we need a smaller reference library which is strictly the PEPs to allow distlib to experiment. If it's experimentations turns out to be good and useful we can make PEPs for them and add them to the reference library.

Right. distlib can be a candidate for stdlib inclusion, or it can be a vehicle for distributing experimental features not covered by any PEP, it can't be both at the same time. Cheers, Nick.

On 22 August 2013 08:12, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

I've been up front about what's in distlib all along - check the overview page in the distlib docs. Above all, I want the stuff I do to be *useful*, rather than tick boxes here and there.

Right, you didn't do anything wrong, I just wasn't really paying attention because promoting distlib adoption has been firmly in the "later" bucket for me, after the setuptools rehabilitation, pip bootstrapping, etc. I can't fault you for not realising I believe something I had never really written down anywhere :)

Nick Coghlan <ncoghlan <at> gmail.com> writes:

...

Right. distlib can be a candidate for stdlib inclusion, or it can be a vehicle for distributing experimental features not covered by any PEP, it can't be both at the same time.

How is it that you're ready to call time on it now, already, even though you've told me standardisation is not something to be even considered until Python 3.5? Of the two options quoted above, you had already decided that it can't be the former in the short to medium term, and now you're saying you don't want it to be the latter either? I'm confused. I'd certainly like to keep making distlib and distil more useful.

I previously thought distlib was going to be the repository for the agreed, stable, "this is going to happen" stuff. It's OK that I was wrong - I think you're right that somewhere is needed as an experimental location to show some of the *possibilities* of the new metadata, and to seed ideas for making it into the eventual standard base that people can assume is readily available. What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all. I'm actually OK with that, since it means we can aim for a self-contained pip that serves as the basis for the rest of the ecosystem, and is upgraded on the *pip* update cycle, rather than CPython's. I'd been wondering how we'd avoid the "stale standard library support" issue we ran into with distutils, and I think the pip bootstrapping proposals give us the answer: we don't make the core distribution infrastructure part of the standard library, we make it part of *pip*. One of the key concepts of the bootstrapping idea is that CPython maintenance releases will bundle newer versions of pip, and also that pip will be able to upgrade itself in place, so if people need newer distribution infrastructure "upgrade pip" is a much lower risk proposition than "upgrade to a newer version of Python". Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as "piplib". The bootstrapping would then provide the pip CLI and the common utilities in piplib, and that would be the more conservative core, leaving distlib and distil free to push the boundaries with experimental features that go beyond what the agreed standards currently support. As ideas from distlib make their way into the relevant PEPs and hence into piplib, then they will become available by default, but in the meantime, people would be able to do a build dependency on distlib to get the experimental features. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Disclaimer: everything I say below about pip is ultimately up to the pip devs. I'm just pointing out what I think makes sense, and my reading of Donald's comments means that I expect he would feel the same way. On 22 August 2013 17:22, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Nick Coghlan <ncoghlan <at> gmail.com> writes:

...

What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this

If that "something else" you're thinking of is something that is supposed to live in the stdlib, then I see no reason why a subset of distlib couldn't be that something else, since stdlib changes are not on the table for 3.4. I certainly have never envisaged that distlib would be adopted wholesale into the stdlib (if at all) without peer review and any changes coming out of that.

I *can't* tell people "we're going to migrate to distlib as the reference packaging infrastructure implementation" when I really mean "we're going to migrate to some as yet undefined subset of distlib, so 'import distlib' won't be the long term answer". Incorporating only a subset of an existing published API into the standard library is a mistake - the PyXML debacle shows us that. If the API is different (even if that means a strict subset), then it needs a different name.

...

common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all.

I honestly think you're making a bit too much of the "experimental" label here, even though it is a label that I use myself. For me, that label is most appropriate for the extended metadata that I collect from PyPI and which is the basis for distlib's smarter dependency resolution.

If your concerns are about instability due to experimental features (and I quite understand the importance of stability in packaging), then there's nothing stopping anyone doing a technical review of distlib to see what any actual risks are. Indeed, I've invited such review from day one.

It has nothing to do with code quality, and everything to do with being able to explain the migration plan to people. I *can* say to them "pip is going to cherry pick parts of distlib and potentially other libraries and make them available as 'piplib', which will be installed automatically when you install pip". At the moment, I no longer feel I can say "distlib will become the reference implementation". Note that there's also the bootstrapping issue with having pip depend on an external library: having the core library *in pip* makes that problem go away.

...

Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as "piplib". The

I think this would be a mistake, and it seems a little early to make this sort of decision. You've given me to understand that pip could at some future point use (some subset of) distlib under the covers, with compatibility maintained at the CLI level. If that is still the case, then I don't see much value in having two lib layers.

When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I now think it makes more sense for pip to migrate to a more tightly constrained public library API, that doesn't go beyond the documented metadata standards (except for legacy compatibility reasons).

Like setuptools, pip has done sterling service, but it's not a codebase I'd like to see become the basis for our long-term packaging infrastructure. I don't mean to offend anybody by saying that - it's just software that has grown organically over time and IMO there will be technical debt to repay if we go down the route of exposing bits of it as Python APIs.

I don't expect the contents of piplib to match the contents of the existing pip module. This is about enabling a gradual refactoring over to a cleaner core library with a public API, rather than a big bang migration to an alternative solution like distlib.

It certainly feels like you're side-lining distlib, or planning to, whether or not that's the message you're intending to send. No matter :-)

I don't currently believe your plans for distlib and my plans for the standard library software distribution support (whether directly in the standard library or via the pip bootstrapping) are compatible. If I am correct, then distlib remains extemely valuable in that scenario, but the nature of its role changes. I want a completely barebones "absolutely no features that aren't covered by an Accepted PEP" library as a candidate for future inclusion. Such a library *cannot* be particularly useful at this point in time, because the metadata 2.0 PEPs are nowhere near complete enough for that. By contrast, you understandably wish for distlib to be useful *now*, which means running ahead of the standardisation process, filling in missing features as needed. Assuming the pip devs are amenable (and given Donald's comments in this thread, I expect they will be), making the decision now that pip *will not* bundle distlib directly, but instead will create its own support library means we have a clear path forward for defining the "suitable subset of distlib (and perhaps other libraries)" that will become the "available by default" core library for installation tools, while leaving you free to make distlib as useful as it can be in the near term, even if it takes the standardisation process a while to catch up. The eventual core API *probably won't support* the legacy metadata formats, instead leaving that to setuptools and/or distlib. It will probably depend on the upgraded PyPI APIs as well (once they're defined). Ideally, this will give us at least two competing implementations on the metadata 2.0 production side (setuptools/bdist_wheel and distlib/distil) and two on the consumption side (piplib/pip and distlib/distil), so the various combinations should help us ensure we've eliminated most of the ambiguity from the specifications and aren't going to the end up with excessively high levels of implementation defined behaviour again. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia On 22 August 2013 17:22, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Nick Coghlan <ncoghlan <at> gmail.com> writes:

...

I previously thought distlib was going to be the repository for the agreed, stable, "this is going to happen" stuff. It's OK that I was wrong - I think you're right that somewhere is needed as an experimental location to show some of the *possibilities* of the new metadata, and to seed ideas for making it into the eventual standard base that people can assume is readily available.

It's not just about completely new, experimental stuff. For example, the resources functionality isn't completely new territory. The PyPI interfacing is (IMO) a saner API than the one in distutils2. A better Windows story (for when launcher support when py.exe can't be used) is also not rocket science.

...

What that means though, is we need *something else* that indicates the common core that people can assume will always be available. It's this

If that "something else" you're thinking of is something that is supposed to live in the stdlib, then I see no reason why a subset of distlib couldn't be that something else, since stdlib changes are not on the table for 3.4. I certainly have never envisaged that distlib would be adopted wholesale into the stdlib (if at all) without peer review and any changes coming out of that.

...

common core which pip will need to factor out to remove their dependency on setuptools, rather than adopting distlib wholesale, experimental features and all.

I honestly think you're making a bit too much of the "experimental" label here, even though it is a label that I use myself. For me, that label is most appropriate for the extended metadata that I collect from PyPI and which is the basis for distlib's smarter dependency resolution.

If your concerns are about instability due to experimental features (and I quite understand the importance of stability in packaging), then there's nothing stopping anyone doing a technical review of distlib to see what any actual risks are. Indeed, I've invited such review from day one.

...

Currently, pip doesn't expose a programmatic API. I suggested to Donald that it may make sense to start exposing one as "piplib". The

I think this would be a mistake, and it seems a little early to make this sort of decision. You've given me to understand that pip could at some future point use (some subset of) distlib under the covers, with compatibility maintained at the CLI level. If that is still the case, then I don't see much value in having two lib layers.

Like setuptools, pip has done sterling service, but it's not a codebase I'd like to see become the basis for our long-term packaging infrastructure. I don't mean to offend anybody by saying that - it's just software that has grown organically over time and IMO there will be technical debt to repay if we go down the route of exposing bits of it as Python APIs.

It certainly feels like you're side-lining distlib, or planning to, whether or not that's the message you're intending to send. No matter :-)

Regards,

Vinay Sajip

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 23 August 2013 00:19, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Nick Coghlan <ncoghlan <at> gmail.com> writes:

...

When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused.

I can certainly see that there are ways to avoid confusion. But never mind, I see that you've made your decision. I would have hoped for a more transparent decision process, but that's probably due to my slowness of uptake.

The only decision I've made is to stop saying "distlib is the future", since that is now in doubt, and I certainly don't have the time available or expertise needed to review all the APIs that have been added to it (I thought it was just the four distutils2 interfaces that almost made it into Python 3.3 and that all your experimental interfaces were in distil, not distlib. While there was plenty of evidence to indicate I was wrong in that belief, I wasn't paying proper attention to it and it didn't properly register until it came up in this thread). The next step is up to the pip folks - if they think adopting distlib wholesale makes sense for them, fine, I have no direct say in that. If they decide to make a "piplib" instead, to expose a public API for an updated version of pip's own infrastructure (perhaps derived from distlib), that's fine by me, too. The only absolute in this space relates to the default installation toolchain: it *will* be pip. Unlike setutptools as a build system, I consider easy_install irredeemable as an installer (from a social perspective), and there's no way I would ever inflict yet another change of recommended client on the community. Given that, any future changes to the core infrastructure will be heavily influenced by the technical choices of the pip developers. Other tools will exist around that, since packaging is a complex topic where "one size" really doesn't fit all, but pip will be the centrepiece. Cheers, Nick.

On Wed, Aug 14, 2013 at 9:58 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)?

I meant that it has a #! line before the zip part, so that the launcher knows what Python to invoke. There are also some challenges for older Pythons to invoke __main__, since the normal Python import machinery frowns on reloading __main__. I expect the zip would need an extra __main.py stub to bootstrap the loading of __main__, and then invoke python with something like '-c "__import__('sys').path[0:0]=['/path/to','path/to/exe'']; __import__('__main').go()"'. (It can't have the import run the app as a side effect, because otherwise the import lock will be held, leading to Bad Things in multi-threaded apps.)

This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding?

I just meant that for cases where there's only one script, or where you are doing a custom-built application. This also becomes The One Obvious Way to do py2exe-like things.

How would such an offset be used? Are you saying the -c scriptlet would use that offset to extract the script? Or do you mean something else?

Extract the script by seeking to the offset and reading it. It's far from ideal, though; the .zip is much better since everything back to 2.3 can support it in some fashion.