On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs <jaraco@jaraco.com> wrote:
-----Original Message----- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco.com@python.org] On Behalf Of PJ Eby Sent: Monday, 12 August, 2013 11:22
On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 12 August 2013 14:01, PJ Eby <pje@telecommunity.com> wrote:
As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content.
Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue.
My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable:
1. Disparity with Unix. Better parity means cleaner code, easier documentation, and less confusion moving from platform to platform. 2. Executables that look like installers. If a launcher executable is used and Windows detects that it "looks like" an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 5. Platform support. We're fortunate that Windows is one of the most stable binary platforms out there. Nevertheless, Setuptools recently got support for AMD binaries in the launcher. By relying on an external launcher, the launcher becomes responsible for platform support. 6. Two to three files to do the job of one. In fact, the "job" isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. 7. Obfuscation of purpose. A single script pretty directly communicates its purpose. When there are multiple files, it's not obvious why they exist or what their purpose is. Indeed, I went years without realizing we had an open issue in Distribute due to a missing manifest (which was fixed in Setuptools), all because I used the 64-bit executable. While it may take some time for the community to learn what a '.pyl' is, it's easily documented and simple to grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a side-by-side executable. 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful.
I think the issue of "too many extensions" vs. "source/binary confusion" is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP.
If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, "hey, run this as a console app via PyLauncher", not that it's a "script" (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile.
If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point.
PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further.
But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT.
Also, in my mind, this approach is most directly addressing the fundamental challenge (distinguishing a (executable) script from a module) in much the way Unix has previously enjoyed.
Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it.
(The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.)
I'm warming up to this idea a bit, especially how it supports the most elegant approach but degrades gracefully. Some questions that arise:
Where would Setuptools expect to find these launchers? Would it expect them to be present on the system?
It could, which would incidentally would address your issue #8 (people whining about Windows in their Linux). ;-) Basically, if the launcher is globally installed, you don't need to copy unless you're trying to be compatible with shells that don't support PATHEXT properly. If the launcher isn't installed, you'll need it bundled, or download it on the fly from a binary distribution dependency.
Would it symlink or hardlink them or simply copy?
Hardlinks and symlinks are essentially useless on Windows, so copy.
Is py.exe subject to the 'looks like installer' behavior, such that it would need a manifest?
Yep, but it can be embedded, as Steve points out. The only reason I never did this is because the Force is insufficiently strong in this one. ;-)
I still feel like this approach would require substantial special-casing, but since it provides a transition to the simple, elegant approach, I'm not opposed.
Well, the transition would probably go something like: 1. Start embedding pylauncher (or having a dependency on an egg or wheel that contains the launcher binaries) to use in place of the existing script mechanism 2. Allow making executables without copying the launcher, provided that a global pylauncher is installed w/file association and proper PATHEXT 3. Switch off using launcher copies by default, leave it as a backward compatibility option Basically step 1 gets manifest files and launcher maintenance off of setuptools' plate, which IIUC is mainly what you want.