[Distutils] How to handle launcher script importability?

[PJ Eby]

On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs <jaraco at jaraco.com> wrote:
>
>
>> -----Original Message-----
>> From: Distutils-SIG [mailto:distutils-sig-
>> bounces+jaraco=jaraco.com at python.org] On Behalf Of PJ Eby
>> Sent: Monday, 12 August, 2013 11:22
>>
>> On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore <p.f.moore at gmail.com>
>> wrote:
>> > On 12 August 2013 14:01, PJ Eby <pje at telecommunity.com> wrote:
>> >
>> > As far as zipped Python applications are concerned (pyz), these can be
>> > created by just using a pys file containing a #! line prepended to the
>> > zip file. Certainly, it's a binary file with a filename that would
>> > normally indicate a text file format, but is that any less true on
>> > Unix when users create these files? I don't know what the user
>> > experience with zipped Python applications on Unix is like - I doubt
>> > it's *that* much better than on Windows. Probably the reality is that
>> > nobody uses zipped applications anyway, so the problems haven't been
>> > identified yet. Maybe the pyz PEP would bet better rewritten to
>> > propose providing tools to create and manage zipped Python
>> > applications, but *not* to require new extensions, merely to reuse
>> > existing ones (pys on Windows, no extension on Unix) with binary (zipped)
>> content.
>>
>> Seems reasonable...  but then somebody will need to write another PEP for
>> the file extension(s) issue.
>
> My preference is to reject the idea of the side-by-side executable launcher.
> There are several downsides that I'm trying to avoid by moving away from the
> executable:
>
> 1. Disparity with Unix. Better parity means cleaner code, easier
> documentation, and less confusion moving from platform to platform.
> 2. Executables that look like installers. If a launcher executable is used and
> Windows detects that it "looks like" an installer and it's a 32-bit executable
> and it doesn't have a manifest to disable the functionality, Windows will
> execute the file in a separate UAC context (often in a separate Window).
> 3. Additional files are needed. In particular, due to (2), a manifest must be
> provided for 32-bit executables.
> 4. Word size accounting. It's not clear to me what word size is needed. 32-bit
> may be sufficient, though 64-bit seem to have some advantages: a manifest is
> not needed, and it can match the word size of the installed Python executable
> (for consistency). Setuptools currently provides both (and installs the one
> that matches the Python executable).
> 5. Platform support. We're fortunate that Windows is one of the most stable
> binary platforms out there. Nevertheless, Setuptools recently got support for
> AMD binaries in the launcher. By relying on an external launcher, the launcher
> becomes responsible for platform support.
> 6. Two to three files to do the job of one. In fact, the "job" isn't much more
> than to invoke code elsewhere, so it seems ugly to require as many as three
> files to do the job. Then multiply that by the Python-specific version and you
> have up to six files for a single script.
> 7. Obfuscation of purpose. A single script pretty directly communicates its
> purpose. When there are multiple files, it's not obvious why they exist or
> what their purpose is. Indeed, I went years without realizing we had an open
> issue in Distribute due to a missing manifest (which was fixed in Setuptools),
> all because I used the 64-bit executable. While it may take some time for the
> community to learn what a '.pyl' is, it's easily documented and simple to
> grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a
> side-by-side executable.
> 8. Unwanted content. Some Unix users have complained about finding Windows
> executables in their Linux packages, so now Setuptools has special handling to
> omit the launchers when installed on Unix systems. This is far from beautiful.
>
>> I think the issue of "too many extensions" vs. "source/binary confusion" is
>> going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or
>> some more Windows-specific BDFL For One PEP.
>>
>> If we go with One Extension To Rule Them All, I would actually suggest
>> '.pyl'
>> (for PyLauncher), since really all that extension does is say, "hey, run
>> this as a
>> console app via PyLauncher", not that it's a "script" (which would be
>> assumed to be text).  And that all you can be sure of is that a .pyl files
>> will
>> start with a #! line, and launch whatever other program is specified there,
>> on
>> the contents of the file
>> -- which may actually be a zipfile.
>
> If it's '.py*', I don't see why it's not reasonable to allow omission of the
> shebang, and assume the default python. After encountering and now
> understanding the subtle import semantics, I'm hoping that this new extension
> can also be used in my personal 'scripts' collection to serve the same purpose
> it does for setuptools console entry points. I guess one could require
> #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel
> at all strongly on this point.
>
>> > PS Either the ref file marker approach, or a new Python command line
>> > argument with appropriate behaviour, could avoid the need for even the
>> > pys/pws extension, if people prefer to reduce the number of extensions
>> > claimed still further.
>>
>> But those would only be available for future Python versions.  A file
>> extension would solve the problem upon installing PyLauncher and PATHEXT,
>> at least for those OSes and shells that recognize PATHEXT.
>
> Also, in my mind, this approach is most directly addressing the fundamental
> challenge (distinguishing a (executable) script from a module) in much the way
> Unix has previously enjoyed.
>
>> Hm, here's a side thought: what if PyLauncher added the ability to serve as
>> a
>> script wrapper, just like setuptools' existing wrappers?
>> Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw,
>> and presto!  No PATHEXT compatibility needed, but users could still opt out
>> of using the .exe wrappers if they're sure their shell works right without
>> it.
>>
>> (The wrapper facility would be implemented by simply checking for an
>> adjacent file of matching filename and extension (.pyl for py.exe, .pyw for
>> pyw.exe), and if found, insert that filename as argv[1] before proceeding
>> with the normal launch process.  For efficiency, the file check could be
>> skipped if the executable has its original name, at the minor cost of it not
>> being possible to name a console script 'py' or a windows app 'pyw'.  But
>> that's an optional tweak.)
>
> I'm warming up to this idea a bit, especially how it supports the most elegant
> approach but degrades gracefully. Some questions that arise:
>
> Where would Setuptools expect to find these launchers? Would it expect them to
> be present on the system?

It could, which would incidentally would address your issue #8 (people
whining about Windows in their Linux).  ;-)

Basically, if the launcher is globally installed, you don't need to
copy unless you're trying to be compatible with shells that don't
support PATHEXT properly.  If the launcher isn't installed, you'll
need it bundled, or download it on the fly from a binary distribution
dependency.


> Would it symlink or hardlink them or simply copy?

Hardlinks and symlinks are essentially useless on Windows, so copy.

>  Is
> py.exe subject to the 'looks like installer' behavior, such that it would need
> a manifest?

Yep, but it can be embedded, as Steve points out.  The only reason I
never did this is because the Force is insufficiently strong in this
one.  ;-)


> I still feel like this approach would require substantial special-casing, but
> since it provides a transition to the simple, elegant approach, I'm not
> opposed.

Well, the transition would probably go something like:

1. Start embedding pylauncher (or having a dependency on an egg or
wheel that contains the launcher binaries) to use in place of the
existing script mechanism
2. Allow making executables without copying the launcher, provided
that a global pylauncher is installed w/file association and proper
PATHEXT
3. Switch off using launcher copies by default, leave it as a backward
compatibility option

Basically step 1 gets manifest files and launcher maintenance off of
setuptools' plate, which IIUC is mainly what you want.