[Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/29/2015 3:33 AM, Paul Moore wrote:
> On 28 May 2015 at 22:09, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> This would be something I could use and benefit from immediately upon it
>> being available, so I laud your idea, and hope you have a successful
>> implementation, and look forward to using it.  It would largely replace the
>> need for the py.exe launcher for some classes of applications.
> The following proof-of-concept works as is (based on my pretty minimal
> testing), and only uses the limited API, so it should work with any
> version of Python 3 (I've not tested it with Python 2, but I think the
> only "new" API is PySys_SetArgvEx, which could be replaced with
> PySys_SetArgv at a pinch). Excuse the dreadful coding style and lack
> of error handling, I hacked it up in about an hour :-)

I have no particular interest in Python 2, having started with Python 3, 
and only used Python 2 for cases where dependent packages required it, 
and I've now reached the nirvana of all my dependency packages being 
ported to Python 3, although I have yet to port/test one remaining 
application to prove that. So I only mentioned Python 2 because it still 
could be useful for other people :)

> (Actually, I just tried building on Python 2 - guess what - Unicode
> :-) SetProgramName and SetArgvEx won't take Unicode values. The easy
> fix is just not to use Unicode, the hard one is to do the encoding
> dance, but I'm not going to bother...).

One approach would be to support Unicode arguments only for Python 3, 
but that would really be only paying lip service to Python 2 support.  
Another approach might be to not #define UNICODE for the Python 2 
version, and use the 8-bit Windows APIs, allowing Windows and the C 
runtime to do the encoding dance for you? Although I'm not sure what 
Python 2 requires in that respect.

> #define UNICODE
> #define _UNICODE
> #include <Python.h>
> #include <windows.h>
>
> int
> main()
> {
>      TCHAR program[MAX_PATH];
>      LPWSTR *argv;
>      int argc;
>      PyObject *runpy;
>      PyObject *ret;
>
>      argv = CommandLineToArgvW(GetCommandLineW(), &argc);
>      GetModuleFileName(NULL, program, MAX_PATH);
>      Py_SetProgramName(program);  /* optional but recommended */
>      Py_Initialize();
>      PySys_SetArgvEx(argc, argv, 0);
>      runpy = PyImport_ImportModule("runpy");
>      if (!runpy) PyErr_Print();
>      ret = PyObject_CallMethod(runpy, "run_path", "u", program);
>      if (!ret) PyErr_Print();
>      Py_Finalize();
>      return 0;
> }

That looks interesting, I wonder what compilation environment it would 
need?  I don't think I've even installed a C compiler on my last couple 
boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, 
since I've moved to using Python for anything a 5 line batch file can't 
do...

> One mildly annoying thing is that python3.dll is only installed in
> <python install dir>\DLLs, which typically isn't on PATH.
Ah, linking.... so I guess if I figured out how to create this binary, 
it would contain a reference to python3.dll that would attempt to be 
resolved via the PATH, from what you say, and typically fail, due to 
PATH seldom containing python3.dll.  The python launcher gets around 
that by (1) being installed in %windir%, and going and finding the 
appropriate Python (per its own configuration file, and command line 
parameters), and setting up the path to that Python, which, when 
executed, knows its own directory structure and can thus find its own 
python3.dll.

The launcher, of course, adds an extra layer of process between the 
shell and the program, because it launches the "real" Python executable.

> So actually using the limited API from your own application fails by default.
> Fixing that's mostly a user admin issue, though (and you can just link
> to the full API and avoid the whole problem).

Do I understand correctly that the "user admin issue" means "add the 
appropriate <python install dir>\DLLs to the PATH"?

What I don't understand here is how linking to the full API avoids the 
problem... it must put more python library code into the stub 
executable? Enough to know how to search the registry to find the 
<python install dir> for the version of Python from which the full API 
was obtained? Or something else?

Are there other alternatives?  Assuming that the reference to the 
missing DLL is not required until the point at which a symbol from it is 
first referenced, so that the stub would have some ability to do 
something before that first call, maybe...

 1. The stub above could be enhanced to contained a "hard coded"
    directory that it adds to the PATH itself?
 2. The stub above could be enhanced to define that its first parameter
    is the <python install dir>, and tweak its PATH.
 3. These days, the Python installer does offer to optionally add itself
    to the PATH. Is that sufficient to make the stub work?
 4. The launcher could be used, assuming it is installed, but then you
    don't need a stub, and you get the extra process layer.
 5. stubpy.cmd could be created, a four line batch file below [1], which
    wouldn't require the launcher or its extra process layer, but would
    have to be placed on the PATH itself, or in the directory with the
    stub Python programs.

Only #3 could be construed as "easy" for the "dumb user"... if the 
Python installer offers to add itself to the PATH on "repair" installs, 
particularly (I'm not sure if it does).  Editing the System PATH through 
the  control panel is hard for the "dumb user", not made easier by the 
squinchy text box M$ provides for the editing. Nor is editing the System 
PATH made less error prone by the whole thing being available for 
editing, rather than the "GUI promotors" providing an editing editing 
interface such as displaying each item separately, with checkboxes to 
delete items, or insert items at particular locations, and directory 
selection dialogs rather than typing the desired new path as text.  Hmm. 
Sounds like a good program task for a stub Python program :) Except it 
doesn't bootstrap, unless it lives in <python install dir>.

[1] stubpy.cmd:
@setlocal
@PATH=<python install dir>;%PATH%
@shift
@%*

>
>> Of course, per other disccusions, this doesn't solve the problem for:
>>
>> A) machine without Python installed
>> B) programs that need binary extensions
>>
>> Other discussions have suggested:
>>
>> 3) The stub could offer to download and install Python
>>
>> A corollary:
>>
>> 4) The stub could offer to download and install the needed binary extensions
>> as well as Python. This would require the installation uniformity of
>> something like pip, so perhaps would be restricted to extensions available
>> via pip.  And it would be much enhanced by some technique where the zipapp
>> would contain metadata readable by the stub, that would declare the list of
>> binary extensions required.  Or, of course, it could even declare non-binary
>> extension that are not packaged with the zipapp, if the process is smooth,
>> the modules available via pip, etc., as a tradeoff.
> I'm pretty strongly against downloading interpreters or extensions.
> Apart from the pretty huge added complexity, as a user I'm not sure
> I'd trust a supposedly simple application I'd received if it started
> asking to download stuff unexpectedly...
>
> Paul

Yep. I mostly mentioned them for completeness.

I have no argument with this: installing Python can be a documentation 
thing.  "To use this program, you need to have Python installed, at 
least version N.M."  Maybe also : "and it should be installed on the 
System PATH, or other methods of setting the PATH before running this 
program must be used, or this program should be saved in the <python 
install dir>."

If stub+zip programs need other extensions, it can be documented as a 
batch file that calls pip a sufficient number of times with appropriate 
parameters, or the stub+zip program itself could be written to detect 
the needed but missing extensions, and invoke pip to get them before 
using them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/e73ce9f1/attachment.html>