[Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Paul Moore]

On 28 May 2015 at 17:13, Barry Warsaw <barry at python.org> wrote:
> On May 28, 2015, at 10:37 AM, Donald Stufft wrote:
>
>>I think docker is a pretty crummy answer to Go’s static binaries. What I would
>>love is for Python to get:
>>
>>* The ability to import .so modules via zipzimport (ideally without a
>>temporary   directory, but that might require newer APIs from libc and such).
>
> +1 - Thomas Wouters mentioned at the language summit some work being done on
> glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
> files directly from a zip.  Not sure what the status is of that, but it would
> be a great addition.

+1 but it needs to be cross-platform - py2exe has something similar
for Windows, which we should expect to use if the glibc solution is
Unix-only.

>>* The ability to create a “static” Python that links everything it needs into
>>the binary to do a zipimport of everything else (including the stdlib).
>
> +1

+0 - I suppose it would be nice, but how important is this really? If
all we do is end up with a single massive file instead of a directory,
then I don't see we've gained much.

Smallish C programs tend to hit the 10-100k executable size. Offhand I
don't know how big a typical go "single file executable" is. If I
could bundle up my Python script (something simple, let's say a "Hello
world" for argument's sake) and get a 10-20k single-file executable,
this would be worthwhile. If the resulting exe was 5MB (which is what
today's solutions that bundle the Python DLL and full zipped stdlib
tend to weigh in at) then I'm not interested.

>>*The ability to execute a zipfile that has been concat onto the end of the
>>Python binary.
>
> +1

+1. This would be immensely useful to tack on the front of a pyz
archive. Even just the existing (39K on Windows) python.exe which
relies on the Python installation would be great. Basically it could
replace the setuptools/distlib "script wrapper" executables with
something that doesn't need to run 2 processes just to run a script.

>>We’d probably want some more quality of life improvements around accessing
>>resources from within that zip file as well, but that can be done as a
>>library easier than the above three things can.
>
> E.g. you really should be using the pkg_resources APIs for loading resources
> from your packages, otherwise you're gonna have problems with zip
> executables.  We've talked before about adopting some of these APIs into
> Python's stdlib.  pkgutil is a start, and the higher level APIs from
> pkg_resources should probably go there.

+1. We need the APIs available in the stdlib, then 3rd party libraries
have a solution when users complain "your library isn't zipimport
safe". (Of course "well, we have to support Python 2.7/3.4" remains a
response for some time yet :-() Then we can start working on the
culture change where library authors start expecting their code to be
deployed in single-file "run-from-zip" applications.

Paul