[Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

Thomas Wouters thomas at python.org
Fri May 29 04:04:52 CEST 2015

On Thu, May 28, 2015 at 6:13 PM, Barry Warsaw <barry at python.org> wrote:

> Go seems to be popular where I work.  It is replacing Python in a number of
> places, although Python (and especially Python 3) is still a very important
> part of our language toolbox.
> There are several reasons why Go is gaining popularity.  Single-file
> executables is definitely a reason; it makes deployment very easy, even if
> it
> increases the maintenance burden (e.g. without shared libraries, you have
> multiple copies of things so when a security fix is required for one of
> those
> things you have to recompile the world).
> Start up times and memory footprint are also factors.  Probably not much
> to be
> done about the latter, but perhaps PEP 432 can lead to improvements in the
> former.  (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)
> Certainly better support for multi-cores comes up a lot.  It should be a
> SMoE
> to just get rid of the GIL once and for all <wink>.
> One thing I've seen more than once is that new development happens in
> Python
> until the problem is understood, then the code is ported to Go.  Python's
> short path from idea to working code, along with its ability to quickly
> morph
> as requirements and understanding changes, its batteries included
> philosophy,
> and its "fits-your-brain" consistency are its biggest strengths!
> On May 28, 2015, at 10:37 AM, Donald Stufft wrote:
> >I think docker is a pretty crummy answer to Go’s static binaries. What I
> would
> >love is for Python to get:
> >
> >* The ability to import .so modules via zipzimport (ideally without a
> >temporary   directory, but that might require newer APIs from libc and
> such).
> +1 - Thomas Wouters mentioned at the language summit some work being done
> on
> glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
> files directly from a zip.  Not sure what the status is of that, but it
> would
> be a great addition.

It's dlopen_with_offset:
https://sourceware.org/bugzilla/show_bug.cgi?id=11767. There's also a patch
that's sort-of dlopen_from_memory (dlopen_ehdr), but it requires a lot of
manual setup to map the right bits to the right places; dlopen_with_offset
is a lot simpler.

Building a Python application into a single file doesn't require
dlopen_with_offset, *iff* you build everything from source. It's not easy
to do this -- Python's setup.py and third-party's uses of distutils don't
allow this -- but it's mostly possible using the old Modules/Setup file.
(Or you could do what we routinely do at Google with third-party packages
and re-implement the build in your own build system ;P)

> >* The ability to create a “static” Python that links everything it needs
> into
> >the binary to do a zipimport of everything else (including the stdlib).
This is possible (with some jumping through hoops) using Modules/Setup and
some post-processing of the standard library. It would be a lot easier if
we got rid of distutils for building Python (or for everything) -- or made
it output a Modules/Setup-like file :) (For those who don't remember,
Modules/Setup was the file we used to build stdlib extension modules before
we had distutils, and it's parsed and incorporated into the regular
Makefile. It can build both static and dynamic extension modules.)

> +1
> >*The ability to execute a zipfile that has been concat onto the end of
> the
> >Python binary.

This is already possible, just not with the regular 'python' binary. All it
takes is fifty lines of C or so, a tiny application that embeds Python,
sets sys.path[0] to argv[0], and uses the runpy module to execute something
from the ZIP file. There are some issues with this approach (like what
sys.executable should be :) but they're mostly cosmetic

> +1
> >I think that if we get all of that, we could easily create a single file
> >executable with real, native support from Python by simply compiling
> Python
> >in that static mode and then appending a zip file containing the standard
> >library and any other distributions we need to the end of it.
> >
> >We’d probably want some more quality of life improvements around accessing
> >resources from within that zip file as well, but that can be done as a
> >library easier than the above three things can.
> E.g. you really should be using the pkg_resources APIs for loading
> resources
> from your packages, otherwise you're gonna have problems with zip
> executables.  We've talked before about adopting some of these APIs into
> Python's stdlib.  pkgutil is a start, and the higher level APIs from
> pkg_resources should probably go there.
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/thomas%40python.org

Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/1f9afe42/attachment-0001.html>

More information about the Python-Dev mailing list