[Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Donald Stufft]

On May 28, 2015 at 12:01:22 PM, Barry Warsaw (barry at python.org) wrote:
> On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
>  
> >You don’t need a "fully functioning Python" for a single file binary, you
> >only need enough to actually run your application. For example, if you're
> >making an application that can download files over HTTP, you don't need to
> >include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite,
> >csv, email, mailcap, mailbox, imaplib, nntplib, etc.
>  
> There are actually two related but different use cases to "single file
> executables".
>  
> The first is nicely solved by tools like pex, where you don't need to include
> a fully functional Python at the head of the zip file because the environment
> you're deploying it into will have enough Python to make the zip work. This
> can certainly result in smaller zip files. This is the approach I took with
> Snappy Ubuntu Core support for Python 3, based on the current situation that
> the atomic upgrade client is written in Python 3. If that changes and Python
> 3 is removed from the image, then this approach won't work.
>  
> pex (and others) does a great job at this, so unless there are things better
> refactored into upstream Python, I don't think we need to do much here.

Pex would be improved by having native support for importing .so’s from within
a zipfile via zipimport. It would also be improved by having good, built in
support for extraneous resources in the stdlib too. It’s doing pretty well on
it’s own though besides those quality of life improvements.

>  
> The second use case is as you describe: put a complete functional Python
> environment at the head of the zip file so you don't need anything in the
> target deployment environment. "Complete" can easily mean the entire stdlib,
> and although that would usually be more bloated than you normally need, hey,
> it's just some extra unused bits so who cares? . I think this would be
> an excellent starting point which can be optimized to trim unnecessary bits
> later, maybe by third party tools.
>  
> >Of course deciding which pieces you include in the zip file you're appending
> >to the end of Python is up to whatever tool builds this executable which
> >doesn't need to be part of Python itself. If Python itself gained the ability
> >to operate in that manner than third party tools could handle trying to do
> >the optimizations where it only includes the things it actually needs in the
> >stdlib and excludes things it doesn't. The key thing here is that since
> >you're doing a single file binary, you don't need to have a Python which is
> >suitable to execute random Python code, you only need one that is suitable to
> >execute this particular code so you can specialize what that includes.
>  
> I'd love to see Python itself gain such a tool, but if it had the critical
> pieces to execute in this way, that would enable a common approach to
> supporting this in third party tools, on a variety of platforms.

Right, it would be great to get it built into Python itself, but I consider that
less important than getting the critical pieces into Python. If those pieces are
there then we can iterate outside of the standard library and try different
approaches to *building* such a file, and eventually take a look at the landscape
and bless one approach (or not, if we don’t want to).

>  
> I do think single-file executables are an important piece to Python's
> long-term competitiveness.
>  

I completely agree. I talk to a lot of people about packaging of things, and while
I think there are some serious problems with huge parts of Go’s packaging and
distribution story the static linking and compiling down to a “single” file is not
one of them. People *really* enjoy this and it simplifies a ton of things for people.
They don’t have to worry about making sure a whole set of files are deployed, they
don’t have to worry about what version of Python is installed (or if any version is
installed), they don’t have to worry about what other things have been installed, they
just copy an executable and then they are good to go.

I think we could potentially do *better* than Go in this regard, because we can make
it possible to do both “static” and “dynamic” dependencies, as well as provide the
ecosystem around that to do things like provide visibility into what versions of
libraries are “compiled” into that single file executable, etc.

---  
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA