[Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Wes Turner]

On Thu, May 28, 2015 at 11:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 28 May 2015 at 16:58, Barry Warsaw <barry at python.org> wrote:
> > On May 28, 2015, at 11:39 AM, Donald Stufft wrote:
> >
> >>You don’t need a "fully functioning Python" for a single file binary, you
> >>only need enough to actually run your application. For example, if you're
> >>making an application that can download files over HTTP, you don't need
> to
> >>include parts of the stdlib like xmlrpc, pickle, shelve, marshall,
> sqlite,
> >>csv, email, mailcap, mailbox, imaplib, nntplib, etc.
> >
> > There are actually two related but different use cases to "single file
> > executables".
> >
> > The first is nicely solved by tools like pex, where you don't need to
> include
> > a fully functional Python at the head of the zip file because the
> environment
> > you're deploying it into will have enough Python to make the zip work.
> This
> > can certainly result in smaller zip files.  This is the approach I took
> with
> > Snappy Ubuntu Core support for Python 3, based on the current situation
> that
> > the atomic upgrade client is written in Python 3.  If that changes and
> Python
> > 3 is removed from the image, then this approach won't work.
> >
> > pex (and others) does a great job at this, so unless there are things
> better
> > refactored into upstream Python, I don't think we need to do much here.
>
> One problem with pex is that it doesn't appear to work on Windows (I
> just gave it a try, and got errors because it relies on symlinks).
>
> IMO, any solution to "distributing Python applications" that is
> intended to compete with the idea that "go produces nice single-file
> executables" needs to be cross-platform. At the moment, zipapp (and in
> general, the core support for running applications from a zip file)
> handles this for the case where you're allowed to assume an already
> installed Python interpreter. The proviso here, as Donald pointed out,
> is that it doesn't handle C extensions.
>
> The biggest problem with 3rd-party solutions is that they don't always
> support the full range of platforms that Python supports. That's fine
> for a 3rd party tool, but if we want to have a response to people
> asking how to bundle their application written in Python, we need a
> better answer than "if you're on Windows, use py2exe, or if you're on
> Unix use pex, or maybe..."
>
> Python has core support for the equivalent of Java's jar format in
> zipapp. It's not well promoted (and doesn't support C extensions) but
> it's a pretty viable option for a lot of situations.
>
> > The second use case is as you describe: put a complete functional Python
> > environment at the head of the zip file so you don't need anything in the
> > target deployment environment.  "Complete" can easily mean the entire
> stdlib,
> > and although that would usually be more bloated than you normally need,
> hey,
> > it's just some extra unused bits so who cares? <wink>.  I think this
> would be
> > an excellent starting point which can be optimized to trim unnecessary
> bits
> > later, maybe by third party tools.
>
> Tools like py2exe and cx_Freeze do this, and are pretty commonly used
> on Windows. An obvious example of use is Mercurial. If you're looking
> at this scenario, a good place to start would probably be
> understanding why cx_Freeze isn't more commonly used on Unix (AFAIK,
> it supports Unix, but I've only ever really heard of it being used on
> Windows).
>

Esky https://github.com/cloudmatrix/esky/

* supports "py2exe, py2app, cxfreeze and bbfreeze"
* builds a zip archive containing an .exe
* manages (failed) [auto-]updates

PEX https://pantsbuild.github.io/pex_design.html

* adds an executable header to a (topo-sorted?) ZIP file with a minimal path

* pipsi https://github.com/mitsuhiko/pipsi/blob/master/pipsi.py
  * installs packages with console_scripts into separate virtualenvs with
minimal sys.paths and ~/.local/bin)

At the end of the day I still need packaging or configmgmt or NIX
for checksums (a manifest wrapped around the executable wrapper).


>
> I suspect "single file executables" just aren't viewed as a desirable
> solution on Unix. Although Donald referred to a 4K binary, which
> probably means just a stub exe that depends on system-installed .so
> files, likely including Python (I'm just guessing here). It's easy to
> do something similar on Windows, but it's *not* what most Windows
> users think of when you say a "single file executable for a Python
> program" (because there's no system package manager doing dependencies
> for you).
>

NuGet, Chocolatey, -> OneGet

It's a topologically sorted adjacency list + build + install + uninstall
scripts.


>
> Again, platform-specific answers are one thing, and are relatively
> common, but having a good cross-platform answer at the language level
> (a section on docs.python.org "How to ship your Python program") is
> much harder.
>
> >>Of course deciding which pieces you include in the zip file you're
> appending
> >>to the end of Python is up to whatever tool builds this executable which
> >>doesn't need to be part of Python itself. If Python itself gained the
> ability
> >>to operate in that manner than third party tools could handle trying to
> do
> >>the optimizations where it only includes the things it actually needs in
> the
> >>stdlib and excludes things it doesn't. The key thing here is that since
> >>you're doing a single file binary, you don't need to have a Python which
> is
> >>suitable to execute random Python code, you only need one that is
> suitable to
> >>execute this particular code so you can specialize what that includes.
> >
> > I'd love to see Python itself gain such a tool, but if it had the
> critical
> > pieces to execute in this way, that would enable a common approach to
> > supporting this in third party tools, on a variety of platforms.
>
> Stripping out unused code is a hard problem in a language as dynamic
> as Python. It would be great to see it happen, but I'm not sure how
> much better we can do than existing tools like modulefinder. (consider
> that stripping out parts of the stdlib is the same in principle as
> stripping out unused bits of a 3rd party library like requests - when
> this issue comes up, people often talk about slimming down the stdlib
> to just what's needed, but why not take out the json support from
> requests if you don't use it?)
>

Is this what they do for AppEngine / AppScale Python?


>
> > I do think single-file executables are an important piece to Python's
> > long-term competitiveness.
>
> Agreed. But also, I think that "single-file" executables
> (single-directory, in practice) are *already* important - as I say,
> for projects like Mercurial. Doing better is great, but we could do
> worse than start by asking the Mercurial/TortoiseHg project and others
> what are the problems with the current situation that changes to the
> core could help to improve. I doubt "please make pythonXY.zip 50%
> smaller" would be the key issue :-)
>

"Select your platform" (According to User-Agent)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/2a600729/attachment.html>