On May 28, 2015, at 11:39 AM, Donald Stufft wrote:

You don’t need a "fully functioning Python" for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc.

There are actually two related but different use cases to "single file executables". The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. "Complete" can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? <wink>. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools.

Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes.

I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms. I do think single-file executables are an important piece to Python's long-term competitiveness. Cheers, -Barry

On Thu, May 28, 2015 at 11:38 AM, Paul Moore wrote:

...

On May 28, 2015, at 11:39 AM, Donald Stufft wrote:

...

You don’t need a "fully functioning Python" for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc.

There are actually two related but different use cases to "single file executables".

The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation

On 28 May 2015 at 16:58, Barry Warsaw wrote: that

...

the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work.

pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here.

One problem with pex is that it doesn't appear to work on Windows (I just gave it a try, and got errors because it relies on symlinks).

IMO, any solution to "distributing Python applications" that is intended to compete with the idea that "go produces nice single-file executables" needs to be cross-platform. At the moment, zipapp (and in general, the core support for running applications from a zip file) handles this for the case where you're allowed to assume an already installed Python interpreter. The proviso here, as Donald pointed out, is that it doesn't handle C extensions.

The biggest problem with 3rd-party solutions is that they don't always support the full range of platforms that Python supports. That's fine for a 3rd party tool, but if we want to have a response to people asking how to bundle their application written in Python, we need a better answer than "if you're on Windows, use py2exe, or if you're on Unix use pex, or maybe..."

Python has core support for the equivalent of Java's jar format in zipapp. It's not well promoted (and doesn't support C extensions) but it's a pretty viable option for a lot of situations.

...

The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. "Complete" can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? <wink>. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools.

Tools like py2exe and cx_Freeze do this, and are pretty commonly used on Windows. An obvious example of use is Mercurial. If you're looking at this scenario, a good place to start would probably be understanding why cx_Freeze isn't more commonly used on Unix (AFAIK, it supports Unix, but I've only ever really heard of it being used on Windows).

Esky https://github.com/cloudmatrix/esky/ * supports "py2exe, py2app, cxfreeze and bbfreeze" * builds a zip archive containing an .exe * manages (failed) [auto-]updates PEX https://pantsbuild.github.io/pex_design.html * adds an executable header to a (topo-sorted?) ZIP file with a minimal path * pipsi https://github.com/mitsuhiko/pipsi/blob/master/pipsi.py * installs packages with console_scripts into separate virtualenvs with minimal sys.paths and ~/.local/bin) At the end of the day I still need packaging or configmgmt or NIX for checksums (a manifest wrapped around the executable wrapper).

I suspect "single file executables" just aren't viewed as a desirable solution on Unix. Although Donald referred to a 4K binary, which probably means just a stub exe that depends on system-installed .so files, likely including Python (I'm just guessing here). It's easy to do something similar on Windows, but it's *not* what most Windows users think of when you say a "single file executable for a Python program" (because there's no system package manager doing dependencies for you).

NuGet, Chocolatey, -> OneGet It's a topologically sorted adjacency list + build + install + uninstall scripts.

Again, platform-specific answers are one thing, and are relatively common, but having a good cross-platform answer at the language level (a section on docs.python.org "How to ship your Python program") is much harder.

...

...

Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes.

I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms.

Stripping out unused code is a hard problem in a language as dynamic as Python. It would be great to see it happen, but I'm not sure how much better we can do than existing tools like modulefinder. (consider that stripping out parts of the stdlib is the same in principle as stripping out unused bits of a 3rd party library like requests - when this issue comes up, people often talk about slimming down the stdlib to just what's needed, but why not take out the json support from requests if you don't use it?)

Is this what they do for AppEngine / AppScale Python?

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Agreed. But also, I think that "single-file" executables (single-directory, in practice) are *already* important - as I say, for projects like Mercurial. Doing better is great, but we could do worse than start by asking the Mercurial/TortoiseHg project and others what are the problems with the current situation that changes to the core could help to improve. I doubt "please make pythonXY.zip 50% smaller" would be the key issue :-)

"Select your platform" (According to User-Agent)

On May 28, 2015 at 12:01:22 PM, Barry Warsaw (barry@python.org) wrote:

On May 28, 2015, at 11:39 AM, Donald Stufft wrote:

...

You don’t need a "fully functioning Python" for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc.

There are actually two related but different use cases to "single file executables".

The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work.

pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here.

Pex would be improved by having native support for importing .so’s from within a zipfile via zipimport. It would also be improved by having good, built in support for extraneous resources in the stdlib too. It’s doing pretty well on it’s own though besides those quality of life improvements.

The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. "Complete" can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? . I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools.

...

Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes.

I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms.

Right, it would be great to get it built into Python itself, but I consider that less important than getting the critical pieces into Python. If those pieces are there then we can iterate outside of the standard library and try different approaches to *building* such a file, and eventually take a look at the landscape and bless one approach (or not, if we don’t want to).

I do think single-file executables are an important piece to Python's long-term competitiveness.

I completely agree. I talk to a lot of people about packaging of things, and while I think there are some serious problems with huge parts of Go’s packaging and distribution story the static linking and compiling down to a “single” file is not one of them. People *really* enjoy this and it simplifies a ton of things for people. They don’t have to worry about making sure a whole set of files are deployed, they don’t have to worry about what version of Python is installed (or if any version is installed), they don’t have to worry about what other things have been installed, they just copy an executable and then they are good to go. I think we could potentially do *better* than Go in this regard, because we can make it possible to do both “static” and “dynamic” dependencies, as well as provide the ecosystem around that to do things like provide visibility into what versions of libraries are “compiled” into that single file executable, etc. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 5/28/2015 12:44 PM, Donald Stufft wrote:

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

I completely agree. I talk to a lot of people about packaging of things, and while I think there are some serious problems with huge parts of Go’s packaging and distribution story the static linking and compiling down to a “single” file is not one of them.

How about the following compromise between assuming the presence of installed python and ignoring the possibility of installed python: we provide (as least for Windows) a C startup file that checks for the needed version of python and offers to download and start the installer if not present. A user would see 'Python 3.5 is needed to run this file; shall I download and install it?' just once on a particular machine. -- Terry Jan Reedy

I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? Barry Warsaw wrote:

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? -Chris On Thu, May 28, 2015 at 8:39 AM, Donald Stufft wrote:

On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.dower@microsoft.com) wrote:

...

Donald Stufft wrote:

...

Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need.

For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...):

* python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB

This gets you to ">>>", and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB.

The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.)

Cheers, Steve

You don’t need a "fully functioning Python" for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc.

Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes.

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 28 May 2015 at 17:28, Chris Barker wrote:

On Thu, May 28, 2015 at 9:23 AM, Chris Barker wrote:

...

Barry Warsaw wrote:

...

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable?

oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why "single file" is critical, over "single thing to install" -- like a OS-X app bundle that can just be dragged into the Applications folder.

On Windows, there's a strong interest in "no install" download-and-run applications (.Net has "xcopy deployment", portable applications are a big deal to some people). "Just unzip the distribution somewhere and run the exe" is definitely a selling point for that audience. And "download an exe and just run it" is even more so. But the latter is definitely an incremental improvement - the former is the big deal (IMO). Paul

On Fri, May 29, 2015 at 2:28 AM, Chris Barker wrote:

oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why "single file" is critical, over "single thing to install" -- like a OS-X app bundle that can just be dragged into the Applications folder.

There's also "single thing to uninstall", which IMO is more important. If I download a tiny program that's supposed to just do one tiny thing, and it has to install itself into Program Files, Common Files, Windows\System32, and Documents & Settings\my-user-name\Applications, then I have to hope it has a proper uninstaller. If it's a single executable that just does its stuff (or, failing that, a single zip file that I extract to anywhere and run the program), I can expect that deleting that file (or directory) will get rid of it all. Of course, it's entirely possible that it's gone and left its droppings all over the system, but that's a matter of trust - a legit program won't lie about that. Is this a Windows-specific issue, or is it also intended for Linux and Mac OS, where there'll already be a system Python (so a single-file-executable would be used to be independent of the system Python)? ChrisA

On Fri, May 29, 2015 at 3:20 AM, Donald Stufft wrote:

On May 28, 2015 at 12:54:34 PM, Chris Angelico (rosuav@gmail.com) wrote:

...

Is this a Windows-specific issue, or is it also intended for Linux and Mac OS, where there'll already be a system Python (so a single-file-executable would be used to be independent of the system Python)?

I think it’s an issue for all platforms, even when there is a system Python that can be used.

Here’s why:

* Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed.

* On OS X you have the system Python yes, but that is in an unknown state. It could have any number of changes made to it or things installed or what have you.

* Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3?

* What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected?

The independence argument. Yep, reasonable. The trouble is that if everyone seeks to be independent of the system Python, it makes the system Python redundant with every single Python app; and the bundled Python will have to be platform-specific (not just OS, but architecture, word size, possibly OS version in some cases, etc, etc). And that independence also means you miss out on security/compatibility updates from upstream, so you're having to manage your own updates. I'd still much rather see a small stub that goes and calls on a system Python than something that has to duplicate all of Python into every binary, but it's a choice for the application developer to make. If there can at least be an easy way to strip off the self-executability header and get back the base Python application, that would be a big help - you could have a script that goes through all your executables, looks for the signature that says "hi, I'm a Python 2.7.10 self-runner", and strips it off in favour of a 2.7.11 that fixes some critical issue. Or you could strip that off and run the underlying Python code on a different OS. Binary blobs are so often unworkable. ChrisA

On 29 May 2015 at 09:36, Steven D'Aprano wrote:

The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. Maybe you move in different circles (perhaps more enterprisey?), but I can already imagine the sort of derogatory comments the sys admins I work with would make about this idea.

In my environments, we frequently have ancient versions of RHEL installed, sometimes with no Python at all (IIRC) or nothing better than 2.4. The sysadmins won't install newer versions, as Python isn't formally needed, but we'd happily use it for adhoc admin-style scripts (the alternative is typically shell scripts or nothing). It's not precisely acceptable, but being able to create a single-file small executable in the support user's home directory made from an admin script would be immensely useful. It's hardly a core use case, and we generally just live with shell scripts, but such environments *do* exist :-( Paul

On May 29, 2015 at 4:37:37 AM, Steven D'Aprano (steve@pearwood.info) wrote:

On Thu, May 28, 2015 at 01:20:06PM -0400, Donald Stufft wrote:

...

I think it’s an issue for all platforms, even when there is a system Python that can be used.

Here’s why:

* Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed.

Donald, are you a Linux user? If so, which distro? Because in the Linux world that I'm familiar with, this (and the points you make below) are absolutely not an issue. You let the package manager worry about dependencies:

yum install myapp # Red Hat based distros apt-get install myapp # Debian based distros

will ensure that the right version of Python is installed.

In the circles I move in, installing anything which does not go through the package manager is considered to be quite dubious. In order of preference (best to worst):

- install from official distro repositories; - install from a third-party repo; - install from source; - find an alternative application; - do without; - install from some binary format (also known as "would you like a root kit with that?").

[...]

...

* Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3?

Most recent distros have both a python2 and python3 package, and when building your rpm or deb file, you specify which is your dependency in the normal fashion.

...

* What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected?

Normally you would write for the version of Python provided by the distros you wish to support. In practice that might mean writing hybrid code targetting (say) 2.6 and 2.7, which covers most recent Red Hat and Debian based systems, and anything else, you provide the source code and let the user work it out.

I suppose that in principle you could include whatever version of Python you like *in your application*, but that would be considered an unusual thing to do. I believe that LibreOffice and OpenOffice do that, so they can support Python as a scripting language, but they're generally considered (1) a special case because they're cross-platform, and (2) not "proper" Unix or Linux apps anyway.

The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. Maybe you move in different circles (perhaps more enterprisey?), but I can already imagine the sort of derogatory comments the sys admins I work with would make about this idea.

I use Linux for servers yes, I don't stick with a single Distribution and right now I manage services that are running on CentOS, Ubuntu, Debian, Alpine, and FreeBSD (not a Linux, but w/e).  Here's the thing though, when you make software that is designed for other people to consume you have two choices. Either you have to try to anticipate every single environment that they might possibly run it in and what is available there so that your software either runs there or you can provide instructions on how to run it there, or you need to depend on as little from the OS as possible. An example of a product that does this is Chef, they install their own Ruby and everything but libc into /opt/chef to completely isolate themselves from the host system. I’m told this made things *much* easier for them as they don't really have to worry at all about what's available on the host system, Chef pretty much just works. Another example is one that I personally worked on recently, where the company I worked for wanted to distribute a CLI to our customers which would "just work" that they could use to interact with the service we provided. We were writing this client in Python and it was very painful to satisfy the requirement that it work without Python installed as a single file executable. That was a hard requirement as anything else would be more difficult to distribute to end users and be more likely to break. The entire time I was working on that particular piece I was feeling like I should be suggesting to my manager that we throw away the Python code and just write it in Go. As folks may or may not know, I'm heavily involved in pip which is probably one of the most widely used CLIs written in Python. A single file executable won't help pip, however through my experience there I can tell you that a significant portion of our issues come from random weird differences in how a particular person's Python environment is setup. If someone is distributing a CLI app written in Python, being able to remain independent from the OS eliminates a whole class of problems. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Fri, May 29, 2015 at 7:23 AM, Paul Sokolovsky wrote:

...

An example of a product that does this is Chef, they install their own Ruby and everything but libc into /opt/chef to completely isolate themselves from the host system.

this sounds a bit like what conda does -- install miniconda, and a conda environment set up with a yaml file,, and away you go. not small, but quite self contained, and give you exactly what you want. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On May 28, 2015 at 12:24:42 PM, Chris Barker (chris.barker@noaa.gov) wrote:

I'm confused:

Doesn't py2exe (optionally) create a single file executable?

And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance)

Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib).

I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-)

Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case.

So is something different being asked for here?

All of those solutions “work” to varying degrees of work, almost all of them rely on hacks in order to make things “work” because the ability to do it isn’t built into Python itself. If the critical pieces to execute in this way was built into Python itself, then those tools would work a whole lot better than they currently do.

Barry Warsaw wrote:

...

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable?

The desktop isn’t dying, Mobile is becoming a very important thing of course, but that’s just because people are using devices *more* to account for the use of Mobile, they aren’t really using their Desktop’s less. See: http://blogs.wsj.com/cmo/2015/05/26/mobile-isnt-killing-the-desktop-internet...

And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ?

It’s not the same problem, but it’s also not very relevant. Volunteer time isn’t fungible, you get what people are willing to work on regardless of whether it will help Python as a whole. It’s also not an either/or proposition, we can both improve our ability to develop under iOS/Android/etc and improve our ability to handle desktop applications. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Thu, May 28, 2015 at 3:25 PM Chris Barker wrote:

OK, I'm really confused here:

1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization)

Because Go is much easier to use to write those CLI apps than C or C++. The barrier now is low enough that the ease of development plus the ability to do statically compiled binaries is enticing folks to drop Python and Ruby for Go when making a CLI app.

2) Why the sudden interest in this as core a Python issue? I've been using Python for desktop apps, on primarily Windows and the Mac for years -- and had to deal with py2exe, py2app, etc. forever. And it has been a very very common question on the various mailing lists for ages: how do I deploy this? how do I make it easy to install? The answer from the developers of cPython itself has always been that that's a third party problem -- and go look for py2exe and friends to solve it. And that it is a solved-enough problem. The biggest "unsolved" issues are that you get a really big application.

Anecdotal evidence suggests Go's user base has a decent amount of converts from Python and it's currently the only language that seems to be siphoning people out of our community. You do hear stories of people skipping Python 3 and going to Go, but considering how much more work that is than writing Python 2/3 code I believe that typically happens when the organization wanted to jump ship anyway and the Python 3 transition just gave them an excuse to rewrite their stuff (plus we all know how enticing it can be to play with a shiny new piece of tech if given the chance).

Don't get me wrong -- I've wanted for years for it to be easier to deploy python-based apps as a single thinking for users to easily install and uninstall where they don't need to know it's python -- but what the heck is different now?

Active user loss where the biggest reason people are leaving that we can actively fix now is easy app deployment (the other is performance; some might argue concurrency but concurrent.futures and async/await water that down somewhat for me).

3) There was mention of a platform-neutral way to do this. Isn't that simply impossible? The platforms are different in all the ways that matter for this problem: both technical differences, and conventions. Which isn't to say you couldn't have one API to produce a single "thing" executable, so it would look like one solution for multiple platforms to the user. But the end product should be (would have to be) a different beast altogether.

I think it's to have a single tool to do it for any platform, not to have the technical nuts and bolts be the same necessarily. I think it's also to figure out if there is anything the interpreter and/or stdlib can do to facilitate this. -Brett

On 5/28/2015 4:29 PM, Paul Moore wrote:

On 28 May 2015 at 20:47, Brett Cannon wrote:

...

I think it's to have a single tool to do it for any platform, not to have the technical nuts and bolts be the same necessarily. I think it's also to figure out if there is anything the interpreter and/or stdlib can do to facilitate this.

Precisely. At the moment, the story seems to be "if you're on Windows, use py2exe, if you're on OSX use py2app, or on Unix, ..., or..."

What would be a compelling story is "to build your app into a single file executable, do "python -m build <myapp>". The machinery behind the build can be as different as necessary - but being able to use the same command on every platform is the goal.

The python-based ren'py visual novel development system has something like this When one is ready to publish, there is an easy option to build single-file downloadable redistributables for any or all of Windows, Linux, and Mac. I know it works as far as it goes because I help my wife use the system not for a novel, but a photo-based tutorial. After testing the resulting files with the help of others on linux and mac systems (we developed on Windows), she put the files up on one of her university pages. As far as I know, the build code should be Python, if anyone want to look at it. -- Terry Jan Reedy

On Fri, May 29, 2015 at 07:08:43AM +1000, Nick Coghlan wrote:

On 29 May 2015 05:25, "Chris Barker" wrote:

...

OK, I'm really confused here:

1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization)

For scientific Python folks, the equivalent conversations I have are about Julia.

If you're not used to thinking of Python's competitive position as "best orchestration language, solid competitor in any given niche", then the rise of niche specific competitors like Go & Julia can feel terrifying, as the relatively narrow user base changes the trade-offs you can make in the language & ecosystem design to better optimise them for that purpose.

We've been there before, with the "Ruby is the Python-killer" FUD of a few years ago. If Go is different and does overtake Python, I think it will be due to its privileged position on Android. Personally, I don't pay a lot of attention to language popularity statistics. Who cares whether Python is used by 8% of projects or 10% of projects? Either way, it's huge. But from time to time, it might be useful to look at a few different measurements of popularity. According to CodeEval, Python is still *by far* the most popular language, at 31.2% (second place is below 20%), with Go at #9 with 2.3%. According to Redmonk, Python is stable at #4, while Go has jumped from #21 to #17 in six months. LangPop gives various different measures of popularity, and according to the overall summary, Python is at #6, but Go doesn't appear to be a language they look at. TIOBE has Python moving up two places to #6, and Go (which was the 2009 "Hall Of Fame" winner) doesn't even appear in the top 100. http://blog.codeeval.com/codeevalblog/2015 http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15/ http://langpop.com/ http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html I think there are some exciting and interesting languages coming up: Swift, Julia, Go, Rust and others. Why are we threatened by this? Python makes a wonderful glue language. It would be great for Python to glue to more than just C and Fortran code. For scientific users, imagine being able to call Julia code from Python, and vice versa. Instead of thinking of Go as an opponent to beat, wouldn't it be great to be able to write extensions in a modern language like Go, Rust or D instead of creaky old C with all its safety issues? -- Steve

On Thu, May 28, 2015 at 11:23 AM, Chris Barker wrote:

I'm confused:

Doesn't py2exe (optionally) create a single file executable?

And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance)

Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib).

I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-)

Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case.

So is something different being asked for here?

Barry Warsaw wrote:

...

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable?

Donald mentioned one earlier: command line utilities. I want a single CLI I can deploy to my customers that doesn't make them have to install Python or even know it's Python at all. My users write code in all types of languages on all OSes, but I should be able to produce one thing that they can all use. Donald himself initiated the CLI in particular I'm talking about, but Go is picking up steam here as we have other utilities that quickly solved the "write one thing, every user can run it immediately, no one knows/cares what it's written in" When I worked on Ubuntu One, I was the Windows guy responsible for making sure the end-user experience was the same there as it was on Ubuntu. On Ubuntu we were a part of the base install and didn't have to worry about much. On Windows we had none of that, not even the C runtime, so we had some pre-installer work to do, and then a bunch of py2exe hacking to make everything play nicely and transparently.

On 28 May 2015 at 18:04, Brian Curtin wrote:

Donald mentioned one earlier: command line utilities. I want a single CLI I can deploy to my customers that doesn't make them have to install Python or even know it's Python at all.

Yep, that's the killer for me as well. I know it's unrealistic in some sense, but my benchmark is what does a "Hello, world" program look like? In C, it's a 38K executable with no external dependencies. What does it look like in Python? (I'm not too worried if it's a bit bigger, but if it's a *lot* bigger that starts to be noticeable - "Python generates bloated exes"). What I'd like to be able to do is to write Python ports of a range of core Unix utilities (comm, cut, join, od, seq, tr, ...) and have them be viable alternatives to my current C builds (a series of single-file 100-200k static exes). On 28 May 2015 at 18:15, Chris Angelico wrote:

Unix-like systems have this courtesy of the shebang, so as long as there's some sort of Python installed, people don't need to know or care that /usr/local/bin/mailmail is implemented in Python. Maybe the solution is to push for Windows to always include a Python interpreter, which would allow a tiny stub to go and call on that?

Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a "first-class" executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes. Having said that, the trick of appending a zipfile to an exe (or similar) is already common practice in the Python world, and works really well. Vinay Sanjip's pyzzer is a good example of this approach. On 28 May 2015 at 16:45, Steve Dower wrote:

Agreed, but the minimally functioning Python is barely under 5MB. That will be considered bloated and won't help us compete with Go, so we should find a better way to fix Python application distribution and stop getting so hung up on putting everything into a single executable file.

There's a perception issue here. You can compile C# code into small exes (that's how people think) and "all" you need is the .net runtime installed. If we shipped a "pyc" compiler that "compiled" Python code into small exes that "just needed the Python runtime installed" would that feel the same to people? Would they be happy to view that as comparable to go compiled executables? (I assume go *doesn't* rely on a separate runtime, though). Nevertheless, I would like to understand how Unix can manage to have a Python 3.4.3 binary at 4kb. Does that *really* have no external dependencies (other than the C library)? Are we really comparing like with like here? Paul

On May 28, 2015 at 2:33:25 PM, Carl Meyer (carl@oddbird.net) wrote:

On 05/28/2015 11:52 AM, Paul Moore wrote: [snip]

...

Nevertheless, I would like to understand how Unix can manage to have a Python 3.4.3 binary at 4kb. Does that *really* have no external dependencies (other than the C library)? Are we really comparing like with like here?

I don't know what Donald was looking at, but I'm not seeing anything close to that 4k figure here. (Maybe he's on OS X, where framework builds have a "stub" executable that just execs the real one?)

On my Ubuntu Trusty system, the system Python 3.4 executable is 3.9M, and the one I compiled myself from source, without any special options, is almost 12M. (Not really sure what accounts for that difference - Ubuntu system Python uses shared libraries for more stuff?)

Carl

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

The problem is I'm an idiot and did du -h against ``which python``, which of course resolved to the symlink of python that points to python3.4. The real executable on my OSX box is 2.6M (built using pyenv). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 28 May 2015 at 19:22, Chris Angelico wrote:

...

Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a "first-class" executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes.

I'm aware of that. When I said "a tiny stub", I was thinking in terms of a small executable. The idea is that its sole purpose is to locate Python someplace else, and chain to it; that has to be actual executable code, complete with the 512-byte "MZ" header and everything, to ensure compatibility. But it should be able to be small, tight, and easy to verify correctness of, so there aren't (in theory!) security exploits in the header itself.

OK, cool. I'm sort of working on that as a bit of a side project - a tiny stub exe that you can prepend to a Python zipapp which runs it via the standard embedding APIs. It's little more than an idea at the moment, but I don't think it'll be too hard to implement... Paul

Getting lost as to what thread this belongs in... But another tack to take toward a single executable is Cython's embedding option: https://github.com/cython/cython/wiki/EmbeddingCython This is a quick and dirty way to create a C executable that will then run the cythonized code, all linked to the python run time. At this point, it still requires the python shared lib, and I think any other compiled extension is shared, too. And if you run Cython on all the python code and modules you use, you'll have a LOT of shared libs. But perhaps one could re-do the linking step of all that and get a single compiled exe. and IIUC, the way Windows dll hell works, if you stuff the whole pile into one dir -- you will get a single executable directory, if not a single file. and about a 2X performance boost, as well, when you cythonize pure Python, at least in my limited experience. Just a thought. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 5/29/2015 3:33 AM, Paul Moore wrote:

On 28 May 2015 at 22:09, Glenn Linderman wrote:

...

This would be something I could use and benefit from immediately upon it being available, so I laud your idea, and hope you have a successful implementation, and look forward to using it. It would largely replace the need for the py.exe launcher for some classes of applications. The following proof-of-concept works as is (based on my pretty minimal testing), and only uses the limited API, so it should work with any version of Python 3 (I've not tested it with Python 2, but I think the only "new" API is PySys_SetArgvEx, which could be replaced with PySys_SetArgv at a pinch). Excuse the dreadful coding style and lack of error handling, I hacked it up in about an hour :-)

I have no particular interest in Python 2, having started with Python 3, and only used Python 2 for cases where dependent packages required it, and I've now reached the nirvana of all my dependency packages being ported to Python 3, although I have yet to port/test one remaining application to prove that. So I only mentioned Python 2 because it still could be useful for other people :)

(Actually, I just tried building on Python 2 - guess what - Unicode :-) SetProgramName and SetArgvEx won't take Unicode values. The easy fix is just not to use Unicode, the hard one is to do the encoding dance, but I'm not going to bother...).

One approach would be to support Unicode arguments only for Python 3, but that would really be only paying lip service to Python 2 support. Another approach might be to not #define UNICODE for the Python 2 version, and use the 8-bit Windows APIs, allowing Windows and the C runtime to do the encoding dance for you? Although I'm not sure what Python 2 requires in that respect.

#define UNICODE #define _UNICODE #include #include

int main() { TCHAR program[MAX_PATH]; LPWSTR *argv; int argc; PyObject *runpy; PyObject *ret;

argv = CommandLineToArgvW(GetCommandLineW(), &argc); GetModuleFileName(NULL, program, MAX_PATH); Py_SetProgramName(program); /* optional but recommended */ Py_Initialize(); PySys_SetArgvEx(argc, argv, 0); runpy = PyImport_ImportModule("runpy"); if (!runpy) PyErr_Print(); ret = PyObject_CallMethod(runpy, "run_path", "u", program); if (!ret) PyErr_Print(); Py_Finalize(); return 0; }

One mildly annoying thing is that python3.dll is only installed in <python install dir>\DLLs, which typically isn't on PATH. Ah, linking.... so I guess if I figured out how to create this binary, it would contain a reference to python3.dll that would attempt to be resolved via the PATH, from what you say, and typically fail, due to PATH seldom containing python3.dll. The python launcher gets around

That looks interesting, I wonder what compilation environment it would need? I don't think I've even installed a C compiler on my last couple boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, since I've moved to using Python for anything a 5 line batch file can't do... that by (1) being installed in %windir%, and going and finding the appropriate Python (per its own configuration file, and command line parameters), and setting up the path to that Python, which, when executed, knows its own directory structure and can thus find its own python3.dll. The launcher, of course, adds an extra layer of process between the shell and the program, because it launches the "real" Python executable.

So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem).

Do I understand correctly that the "user admin issue" means "add the appropriate <python install dir>\DLLs to the PATH"? What I don't understand here is how linking to the full API avoids the problem... it must put more python library code into the stub executable? Enough to know how to search the registry to find the <python install dir> for the version of Python from which the full API was obtained? Or something else? Are there other alternatives? Assuming that the reference to the missing DLL is not required until the point at which a symbol from it is first referenced, so that the stub would have some ability to do something before that first call, maybe... 1. The stub above could be enhanced to contained a "hard coded" directory that it adds to the PATH itself? 2. The stub above could be enhanced to define that its first parameter is the <python install dir>, and tweak its PATH. 3. These days, the Python installer does offer to optionally add itself to the PATH. Is that sufficient to make the stub work? 4. The launcher could be used, assuming it is installed, but then you don't need a stub, and you get the extra process layer. 5. stubpy.cmd could be created, a four line batch file below [1], which wouldn't require the launcher or its extra process layer, but would have to be placed on the PATH itself, or in the directory with the stub Python programs. Only #3 could be construed as "easy" for the "dumb user"... if the Python installer offers to add itself to the PATH on "repair" installs, particularly (I'm not sure if it does). Editing the System PATH through the control panel is hard for the "dumb user", not made easier by the squinchy text box M$ provides for the editing. Nor is editing the System PATH made less error prone by the whole thing being available for editing, rather than the "GUI promotors" providing an editing editing interface such as displaying each item separately, with checkboxes to delete items, or insert items at particular locations, and directory selection dialogs rather than typing the desired new path as text. Hmm. Sounds like a good program task for a stub Python program :) Except it doesn't bootstrap, unless it lives in <python install dir>. [1] stubpy.cmd: @setlocal @PATH=<python install dir>;%PATH% @shift @%*

...

Of course, per other disccusions, this doesn't solve the problem for:

A) machine without Python installed B) programs that need binary extensions

Other discussions have suggested:

3) The stub could offer to download and install Python

A corollary:

4) The stub could offer to download and install the needed binary extensions as well as Python. This would require the installation uniformity of something like pip, so perhaps would be restricted to extensions available via pip. And it would be much enhanced by some technique where the zipapp would contain metadata readable by the stub, that would declare the list of binary extensions required. Or, of course, it could even declare non-binary extension that are not packaged with the zipapp, if the process is smooth, the modules available via pip, etc., as a tradeoff. I'm pretty strongly against downloading interpreters or extensions. Apart from the pretty huge added complexity, as a user I'm not sure I'd trust a supposedly simple application I'd received if it started asking to download stuff unexpectedly...

Paul

Yep. I mostly mentioned them for completeness. I have no argument with this: installing Python can be a documentation thing. "To use this program, you need to have Python installed, at least version N.M." Maybe also : "and it should be installed on the System PATH, or other methods of setting the PATH before running this program must be used, or this program should be saved in the <python install dir>." If stub+zip programs need other extensions, it can be documented as a batch file that calls pip a sufficient number of times with appropriate parameters, or the stub+zip program itself could be written to detect the needed but missing extensions, and invoke pip to get them before using them.

I did that once; it wasn't worth it. It was no smaller than what PyInstaller would output and required manually adding in the required modules that weren't in the stdlib, along with any extra DLLs (e.g. the Qt DLLs). On Fri, May 29, 2015 at 4:45 PM, Paul Moore wrote:

On 29 May 2015 at 21:49, Glenn Linderman wrote:

...

That looks interesting, I wonder what compilation environment it would

...

I don't think I've even installed a C compiler on my last couple boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, since I've moved to using Python for anything a 5 line batch file can't do...

...

One mildly annoying thing is that python3.dll is only installed in <python install dir>\DLLs, which typically isn't on PATH.

Ah, linking.... so I guess if I figured out how to create this binary, it would contain a reference to python3.dll that would attempt to be resolved via the PATH, from what you say, and typically fail, due to PATH seldom containing python3.dll. The python launcher gets around that by (1) being installed in %windir%, and going and finding the appropriate Python (per its own configuration file, and command line parameters), and setting up the path to that Python, which, when executed, knows its own directory structure and can thus find its own python3.dll.

The launcher, of course, adds an extra layer of process between the shell and the program, because it launches the "real" Python executable.

...

So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem).

Do I understand correctly that the "user admin issue" means "add the appropriate <python install dir>\DLLs to the PATH"?

What I don't understand here is how linking to the full API avoids the problem... it must put more python library code into the stub executable? Enough to know how to search the registry to find the

need? dir>

...

for the version of Python from which the full API was obtained? Or something else?

Sorry, I assumed more Windows/C knowledge than you have.

I'll work on this and produce proper binaries in due course, so you can always wait for them. But you can build the stub with pretty much anything, I suspect - I managed with MSVC 2010 and mingw. I'll add some build docs and get it on github.

Using mingw

gcc -Wall -O2 -o stub.exe stub.c -I <python home>\Include C:\Windows\system32\python34.dll strip -s stub.exe

Using MSVC

cl /Festub.exe /O2 stub.c /I<python home>\Include <python home>\libs\python34.lib

Regarding the DLLs, yes the "user admin issue" is adding the right directory to PATH. I used the phrase "admin issue" as it's the aspect that's likely to be far harder than any of the technical issues :-) The reason using the full API helps is that the full API references python34.dll rather than python3.dll. And the Python installer puts python34.dll on PATH automatically, as it's what the "python" command uses. (For the people with more Windows knowledge, I know this is a simplification, but it's close enough for now).

So there are two options with the code I posted.

1. Build an exe that uses a specific version of Python, but which will "just work" in basically the same way that the "python" command works. 2. Build an exe that works with any version of Python, but requires some setup from the user.

Either approach requires that the Python DLL is on PATH, but that's far more likely with the version-specific one, just because of how the installer does things.

With extra code, the stub could locate an appropriate Python DLL dynamically, which would simplify usage at the cost of a bit of fiddly code in the stub.

This might be a useful addition to the zipapp module for Python 3.6.

Paul

PS Current launchers (py.exe, the entry point launchers from pip/setuptools, etc) tend to spawn the actual python program in a subprocess. I believe there are *technically* some differences in the runtime environment when you use an embedding approach like this, but I don't know what they are, and they probably won't affect 99.9% of users. Lack of support for binary extensions is likely to be way more significant. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

On 29 May 2015 at 23:15, Glenn Linderman wrote:

I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, but I didn't ask the installer to put it there either. So I'm guessing your option 1 assumes asking the Python installer to put it there? Not "automatically" but "on request", I think?

In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... so I'm somewhat unclear on your simplified explanation.

I'm definitely batting zero today :-( OK, let's try to be clear. I typically do "all users" installs. The "for me only" install is slightly different, and the new install for 3.5 may be different again. But what I see is: 1. C:\Python34\DLLs containins python3.dll, which is *never* on PATH (and doesn't need to be for normal use). Anything that wants to use python3.dll needs that directory manually adding to PATH. 2. python34.dll is in C:\Windows\System32. This is always available to all processes, as it's in the Windows system directory. If you say "add Python to my PATH" you get C:\Python34 added to PATH. For a user install, I believe python34.dll may be in there rather than in C:\Windows\system32, so technically, for an app that uses python34.dll to work, you need *either* an admin install, *or* to have done "add Python to PATH". I hope that made sense. Sorry for my garbled previous version. Paul

Paul Moore wrote:

One mildly annoying thing is that python3.dll is only installed in <python install dir>\DLLs, which typically isn't on PATH. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem).

I didn't even notice that 3.4 (and earlier?) were doing that, so I changed/fixed it by accident :) Python 3.5 installs python3.dll alongside python35.dll, so it'll go into the user's Python directory by default or into the system directory for an all-users installation. The embeddable distro includes python3.dll alongside the other DLLs as well. Cheers, Steve

py2exe tends to invoke DLL hell if you have various versions of VS or Office or both installed. Because Windows. On May 28, 2015 11:23:57 AM CDT, Chris Barker wrote:

I'm confused:

Doesn't py2exe (optionally) create a single file executable?

And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance)

Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib).

I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-)

Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case.

So is something different being asked for here?

Barry Warsaw wrote:

...

...

I do think single-file executables are an important piece to Python's long-term competitiveness.

Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable?

And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ?

-Chris

On Thu, May 28, 2015 at 8:39 AM, Donald Stufft wrote:

...

On May 28, 2015 at 11:30:37 AM, Steve Dower

...

wrote:

...

Donald Stufft wrote:

...

Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need.

For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...):

* python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB

This gets you to ">>>", and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB.

The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.)

Cheers, Steve

You don’t need a "fully functioning Python" for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc.

Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in

(steve.dower@microsoft.com) the

...

stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes.

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

...

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

------------------------------------------------------------------------

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.