Greetings list,

What's the advantage of having it as an official part, rather than remaining a third-party tool?

Easy generation of executables. Installing a version of Python gives you a complete suite

Producing native executables is an attractive nuisance. It doesn't actually prevent people from disassembling your code (many MANY people seem to think that it does),

The purpose is to create a packed state, not obfuscation. The ability to just run your program without worrying about further installation

it locks in a particular Python version, it locks in an OS architecture, it locks in everything that you shouldn't be locking in.

Putting that sort of thing into the standard

That's the purpose of executables. library will encourage people to use it when they really shouldn't. Explanations appreciated

Also, blessing one particular executable builder means that all others become second-class citizens. Is PyInstaller so much better than everything else?

It's quite good: "PyInstaller freezes (packages) Python applications into stand-alone executables, under Windows, GNU/Linux, Mac OS X, FreeBSD, Solaris and AIX. " This seems a more viable option than creating such a utility from scratch

It also would force PyInstaller to be released on exactly the same cadence as the Python that it comes with. It would be impossible for PyInstaller to release a hotfix between Python releases, or to add features and functionality to an already-released Python version.

Same as all other integrated libs, though i do agree that with executable generators you have to be more careful Kind Regards, Abdur-Rahmaan Janhangeer about | blog github Mauritius

Greetings list,

The purpose of executables is to make it harder to apply bug fix releases of Python? I thought that was an unwanted side effect.

I think you mis-understood me by me making myself not clear I mean bundling this library: https://github.com/pyinstaller/pyinstaller with CPython distributions. Not program executables Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius

Greetings list,

Suppose you distribute a .py script to a million people. Your script is faulty due to a bug in the Python interpreter or std lib. But you don't need to do anything to patch your script: you just tell people to upgrade to the latest version of Python where the bug is fixed. Or you say nothing at all, and when the user's get their mandatory OS-supported upgrade, including Python, it fixes itself.

From experience it's not the only source of bug. I normally package apps with 3rd party libraries. Libs also can contain bugs. Then along the same line i can tell people to upgrade the lib to the desired version. This is true. However it boils down to whether people want executables or not. The purpose many libs exist shows that there is a need for generating native executables It's up to the developer i guess, how much Python-aware he wants the target environment to be, considering that he has the ability to tune. Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius

Disclaimer: I'm not arguing for or against having pyinstaller or the like in the stdlib. Probably slightly against. Anyway... On 19Nov2020 15:59, Chris Angelico <rosuav@gmail.com> wrote:

Producing native executables is an attractive nuisance. [...stuff...]

To me, the beauty of things like pyinstaller is that they produce a _complete_ package/bundle/executable which you can just use. More to the point, that you can hand to someone else to use. I've been using py2app to build a Python+PyQt app for a Mac for a client. It makes a working MacOS .app that contains your dependencies (python itself, various libraries), and on MacOS an app is a complete package -- it contains a bin directory and the associated assets (libraries, static assets like images, what have you). There's a standard layout and the MacOS runtime linker has cool hooks to let you specify library paths _relative_ to the executable. Which means you can plonk the .app down anywhere and it just works. Getting that right yourself is _hard_. Well, I found it hard to debug when there were problems. In particular, the "find all the dependences and bundle them so that they still work" is hard. A tool which "builds an app" is very very useful - you can hand your client something whose install is "copy it here" (on a Mac, usually /Applications). And they're golden. Anyway, this is a long winded way of saying that your misgivings about executables (fixed source code and/or python version) are often a fair price to pay to "just give it to someone" and have it work. If it needs an update or bugfix I can just give them a newer one. BTW, py2app also has a "dev mode", a bit like pip's -e option, to use the "live" source tree instead of a fixed bundled source. Cheers, Cameron Simpson <cs@cskk.id.au>

On 20/11/20 12:24 am, Chris Angelico wrote:

Have you considered the value of using zipapp

You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it.

It won't have all the properties of an app bundle on MacOSX, though. -- Greg

On 11/19/2020 2:17 PM, Steven D'Aprano wrote:

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it. It won't have all the properties of an app bundle on MacOSX, though. Firstly, does that matter? And secondly, what would it take to give it

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote: those additional properties?

It does matter. On Windows, for example, you can't use subprocess.run on a zipapp. See https://docs.python.org/3/library/zipapp.html#making-a-windows-executable. This is similar (but different) from not using .bat file wrappers for console entry points via pip, and instead creating a .exe file. But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained. Eric

On 20/11/20 11:32 am, Chris Angelico wrote:

It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything". But what happens when there's a security patch for Python? Are they going to continue to not > think about the dependency?

There seems to be an assumption here that having a natively installable app automatically means bundling everything, including the Python executable. It doesn't have to be that way. There's no reason the installed app can't use a previously installed Python or other libraries. I still think it would be useful to have some kind of basic installer-creating functionality in the stdlib. Not bundling Python could be the default, or even the only option if you're that concerned about it. -- Greg

On Thu, 19 Nov 2020 at 22:55, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 20/11/20 11:32 am, Chris Angelico wrote:

...

It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything". But what happens when there's a security patch for Python? Are they going to continue to not > think about the dependency?

There seems to be an assumption here that having a natively installable app automatically means bundling everything, including the Python executable. It doesn't have to be that way. There's no reason the installed app can't use a previously installed Python or other libraries.

Agreed, although there are use cases for bundling.

I still think it would be useful to have some kind of basic installer-creating functionality in the stdlib. Not bundling Python could be the default, or even the only option if you're that concerned about it.

I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms. But I *would* support building an "unzip and run" style of distribution. It would be possible to design/develop the interface outside of the stdlib, but as the person who wrote zipapp, I'd be completely in favour of making it part of zipapp once the details are thrashed out. On Thu, 19 Nov 2020 at 22:59, Chris Angelico <rosuav@gmail.com> wrote:

You mean... zipapp?

No. Definitely not. There are fundamental differences between .exe files on Windows and other filetypes. And people have already explained that MacOS app bundles are different from zipapps. You can argue against providing "native bundles" if you want (I'll disagree with you, but that's fine) but please stop insisting we mean zipapps. We don't. I certainly don't, and I should know as I was involved in the zipapp design. On Thu, 19 Nov 2020 at 23:12, Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Nov 20, 2020 at 10:05 AM Eric V. Smith <eric@trueblade.com> wrote:

...

I just don't think we need to be immediately dismissive of people's desire to create a platform native executable that disguises the fact that the code is written in Python.

I never said it shouldn't happen. But having zipapp in the standard library and all native executable creators as third party apps gives a clear indication that a .pyz file should be considered first, and a native executable only if that isn't suitable.

And I don't agree with that. If anything, I believe that .pyz zipapps are the attractive nuisance. People on Windows (to give a specific example, me, who should know better!!!) try to use them, and then trip up over the fact that you really can't persuade the OS to treat anything other than an actual .exe file as if it were a native executable. You can get it to *almost* work, and almost is often sufficient, but it's not seamless, and in my experience, you usually end up hitting an edge case where things go wrong. Maybe you need to run the application via subprocess, maybe you didn't set PATHEXT in your shell, maybe the filetype associations are incorrect. But what you end up with is someone discovering that the file you sent with doesn't work "because it's written in Python, not in <pick another language that writes native exes>". Paul

On 20/11/20 12:24 pm, Paul Moore wrote:

I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms.

By "installer" I just mean something that gets installed the way users expect on that platform. On Windows that means something you run that puts the right things in the right places. On MacOSX it means an app bundle that you just copy onto your disk. On Linux it probably means creating a package for whatever package manager the distro uses (not really familiar with installing desktop apps on Linux). So yes, if the usual convention on your platform means interacting with some platform-provided mechanism, then that's what needs to happen. -- Greg

On Thu, Nov 19, 2020 at 11:24:32PM +0000, Paul Moore wrote:

I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms. But I *would* support building an "unzip and run" style of distribution.

Don't zip files support self-extracting archives? Google says that the most common archivers (WinZip, 7-Zip, WinRar, etc) do, and I've seen people talking about making self-extracting archives on Ubuntu. The low-hanging fruit here is probably for Python's zip library to support self-extracting archives for the most common platforms. -- Steve

Am 20.11.20 um 11:01 schrieb Paul Moore:

1. Add the ability for zipapp to prepend a launcher for Windows, so zipapps can be built that "run natively" on Windows (Unix and MacOS don't need this, because for them, shebangs are handled by the OS, not by the language runtime). It *might* be possible to use the py.exe launcher, if not the sample code in the zipapp docs is probably sufficient. 2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module.

So basically re-implement py2exe, py2app, PyInstaller, etc?! -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

On Sun, Nov 22, 2020 at 6:34 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

Am 20.11.20 um 11:01 schrieb Paul Moore:

...

...

2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module.

Definitely not. I'm thinking of something *far* more limited. Specifically, things I'm not suggesting be in the stdlib:

1. Bundling the Python binaries. 2. Stripping out unneeded modules. 3. Executables forms other than "single file". 4. Support for C extensions.

That last one if VERY limiting :-( But anyway, I haven't used it in a long time, but py2app has (or had) and "alias" mode, that would would make a app bundle that references the installed python. I think it would not take much to make it include all non-stdlib code -- or, even easier, to include what the user specifies. But I'm not sure of the utility of this really -- Linux is the only platform that you can expect a consistent "system Python" on. (Apple has supplied one for years, but it's never been properly maintained and suitable to use this way) I'm curious about zipapp -- I've heard of it, but never tried to use it -- does it get much use in the wild? My feeling is that it hits middle ground that isn't very useful. If you can count on your users having a proper Python installation ,then they can use pip to install your package and run your scripts. If they can't do that, then they likely need a full bundle. But I could be wrong there. -CHB

I would be open to using the "embedded distribution" on Windows to ship such an executable with a dedicated Python interpreter, but unless CPython starts distributing an embedded distribution for other platforms, that's as far as I'd go.

Paul

Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4IEG7I... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Mon, Nov 23, 2020 at 12:09 AM Paul Moore <p.f.moore@gmail.com> wrote:

On Mon, 23 Nov 2020 at 03:37, Christopher Barker <pythonchb@gmail.com> wrote:

...

My feeling is that it hits middle ground that isn't very useful. If you can count on your users having a proper Python installation ,then they can use pip to install your package and run your scripts.

If they can't do that, then they likely need a full bundle.

But I could be wrong there.

You're more or less right, although your perspective on what's useful may be biased a little by the fact that (I believe) you're a heavy numpy user.

indeed I am. But also, where I need this sort of thing is for desktop GUIs (wxPython in my case), another heavy user of compiled extensions. I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package?? Hmm --maybe 'cause you can bundle specific dependency versions? In that case, conda-execute should be part of the discussion. What it does is fire up a conda environment, and then run your script in it. BTW, this all reminds me of a discussion a long while back in the wxPython community -- we thought it would be nice to have a wxPython run-time -- essentially a Python install with wxPython bundled in, so you could make little GUI apps without bundling that massive package. But one trick was that while wxPython was the big one, any given application might require who knows what other dependency as well. I suppose sipassp could help there. And none of us got around to actually implementing it anyway :-( -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 24/11/20 9:05 pm, Chris Angelico wrote:

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top) ...

But to most people a "proper point and click GUI thing" includes having an actual GUI, which is somewhere between very difficult and impossible to do well without C extensions that don't come with a standard Python installation. -- Greg

On 25/11/20 12:14 pm, Chris Angelico wrote:

If you want a perfectly out-of-the-box app, you're probably going to have to stick to Tkinter.

Which is only bundled with Python on Windows, as far as I know. -- Greg

On 2020-11-24 00:05, Chris Angelico wrote:

...

...

I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package??

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top), and if you can't do that, bundle it into a .pyz with zipapp and, again, put a shebang for posix platforms. Windows, if the py.exe launcher is installed, will happily let you double-click on a .py or .pyz and it'll run just fine.

I think by "proper point-and-click GUI thing" he meant the app itself is a GUI app, not just "can be launched via the OS GUI". At least, that is what I would mean by that, and that is what can't be done without C extensions, because most GUI toolkits require C extensions. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 2020-11-24 16:47, Chris Angelico wrote:

On Wed, Nov 25, 2020 at 10:29 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

...

On 2020-11-24 00:05, Chris Angelico wrote:

...

...

...

I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package??

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top), and if you can't do that, bundle it into a .pyz with zipapp and, again, put a shebang for posix platforms. Windows, if the py.exe launcher is installed, will happily let you double-click on a .py or .pyz and it'll run just fine.

I think by "proper point-and-click GUI thing" he meant the app itself is a GUI app, not just "can be launched via the OS GUI". At least, that is what I would mean by that, and that is what can't be done without C extensions, because most GUI toolkits require C extensions.

Python ships with one GUI toolkit (Tkinter). If you can't install anything else, you at least get that.

That's not even close to sufficient. I realize that we're diverging a bit from the original thread here, but basically my position on this is that what would be good is a full solution. Not another half solution or three-quarters solution. What I mean by a full solution is I should be able to write any Python app using any libraries I want and anyone can install it on any OS without knowing anything about Python or any of those libraries, with no limitations except that the libraries my app uses have to exist for the platform they install it on. So if, say, PyQt actually isn't available for your platform, okay, you're out of luck, but if it is available there should be no packaging/distribution/installation obstacle that prevents it from being used in an app that appears to the user as an ordinary native OS app. These other alternatives of "well no but you can do X. . ." are, from my perspective, pointless. There are already lots of half-solutions that allow some combinations of things but not others, and we don't need another one of those. What we need is a full solution. The great thing about Python is ALL the things you can with Python, and what would be good is a solution that lets you do ALL the things you can do with Python, but as regular transparently-OS-installable-and-usable apps. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Tue, 24 Nov 2020 at 07:47, Christopher Barker <pythonchb@gmail.com> wrote:

indeed I am. But also, where I need this sort of thing is for desktop GUIs (wxPython in my case), another heavy user of compiled extensions.

I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package??

For me (on a purely personal basis) it would allow me to install command line tools *without* having them tied to a specific Python installation. So, for example, I can install a tool like tox or nikola and *not* need to reinstall it every time I upgrade my main Python installation. (Even if I put it in a virtualenv, or something like pipx, that's tied to the system Python). I don't (personally) want a Python interpreter bundled (more accurately, I sometimes do, but not always). But zipapps fail for me because they aren't exes - and I regularly trip up over that problem. To give a very explicit example of what I'm personally thinking of, imagine writing the "diff" command in Python. How would you deploy it so that people could use it as easily as a native executable build? Languages like go and rust are competing in this area, and I'd like to be able to use Python for this sort of thing.

Hmm --maybe 'cause you can bundle specific dependency versions?

That's part of it, as well. But I don't often hit version conflicts, so it's minor.

In that case, conda-execute should be part of the discussion. What it does is fire up a conda environment, and then run your script in it.

The problem is for me, that I actively dislike conda. If I can set up conda-execute as a standalone Python application and use it without knowing it's conda under the hood, then it might be useful. If I can't, then it's basically demonstrated that it can't bundle *itself* the way I want it to bundle my code, so I guess that answers my question as to whether it's useful for me :-) But that does raise a good point. What I'm looking for is essentially the equivalent of `cl /Femyapp.exe myapp.c`. A standalone command that I can run, which creates an executable from my Python code, which I can copy and run anywhere. I don't mind "needing a Python runtime" installed on the target, but I *do* mind if it has to be a specific version (If I write my code on Python 3.8, I want the same exe to run when I upgrade my PC to 3.9). If I try to describe my personal checklist of features, it basically comes down to: * The installer itself must be bundled as a standalone command. I don't want to have to install the installer into every project's venv. * An installer that can't bundle itself makes me suspicious. I don't *actually* insist on "the installer must have created its own executable", but I'd want to know why if it wasn't... * The resulting compiled apps must be Windows native exes (I don't currently have a need for other platforms, but I'd want whatever "native binary" meant there as well). * The compiled apps must depend on nothing but a Python interpreter (and must not be tied to a specific version - only the app's own constraints should apply there). * Must be relatively mature/robust - I don't want to interrupt deploying my script to raise a bug on the installer and wait for a fix. Things I'd like to have: * A mode that *does* use a separately-installed Python interpreter. Most of the time I don't need the interpreter bundled, and I'd like to avoid the extra overheads. * Support for C extensions. Pure python only *is* a real limitation. * A "robust" mode that doesn't try to strip out "unused stuff". I don't need this to be the default, but I want it available. I have no wish to find out months after I deployed the script that a dependency used some weird dynamic loading feature that the installer didn't know about, and my app broke because a plugin was missing. * Related to this, I don't need the installer to guess what packages I'm using. I'm perfectly willing to tell it. And I definitely don't want the installer to look at my currently active environment - there could be all sorts of junk in there. A tool like PyInstaller is pretty close to this. The major problems for me are that it doesn't come as a standalone tool, and it has no "depend on a system Python" mode.

BTW, this all reminds me of a discussion a long while back in the wxPython community -- we thought it would be nice to have a wxPython run-time -- essentially a Python install with wxPython bundled in, so you could make little GUI apps without bundling that massive package. But one trick was that while wxPython was the big one, any given application might require who knows what other dependency as well. I suppose sipassp could help there.

And none of us got around to actually implementing it anyway :-(

... and that's the main problem with *all* of these ideas. No-one is actually implementing them. (FWIW, over the weekend I looked at adding "run an appended zip file" functionality to the Python launcher, and while it's not too difficult in principle, my C skills, and my enthusiasm for things like manual memory management and error handling, have dropped to the point where I really couldn't be bothered spending my free time on it :-() Probably because, most of the time it genuinely doesn't matter. Sharing Python code as scripts, or via `pip install` isn't that bad for most use cases. Paul

On Tue, 24 Nov 2020 at 10:18, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Mon, 23 Nov 2020 08:09:07 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

But it's not as limiting as you suggest - it *does* preclude most scientific use (because of numpy etc) but (for example) a large number of web libraries are pure Python.

Not sure what you mean here, but while Web frameworks themselves may be pure Python, you can have C accelerators in a template engine or in a ORM layer. Also, the database driver most likely isn't in pure Python (if you want it to be performant anyway).

All I meant was that how limiting it is depends on what type of application you're trying to write. But there's no question that not being able to use C extensions limits use (I work on pip, which cannot rely on C extensions for other reasons, so I'm pretty familiar with (a) how much of a nuisance it is, and (b) how far you can get even with that limitation). Paul

On 24 Nov 2020, at 11:58, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Tue, 24 Nov 2020 10:23:10 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

On Tue, 24 Nov 2020 at 10:18, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Mon, 23 Nov 2020 08:09:07 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

But it's not as limiting as you suggest - it *does* preclude most scientific use (because of numpy etc) but (for example) a large number of web libraries are pure Python.

Not sure what you mean here, but while Web frameworks themselves may be pure Python, you can have C accelerators in a template engine or in a ORM layer. Also, the database driver most likely isn't in pure Python (if you want it to be performant anyway).

All I meant was that how limiting it is depends on what type of application you're trying to write.

Sure, but the number of applications which don't depend whatsoever on non-stdlib C extensions is probably much smaller than you were trying to say. You should not fool yourself: the suggested "zipapp" (does it exist already? I lost track of the number of weird things that have been implemented in core Python in the name of packaging) would mostly be a non-solution.

This already exists, see <https://docs.python.org/3/library/zipapp.html <https://docs.python.org/3/library/zipapp.html>> It is basically a zipped up directory with an ``__main__.py``. This works for some applications, but is not comparable to py2exe/py2app/pyinstaller/…. IMHO any changes to CPython for this should be solutions for clear problems that are best fixed in the core, preferably along side changes to the prospective users of those changes to ensure that the changes are actually useful. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

Also, virtualenv is distributed as a zipapp these days. Paul On Mon, 23 Nov 2020 at 08:44, Steven D'Aprano <steve@pearwood.info> wrote:

On Sun, Nov 22, 2020 at 07:37:35PM -0800, Christopher Barker wrote:

...

I'm curious about zipapp -- I've heard of it, but never tried to use it -- does it get much use in the wild?

I am quite confident that one of the most used, if controversial, Python applications in the wild is a zipapp on Linux systems.

http://youtube-dl.org/

-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/2NNFSC... Code of Conduct: http://python.org/psf/codeofconduct/

On 2020-11-22 06:32, Paul Moore wrote:

Definitely not. I'm thinking of something*far* more limited. Specifically, things I'm not suggesting be in the stdlib:

1. Bundling the Python binaries. 2. Stripping out unneeded modules. 3. Executables forms other than "single file". 4. Support for C extensions.

I'd say there's no use in having such a tool if it doesn't support C extensions. Without C extensions you can't include most GUI toolkits like Qt, which are key to making a "real app" in many cases. That alone is enough to kill it for me. And if that wasn't damaging enough, you also lose the whole scipy stack. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 11/19/2020 5:32 PM, Chris Angelico wrote:

On Fri, Nov 20, 2020 at 9:22 AM Eric V. Smith <eric@trueblade.com> wrote:

...

On 11/19/2020 2:17 PM, Steven D'Aprano wrote:

...

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it. It won't have all the properties of an app bundle on MacOSX, though. Firstly, does that matter? And secondly, what would it take to give it

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote: those additional properties? It does matter. On Windows, for example, you can't use subprocess.run on a zipapp. See https://docs.python.org/3/library/zipapp.html#making-a-windows-executable. This is similar (but different) from not using .bat file wrappers for console entry points via pip, and instead creating a .exe file.

But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained.

And that's exactly why I say it's an attractive nuisance. It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything". But what happens when there's a security patch for Python? Are they going to continue to not think about the dependency? Until you (manually) push out an update, they can't do a thing to fix the problem. And if they get multiple of these statically-linked Python executables, they have to have independent duplicates of all of Python, each one a snapshot at the version that someone chose to publish.

I think I'm perfectly capable of making these decisions for my use cases. I don't consider this an attractive nuisance, I consider it a feature. It's really no different from any other static vs. dynamic linking scenario. To my knowledge, the world has not abandoned static linking (although I'll admit I might live in a bubble that could have missed this trend). Static linking has its uses.

Python can be installed from the Microsoft App Store or whatever it's called. Just get people to do that and then double click on the .pyz file. Is it really that hard? In some corporate scenarios, yes, it is hard or likely impossible. There are no users on the systems I'm concerned about, so no one to double click anything. Instead, the programs need to be invoked with the moral equivalent of subprocess.run from other (non-Python) software. (Yes, I know that there will be specific situations where that's not possible. That's why the various .exe creation tools need to exist. It does NOT mean they need to be in the stdlib.)

That I mostly agree with, given what I know about the state of the art in this space. But if there was a mature tool that could be included (for all of the various reasons given elsewhere), I might be convinced otherwise. I'm not familiar with the various 3rd party options to know which one(s) to turn to or recommend under various scenarios. Usually I just give up on Python in the cases I mention above. It would be good if Python had a better, or at least more widely known, story here. I just don't think we need to be immediately dismissive of people's desire to create a platform native executable that disguises the fact that the code is written in Python. Eric

On Thu, 19 Nov 2020 at 23:13, Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Nov 20, 2020 at 10:05 AM Eric V. Smith <eric@trueblade.com> wrote:

...

I just don't think we need to be immediately dismissive of people's desire to create a platform native executable that disguises the fact that the code is written in Python.

I never said it shouldn't happen. But having zipapp in the standard library and all native executable creators as third party apps gives a clear indication that a .pyz file should be considered first, and a native executable only if that isn't suitable.

In my experience there is a wide spectrum of different end "users" to be targeted when thinking about packaging an application. I quite often find myself in the situation where I'm providing code to people who are also able to write code and I can just point them to github or wherever and they can read the README and get everything set up. At the other extreme I work with people who have no idea what "Python" is and might struggle to use my application even if it was actually configured and installed on their computer and in the applications menu, had great instructions etc. Somewhere in this spectrum there is a usecase for zipapp which is something like "all my users have Python already installed but do not necessarily know how to use it". I haven't personally found a situation where this applies and haven't used zipapp as a result. Packaging applications is something where Python's usability could be improved enormously. Other languages also suffer around this in different ways but if Python had a simple and reliable (if not full featured approach) then that would be very useful. There has been a lot of progress with packaging and distributing Python libraries which is very welcome but support for distributing even very simple applications is lacking. Oscar

On 19Nov2020 18:03, Eric V. Smith <eric@trueblade.com> wrote:

On 11/19/2020 5:32 PM, Chris Angelico wrote:

...

On Fri, Nov 20, 2020 at 9:22 AM Eric V. Smith <eric@trueblade.com> wrote:

...

But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained.

And that's exactly why I say it's an attractive nuisance. It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything".

I remain in Eric's camp here. Though thanks for informing me of zipapp. If I'm making a tool for a client or end users, _they do not know or care that it is written in Python_. They want a MacOS app or Windows .exe or whatever platform standard application which they can _just invoke_ like any other opaque application. Often, this should work on anyone's machine. So you can't assume: - particular libraries are present, particularly third party libraries (like PyQt in my personal case) - particular Python interpreters are present (if any) So bundling is often _highly desireable_. As Eric says, it's in the same vein as static linking - the required stuff is part of what you ship. With .apps (and presumably in some way in Windows) you get to supply file bindings, icons, etc. The standard natiuve experience. Back to MacOS .apps (my Windows knowledge is thin to the point of transparency), you _can_ make thin .apps, which might include a zipapp to run against the system Python. But would it contain the third party PyPI modules? The third party libraries? A sufficiently modern Python for the code? If not, in some ways I may as well just ship a small shell script which builds a venv and fills it in on first use. Static linking and application dependency bundling is _not_ an attractive nuisance, it is a common and robust way to ship something which Just Works. It may not be necessary in you environment or use cases, but in the very common case where you want users to just use something without installing a development environment to support it, it is the go. Cheers, Cameron Simpson <cs@cskk.id.au>

On Fri, Nov 20, 2020 at 6:06 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

Yes, that's correct. All of what you described is how ordinary apps work. If I install a program and it has a bug or security vulnerability, then I am affected by that vulnerability. Having a way to install a Python program as a program in its own right means that it will also work that way. So what? That's how programs work. The fact that my program happens to depend on Python under the hood doesn't magically make it any different than other programs. So, again, an executable-creating library is just a way to make Python programs work like ordinary programs work.

You've mentioned this objection at least twice now and I still don't see it having any real relevance. All kinds of programs have bugs and vulnerabilities. There is no special reason why someone should expect a program to shield them from bugs or vulnerabilities in that program's underlying components, whether that program is written in Python or any other language.

So what you're saying is: "Everyone else who distributes native executables has these problems, so Python apps distributed as native executables will have these problems". Yes. Of course they will. But a Python app distributed as a .py file or a .pyz archive *won't* have these problems. Is that of no value? The special reason is the entire point of language interpreters. Let's suppose that there's a vulnerability discovered in the V8 JavaScript interpreter (the one behind Node.js and Google Chrome and such). Does everyone who's ever published a web app now have to push out a new version? Certainly not, and I think many web devs would be offended at the mere suggestion. They expect that a browser update will automatically fix it, and it should! Why should Python apps *not* take advantage of this separation? You've mentioned this objection to my objection multiple times too, and I don't understand why you think that more vulnerabilities isn't a problem. There is no special reason why someone should expect a program to have more bugs or vulnerabilities because it's distributed as an app rather than made available through a web browser (aside from the restrictions of web browsers themselves, of course, but that's beside the point). ChrisA

On Fri, Nov 20, 2020 at 01:06:11AM -0800, Brendan Barnwell wrote:

So are you suggesting that every single app should always be distributed as source code, run by a separate interpreter that users install separately?

My system has approximately sixty gajillion[1] apps installed that all rely on the same three or four runtime interpreters, mostly Perl, Python, and bash. I don't have sixty gajillion copies of those interpreters. At no point has Chris suggested that static bundling of stand-alone applications should be prohibited, or that every script should be distributed with its own copy of its runtime environment. The first is an extreme position that Chris never took, and the second is just silly, and would go against Chris' argument. I hate it when people misuse the term "straw man" on the internet, which they often do, but I think in this case the term is apt. Chris' actual position is much less extreme, and more reasonable, than the exaggerated version you make. The bottom line here is not whether or not there are use-cases for bundled stand-alone applications. We might disagree about what those use-cases are and how common they are, but they certainly exist. The question here is whether those use-cases would justify including a project of the complexity of PyInstaller, cx_Freeze, py2exe, etc. [1] A number somewhere between 100 and 100,000 *wink* -- Steve

On Fri, 20 Nov 2020 at 10:55, Abdur-Rahmaan Janhangeer <arj.python@gmail.com> wrote:

After the above thread I contacted a number of Zipapp-related persons including the author of Shiv ( Loren Carvalho ) who was kind enough to reply. Along the conversation he stated that

...

In fact, shiv itself sort of breaks a fundamental assumption about zipapps: that they remain zipped! shiv pyz files extract themselves on execution, for a myriad of reasons.

Shiv is indeed a very good solution. I've used it myself a lot (almost entirely for command line utilities, not for "something you can double click on to run it", though). My observations are that: 1. The fact that it builds .pyz files rather than .exe files *is* a problem for me. It's solvable by prepending a small exe header (as per the recipe in the zipapp docs) but making that step easier would be a significant improvement. 2. The way shiv unzips things is relatively seamless, and needed in order to support C extensions. Loading shared libraries direct from a zip file is very definitely *not* simple, and is one of the things that makes a project like pyinstaller complex to maintain and support. Shiv takes a sensible "practicality vs purity" trade-off here, IMO. 3. Even taking the above into account, there are some things that don't work perfectly - look at the -E option to shiv, which was something added based on experiences I had, to deal with tools like tox, which expect their dependencies to be visible in subprocesses.

Considering the 3 enhancements proposed by Mr Moore, i'd say that zipapps needs a tidbit twerking to be a great packaging solution.

I don't think zipapps will ever be a total solution. But they *can* be very good if they suit your particular requirements. And many people have fairly simple needs, which suit the capabilities of zipapps.

A successful zip-based packaged format is the JAR (https://en.wikipedia.org/wiki/JAR_(file_format)). JSmooth is a tool for converting JARs into EXEs https://en.wikipedia.org/wiki/JSmooth

Jar format has similar problems to zipapps regarding native libraries, I believe. However, Java's relentless drive to have "100% pure Java" solutions, along with a lot of corporate backing, mitigates this significantly. I don't think anyone would seriously consider using a Postgres driver written in "Pure Python" over a C extension, for example. And let's not even contemplate what a "100% Pure Python" version of numpy would be like :-)

Might give some leads

Jar files were certainly an example that motivated zipimport and the whole import hook machinery that underpins zipapps. If you're interested in this type of technology, TCL "Starkits" (https://www.tcl.tk/starkits/) are another interesting example, based on having the embedded code in a database (Metakit) rather than an archive format (zip). I've often thought about writing an import hook to handle putting packages in an embedded sqlite database, but it's never looked like it would add sufficient extra value over zipfiles to seem worth doing anything about it :-(

Though this thread suggests adding the natural abilities of turning python projects into executables, doing so via zipapps is an option i haven't considered. And it does not seem like it's for tomorrow.

IMO zipapps are definitely an under-appreciated option (of course, I would say that :-)) I think the problems with cases where they aren't a good solution, as well as past experience with the (similar, but more ambitious in ways that didn't work out) egg format has hindered their adoption somewhat. Paul

On Fri, 20 Nov 2020 at 08:55, Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Nov 20, 2020 at 6:06 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

...

You've mentioned this objection at least twice now and I still don't see it having any real relevance. All kinds of programs have bugs and vulnerabilities. There is no special reason why someone should expect a program to shield them from bugs or vulnerabilities in that program's underlying components, whether that program is written in Python or any other language.

So what you're saying is: "Everyone else who distributes native executables has these problems, so Python apps distributed as native executables will have these problems". Yes. Of course they will. But a Python app distributed as a .py file or a .pyz archive *won't* have these problems. Is that of no value?

Of course it is. But it's not the *only* consideration. What you seem to be doing is dismissing any possibility that in some circumstances, the balance is in favour of bundled executables over zipapps. The discussion here (at least this part of it) is about those situations where zipapps aren't a useful solution. So *by definition*, zipapps aren't relevant as an option in that case. You may not think that any such cases exist. Fair enough. But those of us that do are trying to discuss ways to handle those situations, and saying "that will never happen" or equivalently "do you mean zipapps" repeatedly, isn't helping.

The special reason is the entire point of language interpreters. Let's suppose that there's a vulnerability discovered in the V8 JavaScript interpreter (the one behind Node.js and Google Chrome and such). Does everyone who's ever published a web app now have to push out a new version? Certainly not, and I think many web devs would be offended at the mere suggestion. They expect that a browser update will automatically fix it, and it should! Why should Python apps *not* take advantage of this separation?

It's hardly the *entire* point, but I agree it's a benefit of them. And web development is *far* from being the only use for Python. And tools like VS Code (written with Node.js, I believe) bundle the V8 engine, so contrary to the point you're trying to make, Javascript could be viewed as an example of why Python needs a way to bundle apps... Paul

On Sat, Nov 21, 2020 at 1:35 PM Christopher Barker <pythonchb@gmail.com> wrote:

I'm not sure I should get involved in this, but ....

On Fri, Nov 20, 2020 at 12:53 AM Chris Angelico <rosuav@gmail.com> wrote:

...

Let's suppose that there's a vulnerability discovered in the V8 JavaScript interpreter (the one behind Node.js and Google Chrome and such). Does everyone who's ever published a web app now have to push out a new version?

No, but anyone that has published an Electron app does.

Yes, and that has been a problem for me. Electron apps whose authors haven't released an update. I am unable to run them in a safer interpreter, and am forced to either continue using the one that was bundled, or stop using the app. That is *the entire point*.

Oh, and Chrome itself needs to be updated -- only on what, millions of machines? V8 is bundled with Chrome -- you know, kind of like a PyInstaller app bundles Python ;-)

Uhhh... no, that's kind of like how Python bundles Python. That's not bundling *an app*. You update Chrome once, and every app is updated. Again, thank you for restating my point, but trying to make it sound like a counter-argument.

Chris A: I ask you to let this go -- Python can be used for many different kinds of application development. And those different use cases have different needs. And Desktop GUI applications, in particular, really, really, do need a "bundling" system -- at least on the Mac and Windows.

But do all bundles have to be native executables? PLEASE reread the above posts. I have never said that native executables should never exist, but I have said that many MANY applications can and should be distributed such that they use a separate installer.

It's clear that that's not the realm you work in, which is fine, and as more and more stuff moves to the Web, there is less need for Desktop apps, but the need is still there, and PyInstaller, and Py2app and py2exe before it (and still) are absolutely critical tools in that space.

I've written desktop apps. I've distributed bundles before. I've done the "this has to be a one-click thing" thing. I do know what I'm talking about here. And no, native executables are usually NOT the only way to achieve "double click on this thing to make it run".

I've been using Python for over twenty years, and discovering py2exe and py2app way back then made it possible for me to deliver multiple products that I simply could not have done without them. (and still do). Our users have NO IDEA that they are built with Python, and that's perfect -- they can just download, install, and have a working, native, desktop app. And it will keep working, even when their sysadmin tells them they can't have Python 2 installed anymore because it's no longer maintained.

But they ARE allowed to continue using a Python 2 that was bundled with something? Is that how it goes - you're not allowed to use a legacy interpreter if you're aware of it, but it's fine as long as it's hidden behind a binary that nobody can touch? Please. Before you rehash all my arguments back at me again, READ THE THREAD. I have never said that native executables shouldn't ever exist. I have never said that zipapp is the one true way to distribute code. I have never once hinted at the notion that all users are idiots who can only ever run a .exe file, nor have I said that everyone should open source their code, or a host of other arguments that I haven't made. Please, can people stop answering arguments that haven't been made, and actually respond to what has? ChrisA

On Sat, Nov 21, 2020 at 3:35 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

So let me ask this: In what circumstances do you think producing native executables IS a bad thing, and in what circumstances do you think it ISN'T a bad thing, and why do you think including such functionality in the stdlib would encourage the former more than the latter?

It's a bad thing any time it isn't actually necessary, and it's a good thing only when it is actually necessary. It's not my place to argue other people's use cases in specifics, but I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work. Under what circumstances should you use subprocesses and pipes in a Python program, and under what circumstances should you just keep everything inside a single process? Should you just always use boilerplate that runs everything in multiple processes in case you need to run it across multiple CPU cores? No. Does this mean I'm arguing against ever using multiple processes? No. Use the simpler options until you can't use them. Then use the more complicated options. ChrisA

On Sat, Nov 21, 2020 at 4:07 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

On 2020-11-20 20:53, Chris Angelico wrote:

...

...

...

So let me ask this: In what circumstances do you think producing native executables IS a bad thing, and in what circumstances do you think it ISN'T a bad thing, and why do you think including such functionality in the stdlib would encourage the former more than the latter?

It's a bad thing any time it isn't actually necessary, and it's a good

On Sat, Nov 21, 2020 at 3:35 PM Brendan Barnwell<brenbarn@brenbarn.net> wrote: thing only when it is actually necessary. It's not my place to argue other people's use cases in specifics, but I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work.

Okay, but that doesn't answer the second part of my question. You were saying earlier that you wanted people to read your posts and not claim that you're saying native executables shouldn't exist. So I'm trying to bring the discussion back to the original more narrow topic, which is the pros and cons of including native executable creation in the stdlib.

Just saying "it's bad if it's not necessary" is a cop-out in this context. The question is why do you think that including such functionality in the stdlib is bad. Your response here is saying it's up to other people to decide whether to use it or not, but that doesn't explain why you think it shouldn't be in the stdlib. Everything in the stdlib can be used if people think it's necessary and not used if they don't.

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one? ChrisA

On Sun, Nov 22, 2020 at 12:56:01PM +1100, Cameron Simpson wrote:

On 21Nov2020 17:54, Chris Angelico <rosuav@gmail.com> wrote:

...

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one?

Well, for my personal example, an in house tool for an in house task, the users _can_ copy a Mac .app into their Applications folder. I _do not_ and would not expect them to (likely choice): - install macports or homebrew - install python - install source code and libraries (needed PyQt)

They're end users; some are (variously) technical and some aren't, but none should need to be technical. I want them to copy an app to a new machine and be happy - drag'n'drop a single thing.

Sure, but why does PyInstaller have to be in the std lib for you to support this use-case? That was Chris' question: Under what circumstances does somebody have to make a native application but **cannot use one of the tools on PyPI to do it**? (Or the freeze tool.) -- Steve

On 22Nov2020 13:22, Steven D'Aprano <steve@pearwood.info> wrote:

On Sun, Nov 22, 2020 at 12:56:01PM +1100, Cameron Simpson wrote:

...

On 21Nov2020 17:54, Chris Angelico <rosuav@gmail.com> wrote:

...

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one?

Well, for my personal example, an in house tool for an in house task, the users _can_ copy a Mac .app into their Applications folder. I _do not_ and would not expect them to (likely choice): - install macports or homebrew - install python - install source code and libraries (needed PyQt)

They're end users; some are (variously) technical and some aren't, but none should need to be technical. I want them to copy an app to a new machine and be happy - drag'n'drop a single thing.

Sure, but why does PyInstaller have to be in the std lib for you to support this use-case?

It doesn't. I'm not arguing for that; I'm at least slightly against it in fact. I'm _still_ arguing against Chris' apparent position that wanting to make a standalone bundle is pretty much a null situation. I think I've let his "attractive nuisance" description get under my skin. Still, as you say:

That was Chris' question: Under what circumstances does somebody have to make a native application but **cannot use one of the tools on PyPI to do it**? (Or the freeze tool.)

It is true that I, at least, am not in that position: needing to make a standalone app and unable to access PyPI to get tools to build it. I may have misconstrued Chris to mean my users rather than myself. (His full question is still quoted at the top of this message for reference.) And on that basis I withdraw my example. But it definitely applies to my users: thier install process should be a trivial copy-this-app experience. That does require a complete bundle. Cheers, Cameron Simpson <cs@cskk.id.au>

Instead of adding pyInstaller, why not link with libpython as in the embedding section of the documentation? -- H On Fri, 20 Nov 2020 at 21:25, Abdur-Rahmaan Janhangeer <arj.python@gmail.com> wrote:

Greetings list,

...

Use the simpler options until you can't use them. Then use the more complicated options.

The only thing i don't trust Mr Chris on is packaging. His technical Py views are sound and great. I have the impression that concerning packaging the view is: we can do with what we have, no need to add more. When enhancing Zipapps was proposed, if i remember correctly, the view was along the lines of: "Why Zipapp when you can double click on a Py file". Now concerning native executables, the view runs along the lines of "at least consider Zipapps".

...

I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work.

I have the impression that everybody talking about native executables here were talking about an optional way of packaging apps which help people and not as the preferred, standard and de facto way.

Kind Regards,

Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Q52PIV... Code of Conduct: http://python.org/psf/codeofconduct/

-- OpenPGP: https://hasan.d8u.us/openpgp.asc If you wish to request my time, please do so using *bit.ly/hd1AppointmentRequest <http://bit.ly/hd1AppointmentRequest>*. Si vous voudrais faire connnaisance, allez a *bit.ly/hd1AppointmentRequest <http://bit.ly/hd1AppointmentRequest>*. <https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1>Sent from my mobile device Envoye de mon portable

On Sat, Nov 21, 2020 at 07:16:22AM -0500, Edwin Zimmerman wrote:

On 11/20/2020 11:53 PM, Chris Angelico wrote:

...

[snip] Use the simpler options until you can't use them. Then use the more complicated options.

Yea, use the simpler options.  This is why I have switched from Python to C# when writing desktop apps, simply because distributing a single exe file is easier at all levels than distributing a complete install of Python.

Decrease in effort packaging your app as an exe file: -10% Increase in effort writing your app in C# in the first place: +800% *semi-wink* But seriously... 1. *None of* the stdlib freeze tool, or any of the third-party projects cx_Freeze, py2app, py2exe or PyInstaller, work for you? 2. Python is pre-installed on pretty much all Linux and BSD systems, and Mac OS, and is trivially installed on Windows from the Microsoft app store. Getting a complete install of Python couldn't be simpler. Where does the process let you down? -- Steve

On Mon, 23 Nov 2020 at 10:39, Mathew Elman <mathew.elman@ocado.com> wrote:

I would argue this sounds like a case for a "python_runner", i.e. a lightweight python vm that can run python apps, e.g. zipapps. Something like the blender_runner for blender. Making it explicitly an app that runs via some other engine.

It would then be a central entry-point that can be updated to a later python and all the apps update with it. It would have to provide a utility for desktop shortcuts to feel like the python app was launching natively.

That's essentially like Java's JRE. For what it's worth, on my PC the JRE is 196M in size. Whereas a full Python distribution is only 94M, and the embedded distribution is 15M. So I think Python already has that, more or less. But my experience with the JRE is that very few applications actually use the "centralised" JRE, preferring to ship an embedded JRE. So the evidence from the Java world is that managing a central "runtime engine" is not actually as convenient as we're assuming, and the arguments for it simply aren't compelling in the real world. (Unix may have a different picture - there's a much stronger culture there of "depend on system packages" :shrug:) Paul

On Mon, 23 Nov 2020 at 11:32, Mathew Elman <mathew.elman@ocado.com> wrote:

I imagine there would be a way to have an install install the runner if there is not one on the machine, and use the existing one if there is, and creating a venv in either case. Meaning that using the equivalent of an embedded JRE would be obsolete, no?

Maybe? No-one has actually done that, so either it's harder than you're imagining, or no-one with the knowledge to do it is actually that interested in doing anything about it...

I suppose my point is that having a utility for launching a python app from the desktop may be sufficient for a lot of use cases.

I'm not sure if desktop apps or command line apps are the most important, TBH. Probably depends on the user base. But in general, yes. Paul

I suppose functionally there may be little difference, but having an explicit "runner" would allow two things: 1. the runner and stdlib for it could be in a compress format itself since it doesn't need to provide any utility for editing or navigating human readable modules. and so lighter 2. it would handle "venv"ing and creating desktop shortcuts etc that would just start the python app in the engine. The reason I mention this here, is because others have said that including functionality to deal with this sort of thing for _all_ OS would be heavy, where as here it could be more lightweight and even not part of the standard python interpreter, just the runner. Also on systems with python already installed it could be told to use the existing python install, if that was desired.

On 2020-11-20 01:51, Steven D'Aprano wrote:

If that's really what you want, you probably should look at making a way to run Python apps in the browser. Everyone has an OS, everyone has a browser, GUI browsers have similar looking look-and-feels, the days when devs assumed Internet Explorer are long gone.

I'm as steadfastly negative about browser apps as Chris A. is about native executables. :-) I realize this is an unpopular view, but I think the browser is a terrible app platform. It provides no user-configurable widget sets, so every app looks the way the app "designer" came up with, rather than all apps using a common look and feel. It has crippled access to the filesystem. Worst of all, the "app platform" is inextricably bound to a particular GUI app which cannot be versioned or used independently of the interpreter that runs other people's web apps. That is, if I like the interface of Firefox itself, but then they change it, I have no way to use the old Firefox interface while using the upgraded app engine (i.e., JS interpreter) because they're joined at the hip. All this makes using web apps a very frustrating experience for me. From my perspective bundling and locking and all those things have a very positive aspect, which is that they allow the user to "freeze" their workflow with a given tool and not be subject to changes foisted upon them by the authors. This wouldn't be such a problem if app authors were more disciplined about separating concerns, so that you could always get a security fix for some hidden bug without changing anything about how the app works from a user perspective. But since app authors don't do that, it's critical for me as a user to have a way to protect myself from an app author who suddenly decides their app should change and pulls the rug out from under my workflow. Now, it's true that something that provided the app-platform side of the browser without being tied to a particular interface would be great. That sounds more like the recent spinoff thread about a Python "app store". I think that would be even better than a native-executable-creation library, if we can get it. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 11/20/20 8:41 PM, Brendan Barnwell wrote:

On 2020-11-20 01:51, Steven D'Aprano wrote:

...

If that's really what you want, you probably should look at making a way to run Python apps in the browser. Everyone has an OS, everyone has a browser, GUI browsers have similar looking look-and-feels, the days when devs assumed Internet Explorer are long gone.

I'm as steadfastly negative about browser apps as Chris A. is about native executables.  :-)

+1000 -- ~Ethan~

On 19 Nov 2020, at 20:17, Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote:

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp

...

You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it.

It is questionable if not including the interpreter is an advantage. One use case for tools like pyinstaller to to ship applications to end users. Having to install some kind of runtime before being able to use an application is not an advantage in that case.

...

It won't have all the properties of an app bundle on MacOSX, though.

Firstly, does that matter? And secondly, what would it take to give it those additional properties?

It does if you want to create a GUI application that feels like a native application. A native bundle can have an icon, file associations, be signed, …. Adding those requires shipping as a .app bundle, which is a directory with a particular structure. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On Thu, 19 Nov 2020 at 07:42, Stéfane Fermigier <sf@fermigier.com> wrote:

But cooperation between the PyInstaller team and the Python Packaging Authority, if this doesn't happen already, could probably help.

It doesn't, but that is simply because the PyInstaller team have never reached out *asking* to be involved with the PyPA. I for one would be delighted to see installer projects (PyInstaller, cx_freeze, py2exe, probably others too) getting involved, discussing interoperability, working out better ways of addressing the needs of people who want "standalone" Python applications, etc. Paul

Hi Abdur, On 19.11.2020 10:02, Abdur-Rahmaan Janhangeer wrote:

Before asking *us*, you ought to ask what the PyInstaller developers think of the idea of:

- relinquishing copyright to the PSF; - operating under the control of the Python core developers and steering   council, under their terms; - releasing versions under the schedule of the Python interpreter; - under CPython's rules about backwards compatibility and new features;

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

I think you misunderstood Steven's questions. The PSF requires that contributors sign a contributor agreement for any code which goes into the stdlib (or Python in general). Since PyInstaller is GPLed, it cannot be added to the stdlib without the copyright owners giving the PSF permission to relicense the code under the PSF license (or any other open source license as per the contributor agreement). Only the copyright owners can make this call. Note that this does not mean "relinquishing" the copyright as Steven put it. The copyright owners keep their copyright. They only give permission specifically to the PSF to relicense the code. The other points Steve gave are important as well, since continuing the development of PyInstaller within the context of Python's stdlib means that they would have adhere to the processes we have for this. IMO, PyInstaller is a great tool, but adding it to the Python stdlib would not necessarily be an advantage, since it's development would then be tied to Python's release cycle, which reduces the flexibility the maintainers have in e.g. providing fixes quickly. Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 19 2020)

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

Summary up to now: - Must ask permission to be integrated - If integrated, tied to CPython's release cycle - They can ask the PSF for grants - It would be useful to cooperate on possible changes to CPython and the packaging landscape to make it easier to write tools like this. - Consider zipapp - there could be something in the std-lib that allowed packaging into an executable but with some limitations - transforming zipapps into executables https://docs.python.org/3/library/zipapp.html#making-a-windows-executable As for Zipapp replacing native executables, well this is not really the thread for it. Well i think i'll try to contact the PyInstaller team to see what they say Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius On Thu, Nov 19, 2020 at 1:29 PM M.-A. Lemburg <mal@egenix.com> wrote:

Hi Abdur,

On 19.11.2020 10:02, Abdur-Rahmaan Janhangeer wrote:

...

Before asking *us*, you ought to ask what the PyInstaller developers think of the idea of:

- relinquishing copyright to the PSF; - operating under the control of the Python core developers and

steering

...

council, under their terms; - releasing versions under the schedule of the Python interpreter; - under CPython's rules about backwards compatibility and new

features;

...

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

I think you misunderstood Steven's questions.

The PSF requires that contributors sign a contributor agreement for any code which goes into the stdlib (or Python in general).

Since PyInstaller is GPLed, it cannot be added to the stdlib without the copyright owners giving the PSF permission to relicense the code under the PSF license (or any other open source license as per the contributor agreement).

Only the copyright owners can make this call.

Note that this does not mean "relinquishing" the copyright as Steven put it. The copyright owners keep their copyright. They only give permission specifically to the PSF to relicense the code.

The other points Steve gave are important as well, since continuing the development of PyInstaller within the context of Python's stdlib means that they would have adhere to the processes we have for this.

IMO, PyInstaller is a great tool, but adding it to the Python stdlib would not necessarily be an advantage, since it's development would then be tied to Python's release cycle, which reduces the flexibility the maintainers have in e.g. providing fixes quickly.

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

Cheers, -- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2020)

...

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

---

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

- Some people, including me, don't think at this point this is a good idea to integrate PyInstaller in the Python code base.

The maintainer of Py2App said

FWIW I don’t think that bundling any of these tools with Python is useful at this time.

Which i overlooked as it was not the maintainer of PyInstaller speaking and i did not see any elaborattion Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius

On 20/11/20 9:17 pm, Andrew Svetlov wrote:

Digging into the problem more, I've figured out that PyInstaller has hooks <https://github.com/pyinstaller/pyinstaller/tree/develop/PyInstaller/hooks> for a bunch of popular libraries to make them work.

I've seen this sort of thing in other app bundlers too, e.g. last time I looked py2app and py2exe had a bunch of special casing for various libraries. This is quite a big problem, IMO. It makes these tools very flakey. What is it about app bundling in Python that makes these things necessary? Something seems very wrong somewhere. -- Greg

On 20 Nov 2020, at 15:03, Paul Moore <p.f.moore@gmail.com> wrote:

On Fri, 20 Nov 2020 at 13:47, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

...

I've seen this sort of thing in other app bundlers too, e.g. last time I looked py2app and py2exe had a bunch of special casing for various libraries.

This is quite a big problem, IMO. It makes these tools very flakey.

What is it about app bundling in Python that makes these things necessary? Something seems very wrong somewhere.

To be honest, I think it's more a case of a "clever idea" that everyone picked up on without really thinking about.

Once upon a time minimising bundled packages was a good thing due to having smallers disks. That’s getting less relevant every year. That said, some packages are still large and of the “kitchen sink” variety.

For my own purposes, what I *actually* want is to specify a list of 3rd party packages (in pip terms, a "requirements file") and then have the installer bundle those, plus their dependencies, plus the stdlib. I *don't* want clever logic to decide how to strip out "unused" bits. I'm more than happy to trade application size for reliability. But for some unknown reason, no installer tool even offers this more of operation as an option.

I think we'd be in a much better place (not great, but better) if application packaging tools worked on a principle of "tell me what you want, and I'll bundle it" rather than trying to infer the answer by inspecting code.

DWIM is also nice to have :-) I’m slowly working on a redesign of py2app that will something like this, with optional stripping of dependencies. This will likely still have a dependency checker by default, but one that deduces the contents of the “requirements.txt” file instead of at the module level. At the current rate of work I’ll be retired before that’s finished though. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ <https://blog.ronaldoussoren.net/>

On Fri, Nov 20, 2020 at 4:18 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 21/11/20 3:03 am, Paul Moore wrote:

...

For my own purposes, what I *actually* want is to specify a list of 3rd party packages ... I *don't* want clever logic to decide how to strip out "unused" bits.

I'm pretty sure that you can do that with PyInstller -- you certainly could with py2app. You simply specify these particular packages are to be fully copied over. Last time I worked with PyInstaller, it was a little clunkier, but I think you could still do it. With venvs, it seems like it should be possible to have a very

simple tool that just packages up everything in your venv. Such a tool ought to be quite small and shouldn't require anything like the amount of maintenance that PyInstaller and its ilk seem to need.

Am I wrong about this?

I don't think you are. And conda is similar -- there is even the "conda constructor" that will build an installer for a conda environment that meets your specs. But the fact is that you get a LOT of extra stuff along with what you need. I've had a lot of complaints as it is with my app bundles being pretty darn large (all of scipy anyone?). Earlier in this thread, I thought that the idea that the "installer" folks should work more with the PyPA folks didn't make sense - they are really trying to solve different problems. But I now am thinking that PyPA could, in fact, make it a lot easier for bundlers, by providing a standard mechanism to specify what is really needed at run time for a particular package. Maybe. Though frankly, after having been involved in this for decades (less recently), I've found that the bulk of the Python community is disinterested (if not hostile to) addressing this particular problem. Given how focus has shifted even more to web apps over the years, I don't really see that changing. But maybe I'm wrong -- I hope so! BTW, this is not the least bit a new idea. Anyone remember the old "freeze": https://wiki.python.org/moin/Freeze and https://docs.python.org/2.7/faq/programming.html?highlight=freeze#how-can-i-... It came with the standard library :-) And here's a somewhat newer summary: https://docs.python-guide.org/shipping/freezing/ -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Sat, Nov 21, 2020 at 4:04 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

...

conda is similar -- there is even the "conda constructor" that will build an installer for a conda environment that meets your specs. But the fact is that you get a LOT of extra stuff along with what you need.

My thinking is that you would only add things to the venv that your app needs, as and when you discover that they're needed. E.g. you wouldn't add the whole of PyObjC at once, just the submodules you use.

but unfortunately, that's not how most python packages are set up -- you install the whole thing at once. As an example, it's really tricky to use even one function from scipy without installing the whole thing. And that's in fact where the complications come in for the bundlers -- if you use PIL, you get all of tkInter. If you use matplotlib, you may get all of QT and who knows what other back ends. And there is the other complication that if you have a virtualenv or conda environment, you still need a single file that looks like an application to start the whole thing up -- with an icon and all that. I recently revisited this for an application that used a Browser interface with a Python server that we wanted to bundle as a desktop app (using Electron to provide the app wrapper and browser). conda constructor really didn't help -- it was easier to use PyInstaller for the Python part, and we got amore compact system. -CHB

While this would be more work for the developer, it would allow the packaging tool to be far simpler and easier to maintain.

Another benefit of working this way is that it's easier to be sure that the package you create will work on machines other than the one you developed it on. If it runs in your development environment, it should run elsewhere too.

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T27AYT... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Mon, Nov 23, 2020 at 11:56:47AM +1300, Greg Ewing wrote:

On 22/11/20 4:31 pm, Christopher Barker wrote:

...

unfortunately, that's not how most python packages are set up -- you install the whole thing at once. As an example, it's really tricky to use even one function from scipy without installing the whole thing.

Something needs to change about how Python packages are set up, then. Most other software development systems manage to strip out unused stuff without needing a bunch of flakey heuristics for each individual library.

In a language without declarations, how do you know that something is unused? One can, I guess, look for imports, but the presence of even a single eval or exec opens the flood-gates. And then there could be code that looks at sys.modules directly. Possibly via an alias: # Module spam from sys import modules as bunchofstuff # Module eggs from spam import bunchofstuff re = bunchofstuff['re'] I trust you're not going to say that if you write obfuscated code like that, you deserve whatever pain you get :-) -- Steve

On 22 Nov 2020, at 01:01, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 21/11/20 3:59 pm, Christopher Barker wrote:

...

On Fri, Nov 20, 2020 at 4:18 PM Greg Ewing <greg.ewing@canterbury.ac.nz With venvs, it seems like it should be possible to have a very simple tool that just packages up everything in your venv.

conda is similar -- there is even the "conda constructor" that will build an installer for a conda environment that meets your specs. But the fact is that you get a LOT of extra stuff along with what you need.

My thinking is that you would only add things to the venv that your app needs, as and when you discover that they're needed. E.g. you wouldn't add the whole of PyObjC at once, just the submodules you use.

While this would be more work for the developer, it would allow the packaging tool to be far simpler and easier to maintain.

That would be nice to have, but there’s no incentive for developers of packages to do this. Splitting up (large) PyPI packages is work with no clear advantages and results in more work unless you develop the multiple packages in lock step (that is, treat it as a single project other than for the release artefacts on PyPI) It is probably easier to develop a metadata standard that specifies how packages can be split up by app bundling tools. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On 20 Nov 2020, at 14:45, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

On 20/11/20 9:17 pm, Andrew Svetlov wrote:

...

Digging into the problem more, I've figured out that PyInstaller has hooks <https://github.com/pyinstaller/pyinstaller/tree/develop/PyInstaller/hooks> for a bunch of popular libraries to make them work.

I've seen this sort of thing in other app bundlers too, e.g. last time I looked py2app and py2exe had a bunch of special casing for various libraries.

This is quite a big problem, IMO. It makes these tools very flakey.

What is it about app bundling in Python that makes these things necessary? Something seems very wrong somewhere.

The runtime environment for bundles applications is slightly different than that of a regular install. Two major reasons for hooks in py2app: - Py2app tries to bundle python libraries in a zipfile, but not all libraries support that - Py2app tries to minimise the amount of modules included in the app bundle by looking at the dependency graph through import statements. This can require manual adjustment, either because of hidden imports (in an extension or by use of the __import__ function), or because modules have optional dependencies that aren’t used (a dependency between pydoc and tkinter is a good example of that) There’s bound to be stuff that currently needs hooks, that could be replaced by some (to be designed) generic solution. I’m not sure about the other bundling projects, but I expect that all of them are run by volunteers that have other projects and limited time. A bit like just about every project in the packaging space. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LNAZLM... Code of Conduct: http://python.org/psf/codeofconduct/

On 20 Nov 2020, at 08:13, Abdur-Rahmaan Janhangeer <arj.python@gmail.com> wrote:

...

- Some people, including me, don't think at this point this is a good idea to integrate PyInstaller in the Python code base.

The maintainer of Py2App said

...

FWIW I don’t think that bundling any of these tools with Python is useful at this time.

Which i overlooked as it was not the maintainer of PyInstaller speaking and i did not see any elaborattion

Wait a minute. Didn’t you ask them first?? Discussing the inclusion of a package without first asking the maintainer of a package about their opinion is not very productive at best. There are various reasons for not shipping these tools at this time: 1. None of them are clearly the best option (not that this is a sufficient reason for inclusion) 2. The tooling is not “done” in any way, including it in the stdlib would tie the release cycle to that of CPython itself 3. The tools need to adjust for changes in the underlying platform, which might require additional features 4. Some of the tools contain logic to coax 3th-party libraries to work correctly in a packaged application and need to adjust for changes in those libraries Cooperation and standardisation might be useful in for the last item, as well as for any changes needed in CPython to make it easier to use for these tools. This last one is similar to virtual environments: those were developed outside of the stdlib and at some point a stdlib version added low-level functionality that made writing tools like virtualenv easier. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

Greetings, I suggest: -- To setup PyInstaller funding via the PSF like the pallets does can be a first step to something more concrete -- To let the PSF take over the lib (requires the PSF and your approval) Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://abdur-rahmaanj.github.io/> github <https://github.com/Abdur-RahmaanJ> Mauritius On Sun, Nov 22, 2020 at 5:38 PM Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:

Am 19.11.20 um 10:29 schrieb M.-A. Lemburg:

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

I tried this without much success.

- The PSF offered to retweet my tweets - antic and no much of a help. - Python Software Verband did not even answer (AFAIR) - Major companies did not even answer. Those two or three which did answer, are not willed to give money.

-- Regards Hartmut Goebel

| Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XOLWIJ... Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Nov 19, 2020 at 01:02:17PM +0400, Abdur-Rahmaan Janhangeer wrote:

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

The Steering Council has the final say, but they don't have the only say or even the first say.

Next time someone suggests a new feature i would ask him why are you asking *us*?

For a new feature that is created from nothing, Python-Ideas is one of the best places to begin the discussion. But if you are talking about taking an existing project owned by somebody else, the first step probably should be to ask them first if they would consider moving into the std lib. If you start the conservation here, we will spend many emails discussing the pros and cons of the idea, people will argue for and against, the could be dozens or hundreds of emails and countless person-hours of thought and effort put into the discussion, and then we ask the code owners and they say "No way!". What a waste of time and effort that would be. So that's why we should start by checking with the code owners first. Even if we could legally take their code without their approval, we probably would not do so because that would be rude. -- Steve