On Thu, Nov 19, 2020 at 3:25 PM Abdur-Rahmaan Janhangeer wrote:

Greetings list,

What do you think of adding PyInstaller as an official part of CPython? Among the different native exports options, PyInstaller holds a nice track of clean delivery.

What's the advantage of having it as an official part, rather than remaining a third-party tool? Producing native executables is an attractive nuisance. It doesn't actually prevent people from disassembling your code (many MANY people seem to think that it does), it locks in a particular Python version, it locks in an OS architecture, it locks in everything that you shouldn't be locking in. Putting that sort of thing into the standard library will encourage people to use it when they really shouldn't. Also, blessing one particular executable builder means that all others become second-class citizens. Is PyInstaller so much better than everything else? It also would force PyInstaller to be released on exactly the same cadence as the Python that it comes with. It would be impossible for PyInstaller to release a hotfix between Python releases, or to add features and functionality to an already-released Python version. Against all of these disadvantages, what would be gained by bringing it into core? ChrisA

On Thu, Nov 19, 2020 at 5:39 PM Abdur-Rahmaan Janhangeer wrote:

Greetings list,

...

What's the advantage of having it as an official part, rather than remaining a third-party tool?

Easy generation of executables. Installing a version of Python gives you a complete suite

...

Producing native executables is an attractive nuisance. It doesn't actually prevent people from disassembling your code (many MANY people seem to think that it does),

The purpose is to create a packed state, not obfuscation. The ability to just run your program without worrying about further installation

...

it locks in a particular Python version, it locks in an OS architecture, it locks in everything that you shouldn't be locking in.

That's the purpose of executables.

The purpose of executables is to make it harder to apply bug fix releases of Python? I thought that was an unwanted side effect. ChrisA

Greetings list,

The purpose of executables is to make it harder to apply bug fix releases of Python? I thought that was an unwanted side effect.

I think you mis-understood me by me making myself not clear I mean bundling this library: https://github.com/pyinstaller/pyinstaller with CPython distributions. Not program executables Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

Hi, I haven't tried PyInstaller (yet) but I believe it addresses a useful need. IIUC, PyInstaller helps create single-file standalone executables for Python apps, which can be convenient for several use cases (both for server and desktop apps). If this is correct, then it's name might be a bit misleading as it's not an "installer" in the sense of https://en.wikipedia.org/wiki/Installation_(computer_programs) I don't believe the project should belong to the standard library, though. But cooperation between the PyInstaller team and the Python Packaging Authority, if this doesn't happen already, could probably help. Regards, S. On Thu, Nov 19, 2020 at 5:24 AM Abdur-Rahmaan Janhangeer < arj.python@gmail.com> wrote:

Greetings list,

What do you think of adding PyInstaller as an official part of CPython? Among the different native exports options, PyInstaller holds a nice track of clean delivery.

Instead of the project battling for its survival as any other project, if the community finds it useful enough, the devs can get some peace of mind.

The stereotyped idea about languages like Python is that they don't by default produce executables compared to let's say Go. Adding the native ability to produce executables can be a turning point for Python. In the process, PyInstaller might become even better.

Sure it will need people who are well versed in at least 3 operating systems to maintain the project. It will add to the testing load but if it's worth it, it's worth it. Though for a good DevOps team, set up is not a concern. However, having an executable makes the task easy as you just ship a final product.

I don't exactly know how executables will interact with the WSGI protocol but these concerns are for after and is an issue which does not concern integrating a library per se.

Kind Regards,

Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MCIPDY... Code of Conduct: http://python.org/psf/codeofconduct/

-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/

On Thu, Nov 19, 2020 at 08:22:40AM +0400, Abdur-Rahmaan Janhangeer wrote:

What do you think of adding PyInstaller as an official part of CPython?

Before asking *us*, you ought to ask what the PyInstaller developers think of the idea of: - relinquishing copyright to the PSF; - operating under the control of the Python core developers and steering council, under their terms; - releasing versions under the schedule of the Python interpreter; - under CPython's rules about backwards compatibility and new features; etc. Unless the PyInstaller devs are willing to do all that, there is no point to even think about moving development under CPython. We're not going to just take the project away from them. -- Steve

Greetings list,

Suppose you distribute a .py script to a million people. Your script is faulty due to a bug in the Python interpreter or std lib. But you don't need to do anything to patch your script: you just tell people to upgrade to the latest version of Python where the bug is fixed. Or you say nothing at all, and when the user's get their mandatory OS-supported upgrade, including Python, it fixes itself.

From experience it's not the only source of bug. I normally package apps with 3rd party libraries. Libs also can contain bugs. Then along the same line i can tell people to upgrade the lib to the desired version. This is true. However it boils down to whether people want executables or not. The purpose many libs exist shows that there is a need for generating native executables It's up to the developer i guess, how much Python-aware he wants the target environment to be, considering that he has the ability to tune. Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

Disclaimer: I'm not arguing for or against having pyinstaller or the like in the stdlib. Probably slightly against. Anyway... On 19Nov2020 15:59, Chris Angelico wrote:

Producing native executables is an attractive nuisance. [...stuff...]

To me, the beauty of things like pyinstaller is that they produce a _complete_ package/bundle/executable which you can just use. More to the point, that you can hand to someone else to use. I've been using py2app to build a Python+PyQt app for a Mac for a client. It makes a working MacOS .app that contains your dependencies (python itself, various libraries), and on MacOS an app is a complete package -- it contains a bin directory and the associated assets (libraries, static assets like images, what have you). There's a standard layout and the MacOS runtime linker has cool hooks to let you specify library paths _relative_ to the executable. Which means you can plonk the .app down anywhere and it just works. Getting that right yourself is _hard_. Well, I found it hard to debug when there were problems. In particular, the "find all the dependences and bundle them so that they still work" is hard. A tool which "builds an app" is very very useful - you can hand your client something whose install is "copy it here" (on a Mac, usually /Applications). And they're golden. Anyway, this is a long winded way of saying that your misgivings about executables (fixed source code and/or python version) are often a fair price to pay to "just give it to someone" and have it work. If it needs an update or bugfix I can just give them a newer one. BTW, py2app also has a "dev mode", a bit like pip's -e option, to use the "live" source tree instead of a fixed bundled source. Cheers, Cameron Simpson

On Thu, Nov 19, 2020 at 01:02:17PM +0400, Abdur-Rahmaan Janhangeer wrote:

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

The Steering Council has the final say, but they don't have the only say or even the first say.

Next time someone suggests a new feature i would ask him why are you asking *us*?

For a new feature that is created from nothing, Python-Ideas is one of the best places to begin the discussion. But if you are talking about taking an existing project owned by somebody else, the first step probably should be to ask them first if they would consider moving into the std lib. If you start the conservation here, we will spend many emails discussing the pros and cons of the idea, people will argue for and against, the could be dozens or hundreds of emails and countless person-hours of thought and effort put into the discussion, and then we ask the code owners and they say "No way!". What a waste of time and effort that would be. So that's why we should start by checking with the code owners first. Even if we could legally take their code without their approval, we probably would not do so because that would be rude. -- Steve

On 19 Nov 2020, at 08:48, Paul Moore wrote:

On Thu, 19 Nov 2020 at 07:42, Stéfane Fermigier wrote:

...

But cooperation between the PyInstaller team and the Python Packaging Authority, if this doesn't happen already, could probably help.

It doesn't, but that is simply because the PyInstaller team have never reached out *asking* to be involved with the PyPA. I for one would be delighted to see installer projects (PyInstaller, cx_freeze, py2exe, probably others too) getting involved, discussing interoperability, working out better ways of addressing the needs of people who want "standalone" Python applications, etc.

+1 (as the maintainer of py2app, yet another tool in this space). FWIW I don’t think that bundling any of these tools with Python is useful at this time. It would be useful to cooperate on possible changes to CPython and the packaging landscape to make it easier to write tools like this. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On 20/11/20 12:24 am, Chris Angelico wrote:

Have you considered the value of using zipapp

You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it.

It won't have all the properties of an app bundle on MacOSX, though. -- Greg

On Thu, 19 Nov 2020 at 15:03, Mathew Elman wrote:

Perhaps there could be something in the std-lib that allowed packaging into an executable but with some limitations, as a toy example: only supporting the std-lib dependencies. There is some precedence for minimal implementations existing in std-lib and third party libraries being more capable e.g. simple http server. So something like PyInstaller would be for when you need something more powerful, like support for third party libraries in the toy example limitation.

This could amount to adding certain extra features/behaviour to the zipapp module and a lightweight python_runner app (something like blender_runner for blender)

This would be a useful addition, but ultimately someone has to contribute the code. The recipe in the zipapp docs here https://docs.python.org/3/library/zipapp.html#making-a-windows-executable shows how you can make a standalone exe on Windows, but it's not exactly packaged up in a user friendly form, and I never really had the time to address that problem. And in any case, I don't know enough to do something similar for other operating systems (where there's no equivalent of the embedded distribution). Maybe someone could build a 3rd party implementation, and offer it for stdlib inclusion - but I'm not sure it's going to happen any other way. Paul

On 11/19/2020 2:17 PM, Steven D'Aprano wrote:

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it. It won't have all the properties of an app bundle on MacOSX, though. Firstly, does that matter? And secondly, what would it take to give it

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote: those additional properties?

It does matter. On Windows, for example, you can't use subprocess.run on a zipapp. See https://docs.python.org/3/library/zipapp.html#making-a-windows-executable. This is similar (but different) from not using .bat file wrappers for console entry points via pip, and instead creating a .exe file. But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained. Eric

On Fri, Nov 20, 2020 at 9:22 AM Eric V. Smith wrote:

On 11/19/2020 2:17 PM, Steven D'Aprano wrote:

...

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it. It won't have all the properties of an app bundle on MacOSX, though. Firstly, does that matter? And secondly, what would it take to give it

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote: those additional properties?

It does matter. On Windows, for example, you can't use subprocess.run on a zipapp. See https://docs.python.org/3/library/zipapp.html#making-a-windows-executable. This is similar (but different) from not using .bat file wrappers for console entry points via pip, and instead creating a .exe file.

But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained.

And that's exactly why I say it's an attractive nuisance. It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything". But what happens when there's a security patch for Python? Are they going to continue to not think about the dependency? Until you (manually) push out an update, they can't do a thing to fix the problem. And if they get multiple of these statically-linked Python executables, they have to have independent duplicates of all of Python, each one a snapshot at the version that someone chose to publish. Python can be installed from the Microsoft App Store or whatever it's called. Just get people to do that and then double click on the .pyz file. Is it really that hard? (Yes, I know that there will be specific situations where that's not possible. That's why the various .exe creation tools need to exist. It does NOT mean they need to be in the stdlib.) ChrisA

On Fri, Nov 20, 2020 at 9:57 AM Greg Ewing wrote:

On 20/11/20 11:32 am, Chris Angelico wrote:

...

It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything". But what happens when there's a security patch for Python? Are they going to continue to not > think about the dependency?

There seems to be an assumption here that having a natively installable app automatically means bundling everything, including the Python executable. It doesn't have to be that way. There's no reason the installed app can't use a previously installed Python or other libraries.

I still think it would be useful to have some kind of basic installer-creating functionality in the stdlib. Not bundling Python could be the default, or even the only option if you're that concerned about it.

You mean... zipapp? ChrisA

On Fri, Nov 20, 2020 at 10:05 AM Eric V. Smith wrote:

I just don't think we need to be immediately dismissive of people's desire to create a platform native executable that disguises the fact that the code is written in Python.

I never said it shouldn't happen. But having zipapp in the standard library and all native executable creators as third party apps gives a clear indication that a .pyz file should be considered first, and a native executable only if that isn't suitable. ChrisA

On 19Nov2020 18:03, Eric V. Smith wrote:

On 11/19/2020 5:32 PM, Chris Angelico wrote:

...

On Fri, Nov 20, 2020 at 9:22 AM Eric V. Smith wrote:

...

But even that doesn't help with the use case of wanting an executable without having to install Python first. I've had need to ship an executable to users who have no idea what Python is. I just want a normal Windows .exe file that's self contained.

And that's exactly why I say it's an attractive nuisance. It's very tempting to think "oh, I could just make it easier for my users, and then they don't have to think about anything".

I remain in Eric's camp here. Though thanks for informing me of zipapp. If I'm making a tool for a client or end users, _they do not know or care that it is written in Python_. They want a MacOS app or Windows .exe or whatever platform standard application which they can _just invoke_ like any other opaque application. Often, this should work on anyone's machine. So you can't assume: - particular libraries are present, particularly third party libraries (like PyQt in my personal case) - particular Python interpreters are present (if any) So bundling is often _highly desireable_. As Eric says, it's in the same vein as static linking - the required stuff is part of what you ship. With .apps (and presumably in some way in Windows) you get to supply file bindings, icons, etc. The standard natiuve experience. Back to MacOS .apps (my Windows knowledge is thin to the point of transparency), you _can_ make thin .apps, which might include a zipapp to run against the system Python. But would it contain the third party PyPI modules? The third party libraries? A sufficiently modern Python for the code? If not, in some ways I may as well just ship a small shell script which builds a venv and fills it in on first use. Static linking and application dependency bundling is _not_ an attractive nuisance, it is a common and robust way to ship something which Just Works. It may not be necessary in you environment or use cases, but in the very common case where you want users to just use something without installing a development environment to support it, it is the go. Cheers, Cameron Simpson

On 20/11/20 12:24 pm, Paul Moore wrote:

I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms.

By "installer" I just mean something that gets installed the way users expect on that platform. On Windows that means something you run that puts the right things in the right places. On MacOSX it means an app bundle that you just copy onto your disk. On Linux it probably means creating a package for whatever package manager the distro uses (not really familiar with installing desktop apps on Linux). So yes, if the usual convention on your platform means interacting with some platform-provided mechanism, then that's what needs to happen. -- Greg

On 11/19/2020 6:24 PM, Paul Moore wrote:

On Thu, 19 Nov 2020 at 22:55, Greg Ewing wrote:

...

I still think it would be useful to have some kind of basic installer-creating functionality in the stdlib. Not bundling Python could be the default, or even the only option if you're that concerned about it. I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms. But I *would* support building an "unzip and run" style of distribution.

Once  you have a native executable, a whole range of platform-specific installers is available to you. I don't think we need to invent anything new in this space. Eric

Summary up to now: - Must ask permission to be integrated - If integrated, tied to CPython's release cycle - They can ask the PSF for grants - It would be useful to cooperate on possible changes to CPython and the packaging landscape to make it easier to write tools like this. - Consider zipapp - there could be something in the std-lib that allowed packaging into an executable but with some limitations - transforming zipapps into executables https://docs.python.org/3/library/zipapp.html#making-a-windows-executable As for Zipapp replacing native executables, well this is not really the thread for it. Well i think i'll try to contact the PyInstaller team to see what they say Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius On Thu, Nov 19, 2020 at 1:29 PM M.-A. Lemburg wrote:

Hi Abdur,

On 19.11.2020 10:02, Abdur-Rahmaan Janhangeer wrote:

...

Before asking *us*, you ought to ask what the PyInstaller developers think of the idea of:

- relinquishing copyright to the PSF; - operating under the control of the Python core developers and

steering

...

council, under their terms; - releasing versions under the schedule of the Python interpreter; - under CPython's rules about backwards compatibility and new

features;

...

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

I think you misunderstood Steven's questions.

The PSF requires that contributors sign a contributor agreement for any code which goes into the stdlib (or Python in general).

Since PyInstaller is GPLed, it cannot be added to the stdlib without the copyright owners giving the PSF permission to relicense the code under the PSF license (or any other open source license as per the contributor agreement).

Only the copyright owners can make this call.

Note that this does not mean "relinquishing" the copyright as Steven put it. The copyright owners keep their copyright. They only give permission specifically to the PSF to relicense the code.

The other points Steve gave are important as well, since continuing the development of PyInstaller within the context of Python's stdlib means that they would have adhere to the processes we have for this.

IMO, PyInstaller is a great tool, but adding it to the Python stdlib would not necessarily be an advantage, since it's development would then be tied to Python's release cycle, which reduces the flexibility the maintainers have in e.g. providing fixes quickly.

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

Cheers, -- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2020)

...

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

---

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

I think you've missed a couple of points (important, IMHO) in your summary: - Some people, including me, don't think at this point this is a good idea to integrate PyInstaller in the Python code base. - There is room for cooperation between the PyInstaller developers (and users) and the Python Packaging Authority *Anyone is welcome to (while following the :ref:`Code of Conduct`_

https://www.pypa.io/en/latest/members/#id3) contribute patches, bug reports, feature requests, ideas, questions, answers, and similar information in our GitHub organization https://github.com/pypa and Bitbucket organization https://bitbucket.org/pypa repositories, and discuss issues and plans with us in the Packaging category on discuss.python.org https://discuss.python.org/c/packaging and on the distutils-sig mailing list http://mail.python.org/mailman/listinfo/distutils-sig.*

S. On Fri, Nov 20, 2020 at 7:07 AM Abdur-Rahmaan Janhangeer < arj.python@gmail.com> wrote:

Summary up to now:

- Must ask permission to be integrated - If integrated, tied to CPython's release cycle - They can ask the PSF for grants - It would be useful to cooperate on possible changes to CPython and the packaging landscape to make it easier to write tools like this. - Consider zipapp - there could be something in the std-lib that allowed packaging into an executable but with some limitations - transforming zipapps into executables https://docs.python.org/3/library/zipapp.html#making-a-windows-executable

As for Zipapp replacing native executables, well this is not really the thread for it.

Well i think i'll try to contact the PyInstaller team to see what they say

Kind Regards,

Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

On Thu, Nov 19, 2020 at 1:29 PM M.-A. Lemburg wrote:

...

Hi Abdur,

On 19.11.2020 10:02, Abdur-Rahmaan Janhangeer wrote:

...

Before asking *us*, you ought to ask what the PyInstaller developers think of the idea of:

- relinquishing copyright to the PSF; - operating under the control of the Python core developers and

steering

...

council, under their terms; - releasing versions under the schedule of the Python interpreter; - under CPython's rules about backwards compatibility and new

features;

...

Thank you for your input Mr Steven. If we go along the same lines, i should begin checking whether anyone who replies forms part of the SC or not, whether they have the right or not to reply to this thread etc.

I think you misunderstood Steven's questions.

The PSF requires that contributors sign a contributor agreement for any code which goes into the stdlib (or Python in general).

Since PyInstaller is GPLed, it cannot be added to the stdlib without the copyright owners giving the PSF permission to relicense the code under the PSF license (or any other open source license as per the contributor agreement).

Only the copyright owners can make this call.

Note that this does not mean "relinquishing" the copyright as Steven put it. The copyright owners keep their copyright. They only give permission specifically to the PSF to relicense the code.

The other points Steve gave are important as well, since continuing the development of PyInstaller within the context of Python's stdlib means that they would have adhere to the processes we have for this.

IMO, PyInstaller is a great tool, but adding it to the Python stdlib would not necessarily be an advantage, since it's development would then be tied to Python's release cycle, which reduces the flexibility the maintainers have in e.g. providing fixes quickly.

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

Cheers, -- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2020)

...

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

---

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EFFD4J... Code of Conduct: http://python.org/psf/codeofconduct/

-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/

I'm not PyInstaller user but communicated with this tool about a year ago. The story is: I'm the maintainer of multidict library and once I had a bug report like "multidict 4.7.1 causes segfault when bundled with PyInstaller". The report author said that the previous "multidict" version works fine. Sure, I fixed this annoying error by pull request https://github.com/aio-libs/multidict/pull/438. In fact, the fix just switches off C Accelerators for the library allowing PyInstaller to import the Pure Python version. It works but the performance slowdown is an order of tens. Digging into the problem more, I've figured out that PyInstaller has hooks https://github.com/pyinstaller/pyinstaller/tree/develop/PyInstaller/hooks for a bunch of popular libraries to make them work. These hooks do a dirty monkey patching. Sure, "multidict" is not covered by builtin hooks and had a problem. As a "multidict" maintainer I'm not interested in supporting the required patch in PyInstaller, switching to a slow-but-working version is totally fine with me. The story for "multidict" is done. With the CPython developer hat on, I very doubt if PyInstaller can be included in the standard library. The project exists and it is useful for many people, this is great. But it requires maintaining very many hooks for every non-trivial library to keep the PyInstaller working. This means that we, CPython core developers, should carry this maintenance burden if PyInstaller becomes a part of CPython. We will need to adopt and patch an endless list of libraries that are fine and works perfectly themself but have problems when packed with PyInstaller. Sorry, I think it never happened. Say again, PyInstaller exists. It is available on PyPI and can be installed easily by pip. PyInstaller does amazing work. But by the PyInstaller nature, it requires the maintenance of very many hooks for every non-trivial Python library. This raises an extremely high bar for the inclusion of the library into the standard batteries. Better to keep it as-is, as a third-party tool installable from PyPI. On Fri, Nov 20, 2020 at 9:14 AM Abdur-Rahmaan Janhangeer < arj.python@gmail.com> wrote:

...

- Some people, including me, don't think at this point this is a good idea to integrate PyInstaller in the Python code base.

The maintainer of Py2App said

...

FWIW I don’t think that bundling any of these tools with Python is useful at this time.

Which i overlooked as it was not the maintainer of PyInstaller speaking and i did not see any elaborattion

Kind Regards,

Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FEFBAI... Code of Conduct: http://python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov

On 2020-11-20 00:53, Chris Angelico wrote:

So what you're saying is: "Everyone else who distributes native executables has these problems, so Python apps distributed as native executables will have these problems". Yes. Of course they will. But a Python app distributed as a .py file or a .pyz archive*won't* have these problems. Is that of no value?

So are you suggesting that every single app should always be distributed as source code, run by a separate interpreter that users install separately? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Fri, 20 Nov 2020 at 08:55, Chris Angelico wrote:

On Fri, Nov 20, 2020 at 6:06 PM Brendan Barnwell wrote:

...

You've mentioned this objection at least twice now and I still don't see it having any real relevance. All kinds of programs have bugs and vulnerabilities. There is no special reason why someone should expect a program to shield them from bugs or vulnerabilities in that program's underlying components, whether that program is written in Python or any other language.

So what you're saying is: "Everyone else who distributes native executables has these problems, so Python apps distributed as native executables will have these problems". Yes. Of course they will. But a Python app distributed as a .py file or a .pyz archive *won't* have these problems. Is that of no value?

Of course it is. But it's not the *only* consideration. What you seem to be doing is dismissing any possibility that in some circumstances, the balance is in favour of bundled executables over zipapps. The discussion here (at least this part of it) is about those situations where zipapps aren't a useful solution. So *by definition*, zipapps aren't relevant as an option in that case. You may not think that any such cases exist. Fair enough. But those of us that do are trying to discuss ways to handle those situations, and saying "that will never happen" or equivalently "do you mean zipapps" repeatedly, isn't helping.

The special reason is the entire point of language interpreters. Let's suppose that there's a vulnerability discovered in the V8 JavaScript interpreter (the one behind Node.js and Google Chrome and such). Does everyone who's ever published a web app now have to push out a new version? Certainly not, and I think many web devs would be offended at the mere suggestion. They expect that a browser update will automatically fix it, and it should! Why should Python apps *not* take advantage of this separation?

It's hardly the *entire* point, but I agree it's a benefit of them. And web development is *far* from being the only use for Python. And tools like VS Code (written with Node.js, I believe) bundle the V8 engine, so contrary to the point you're trying to make, Javascript could be viewed as an example of why Python needs a way to bundle apps... Paul

On Fri, 20 Nov 2020 at 09:12, Steven D'Aprano wrote:

On Thu, Nov 19, 2020 at 11:24:32PM +0000, Paul Moore wrote:

...

I'm not sure about an installer. To me that means integrating with system "installed apps" mechanisms. But I *would* support building an "unzip and run" style of distribution.

Don't zip files support self-extracting archives? Google says that the most common archivers (WinZip, 7-Zip, WinRar, etc) do, and I've seen people talking about making self-extracting archives on Ubuntu.

Yes.

The low-hanging fruit here is probably for Python's zip library to support self-extracting archives for the most common platforms.

That would involve bundling native "zip header" self-extractors for all platforms. It would be an interesting addition to the zipfile module, but I'm not sure it's "low hanging" fruit. On the other hand, I did a quick check of zip tools (on Windows) and it's surprisingly hard to find tools that convert an existing zip into a self-extracting exe, so having the functionality in Python would be good... Ironically, the "executable stub" needed to prepend to a zipfile can't easily be written in Python, because it's a case where a native exe is absolutely necessary... For me, the low-hanging fruit still seems to be to add a couple of features: 1. Add the ability for zipapp to prepend a launcher for Windows, so zipapps can be built that "run natively" on Windows (Unix and MacOS don't need this, because for them, shebangs are handled by the OS, not by the language runtime). It *might* be possible to use the py.exe launcher, if not the sample code in the zipapp docs is probably sufficient. 2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module. 3. (Maybe not as low hanging, and something of a side issue) Add something to zipapp that uses pip to bundle 3rd party libraries. This is fragile, though, because zipapps don't handle C extensions. External tools like shiv do this better. One of these days, I'm just going to bite the bullet and do number (1), and maybe (3). But (1) needs me to dust off my C skills, which are very rusty these days :-( The harder question is around bundling an interpreter with a zipapp. That needs us to actually *have* a version of the interpreter intended for bundling, and that only exists right now on Windows. So while I'd like to see it for Windows, I think it's a platform-specific feature at least for now. And maybe that's where some of the differences in opinion come from (bundling an interpreter is much less frowned on, for better or worse, in Windows environments than it is on Linux). Paul

In fact, shiv itself sort of breaks a fundamental assumption about zipapps:

Greetings list, Since the thread is revolving around Zipapps, this thread might be useful: https://mail.python.org/archives/list/python-ideas@python.org/thread/EMP4PMT... The elaboration by Andrew Svetlov gives a full idea of how PyInstaller works. Not very clean for sure, though i have not seen other approaches. On the other hands if the use case is strong enough then it warrants it's use. After the above thread I contacted a number of Zipapp-related persons including the author of Shiv ( Loren Carvalho ) who was kind enough to reply. Along the conversation he stated that that they remain zipped! shiv pyz files extract themselves on execution, for a myriad of reasons. Considering the 3 enhancements proposed by Mr Moore, i'd say that zipapps needs a tidbit twerking to be a great packaging solution. A successful zip-based packaged format is the JAR ( https://en.wikipedia.org/wiki/JAR_(file_format)). JSmooth is a tool for converting JARs into EXEs https://en.wikipedia.org/wiki/JSmooth Might give some leads Though this thread suggests adding the natural abilities of turning python projects into executables, doing so via zipapps is an option i haven't considered. And it does not seem like it's for tomorrow. Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

On 19 Nov 2020, at 20:17, Steven D'Aprano wrote:

On Fri, Nov 20, 2020 at 02:05:02AM +1300, Greg Ewing wrote:

...

On 20/11/20 12:24 am, Chris Angelico wrote:

...

Have you considered the value of using zipapp

...

You get all the advantages of a single runnable file, and all the advantages of NOT including the full Python interpreter with it.

It is questionable if not including the interpreter is an advantage. One use case for tools like pyinstaller to to ship applications to end users. Having to install some kind of runtime before being able to use an application is not an advantage in that case.

...

It won't have all the properties of an app bundle on MacOSX, though.

Firstly, does that matter? And secondly, what would it take to give it those additional properties?

It does if you want to create a GUI application that feels like a native application. A native bundle can have an icon, file associations, be signed, …. Adding those requires shipping as a .app bundle, which is a directory with a particular structure. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On 20 Nov 2020, at 08:13, Abdur-Rahmaan Janhangeer wrote:

...

- Some people, including me, don't think at this point this is a good idea to integrate PyInstaller in the Python code base.

The maintainer of Py2App said

...

FWIW I don’t think that bundling any of these tools with Python is useful at this time.

Which i overlooked as it was not the maintainer of PyInstaller speaking and i did not see any elaborattion

Wait a minute. Didn’t you ask them first?? Discussing the inclusion of a package without first asking the maintainer of a package about their opinion is not very productive at best. There are various reasons for not shipping these tools at this time: 1. None of them are clearly the best option (not that this is a sufficient reason for inclusion) 2. The tooling is not “done” in any way, including it in the stdlib would tie the release cycle to that of CPython itself 3. The tools need to adjust for changes in the underlying platform, which might require additional features 4. Some of the tools contain logic to coax 3th-party libraries to work correctly in a packaged application and need to adjust for changes in those libraries Cooperation and standardisation might be useful in for the last item, as well as for any changes needed in CPython to make it easier to use for these tools. This last one is similar to virtual environments: those were developed outside of the stdlib and at some point a stdlib version added low-level functionality that made writing tools like virtualenv easier. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

On Fri, 20 Nov 2020 at 13:47, Greg Ewing wrote:

I've seen this sort of thing in other app bundlers too, e.g. last time I looked py2app and py2exe had a bunch of special casing for various libraries.

This is quite a big problem, IMO. It makes these tools very flakey.

What is it about app bundling in Python that makes these things necessary? Something seems very wrong somewhere.

To be honest, I think it's more a case of a "clever idea" that everyone picked up on without really thinking about. For my own purposes, what I *actually* want is to specify a list of 3rd party packages (in pip terms, a "requirements file") and then have the installer bundle those, plus their dependencies, plus the stdlib. I *don't* want clever logic to decide how to strip out "unused" bits. I'm more than happy to trade application size for reliability. But for some unknown reason, no installer tool even offers this more of operation as an option. I think we'd be in a much better place (not great, but better) if application packaging tools worked on a principle of "tell me what you want, and I'll bundle it" rather than trying to infer the answer by inspecting code. Paul

On 20 Nov 2020, at 15:03, Paul Moore wrote:

On Fri, 20 Nov 2020 at 13:47, Greg Ewing wrote:

...

I've seen this sort of thing in other app bundlers too, e.g. last time I looked py2app and py2exe had a bunch of special casing for various libraries.

This is quite a big problem, IMO. It makes these tools very flakey.

What is it about app bundling in Python that makes these things necessary? Something seems very wrong somewhere.

To be honest, I think it's more a case of a "clever idea" that everyone picked up on without really thinking about.

Once upon a time minimising bundled packages was a good thing due to having smallers disks. That’s getting less relevant every year. That said, some packages are still large and of the “kitchen sink” variety.

For my own purposes, what I *actually* want is to specify a list of 3rd party packages (in pip terms, a "requirements file") and then have the installer bundle those, plus their dependencies, plus the stdlib. I *don't* want clever logic to decide how to strip out "unused" bits. I'm more than happy to trade application size for reliability. But for some unknown reason, no installer tool even offers this more of operation as an option.

I think we'd be in a much better place (not great, but better) if application packaging tools worked on a principle of "tell me what you want, and I'll bundle it" rather than trying to infer the answer by inspecting code.

DWIM is also nice to have :-) I’m slowly working on a redesign of py2app that will something like this, with optional stripping of dependencies. This will likely still have a dependency checker by default, but one that deduces the contents of the “requirements.txt” file instead of at the module level. At the current rate of work I’ll be retired before that’s finished though. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/ https://blog.ronaldoussoren.net/

On 21/11/20 3:03 am, Paul Moore wrote:

For my own purposes, what I *actually* want is to specify a list of 3rd party packages ... I *don't* want clever logic to decide how to strip out "unused" bits.

I concur. With venvs, it seems like it should be possible to have a very simple tool that just packages up everything in your venv. Such a tool ought to be quite small and shouldn't require anything like the amount of maintenance that PyInstaller and its ilk seem to need. Am I wrong about this? -- Greg

On Fri, Nov 20, 2020 at 4:18 PM Greg Ewing wrote:

On 21/11/20 3:03 am, Paul Moore wrote:

...

For my own purposes, what I *actually* want is to specify a list of 3rd party packages ... I *don't* want clever logic to decide how to strip out "unused" bits.

I'm pretty sure that you can do that with PyInstller -- you certainly could with py2app. You simply specify these particular packages are to be fully copied over. Last time I worked with PyInstaller, it was a little clunkier, but I think you could still do it. With venvs, it seems like it should be possible to have a very

simple tool that just packages up everything in your venv. Such a tool ought to be quite small and shouldn't require anything like the amount of maintenance that PyInstaller and its ilk seem to need.

Am I wrong about this?

I don't think you are. And conda is similar -- there is even the "conda constructor" that will build an installer for a conda environment that meets your specs. But the fact is that you get a LOT of extra stuff along with what you need. I've had a lot of complaints as it is with my app bundles being pretty darn large (all of scipy anyone?). Earlier in this thread, I thought that the idea that the "installer" folks should work more with the PyPA folks didn't make sense - they are really trying to solve different problems. But I now am thinking that PyPA could, in fact, make it a lot easier for bundlers, by providing a standard mechanism to specify what is really needed at run time for a particular package. Maybe. Though frankly, after having been involved in this for decades (less recently), I've found that the bulk of the Python community is disinterested (if not hostile to) addressing this particular problem. Given how focus has shifted even more to web apps over the years, I don't really see that changing. But maybe I'm wrong -- I hope so! BTW, this is not the least bit a new idea. Anyone remember the old "freeze": https://wiki.python.org/moin/Freeze and https://docs.python.org/2.7/faq/programming.html?highlight=freeze#how-can-i-... It came with the standard library :-) And here's a somewhat newer summary: https://docs.python-guide.org/shipping/freezing/ -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Sat, Nov 21, 2020 at 1:35 PM Christopher Barker wrote:

I'm not sure I should get involved in this, but ....

On Fri, Nov 20, 2020 at 12:53 AM Chris Angelico wrote:

...

Let's suppose that there's a vulnerability discovered in the V8 JavaScript interpreter (the one behind Node.js and Google Chrome and such). Does everyone who's ever published a web app now have to push out a new version?

No, but anyone that has published an Electron app does.

Yes, and that has been a problem for me. Electron apps whose authors haven't released an update. I am unable to run them in a safer interpreter, and am forced to either continue using the one that was bundled, or stop using the app. That is *the entire point*.

Oh, and Chrome itself needs to be updated -- only on what, millions of machines? V8 is bundled with Chrome -- you know, kind of like a PyInstaller app bundles Python ;-)

Uhhh... no, that's kind of like how Python bundles Python. That's not bundling *an app*. You update Chrome once, and every app is updated. Again, thank you for restating my point, but trying to make it sound like a counter-argument.

Chris A: I ask you to let this go -- Python can be used for many different kinds of application development. And those different use cases have different needs. And Desktop GUI applications, in particular, really, really, do need a "bundling" system -- at least on the Mac and Windows.

But do all bundles have to be native executables? PLEASE reread the above posts. I have never said that native executables should never exist, but I have said that many MANY applications can and should be distributed such that they use a separate installer.

It's clear that that's not the realm you work in, which is fine, and as more and more stuff moves to the Web, there is less need for Desktop apps, but the need is still there, and PyInstaller, and Py2app and py2exe before it (and still) are absolutely critical tools in that space.

I've written desktop apps. I've distributed bundles before. I've done the "this has to be a one-click thing" thing. I do know what I'm talking about here. And no, native executables are usually NOT the only way to achieve "double click on this thing to make it run".

I've been using Python for over twenty years, and discovering py2exe and py2app way back then made it possible for me to deliver multiple products that I simply could not have done without them. (and still do). Our users have NO IDEA that they are built with Python, and that's perfect -- they can just download, install, and have a working, native, desktop app. And it will keep working, even when their sysadmin tells them they can't have Python 2 installed anymore because it's no longer maintained.

But they ARE allowed to continue using a Python 2 that was bundled with something? Is that how it goes - you're not allowed to use a legacy interpreter if you're aware of it, but it's fine as long as it's hidden behind a binary that nobody can touch? Please. Before you rehash all my arguments back at me again, READ THE THREAD. I have never said that native executables shouldn't ever exist. I have never said that zipapp is the one true way to distribute code. I have never once hinted at the notion that all users are idiots who can only ever run a .exe file, nor have I said that everyone should open source their code, or a host of other arguments that I haven't made. Please, can people stop answering arguments that haven't been made, and actually respond to what has? ChrisA

On 2020-11-20 01:51, Steven D'Aprano wrote:

If that's really what you want, you probably should look at making a way to run Python apps in the browser. Everyone has an OS, everyone has a browser, GUI browsers have similar looking look-and-feels, the days when devs assumed Internet Explorer are long gone.

I'm as steadfastly negative about browser apps as Chris A. is about native executables. :-) I realize this is an unpopular view, but I think the browser is a terrible app platform. It provides no user-configurable widget sets, so every app looks the way the app "designer" came up with, rather than all apps using a common look and feel. It has crippled access to the filesystem. Worst of all, the "app platform" is inextricably bound to a particular GUI app which cannot be versioned or used independently of the interpreter that runs other people's web apps. That is, if I like the interface of Firefox itself, but then they change it, I have no way to use the old Firefox interface while using the upgraded app engine (i.e., JS interpreter) because they're joined at the hip. All this makes using web apps a very frustrating experience for me. From my perspective bundling and locking and all those things have a very positive aspect, which is that they allow the user to "freeze" their workflow with a given tool and not be subject to changes foisted upon them by the authors. This wouldn't be such a problem if app authors were more disciplined about separating concerns, so that you could always get a security fix for some hidden bug without changing anything about how the app works from a user perspective. But since app authors don't do that, it's critical for me as a user to have a way to protect myself from an app author who suddenly decides their app should change and pulls the rug out from under my workflow. Now, it's true that something that provided the app-platform side of the browser without being tied to a particular interface would be great. That sounds more like the recent spinoff thread about a Python "app store". I think that would be even better than a native-executable-creation library, if we can get it. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 2020-11-20 20:53, Chris Angelico wrote:

...

...

So let me ask this: In what circumstances do you think producing native executables IS a bad thing, and in what circumstances do you think it ISN'T a bad thing, and why do you think including such functionality in the stdlib would encourage the former more than the latter?

It's a bad thing any time it isn't actually necessary, and it's a good

On Sat, Nov 21, 2020 at 3:35 PM Brendan Barnwell wrote: thing only when it is actually necessary. It's not my place to argue other people's use cases in specifics, but I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work.

Okay, but that doesn't answer the second part of my question. You were saying earlier that you wanted people to read your posts and not claim that you're saying native executables shouldn't exist. So I'm trying to bring the discussion back to the original more narrow topic, which is the pros and cons of including native executable creation in the stdlib. Just saying "it's bad if it's not necessary" is a cop-out in this context. The question is why do you think that including such functionality in the stdlib is bad. Your response here is saying it's up to other people to decide whether to use it or not, but that doesn't explain why you think it shouldn't be in the stdlib. Everything in the stdlib can be used if people think it's necessary and not used if they don't.

Under what circumstances should you use subprocesses and pipes in a Python program, and under what circumstances should you just keep everything inside a single process? Should you just always use boilerplate that runs everything in multiple processes in case you need to run it across multiple CPU cores? No. Does this mean I'm arguing against ever using multiple processes? No.

This example seems to quite contradict the view that executable bundling shouldn't be in the stdlib. You're saying "don't create a native executable unless necessary" and "don't use multiple processes unless necessary". But we do have multiprocessing in the stdlib. Why is including a native executable creator in the stdlib any more insidious than including multiprocessing? If people aren't throwing multiprocessing boilerplate in where they don't need it, why should we assume they'll throw in native-executable boilerplate where they don't need it? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

Greetings list,

Use the simpler options until you can't use them. Then use the more complicated options.

The only thing i don't trust Mr Chris on is packaging. His technical Py views are sound and great. I have the impression that concerning packaging the view is: we can do with what we have, no need to add more. When enhancing Zipapps was proposed, if i remember correctly, the view was along the lines of: "Why Zipapp when you can double click on a Py file". Now concerning native executables, the view runs along the lines of "at least consider Zipapps".

I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work.

I have the impression that everybody talking about native executables here were talking about an optional way of packaging apps which help people and not as the preferred, standard and de facto way. Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

On Sat, Nov 21, 2020 at 4:07 PM Brendan Barnwell wrote:

On 2020-11-20 20:53, Chris Angelico wrote:

...

...

...

So let me ask this: In what circumstances do you think producing native executables IS a bad thing, and in what circumstances do you think it ISN'T a bad thing, and why do you think including such functionality in the stdlib would encourage the former more than the latter?

It's a bad thing any time it isn't actually necessary, and it's a good

On Sat, Nov 21, 2020 at 3:35 PM Brendan Barnwell wrote: thing only when it is actually necessary. It's not my place to argue other people's use cases in specifics, but I'm just saying that the default should be to NOT bundle the interpreter, and you only reach for a native executable if that doesn't work.

Okay, but that doesn't answer the second part of my question. You were saying earlier that you wanted people to read your posts and not claim that you're saying native executables shouldn't exist. So I'm trying to bring the discussion back to the original more narrow topic, which is the pros and cons of including native executable creation in the stdlib.

Just saying "it's bad if it's not necessary" is a cop-out in this context. The question is why do you think that including such functionality in the stdlib is bad. Your response here is saying it's up to other people to decide whether to use it or not, but that doesn't explain why you think it shouldn't be in the stdlib. Everything in the stdlib can be used if people think it's necessary and not used if they don't.

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one? ChrisA

On Sat, Nov 21, 2020 at 07:16:22AM -0500, Edwin Zimmerman wrote:

On 11/20/2020 11:53 PM, Chris Angelico wrote:

...

[snip] Use the simpler options until you can't use them. Then use the more complicated options.

Yea, use the simpler options.  This is why I have switched from Python to C# when writing desktop apps, simply because distributing a single exe file is easier at all levels than distributing a complete install of Python.

Decrease in effort packaging your app as an exe file: -10% Increase in effort writing your app in C# in the first place: +800% *semi-wink* But seriously... 1. *None of* the stdlib freeze tool, or any of the third-party projects cx_Freeze, py2app, py2exe or PyInstaller, work for you? 2. Python is pre-installed on pretty much all Linux and BSD systems, and Mac OS, and is trivially installed on Windows from the Microsoft app store. Getting a complete install of Python couldn't be simpler. Where does the process let you down? -- Steve

On 21Nov2020 17:54, Chris Angelico wrote:

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one?

Well, for my personal example, an in house tool for an in house task, the users _can_ copy a Mac .app into their Applications folder. I _do not_ and would not expect them to (likely choice): - install macports or homebrew - install python - install source code and libraries (needed PyQt) They're end users; some are (variously) technical and some aren't, but none should need to be technical. I want them to copy an app to a new machine and be happy - drag'n'drop a single thing. Cheers, Cameron Simpson

On Sat, Nov 21, 2020 at 4:04 PM Greg Ewing wrote:

...

conda is similar -- there is even the "conda constructor" that will build an installer for a conda environment that meets your specs. But the fact is that you get a LOT of extra stuff along with what you need.

My thinking is that you would only add things to the venv that your app needs, as and when you discover that they're needed. E.g. you wouldn't add the whole of PyObjC at once, just the submodules you use.

but unfortunately, that's not how most python packages are set up -- you install the whole thing at once. As an example, it's really tricky to use even one function from scipy without installing the whole thing. And that's in fact where the complications come in for the bundlers -- if you use PIL, you get all of tkInter. If you use matplotlib, you may get all of QT and who knows what other back ends. And there is the other complication that if you have a virtualenv or conda environment, you still need a single file that looks like an application to start the whole thing up -- with an icon and all that. I recently revisited this for an application that used a Browser interface with a Python server that we wanted to bundle as a desktop app (using Electron to provide the app wrapper and browser). conda constructor really didn't help -- it was easier to use PyInstaller for the Python part, and we got amore compact system. -CHB

While this would be more work for the developer, it would allow the packaging tool to be far simpler and easier to maintain.

Another benefit of working this way is that it's easier to be sure that the package you create will work on machines other than the one you developed it on. If it runs in your development environment, it should run elsewhere too.

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T27AYT... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 22Nov2020 13:22, Steven D'Aprano wrote:

On Sun, Nov 22, 2020 at 12:56:01PM +1100, Cameron Simpson wrote:

...

On 21Nov2020 17:54, Chris Angelico wrote:

...

The range of people who (a) cannot install from PyPI and can only use the stdlib, and (b) cannot deploy with a .pyz and must deploy an .exe, is extremely narrow. In what situation do you have to make a native executable but cannot get a tool from PyPI to make one?

Well, for my personal example, an in house tool for an in house task, the users _can_ copy a Mac .app into their Applications folder. I _do not_ and would not expect them to (likely choice): - install macports or homebrew - install python - install source code and libraries (needed PyQt)

They're end users; some are (variously) technical and some aren't, but none should need to be technical. I want them to copy an app to a new machine and be happy - drag'n'drop a single thing.

Sure, but why does PyInstaller have to be in the std lib for you to support this use-case?

It doesn't. I'm not arguing for that; I'm at least slightly against it in fact. I'm _still_ arguing against Chris' apparent position that wanting to make a standalone bundle is pretty much a null situation. I think I've let his "attractive nuisance" description get under my skin. Still, as you say:

That was Chris' question: Under what circumstances does somebody have to make a native application but **cannot use one of the tools on PyPI to do it**? (Or the freeze tool.)

It is true that I, at least, am not in that position: needing to make a standalone app and unable to access PyPI to get tools to build it. I may have misconstrued Chris to mean my users rather than myself. (His full question is still quoted at the top of this message for reference.) And on that basis I withdraw my example. But it definitely applies to my users: thier install process should be a trivial copy-this-app experience. That does require a complete bundle. Cheers, Cameron Simpson

Am 19.11.20 um 10:29 schrieb M.-A. Lemburg:

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

I tried this without much success. * The PSF offered to retweet my tweets - antic and no much of a help. * Python Software Verband did not even answer (AFAIR) * Major companies did not even answer. Those two or three which did answer, are not willed to give money. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

Am 20.11.20 um 11:01 schrieb Paul Moore:

1. Add the ability for zipapp to prepend a launcher for Windows, so zipapps can be built that "run natively" on Windows (Unix and MacOS don't need this, because for them, shebangs are handled by the OS, not by the language runtime). It *might* be possible to use the py.exe launcher, if not the sample code in the zipapp docs is probably sufficient. 2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module.

So basically re-implement py2exe, py2app, PyInstaller, etc?! -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

On Sun, 22 Nov 2020 at 11:32, Hartmut Goebel wrote:

Am 20.11.20 um 11:01 schrieb Paul Moore:

...

1. Add the ability for zipapp to prepend a launcher for Windows, so zipapps can be built that "run natively" on Windows (Unix and MacOS don't need this, because for them, shebangs are handled by the OS, not by the language runtime). It *might* be possible to use the py.exe launcher, if not the sample code in the zipapp docs is probably sufficient. 2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module.

So basically re-implement py2exe, py2app, PyInstaller, etc?!

Definitely not. I'm thinking of something *far* more limited. Specifically, things I'm not suggesting be in the stdlib: 1. Bundling the Python binaries. 2. Stripping out unneeded modules. 3. Executables forms other than "single file". 4. Support for C extensions. I would be open to using the "embedded distribution" on Windows to ship such an executable with a dedicated Python interpreter, but unless CPython starts distributing an embedded distribution for other platforms, that's as far as I'd go. Paul Paul

On Sun, Nov 22, 2020 at 6:34 AM Paul Moore wrote:

...

Am 20.11.20 um 11:01 schrieb Paul Moore:

...

...

2. Add something that makes writing MacOS "app bundles" from zipapps easier. That could be part of the zipapp module, or a standalone module.

Definitely not. I'm thinking of something *far* more limited. Specifically, things I'm not suggesting be in the stdlib:

1. Bundling the Python binaries. 2. Stripping out unneeded modules. 3. Executables forms other than "single file". 4. Support for C extensions.

That last one if VERY limiting :-( But anyway, I haven't used it in a long time, but py2app has (or had) and "alias" mode, that would would make a app bundle that references the installed python. I think it would not take much to make it include all non-stdlib code -- or, even easier, to include what the user specifies. But I'm not sure of the utility of this really -- Linux is the only platform that you can expect a consistent "system Python" on. (Apple has supplied one for years, but it's never been properly maintained and suitable to use this way) I'm curious about zipapp -- I've heard of it, but never tried to use it -- does it get much use in the wild? My feeling is that it hits middle ground that isn't very useful. If you can count on your users having a proper Python installation ,then they can use pip to install your package and run your scripts. If they can't do that, then they likely need a full bundle. But I could be wrong there. -CHB

I would be open to using the "embedded distribution" on Windows to ship such an executable with a dedicated Python interpreter, but unless CPython starts distributing an embedded distribution for other platforms, that's as far as I'd go.

Paul

Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4IEG7I... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

3. Executables forms other than "single file".

Greetings list, 4. Support for C extensions. @CHB, @Barnwell I haven't replied in full to a previous email of Mr Moore, but the no support for C extensions is a crucial point for packaging. If the stdlib has to support a packaging tool, it has to not provide support for C extensions. It has to do with the nature of Python. The indecisiveness of to include or not C extensions halted the packaging issue. Python is very backwards (in terms of packaging) compared to where it should have been given it's maturity and adoption index. C is not in the spirit of Python. Currently, the only tool included in the stdlib for packaging is Zipapps. As such i am 100% for Zipapps not to include C extensions and to be in a single file. This is in the spirit of packaging for VM-based languages. In other words we must enhance Zipapps as far as possible. The branding of language X runs on billions of devices is as true as it is for Python. We just have to enhance Zipapp more. Notes: - Single files: one must be aware has some severe limitations such as you cannot pack sqlite dbs and save your changes. You can only read data. This applies not only for Zipapps but also for .exe and even jar files. You have to include the path to the resources. This is a reasonable choice. This is why many projects has the structure: folder1/ folder2/ folder3/ main_compressed_entry_point The resources are expressed as relative paths and the tool, much like PyInstaller does, gives you your project with codefiles compressed while keeping such a structure - C extensions: Everybody is welcomed to find ideas on how to execute C extensions while maintaining a Zipped state. In the lack of such a solution, it's better to Zipapps to leave it out. That's why including the ability to have native executables is a must for Python. Python is different from other languages in the sense that for a dynamic lang, it has close ties with C which gives it fearful powers. To sum up: For pure Py projects, use Zipapps. For projects including C-extensions use the tool for native executables. Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

Greetings, I suggest: -- To setup PyInstaller funding via the PSF like the pallets does can be a first step to something more concrete -- To let the PSF take over the lib (requires the PSF and your approval) Kind Regards, Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius On Sun, Nov 22, 2020 at 5:38 PM Hartmut Goebel wrote:

Am 19.11.20 um 10:29 schrieb M.-A. Lemburg:

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

I tried this without much success.

- The PSF offered to retweet my tweets - antic and no much of a help. - Python Software Verband did not even answer (AFAIR) - Major companies did not even answer. Those two or three which did answer, are not willed to give money.

-- Regards Hartmut Goebel

| Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XOLWIJ... Code of Conduct: http://python.org/psf/codeofconduct/

Also, virtualenv is distributed as a zipapp these days. Paul On Mon, 23 Nov 2020 at 08:44, Steven D'Aprano wrote:

On Sun, Nov 22, 2020 at 07:37:35PM -0800, Christopher Barker wrote:

...

I'm curious about zipapp -- I've heard of it, but never tried to use it -- does it get much use in the wild?

I am quite confident that one of the most used, if controversial, Python applications in the wild is a zipapp on Linux systems.

http://youtube-dl.org/

-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/2NNFSC... Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, 23 Nov 2020 at 10:39, Mathew Elman wrote:

I would argue this sounds like a case for a "python_runner", i.e. a lightweight python vm that can run python apps, e.g. zipapps. Something like the blender_runner for blender. Making it explicitly an app that runs via some other engine.

It would then be a central entry-point that can be updated to a later python and all the apps update with it. It would have to provide a utility for desktop shortcuts to feel like the python app was launching natively.

That's essentially like Java's JRE. For what it's worth, on my PC the JRE is 196M in size. Whereas a full Python distribution is only 94M, and the embedded distribution is 15M. So I think Python already has that, more or less. But my experience with the JRE is that very few applications actually use the "centralised" JRE, preferring to ship an embedded JRE. So the evidence from the Java world is that managing a central "runtime engine" is not actually as convenient as we're assuming, and the arguments for it simply aren't compelling in the real world. (Unix may have a different picture - there's a much stronger culture there of "depend on system packages" :shrug:) Paul

I suppose functionally there may be little difference, but having an explicit "runner" would allow two things: 1. the runner and stdlib for it could be in a compress format itself since it doesn't need to provide any utility for editing or navigating human readable modules. and so lighter 2. it would handle "venv"ing and creating desktop shortcuts etc that would just start the python app in the engine. The reason I mention this here, is because others have said that including functionality to deal with this sort of thing for _all_ OS would be heavy, where as here it could be more lightweight and even not part of the standard python interpreter, just the runner. Also on systems with python already installed it could be told to use the existing python install, if that was desired.

That's essentially like Java's JRE. For what it's worth, on my PC the JRE is 196M in size. Whereas a full Python distribution is only 94M, and the embedded distribution is 15M. So I think Python already has that, more or less.

I hadn't realised that, so thanks :)

But my experience with the JRE is that very few applications actually use the "centralised" JRE, preferring to ship an embedded JRE. So the evidence from the Java world is that managing a central "runtime engine" is not actually as convenient as we're assuming, and the arguments for it simply aren't compelling in the real world. (Unix may have a different picture - there's a much stronger culture there of "depend on system packages" :shrug:) Paul

I imagine there would be a way to have an install install the runner if there is not one on the machine, and use the existing one if there is, and creating a venv in either case. Meaning that using the equivalent of an embedded JRE would be obsolete, no? I suppose my point is that having a utility for launching a python app from the desktop may be sufficient for a lot of use cases.

I cannot speak for the Board, since I'm not currently on it. I think it's *probable* they'd be willing to be a fiscal sponsor for PyInstaller, as they have for some other things, but only if the developers really want that. All that really gets is an ability for donators to flag funds as pass-through to a specific project or organization, it doesn't make money in itself. I find it enormously unlikely PSF would want to take over the library (or any other library). The PSF grant budget has diminished quite a bit with Covid-19 disrupting the PyCon conference that was the main source of income. Grants still exist, but they are selective. If you do want to contact them, the address is: PSF Board < psf-board@python.org> On Mon, Nov 23, 2020 at 8:33 AM Abdur-Rahmaan Janhangeer < arj.python@gmail.com> wrote:

Greetings,

I suggest:

-- To setup PyInstaller funding via the PSF like the pallets does can be a first step to something more concrete -- To let the PSF take over the lib (requires the PSF and your approval)

Kind Regards,

Abdur-Rahmaan Janhangeer about https://compileralchemy.github.io/ | blog https://abdur-rahmaanj.github.io/ github https://github.com/Abdur-RahmaanJ Mauritius

On Sun, Nov 22, 2020 at 5:38 PM Hartmut Goebel < h.goebel@crazy-compilers.com> wrote:

...

Am 19.11.20 um 10:29 schrieb M.-A. Lemburg:

Since the project appears to be struggling a bit, it may be worthwhile having the project owners ask the PSF or major company users for a grant.

I tried this without much success.

- The PSF offered to retweet my tweets - antic and no much of a help. - Python Software Verband did not even answer (AFAIR) - Major companies did not even answer. Those two or three which did answer, are not willed to give money.

-- Regards Hartmut Goebel

| Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XOLWIJ... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3526D5... Code of Conduct: http://python.org/psf/codeofconduct/

-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.

On Tue, Nov 24, 2020 at 6:47 PM Christopher Barker wrote:

On Mon, Nov 23, 2020 at 12:09 AM Paul Moore wrote:

...

On Mon, 23 Nov 2020 at 03:37, Christopher Barker wrote:

...

My feeling is that it hits middle ground that isn't very useful. If you can count on your users having a proper Python installation ,then they can use pip to install your package and run your scripts.

If they can't do that, then they likely need a full bundle.

But I could be wrong there.

You're more or less right, although your perspective on what's useful may be biased a little by the fact that (I believe) you're a heavy numpy user.

indeed I am. But also, where I need this sort of thing is for desktop GUIs (wxPython in my case), another heavy user of compiled extensions.

I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package??

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top), and if you can't do that, bundle it into a .pyz with zipapp and, again, put a shebang for posix platforms. Windows, if the py.exe launcher is installed, will happily let you double-click on a .py or .pyz and it'll run just fine. (Annoyingly, you cannot simply exec to this sort of thing on Windows, since it's only a GUI feature. But I believe you can use ShellExecute to invoke them. Can't wait on their output though.) ChrisA

On Mon, 23 Nov 2020 08:09:07 +0000 Paul Moore wrote:

But it's not as limiting as you suggest - it *does* preclude most scientific use (because of numpy etc) but (for example) a large number of web libraries are pure Python.

Not sure what you mean here, but while Web frameworks themselves may be pure Python, you can have C accelerators in a template engine or in a ORM layer. Also, the database driver most likely isn't in pure Python (if you want it to be performant anyway). Regards Antoine.

On Tue, 24 Nov 2020 10:23:10 +0000 Paul Moore wrote:

On Tue, 24 Nov 2020 at 10:18, Antoine Pitrou wrote:

...

On Mon, 23 Nov 2020 08:09:07 +0000 Paul Moore wrote:

...

But it's not as limiting as you suggest - it *does* preclude most scientific use (because of numpy etc) but (for example) a large number of web libraries are pure Python.

Not sure what you mean here, but while Web frameworks themselves may be pure Python, you can have C accelerators in a template engine or in a ORM layer. Also, the database driver most likely isn't in pure Python (if you want it to be performant anyway).

All I meant was that how limiting it is depends on what type of application you're trying to write.

Sure, but the number of applications which don't depend whatsoever on non-stdlib C extensions is probably much smaller than you were trying to say. You should not fool yourself: the suggested "zipapp" (does it exist already? I lost track of the number of weird things that have been implemented in core Python in the name of packaging) would mostly be a non-solution. Regards Antoine.

On Mon, Nov 23, 2020 at 11:56:47AM +1300, Greg Ewing wrote:

On 22/11/20 4:31 pm, Christopher Barker wrote:

...

unfortunately, that's not how most python packages are set up -- you install the whole thing at once. As an example, it's really tricky to use even one function from scipy without installing the whole thing.

Something needs to change about how Python packages are set up, then. Most other software development systems manage to strip out unused stuff without needing a bunch of flakey heuristics for each individual library.

In a language without declarations, how do you know that something is unused? One can, I guess, look for imports, but the presence of even a single eval or exec opens the flood-gates. And then there could be code that looks at sys.modules directly. Possibly via an alias: # Module spam from sys import modules as bunchofstuff # Module eggs from spam import bunchofstuff re = bunchofstuff['re'] I trust you're not going to say that if you write obfuscated code like that, you deserve whatever pain you get :-) -- Steve

On Wed, Nov 25, 2020 at 10:13 AM Greg Ewing wrote:

On 24/11/20 9:05 pm, Chris Angelico wrote:

...

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top) ...

But to most people a "proper point and click GUI thing" includes having an actual GUI, which is somewhere between very difficult and impossible to do well without C extensions that don't come with a standard Python installation.

True. If you want a perfectly out-of-the-box app, you're probably going to have to stick to Tkinter. ChrisA

On 25/11/20 11:48 am, Steven D'Aprano wrote:

In a language without declarations, how do you know that something is unused?

Personally, I don't mind if I have to *tell* it what I'm using. I don't insist on the stripping-out being automatic, and I would actually prefer it not to be. To support that, I want a system that lets me specify exactly what I'm using, during development *and* packaging, both working off the same information. That way I can be sure that if it runs for me, it will also run for my users.

# Module eggs from spam import bunchofstuff re = bunchofstuff['re']

I trust you're not going to say that if you write obfuscated code like that, you deserve whatever pain you get :-)

With the scheme described above, I'll find out during testing if I've forgotten to add re as a dependency. And if I don't, then it's my fault for not testing thoroughly enough. -- Greg

On Wed, Nov 25, 2020 at 10:29 AM Brendan Barnwell wrote:

On 2020-11-24 00:05, Chris Angelico wrote:

...

...

...

I'm still confused what the point is of a zipapp, if it can't be a proper point and click GUI thing, and it can't use any compiled extensions. How it is it better than a console_script and a pip-installed package??

It CAN be a proper point-and-click GUI thing. You can have a fully executable Python script if it has no dependencies (just distribute a single .py file with a shebang at the top), and if you can't do that, bundle it into a .pyz with zipapp and, again, put a shebang for posix platforms. Windows, if the py.exe launcher is installed, will happily let you double-click on a .py or .pyz and it'll run just fine.

I think by "proper point-and-click GUI thing" he meant the app itself is a GUI app, not just "can be launched via the OS GUI". At least, that is what I would mean by that, and that is what can't be done without C extensions, because most GUI toolkits require C extensions.

Python ships with one GUI toolkit (Tkinter). If you can't install anything else, you at least get that. (On Linux systems, it's sometimes in a separate package, but so are other parts of Python too, so it's hard to predict. I generally assume that, if you have to depend on the standard library only, you can at least depend on python3-all or the equivalent.) ChrisA

On 2020-11-25 00:42, Greg Ewing wrote:

On 25/11/20 12:14 pm, Chris Angelico wrote:

...

If you want a perfectly out-of-the-box app, you're probably going to have to stick to Tkinter.

Which is only bundled with Python on Windows, as far as I know.

It's also bundled with Python on Raspbian.

On 11/20/20 8:41 PM, Brendan Barnwell wrote:

On 2020-11-20 01:51, Steven D'Aprano wrote:

...

If that's really what you want, you probably should look at making a way to run Python apps in the browser. Everyone has an OS, everyone has a browser, GUI browsers have similar looking look-and-feels, the days when devs assumed Internet Explorer are long gone.

I'm as steadfastly negative about browser apps as Chris A. is about native executables.  :-)

+1000 -- ~Ethan~