Using Python for end user applications

On 28 January 2017 at 02:11, C Anthony Risinger <anthony@xtfx.me> wrote:
Picking up on this and the comment you made in the original post
With a still difficult distribution/compatibility story, I've watched dozens of instances where people choose something else, usually Node or Golang.
Can you explain why you recommend against Python, in a bit more detail? If you are an enthusiastic Python user, but you are steering people away from Python, then it would be worth understanding why. As you mention end user applications and distribution, one of my first questions would be what platform you work on. Following on from that, what sort of end user applications are you looking at? If we're talking here about games for iOS, then that's a much different situation than GUI apps for Windows or command line tools for Linux. My personal feeling is that Python happily works in the "Command line tools for Linux" area (except possibly with regard to C extensions where the plethora of Linux ABIs makes things hard). But other areas less so. I've been having good experiences making standalone applications with the new Windows "embedded" distributions, but that is relatively new, and still has a lot of rough edges. I'm working on a project to bundle a working zipapp with the embedded distribution to make a standalone exe - would having something like that make any difference in your environment? So I think it would be good to understand precisely where and why you feel that you need to recommend Go or Node over Python. It's possible that we have to accept that your situation is simply not a use case that Python is well suited for, but equally it may be that there's something we can do. Paul

On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm working on Linux myself, with my typical audience being other developers or occasionally leadership. Builds usually include Windows, but always Linux and OSX. I'd like a one-click solution to: How do I redistribute and successfully install Python, dependencies, and an application with the least possible steps for the end user? For any platform or persona? I prefer dynamic applications redistribute their runtime, and not depend on the system in any significant way aside from major libraries or support files. It's more obnoxious sometimes but I believe it lowers the number of unexpected incidents. I also don't want to setup support infrastructure (pypi proxy cache or self hosted pypi) myself or require coordination with ops teams. Simple authenticated download for publishing is ideal, and direct execution of that download is even better. Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools. I think there is still benefit to empowering a way to "remix python" with your local environment and produce a redistributable binary for downstream users. I have far less experience with Go but I like how easy it is to generate such executables. My personal feeling is that Python happily works in the "Command line
Yes I believe it would. I implemented something to this end for Linux (defunct, no Python 3, uses abandoned PEP 425): https://github.com/anthonyrisinger/zippy about 4-5 years ago for this purpose, before similar technologies such as PEX and zipapp existed. It worked well for the time it was used. The project is dead, but it had a few ideas I think might be worth further exploration. In a nutshell, "zippy" was a build tool (waf and distlib because no interface to pip) that always output a single python binary built from source. All python code (stdlib, deps, app) was appended in a zipfile. All C-extensions (including dotted) were observed during .so generation and re-linked statically into the binary as builtins instead (more on this below). The end result was a self-contained executable that would only recognize it's own stdlib and could properly run embedded C-modules. The implementation also changed entrypoints based on argv[0], so this could thing could perform as a multicall binary like BusyBox (the symlink name was resolved to an entrypoint). It was easy to send this to other developers where they could unpack it for development or use it as-is. A future goal was to allow respinning an unpacked environment into a new redistributable. To make this work, I changed python in a few specific ways: Set LANDMARK file at build time to something other than `os.py`. Python uses this file to recognize the stdlib during early boot. Customizing it [zippy-SOMEHASH.json] ensures a different unpacked build's stdlib is unrecognizable (this happens *often* if you've "installed" a build by unpacking to the default path, then do another build). Add `sys.executable` to the default search path. This ensures python first scans the filesystem for an unpacked environment but always falls back to its own appended zipfile. Combined with custom LANDMARK this made the python installation executable from any on-disk location. Build python locally, install app and all deps into the local build, re-link all C-extensions statically, append zipfile to python binary. I *may* have went overboard, but I generalized a method for statically linking numpy into python, I believe originally described at http://yt.enzotools.org/wiki/CrayXT5Installation . In short, distutils is monkeypatched to trace calls to `unixccompiler.UnixCCompiler.link_shared_object`. A normal object file with strict symbol exports, similar to a shared object, is written alongside the shared object, and Modules/Setup is updated. After deps are installed from pypi, python is re-linked with all the newly traced C-ext static objects. The process exposed a few interesting bugs in python and elsewhere (possibly since fixed) but was known to work for all stdlib C-extensions and every popular library we used throughout the company (psycopg2, librabbitmq, sqlite3, and dozens others were all successfully linked into the same process). This is approximately when Golang starting looking attractive to me. The past few years have seen serious improvements to this space for Python but I think there is a ways to go yet. Python has different challenges and advantages because it's older and dynamic, but I think some activity around all this might make it easier to customize Python builds. -- C Anthony

On Tue, Feb 07, 2017 at 12:05:49AM -0600, C Anthony Risinger wrote:
For Linux, Unix and Mac, where you can generally expect Python is already installed (although perhaps not the most recent version you would like), I think the best way to distribute Python apps is as a zip file: https://www.python.org/dev/peps/pep-0441/ For Windows, it's not *quite* so simple, as the end-user needs to install Python themselves.
I prefer dynamic applications redistribute their runtime,
I don't. That means that when there's a serious security bug in Python, instead of patching it once (the system Python), I have to patch two, ten, a hundred applications, some of which I don't even know are written in Python, some of which may never have a patch released, or if they do, it may be weeks or months later.
Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools.
It's the future! :-) https://circleci.com/blog/its-the-future/ On the other hand: https://circleci.com/blog/it-really-is-the-future/ -- Steve

On 7 February 2017 at 12:33, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul

You can leave python.exe out of your distribution to avoid it showing up on PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then python36.dll you should be able to put them wherever you like. I think what we really want is a self-extractor that "installs" into the user's AppData directory without prompting for admin. I'm seeing more apps about like this and I think it's the right model for Python. We probably want to include the Electron shell as well, or make it trivially easy to add (I've been using a built-in Windows tool that is similar, but it's not ideal). There's a real chance we could have very modern, cross-platform Python apps with this approach. I don't think the tools for my current project will ever see public release (though the project itself should), but the experience has been useful. Thomas's work with pynsist is also informative, at least for the issues on Windows (we're already at the point where things would be easiest if Windows 7 just went away :) ). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" <p.f.moore@gmail.com> Sent: 2/7/2017 5:03 To: "Thomas Kluyver" <thomas@kluyver.me.uk> Cc: "Python-Ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Using Python for end user applications On 7 February 2017 at 12:33, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm working on Linux myself, with my typical audience being other developers or occasionally leadership. Builds usually include Windows, but always Linux and OSX. I'd like a one-click solution to: How do I redistribute and successfully install Python, dependencies, and an application with the least possible steps for the end user? For any platform or persona? I prefer dynamic applications redistribute their runtime, and not depend on the system in any significant way aside from major libraries or support files. It's more obnoxious sometimes but I believe it lowers the number of unexpected incidents. I also don't want to setup support infrastructure (pypi proxy cache or self hosted pypi) myself or require coordination with ops teams. Simple authenticated download for publishing is ideal, and direct execution of that download is even better. Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools. I think there is still benefit to empowering a way to "remix python" with your local environment and produce a redistributable binary for downstream users. I have far less experience with Go but I like how easy it is to generate such executables. My personal feeling is that Python happily works in the "Command line
Yes I believe it would. I implemented something to this end for Linux (defunct, no Python 3, uses abandoned PEP 425): https://github.com/anthonyrisinger/zippy about 4-5 years ago for this purpose, before similar technologies such as PEX and zipapp existed. It worked well for the time it was used. The project is dead, but it had a few ideas I think might be worth further exploration. In a nutshell, "zippy" was a build tool (waf and distlib because no interface to pip) that always output a single python binary built from source. All python code (stdlib, deps, app) was appended in a zipfile. All C-extensions (including dotted) were observed during .so generation and re-linked statically into the binary as builtins instead (more on this below). The end result was a self-contained executable that would only recognize it's own stdlib and could properly run embedded C-modules. The implementation also changed entrypoints based on argv[0], so this could thing could perform as a multicall binary like BusyBox (the symlink name was resolved to an entrypoint). It was easy to send this to other developers where they could unpack it for development or use it as-is. A future goal was to allow respinning an unpacked environment into a new redistributable. To make this work, I changed python in a few specific ways: Set LANDMARK file at build time to something other than `os.py`. Python uses this file to recognize the stdlib during early boot. Customizing it [zippy-SOMEHASH.json] ensures a different unpacked build's stdlib is unrecognizable (this happens *often* if you've "installed" a build by unpacking to the default path, then do another build). Add `sys.executable` to the default search path. This ensures python first scans the filesystem for an unpacked environment but always falls back to its own appended zipfile. Combined with custom LANDMARK this made the python installation executable from any on-disk location. Build python locally, install app and all deps into the local build, re-link all C-extensions statically, append zipfile to python binary. I *may* have went overboard, but I generalized a method for statically linking numpy into python, I believe originally described at http://yt.enzotools.org/wiki/CrayXT5Installation . In short, distutils is monkeypatched to trace calls to `unixccompiler.UnixCCompiler.link_shared_object`. A normal object file with strict symbol exports, similar to a shared object, is written alongside the shared object, and Modules/Setup is updated. After deps are installed from pypi, python is re-linked with all the newly traced C-ext static objects. The process exposed a few interesting bugs in python and elsewhere (possibly since fixed) but was known to work for all stdlib C-extensions and every popular library we used throughout the company (psycopg2, librabbitmq, sqlite3, and dozens others were all successfully linked into the same process). This is approximately when Golang starting looking attractive to me. The past few years have seen serious improvements to this space for Python but I think there is a ways to go yet. Python has different challenges and advantages because it's older and dynamic, but I think some activity around all this might make it easier to customize Python builds. -- C Anthony

On Tue, Feb 07, 2017 at 12:05:49AM -0600, C Anthony Risinger wrote:
For Linux, Unix and Mac, where you can generally expect Python is already installed (although perhaps not the most recent version you would like), I think the best way to distribute Python apps is as a zip file: https://www.python.org/dev/peps/pep-0441/ For Windows, it's not *quite* so simple, as the end-user needs to install Python themselves.
I prefer dynamic applications redistribute their runtime,
I don't. That means that when there's a serious security bug in Python, instead of patching it once (the system Python), I have to patch two, ten, a hundred applications, some of which I don't even know are written in Python, some of which may never have a patch released, or if they do, it may be weeks or months later.
Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools.
It's the future! :-) https://circleci.com/blog/its-the-future/ On the other hand: https://circleci.com/blog/it-really-is-the-future/ -- Steve

On 7 February 2017 at 12:33, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul

You can leave python.exe out of your distribution to avoid it showing up on PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then python36.dll you should be able to put them wherever you like. I think what we really want is a self-extractor that "installs" into the user's AppData directory without prompting for admin. I'm seeing more apps about like this and I think it's the right model for Python. We probably want to include the Electron shell as well, or make it trivially easy to add (I've been using a built-in Windows tool that is similar, but it's not ideal). There's a real chance we could have very modern, cross-platform Python apps with this approach. I don't think the tools for my current project will ever see public release (though the project itself should), but the experience has been useful. Thomas's work with pynsist is also informative, at least for the issues on Windows (we're already at the point where things would be easiest if Windows 7 just went away :) ). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" <p.f.moore@gmail.com> Sent: 2/7/2017 5:03 To: "Thomas Kluyver" <thomas@kluyver.me.uk> Cc: "Python-Ideas" <python-ideas@python.org> Subject: Re: [Python-ideas] Using Python for end user applications On 7 February 2017 at 12:33, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
participants (5)
-
C Anthony Risinger
-
Paul Moore
-
Steve Dower
-
Steven D'Aprano
-
Thomas Kluyver