[Python-ideas] Using Python for end user applications

[C Anthony Risinger]

On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 28 January 2017 at 02:11, C Anthony Risinger <anthony at xtfx.me> wrote:
> > I can't articulate it we'll, or even fully isolate the reasons for it.
> All I
> > really know is how I feel when peers ask me about Python or the reading I
> > get when others speak about their experience using it. Python is
> absolutely
> > one of my favorite languages to write, yet I find myself recommending
> > against it, and watching others do the same. Python comes with caveats
> and
> > detailed explanations out the gate and people simply perceive higher
> > barriers and more chores.
>
> Picking up on this and the comment you made in the original post
>
> > With a still difficult distribution/compatibility story, I've watched
> dozens of instances
> > where people choose something else, usually Node or Golang.
>
> Can you explain why you recommend against Python, in a bit more
> detail? If you are an enthusiastic Python user, but you are steering
> people away from Python, then it would be worth understanding why.
>
> As you mention end user applications and distribution, one of my first
> questions would be what platform you work on. Following on from that,
> what sort of end user applications are you looking at? If we're
> talking here about games for iOS, then that's a much different
> situation than GUI apps for Windows or command line tools for Linux.
>

I'm working on Linux myself, with my typical audience being other
developers or occasionally leadership. Builds usually include Windows, but
always Linux and OSX. I'd like a one-click solution to:

How do I redistribute and successfully install Python, dependencies, and an
application with the least possible steps for the end user? For any
platform or persona?

I prefer dynamic applications redistribute their runtime, and not depend on
the system in any significant way aside from major libraries or support
files. It's more obnoxious sometimes but I believe it lowers the number of
unexpected incidents. I also don't want to setup support infrastructure
(pypi proxy cache or self hosted pypi) myself or require coordination with
ops teams. Simple authenticated download for publishing is ideal, and
direct execution of that download is even better.

Containers are have greatly impacted how people think about distribution.
There is a trend to use fast/small containers as CLI tools. I think there
is still benefit to empowering a way to "remix python" with your local
environment and produce a redistributable binary for downstream users. I
have far less experience with Go but I like how easy it is to generate such
executables.

My personal feeling is that Python happily works in the "Command line
> tools for Linux" area (except possibly with regard to C extensions
> where the plethora of Linux ABIs makes things hard). But other areas
> less so. I've been having good experiences making standalone
> applications with the new Windows "embedded" distributions, but that
> is relatively new, and still has a lot of rough edges. I'm working on
> a project to bundle a working zipapp with the embedded distribution to
> make a standalone exe - would having something like that make any
> difference in your environment?
>

Yes I believe it would. I implemented something to this end for Linux
(defunct, no Python 3, uses abandoned PEP 425):

https://github.com/anthonyrisinger/zippy

about 4-5 years ago for this purpose, before similar technologies such as
PEX and zipapp existed. It worked well for the time it was used.

The project is dead, but it had a few ideas I think might be worth further
exploration. In a nutshell, "zippy" was a build tool (waf and distlib
because no interface to pip) that always output a single python binary
built from source. All python code (stdlib, deps, app) was appended in a
zipfile. All C-extensions (including dotted) were observed during .so
generation and re-linked statically into the binary as builtins instead
(more on this below).

The end result was a self-contained executable that would only recognize
it's own stdlib and could properly run embedded C-modules. The
implementation also changed entrypoints based on argv[0], so this could
thing could perform as a multicall binary like BusyBox (the symlink name
was resolved to an entrypoint). It was easy to send this to other
developers where they could unpack it for development or use it as-is. A
future goal was to allow respinning an unpacked environment into a new
redistributable.

To make this work, I changed python in a few specific ways:

Set LANDMARK file at build time to something other than `os.py`. Python
uses this file to recognize the stdlib during early boot. Customizing it
[zippy-SOMEHASH.json] ensures a different unpacked build's stdlib is
unrecognizable (this happens *often* if you've "installed" a build by
unpacking to the default path, then do another build).

Add `sys.executable` to the default search path. This ensures python first
scans the filesystem for an unpacked environment but always falls back to
its own appended zipfile. Combined with custom LANDMARK this made the
python installation executable from any on-disk location.

Build python locally, install app and all deps into the local build,
re-link all C-extensions statically, append zipfile to python binary. I
*may* have went overboard, but I generalized a method for statically
linking numpy into python, I believe originally described at
http://yt.enzotools.org/wiki/CrayXT5Installation . In short, distutils is
monkeypatched to trace calls to
`unixccompiler.UnixCCompiler.link_shared_object`. A normal object file with
strict symbol exports, similar to a shared object, is written alongside the
shared object, and Modules/Setup is updated. After deps are installed from
pypi, python is re-linked with all the newly traced C-ext static objects.
The process exposed a few interesting bugs in python and elsewhere
(possibly since fixed) but was known to work for all stdlib C-extensions
and every popular library we used throughout the company (psycopg2,
librabbitmq, sqlite3, and dozens others were all successfully linked into
the same process).

This is approximately when Golang starting looking attractive to me. The
past few years have seen serious improvements to this space for Python but
I think there is a ways to go yet. Python has different challenges and
advantages because it's older and dynamic, but I think some activity around
all this might make it easier to customize Python builds.

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170207/eb7b49ef/attachment-0001.html>