On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 January 2017 at 02:11, C Anthony Risinger <anthony@xtfx.me> wrote:
I can't articulate it we'll, or even fully isolate the reasons for it. All I really know is how I feel when peers ask me about Python or the reading I get when others speak about their experience using it. Python is absolutely one of my favorite languages to write, yet I find myself recommending against it, and watching others do the same. Python comes with caveats and detailed explanations out the gate and people simply perceive higher barriers and more chores.
Picking up on this and the comment you made in the original post
With a still difficult distribution/compatibility story, I've watched dozens of instances where people choose something else, usually Node or Golang.
Can you explain why you recommend against Python, in a bit more detail? If you are an enthusiastic Python user, but you are steering people away from Python, then it would be worth understanding why.
As you mention end user applications and distribution, one of my first questions would be what platform you work on. Following on from that, what sort of end user applications are you looking at? If we're talking here about games for iOS, then that's a much different situation than GUI apps for Windows or command line tools for Linux.
I'm working on Linux myself, with my typical audience being other developers or occasionally leadership. Builds usually include Windows, but always Linux and OSX. I'd like a one-click solution to: How do I redistribute and successfully install Python, dependencies, and an application with the least possible steps for the end user? For any platform or persona? I prefer dynamic applications redistribute their runtime, and not depend on the system in any significant way aside from major libraries or support files. It's more obnoxious sometimes but I believe it lowers the number of unexpected incidents. I also don't want to setup support infrastructure (pypi proxy cache or self hosted pypi) myself or require coordination with ops teams. Simple authenticated download for publishing is ideal, and direct execution of that download is even better. Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools. I think there is still benefit to empowering a way to "remix python" with your local environment and produce a redistributable binary for downstream users. I have far less experience with Go but I like how easy it is to generate such executables. My personal feeling is that Python happily works in the "Command line
tools for Linux" area (except possibly with regard to C extensions where the plethora of Linux ABIs makes things hard). But other areas less so. I've been having good experiences making standalone applications with the new Windows "embedded" distributions, but that is relatively new, and still has a lot of rough edges. I'm working on a project to bundle a working zipapp with the embedded distribution to make a standalone exe - would having something like that make any difference in your environment?
Yes I believe it would. I implemented something to this end for Linux (defunct, no Python 3, uses abandoned PEP 425): https://github.com/anthonyrisinger/zippy about 4-5 years ago for this purpose, before similar technologies such as PEX and zipapp existed. It worked well for the time it was used. The project is dead, but it had a few ideas I think might be worth further exploration. In a nutshell, "zippy" was a build tool (waf and distlib because no interface to pip) that always output a single python binary built from source. All python code (stdlib, deps, app) was appended in a zipfile. All C-extensions (including dotted) were observed during .so generation and re-linked statically into the binary as builtins instead (more on this below). The end result was a self-contained executable that would only recognize it's own stdlib and could properly run embedded C-modules. The implementation also changed entrypoints based on argv[0], so this could thing could perform as a multicall binary like BusyBox (the symlink name was resolved to an entrypoint). It was easy to send this to other developers where they could unpack it for development or use it as-is. A future goal was to allow respinning an unpacked environment into a new redistributable. To make this work, I changed python in a few specific ways: Set LANDMARK file at build time to something other than `os.py`. Python uses this file to recognize the stdlib during early boot. Customizing it [zippy-SOMEHASH.json] ensures a different unpacked build's stdlib is unrecognizable (this happens *often* if you've "installed" a build by unpacking to the default path, then do another build). Add `sys.executable` to the default search path. This ensures python first scans the filesystem for an unpacked environment but always falls back to its own appended zipfile. Combined with custom LANDMARK this made the python installation executable from any on-disk location. Build python locally, install app and all deps into the local build, re-link all C-extensions statically, append zipfile to python binary. I *may* have went overboard, but I generalized a method for statically linking numpy into python, I believe originally described at http://yt.enzotools.org/wiki/CrayXT5Installation . In short, distutils is monkeypatched to trace calls to `unixccompiler.UnixCCompiler.link_shared_object`. A normal object file with strict symbol exports, similar to a shared object, is written alongside the shared object, and Modules/Setup is updated. After deps are installed from pypi, python is re-linked with all the newly traced C-ext static objects. The process exposed a few interesting bugs in python and elsewhere (possibly since fixed) but was known to work for all stdlib C-extensions and every popular library we used throughout the company (psycopg2, librabbitmq, sqlite3, and dozens others were all successfully linked into the same process). This is approximately when Golang starting looking attractive to me. The past few years have seen serious improvements to this space for Python but I think there is a ways to go yet. Python has different challenges and advantages because it's older and dynamic, but I think some activity around all this might make it easier to customize Python builds. -- C Anthony