PEP: Improving Python ZIP Application Support
Python ZIP Application Support - https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr... PEP: 4XX Title: Improving Python ZIP Application Support Author: Daniel Holth <dholth@gmail.com> Status: Draft Type: Standards Track Python-Version: 3.4 Created: 30 March 2013 Post-History: 30 March 2013 Improving Python ZIP Application Support Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be for two reasons: a) users haven’t heard of it because it wasn’t mentioned in earlier versions of Python 2.6 “What’s New” document, and b) Windows users don’t have a file extension (other than .py) to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP applications” and “Windowed Python Zip Applications”, and providing some simple tooling to manage the format. A New Python ZIP Application Extension The Python 3.4 installer will associate .pyz and .pyzw “Python ZIP Applications” with itself so they can be executed by the Windows launcher. A .pyz archive is a console application and a .pyzw archive is a windowed application. This indicates whether the console should appear when running the app. Why not use .zip or .py? Users expect a .zip file would be opened with an archive tool, and users expect .py to be opened with a text editor. Both would be confusing for this use case. For UNIX users, .pyz applications should be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation. #!/usr/bin/env python # This is a Python application stored in a ZIP archive. (binary contents of archive) As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The pyzaa Module This PEP also proposes including a simple application for working with Python ZIP Archives: The Python Zip Application Archiver “pyzaa” (rhymes with “huzzah” or “pizza”). “pyzaa” can archive or extract these files, compile bytecode, and can write the __main__ module if it is not present. Usage python -m pyzaa (pack | unpack | compile) python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c] [-w] [-p interpreter] directory: ZIP the contents of directory as directory.pyz or [-w] directory.pyzw. Adds the executable flag to the archive. -c compile .pyc files and add them to the archive -p interpreter include #!interpreter as the first line of the archive -o path/name archive is written to path/name.pyz[w] instead of dirname. The extension is added if not specified. -m module.submodule:callable __main__.py is written as “import module.submodule; module.submodule.callable()” pyzaa pack will warn if the directory contains C extensions or if it doesn’t contain __main__.py. python -m pyzaa unpack arcname.pyz[w] The archive is unpacked into a directory [arcname] python -m pyzaa compile arcname.pyz[w] The Python files in arcname.pyz[w] are compiled and appended to the ZIP file. References [1] http://bugs.python.org/issue1739468 “Allow interpreter to execute a zip file” Copyright This document has been placed into the public domain.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/30/2013 03:22 PM, Daniel Holth wrote:
Python ZIP Application Support - https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr...
+1,
but I think this actually belongs on python-dev (AFAICT it is unrelated to distutils, setuptools, etc.) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFXbbQACgkQ+gerLs4ltQ5+OgCgqP75xYWG8TGxiT3efvjZWe5U YT4AoKTQJMhkbkU4KBoqmhOCDUl+5/Xo =ERp8 -----END PGP SIGNATURE-----
On Mar 30, 2013 3:23 PM, "Daniel Holth" <dholth@gmail.com> wrote:
Python ZIP Application Support -
https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr...
PEP: 4XX
Title: Improving Python ZIP Application Support
Author: Daniel Holth <dholth@gmail.com>
Status: Draft
Type: Standards Track
Python-Version: 3.4
Created: 30 March 2013
Post-History: 30 March 2013
Improving Python ZIP Application Support
Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules.
This feature is not as popular as it should be for two reasons: a) users haven’t heard of it because it wasn’t mentioned in earlier versions of Python 2.6 “What’s New” document, and b) Windows users don’t have a file extension (other than .py) to associate with the launcher.
This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP applications” and “Windowed Python Zip Applications”, and providing some simple tooling to manage the format.
A New Python ZIP Application Extension
The Python 3.4 installer will associate .pyz and .pyzw “Python ZIP Applications” with itself so they can be executed by the Windows launcher. A .pyz archive is a console application and a .pyzw archive is a windowed application. This indicates whether the console should appear when running the app.
Why not use .zip or .py? Users expect a .zip file would be opened with an archive tool, and users expect .py to be opened with a text editor. Both would be confusing for this use case.
For UNIX users, .pyz applications should be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation.
#!/usr/bin/env python
# This is a Python application stored in a ZIP archive.
... built using pyzaa.
(binary contents of archive)
As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work.
Minimal Tooling: The pyzaa Module
This PEP also proposes including a simple application for working with Python ZIP Archives: The Python Zip Application Archiver “pyzaa” (rhymes with “huzzah” or “pizza”). “pyzaa” can archive or extract these files, compile bytecode, and can write the __main__ module if it is not present.
Usage
python -m pyzaa (pack | unpack | compile)
python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c] [-w] [-p interpreter] directory:
ZIP the contents of directory as directory.pyz or [-w] directory.pyzw. Adds the executable flag to the archive.
-c compile .pyc files and add them to the archive
-p interpreter include #!interpreter as the first line of the archive
Would `/usr/bin/env python` (or python3 depending on interpreter used to compile) be set otherwise? Or how about the specific python version to prevent possible future-compatibility issues (e.g. specifying python3.3)?
-o path/name archive is written to path/name.pyz[w] instead of dirname. The extension is added if not specified.
-m module.submodule:callable __main__.py is written as “import module.submodule; module.submodule.callable()”
pyzaa pack will warn if the directory contains C extensions or if it doesn’t contain __main__.py.
python -m pyzaa unpack arcname.pyz[w]
The archive is unpacked into a directory [arcname]
Is this truly necessary? If it's a zip file any archiving tool can unzip it. Heck, we can add a CLI to the ziofile module if it doesn't already have one.
python -m pyzaa compile arcname.pyz[w]
The Python files in arcname.pyz[w] are compiled and appended to the ZIP file.
I would suggest allowing multiple versions for compilation (when specified). There should also be a check that people don't specify multiple versions that can't exist on the same directory (e.g 2.6 and 2.7). Otherwise the compileall module's CLI handles this and people can call it with different interpreters as necessary. IOW I'm advocating KISS for what the tool does. Since making the zip file is only really tricky bit it should only handle that case. Heck you can make it part of the zipfile module if there is resistance to adding yet another script that python installs (obviously will need a Cheeseshop package for older versions). Otherwise I like everything else. -Brett
References
[1] http://bugs.python.org/issue1739468 “Allow interpreter to execute a zip file”
Copyright
This document has been placed into the public domain. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Mar 30, 2013 at 7:07 PM, Brett Cannon <brett@yvrsfo.ca> wrote:
On Mar 30, 2013 3:23 PM, "Daniel Holth" <dholth@gmail.com> wrote:
Python ZIP Application Support -
https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr...
PEP: 4XX
Title: Improving Python ZIP Application Support
Author: Daniel Holth <dholth@gmail.com>
Status: Draft
Type: Standards Track
Python-Version: 3.4
Created: 30 March 2013
Post-History: 30 March 2013
Improving Python ZIP Application Support
Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules.
This feature is not as popular as it should be for two reasons: a) users haven’t heard of it because it wasn’t mentioned in earlier versions of Python 2.6 “What’s New” document, and b) Windows users don’t have a file extension (other than .py) to associate with the launcher.
This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP applications” and “Windowed Python Zip Applications”, and providing some simple tooling to manage the format.
A New Python ZIP Application Extension
The Python 3.4 installer will associate .pyz and .pyzw “Python ZIP Applications” with itself so they can be executed by the Windows launcher. A .pyz archive is a console application and a .pyzw archive is a windowed application. This indicates whether the console should appear when running the app.
Why not use .zip or .py? Users expect a .zip file would be opened with an archive tool, and users expect .py to be opened with a text editor. Both would be confusing for this use case.
For UNIX users, .pyz applications should be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation.
#!/usr/bin/env python
# This is a Python application stored in a ZIP archive.
... built using pyzaa.
(binary contents of archive)
As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work.
Minimal Tooling: The pyzaa Module
This PEP also proposes including a simple application for working with Python ZIP Archives: The Python Zip Application Archiver “pyzaa” (rhymes with “huzzah” or “pizza”). “pyzaa” can archive or extract these files, compile bytecode, and can write the __main__ module if it is not present.
Usage
python -m pyzaa (pack | unpack | compile)
python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c] [-w] [-p interpreter] directory:
ZIP the contents of directory as directory.pyz or [-w] directory.pyzw. Adds the executable flag to the archive.
-c compile .pyc files and add them to the archive
-p interpreter include #!interpreter as the first line of the archive
Would `/usr/bin/env python` (or python3 depending on interpreter used to compile) be set otherwise? Or how about the specific python version to prevent possible future-compatibility issues (e.g. specifying python3.3)?
-o path/name archive is written to path/name.pyz[w] instead of dirname. The extension is added if not specified.
-m module.submodule:callable __main__.py is written as “import module.submodule; module.submodule.callable()”
pyzaa pack will warn if the directory contains C extensions or if it doesn’t contain __main__.py.
python -m pyzaa unpack arcname.pyz[w]
The archive is unpacked into a directory [arcname]
Is this truly necessary? If it's a zip file any archiving tool can unzip it. Heck, we can add a CLI to the ziofile module if it doesn't already have one.
It is a convenience so that the contents don't wind up in $PWD, I like the idea of dropping it though. I'll be sure to emphasize that these are completely standard ZIP archives with a new extension. I'm pretty sure zipfile has a CLI. The pack command is a convenience on top of zip & cat. Hope we don't need ignore / MANIFEST.in type features...
python -m pyzaa compile arcname.pyz[w]
The Python files in arcname.pyz[w] are compiled and appended to the ZIP file.
I would suggest allowing multiple versions for compilation (when specified). There should also be a check that people don't specify multiple versions that can't exist on the same directory (e.g 2.6 and 2.7). Otherwise the compileall module's CLI handles this and people can call it with different interpreters as necessary.
If that's easy with compileall then great.
IOW I'm advocating KISS for what the tool does. Since making the zip file is only really tricky bit it should only handle that case. Heck you can make it part of the zipfile module if there is resistance to adding yet another script that python installs (obviously will need a Cheeseshop package for older versions).
Otherwise I like everything else.
Thanks!
-Brett
References
[1] http://bugs.python.org/issue1739468 “Allow interpreter to execute a zip file”
Copyright
This document has been placed into the public domain. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Mar 31, 2013 at 9:07 AM, Brett Cannon <brett@yvrsfo.ca> wrote:
I would suggest allowing multiple versions for compilation (when specified). There should also be a check that people don't specify multiple versions that can't exist on the same directory (e.g 2.6 and 2.7). Otherwise the compileall module's CLI handles this and people can call it with different interpreters as necessary.
IOW I'm advocating KISS for what the tool does. Since making the zip file is only really tricky bit it should only handle that case. Heck you can make it part of the zipfile module if there is resistance to adding yet another script that python installs (obviously will need a Cheeseshop package for older versions).
Otherwise I like everything else.
Agreed. However, Tres is right that this is a python-dev PEP rather than a distutils-sig one :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Mar 30, 2013 at 8:22 PM, Daniel Holth <dholth@gmail.com> wrote:
Python ZIP Application Support - https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr... PEP: 4XX Title: Improving Python ZIP Application Support So I guess that this already-available-yet-hidden-or-little-known feature we had since Python 2.6 will be getting a little light.
Let me ask a few silly questions: Does this means that any zip with a __main__.py is de-facto already executable? What about a wheel with a __main__ ? or an egg? Or a source archive where the __main__ calls setup.py install or buildout bootstrap? Is this something to promote? How is this overlapping with other packaging approaches? or possibly replacing them all? -- Philippe Ombredanne +1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
On Mar 31, 2013 1:09 PM, "Philippe Ombredanne" <pombredanne@nexb.com> wrote:
On Sat, Mar 30, 2013 at 8:22 PM, Daniel Holth <dholth@gmail.com> wrote:
Python ZIP Application Support -
https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr...
PEP: 4XX Title: Improving Python ZIP Application Support So I guess that this already-available-yet-hidden-or-little-known feature we had since Python 2.6 will be getting a little light.
Let me ask a few silly questions:
Does this means that any zip with a __main__.py is de-facto already executable? What about a wheel with a __main__ ? or an egg? Or a source archive where the __main__ calls setup.py install or buildout bootstrap? Is this something to promote? How is this overlapping with other packaging approaches? or possibly replacing them all?
Yes regardless of the extension, yes, yes, not really, doesn't overlap much. A __main__ at the root of a wheel or egg would be a problem since it would wind up in site packages, shadowing other mains. Wheel itself uses a __main__ in a sub path of the archive. Python app.zip/subpath - that works too. Generally if you can have an installer you don't want the zip application strategy. It is best for medium complexity scripts or pure python applications where the user just wants to use the software and not build on top of it. We don't want packages to self-install except in special cases. It doesn't leave enough control to the end user.
-- Philippe Ombredanne
+1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
On Sun, Mar 31, 2013 at 1:09 PM, Philippe Ombredanne <pombredanne@nexb.com> wrote:
On Sat, Mar 30, 2013 at 8:22 PM, Daniel Holth <dholth@gmail.com> wrote:
Python ZIP Application Support - https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr... PEP: 4XX Title: Improving Python ZIP Application Support So I guess that this already-available-yet-hidden-or-little-known feature we had since Python 2.6 will be getting a little light.
Let me ask a few silly questions:
Does this means that any zip with a __main__.py is de-facto already executable? Yes.
What about a wheel with a __main__ ? or an egg? Yes.
Or a source archive where the __main__ calls setup.py install or buildout bootstrap?
Yep.
Is this something to promote?
Why not?
How is this overlapping with other packaging approaches? or possibly replacing them all?
It helps make things easier to install that are themselves installation tools. By the way, after some experimenting this morning, I have figured out how to make this work with Python 2.3 and up in a fairly simple way. It turns out that if you stick a .py file on the front of a zipfile, and you end it with a '#' and a few special characters, then when you run the zipfile with 2.6+, the __main__ is executed, but for 2.3-2.5, the .py header is executed. If this header does sys.path.insert(0, __file__), it then can import things from the zipfile. So, with an appropriate header, you can make a one-size-fits-all executable zipfile. In practice, there are a couple of wrinkles. The magic terminator string, at least for the Windows and Linux boxes I've tested so far, is '\n#\x1a\n\x00\n'. But there are more platforms and builds I *haven't* tested on than those I have.
On Sun, Mar 31, 2013 at 5:43 PM, PJ Eby <pje@telecommunity.com> wrote:
In practice, there are a couple of wrinkles. The magic terminator string, at least for the Windows and Linux boxes I've tested so far, is '\n#\x1a\n\x00\n'. But there are more platforms and builds I *haven't* tested on than those I have.
Oops. That's supposed to be '\n\x00\n#\x00\x04\x1a' -- the other string was from an earlier series of tests, before I realized that an x04 in the zipfile itself was doing part of the work for me.
On 30.03.2013 20:22, Daniel Holth wrote:
Python ZIP Application Support - https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnr...
PEP: 4XX
Title: Improving Python ZIP Application Support
Author: Daniel Holth <dholth@gmail.com>
Status: Draft
Type: Standards Track
Python-Version: 3.4
Created: 30 March 2013
Post-History: 30 March 2013
Improving Python ZIP Application Support
Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules.
This feature is not as popular as it should be for two reasons: a) users haven’t heard of it because it wasn’t mentioned in earlier versions of Python 2.6 “What’s New” document, and b) Windows users don’t have a file extension (other than .py) to associate with the launcher.
c) The feature has never really been documented outside the ticket where it got added to Python :-) Ticket for this: http://bugs.python.org/issue17359 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 01 2013)
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2013-03-25: Released mxODBC 3.2.2 ... http://egenix.com/go40 2013-04-10: Python Meeting Duesseldorf ... 9 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Tue, Apr 2, 2013 at 6:12 AM, M.-A. Lemburg <mal@egenix.com> wrote:
c) The feature has never really been documented outside the ticket where it got added to Python :-)
Not true, it's covered in the command line docs at http://docs.python.org/2/using/cmdline.html#interface-options It's just that people generally assume they know how our CLI works, so they never read that. It's also in the runpy.run_path docs, but that's even more obscure.
Ticket for this: http://bugs.python.org/issue17359
I have updated that to reference the existing documentation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Apr 2, 2013 at 6:12 AM, M.-A. Lemburg <mal@egenix.com> wrote:
c) The feature has never really been documented outside the ticket where it got added to Python :-) Not true, it's covered in the command line docs at http://docs.python.org/2/using/cmdline.html#interface-options The doc has a bug BTW: "Changed in version 2.5: Directories and zipfiles containing a __main__.py file at the top level are now considered valid Python
On Tue, Apr 2, 2013 at 12:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: scripts." I tested it and this does not work in Pythin 2.5, only in 2.6 -- Philippe Ombredanne +1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
Nick Coghlan <ncoghlan <at> gmail.com> writes:
Not true, it's covered in the command line docs at http://docs.python. org/2/using/cmdline.html#interface-options
It's just that people generally assume they know how our CLI works, so they never read that.
Well, people won't open a Web browser to read a CLI's documentation when they can simply type "python --help". $ ./python --help usage: ./python [option] ... [-c cmd | -m mod | file | -] [arg] ... [...] file : program read from script file - : program read from stdin (default; interactive mode if a tty) There's nothing about a zip file here ;-) And even the much more comprehensive man page doesn't mention it either: $ grep -i zip Misc/python.man $ Regards Antoine.
participants (8)
-
Antoine Pitrou -
Brett Cannon -
Daniel Holth -
M.-A. Lemburg -
Nick Coghlan -
Philippe Ombredanne -
PJ Eby -
Tres Seaver