PEP 441 - Improving Python ZIP Application Support
I'm looking at putting together a patch for CPython to implement PEP 441. In doing so, there are a few issues that I'd like to raise with the PEP. These are all to do with the supporting app "pyzaa" (IIRC, Nick proposed renaming this to "pyzapp", which I like, but it's not a big deal either way). 1. The PEP says that the application will warn if you try to add C extensions to the archive. In practice, this will be quite hard to do - extensions vary across platforms, and it's not only C extensions that won't work from an archive (consider a DLL with a pure-python ctypes wrapper). We could check the filenames for "known" extensions (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies by platform) but I think that's going to be fragile. IMO, the costs aren't worth the benefits, and I'd like to remove this proposal and simply document that applications packed up with pyzaa need to be tested to ensure they work from a zipfile. 2. The option to compile Python source seems to me to be of limited use. It ties the archive to a particular Python version (more precisely, only one version gets the benefits of precompiled bytecode). It's also tricky to implement, particularly the option to compile an existing archive, because there standard library py_compile and compileall modules work on the filesystem, and for compiling files in an archive it would be better to build the bytecode in memory (no need to use tempfiles). I'd prefer to say that if users want to include bytecode, they can precompile on disk (using compileall) for as many versions as they want. FWIW, Daniel's standalone pyzaa program doesn't currently include either of the above features. As well as removing things, I'd like to add a programmable API for the utility. At the moment I only have a rough idea of how that would work, but I'm thinking of a PyZApp class, which can be created from a filename or an open stream (like a ZipFile object) and which has methods to set the shebang line, setting the main entry point, and adding files or directories. Does that need to go into the PEP or is it OK to review the API as part of the patch? I don't mind either way, but my normal coding process tends to be to fiddle with the API as I code it up and use it in the main app, so I'd normally leave documenting the final API till it's written... If the above seems like a reasonable plan, I'll work up a patch with the intention that it gets approved and implemented in time for 3.5. Paul
On 15 February 2015 at 00:44, Paul Moore <p.f.moore@gmail.com> wrote:
I'm looking at putting together a patch for CPython to implement PEP 441. In doing so, there are a few issues that I'd like to raise with the PEP. These are all to do with the supporting app "pyzaa" (IIRC, Nick proposed renaming this to "pyzapp", which I like, but it's not a big deal either way).
1. The PEP says that the application will warn if you try to add C extensions to the archive. In practice, this will be quite hard to do - extensions vary across platforms, and it's not only C extensions that won't work from an archive (consider a DLL with a pure-python ctypes wrapper). We could check the filenames for "known" extensions (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies by platform) but I think that's going to be fragile. IMO, the costs aren't worth the benefits, and I'd like to remove this proposal and simply document that applications packed up with pyzaa need to be tested to ensure they work from a zipfile.
2. The option to compile Python source seems to me to be of limited use. It ties the archive to a particular Python version (more precisely, only one version gets the benefits of precompiled bytecode). It's also tricky to implement, particularly the option to compile an existing archive, because there standard library py_compile and compileall modules work on the filesystem, and for compiling files in an archive it would be better to build the bytecode in memory (no need to use tempfiles). I'd prefer to say that if users want to include bytecode, they can precompile on disk (using compileall) for as many versions as they want.
FWIW, Daniel's standalone pyzaa program doesn't currently include either of the above features.
"Do the simplest thing that could possibly work" applies, so I'd be OK with dropping both of those features.
As well as removing things, I'd like to add a programmable API for the utility. At the moment I only have a rough idea of how that would work, but I'm thinking of a PyZApp class, which can be created from a filename or an open stream (like a ZipFile object) and which has methods to set the shebang line, setting the main entry point, and adding files or directories. Does that need to go into the PEP or is it OK to review the API as part of the patch? I don't mind either way, but my normal coding process tends to be to fiddle with the API as I code it up and use it in the main app, so I'd normally leave documenting the final API till it's written...
I'd suggest including the API in the PEP, which can co-evolve with the reference implementation. The following suggestions are ones I came up with last time I looked closely at the draft PEP: * use "zipapp" for the support module name, distinct from the command line tool name (think venv vs pyvenv) * expose a zipapp.pack programmatic interface that takes all the input pieces and builds a suitable archive I'd definitely prefer a simple procedural API like that over offering a public stateful object-oriented API, as the latter significantly increases testing complexity without offering a sufficiently compelling gain in expressiveness to justify the additional upfront test development cost and the ongoing maintenance and support burden. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 February 2015 at 05:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'd definitely prefer a simple procedural API like that over offering a public stateful object-oriented API, as the latter significantly increases testing complexity without offering a sufficiently compelling gain in expressiveness to justify the additional upfront test development cost and the ongoing maintenance and support burden.
The main advantage of a class API is the ability to support building an archive from data in memory (or by collecting parts from different areas on the filesystem). That's something I've had a use for on a number of occasions. OTOH, if the pyz/pyzw extensions are already (or will be) registered by the installer then the only remaining feature of this PEP [1] is the archive creation app. And designing that as we go is probably the wrong way to get something in the stdlib. Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? There's already at least pyzaa and pyzzer available... Paul [1] Apart from the benefit of publicising executable zip files as a means of deploying Python code by mentioning them in a "What's new" section :-)
On 15 February 2015 at 08:14, Paul Moore <p.f.moore@gmail.com> wrote:
Maybe it would be better to put something on PyPI and let it develop outside the stdlib first?
The only place where a ".pyz" file can't easily be manipulated with the stdlib zipfile module is in writing a shebang line to the start of the archive (i.e. adding "prefix" bytes before the start of the zipfile data). It would be nice if the ZipFile class supported this (because to do it properly you need access to the file object that the ZipFile object wraps). Would it be reasonable to add methods to the ZipFile class to read and write the prefix data? Paul
On 15.02.15 10:47, Paul Moore wrote:
On 15 February 2015 at 08:14, Paul Moore <p.f.moore@gmail.com> wrote:
Maybe it would be better to put something on PyPI and let it develop outside the stdlib first?
The only place where a ".pyz" file can't easily be manipulated with the stdlib zipfile module is in writing a shebang line to the start of the archive (i.e. adding "prefix" bytes before the start of the zipfile data). It would be nice if the ZipFile class supported this (because to do it properly you need access to the file object that the ZipFile object wraps). Would it be reasonable to add methods to the ZipFile class to read and write the prefix data?
But the stdlib zipfile module supports this. with open(filename, 'wb') as f: f.write(shebang) with zipfile.PyZipFile(f, 'a') as zf: ...
On 15 February 2015 at 16:00, Serhiy Storchaka <storchaka@gmail.com> wrote:
Would it be reasonable to add methods to the ZipFile class to read and write the prefix data?
But the stdlib zipfile module supports this.
with open(filename, 'wb') as f: f.write(shebang) with zipfile.PyZipFile(f, 'a') as zf:
Good point. It's not quite as easy to read the prefix data. I think you can do it by reopening the file, and reading the bytes up to byte min(i.header_offset for i in zf.infolist()). You need to reopen the file, though, and that calculation doesn't work for an empty zipfile (where you need to read to the start of the central directory instead). But it's sort of possible, and it's a pretty specialised requirement anyway (I only really needed it for testing). Thanks, Paul
On 15 February 2015 at 18:14, Paul Moore <p.f.moore@gmail.com> wrote:
OTOH, if the pyz/pyzw extensions are already (or will be) registered by the installer then the only remaining feature of this PEP [1] is the archive creation app. And designing that as we go is probably the wrong way to get something in the stdlib. Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? There's already at least pyzaa and pyzzer available...
Perhaps rather than creating yet another implementation of this, we should just add some "See Also" links to the relevant parts of the command line usage guide and the runpy module documentation? Particularly if the modules are pip-installable and we add the cross-references to the 2.7 and 3.4 docs as well - we didn't install pip by default back when Daniel first wrote PEP 441.
[1] Apart from the benefit of publicising executable zip files as a means of deploying Python code by mentioning them in a "What's new" section :-)
That will be covered to some degree by the mention of the file associations being registered on Windows, and we can try to persuade Larry it's worth mentioning in the release announcement itself :) The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 February 2015 at 08:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later.
Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as "python -m zipapp" (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Paul
On 15 February 2015 at 23:21, Paul Moore <p.f.moore@gmail.com> wrote:
On 15 February 2015 at 08:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later.
Ok, thinking about this a little more.
Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly.
Which brings us full circle. A simple module, executable as "python -m zipapp" (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module.
Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py?
Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult.
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't
If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI.
Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch.
Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 February 2015 at 13:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult.
Fine with me - the wrappers are basically useless on Windows where Tools/Scripts isn't on PATH. Paul
FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that *does* support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopen_with_offset as described by ppluzhnikov in https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should also be possible using dlopen_ehdr/dlopen_phdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding _writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain. It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this. -- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On 15 February 2015 at 16:21, Thomas Wouters <thomas@python.org> wrote:
It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this.
Those cases sound more like things to consider for Python's "executable zip" support. The zipapp module is purely concerned with taking a bunch of stuff and making a zipfile-with-shebang "python-executable zipfile" out of it. Of course if Python needed certain files to be aligned a particular way in order to execute the zipfile, that *would* be something that zipapp should take into account. But I'd say let's wait till zipfile execution needs the feature before adding it to zipapp. Paul
On 15.02.15 18:21, Thomas Wouters wrote:
which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything
It is possible to add this feature to ZipFile. It can be useful because will allow to mmap uncompressed files in ZIP file.
On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 15.02.15 18:21, Thomas Wouters wrote:
which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything
It is possible to add this feature to ZipFile. It can be useful because will allow to mmap uncompressed files in ZIP file.
One of the cool things about ZIP files is that they ignore any data that is not referenced by the index (which is at the end of the file). That is why we can put a #! at the beginning of the file, prefix .exe self-extractors, or put extra data between files in the archive. You could probably even overlap archive members if you were motivated. But it would be best to let the uncompressed extensions be standard ZIP members.
On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote:
FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that *does* support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopen_with_offset as described by ppluzhnikov in https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should also be possible using dlopen_ehdr/dlopen_phdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding _writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain.
So, let's add this to zipimport so we can remove an unfortunate limitation and avoid everyone having to reinvent the wheel! Oh, and the dlopen extension sound pretty cool. Can't wait until Google gets that into glibc so us mere mortals can use it. Seems reasonable to add this to zipimport too, with whatever feature guards are necessary. You'll want to give your future self an opportunity to trot out the keys to the time machine. :)
It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all.
+1
I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this.
+1 Cheers, -Barry
On 17 February 2015 at 19:00, Barry Warsaw <barry@python.org> wrote:
It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all.
+1
+1 from me also. Anything that makes it easier for packages to work under zipimport. But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport). Paul
On Feb 17, 2015, at 08:58 PM, Paul Moore wrote:
But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport).
The actual "import an extension module" support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too. I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me. Cheers, -Barry
On 17 February 2015 at 21:10, Barry Warsaw <barry@python.org> wrote:
The actual "import an extension module" support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too.
Oh, absolutely. As long as the functionality is supported everywhere, the implementation can be whatever is needed.
I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me.
Yeah, it's not at all clear to me what people's intended use cases are for pyz files. Personally, I want them for writing scripts I can copy between my various PCs without hassle. (So single file saves me from having to make sure I have dependencies installed everywhere, but sys.executable ties me to a specific install location). Your use case seems similar (but on Unix, where there's less likelihood that Python will be installed in different places). I don't know if anyone is seeing them as a distribution format (i.e., they put their software up on their website for anyone to use). If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format). Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go. Paul
On Feb 17, 2015, at 09:45 PM, Paul Moore wrote:
If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format).
Agreed!
Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go.
I think that's totally reasonable. Admittedly, I was hijacking your PEP discussion for some pie-in-the-sky musing. :) Cheers, -Barry
Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 15 February 2015 at 23:21, Paul Moore <p.f.moore@gmail.com> wrote:
On 15 February 2015 at 08:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later.
Ok, thinking about this a little more.
Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly.
Which brings us full circle. A simple module, executable as "python -m zipapp" (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module.
Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py?
Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult.
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't
If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI.
Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch.
Sounds good to me.
Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
"Go ahead, make my pep." We should make a python-dev t-shirt with this on it :) Top-posted from my Windows Phone ________________________________ From: Daniel Holth<mailto:dholth@gmail.com> Sent: 2/15/2015 9:46 To: Nick Coghlan<mailto:ncoghlan@gmail.com> Cc: Paul Moore<mailto:p.f.moore@gmail.com>; Steve Dower<mailto:Steve.Dower@microsoft.com>; Python Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, "Nick Coghlan" <ncoghlan@gmail.com<mailto:ncoghlan@gmail.com>> wrote: On 15 February 2015 at 23:21, Paul Moore <p.f.moore@gmail.com<mailto:p.f.moore@gmail.com>> wrote:
On 15 February 2015 at 08:59, Nick Coghlan <ncoghlan@gmail.com<mailto:ncoghlan@gmail.com>> wrote:
The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later.
Ok, thinking about this a little more.
Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly.
Which brings us full circle. A simple module, executable as "python -m zipapp" (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module.
Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py?
Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult.
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't
If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI.
Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch.
Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com<mailto:ncoghlan@gmail.com> | Brisbane, Australia
On 15/02/2015 18:06, Steve Dower wrote:
"Go ahead, make my pep."
We should make a python-dev t-shirt with this on it :)
I'll buy one provided p&p isn't too high :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On 15 February 2015 at 17:46, Daniel Holth <dholth@gmail.com> wrote:
Go ahead, make my pep.
I will appreciate seeing it happen.
Here is a draft update for PEP 441. It's still a work in progress - in particular I want to wait for consensus on the issue of the default interpreter before finalising it. But I thought it would be worth having a full spec available for people. PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth@gmail.com>, Paul Moore <p.f.moore@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support ======================================== Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than .py) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP Applications” and “Windowed Python ZIP Applications”, and providing some simple tooling to manage the format. A New Python ZIP Application Extension ====================================== The Python 3.5 installer will associate .pyz and .pyzw “Python ZIP Applications” with the platform launcher so they can be executed. A .pyz archive is a console application and a .pyzw archive is a windowed application, indicating whether the console should appear when running the app. For UNIX users, .pyz applications should typically be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) However, it is always possible to execute a .pyz application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module ================================== This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. Module Interface ---------------- The zipapp module will provide the following functions: ``pack(target, directory, interpreter=None, main=None)`` Writes an application archive called ``target``, containing the contents of ``directory``. If ``interpreter`` is specified, it will be written to the start of the archive as a shebang line and the file will be made executable (if no interpreter is specified, the shebang line will be omitted). If the directory contains no ``__main__.py`` file, the function will construct a ``__main__.py`` which calls the function specified in the ``main`` argument (which should be in the form ``"pkg.mod:fn"``). It is an error to specify ``main`` if the directory contains a ``__main__.py``, or to omit ``main`` when there is no ``__main__.py`` (as that will result in an archive which has no main function and so cannot be executed). ``get_interpreter(archive)`` Returns the interpreter specified in the shebang line of the archive. If there is no shebang, the function returns None. ``set_interpreter(archive, new_archive, interpreter=None)`` Modifies the archive's shebang line to contain the specified interpreter, and writes the updated archive to new_archive. If the interpreter is None, removes the shebang line. Command Line Usage ------------------ The zipapp module can be run with the python -m flag. The command line interface is as follows:: python -m zipapp [options] directory Create an archive from the contents of the given directory. By default, an archive will be created with the same name as the source directory, with a .pyz extension. The following options can be specified: -o archive The destination archive will have the specified name. -p interpreter The given interpreter will be written to the shebang line of the archive. If this option is not given, the archive will have no shebang line. -m pkg.mod:fn The source directory must not have a __main__.py file. The archiver will write a __main__.py file into the target which calls fn from the module pkg.mod. The behaviour of the command line interface matches that of ``zipapp.pack()``. As noted, the archives are standard zip files, and so can be unpacked using any standard ZIP utility or Python’s zipfile module. FAQ --- Are you sure a standard ZIP utility can handle #! at the beginning? Absolutely. The zipfile specification allows for arbitrary data to be prepended to a zipfile. This feature is commonly used by "self-extracting zip" programs. If your archive program can't handle this, it is a bug in your archive program. Isn’t zipapp just a very thin wrapper over the zipfile module? Yes. If you prefer to build your own Python zip application archives using other tools, they will work just as well. The zipapp module is a convenience, nothing more. Why not use just use a .zip or .py extension? Users expect a .zip file to be opened with an archive tool, and expect a .py file to contain readable text. Both would be confusing for this use case. How does this compete with existing package formats? The sdist, bdist and wheel formats are designed for packaging of modules to be installed into an existing Python installation. They are not intended to be used without installing. The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script. Rejected Proposals ================== Convenience Values for Shebang Lines ------------------------------------ Is it worth having "convenience" forms for any of the common interpreter values? For example, ``-p 3`` meaning the same as ``-p "/usr/bin/env python3"``. It would save a lot of typing for the common cases, as well as giving cross-platform options for people who don't want or need to understand the intricacies of shebang handling on "other" platforms. Downsides are that it's not obvious how to translate the abbreviations. For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3", "python3", or something else? Also, there is no obvious short form for the key case of "/usr/bin/env python" (any available version of Python), which could easily result in scripts being written with overly-restrictive shebang lines. Overall, this seems like there are more problems than benefits, and as a result has been dropped from consideration. Open Questions ============== Default Interpreter ------------------- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. Unix users have problems with this behaviour, as the default for the python command on many distributions is Python 2, and it is felt that this PEP should prefer Python 3 by default. However, using a command of ``python3`` can result in unexpected behaviour for Windows users, where the default behaviour of the launcher for the command "python" is commonly customised by users, but the behaviour of "python3" may not be modified to match. Currently, the principle "in the face of ambiguity, refuse to guess" has been invoked, and archives have no shebang line unless explicitly requested. On Windows, the archives will still be run (with the default Python) by the launcher, and on Unix, the archives can be run by explicitly invoking the desired Python interpreter. This issue is currently under active discussion on python-dev, and the results will be reflected here when consensus has been reached. Command Line Tool to Manage Shebang Lines ----------------------------------------- It is conceivable that users would want to modify the shebang line for an existing archive, or even just display the current shebang line. This is tricky to do so with existing tools (zip programs typically ignore prepended data totally, and text editors can have trouble editing files containing binary data). The zipapp module provides functions to handle the shebang line, but should these be exposed via the command line interface? At the moment, the PEP proposes *not* to provide a command line interface for these functions, as it is not clear how to provide one without the resulting interface being over-complex and potentially confusing. References ========== .. [1] “Allow interpreter to execute a zip file” (http://bugs.python.org/issue1739468) .. [2] “Feature is not documented” (http://bugs.python.org/issue17359) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Feb 16, 2015, at 11:21 PM, Paul Moore wrote:
Default Interpreter -------------------
The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter.
The other reasonable alternative for a default shebang is sys.executable. Cheers, -Barry
On 17 February 2015 at 18:56, Barry Warsaw <barry@python.org> wrote:
The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter.
The other reasonable alternative for a default shebang is sys.executable.
That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5... Paul
On Feb 17, 2015, at 08:53 PM, Paul Moore wrote:
On 17 February 2015 at 18:56, Barry Warsaw <barry@python.org> wrote:
The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter.
The other reasonable alternative for a default shebang is sys.executable.
That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5...
But it doesn't matter (at least to me). The scenarios I want to support don't include building a pyz on my Linux machine, giving it to you, and having you able to run it natively on your Windows machine. There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. Don't discount sys.executable to quickly. :) Cheers, -Barry
On 02/17/2015 01:13 PM, Barry Warsaw wrote:
There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone.
All those locations have Python installed in the same place? Even your phone??
Don't discount sys.executable too quickly. :)
I suppose it could be an option. ;) -- ~Ethan~
On Feb 17, 2015, at 01:25 PM, Ethan Furman wrote:
On 02/17/2015 01:13 PM, Barry Warsaw wrote:
There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone.
All those locations have Python installed in the same place? Even your phone??
But of course! :) Cheers, -Barry
On 17 February 2015 at 21:13, Barry Warsaw <barry@python.org> wrote:
Don't discount sys.executable to quickly. :)
I used pyzzer when it defaulted to sys.executable, and the experience was painful. That's on Windows, and Unix may be different, but let's just say I don't want to go there :-) Paul
On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore <p.f.moore@gmail.com> wrote:
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python)
On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default.
On 15 February 2015 at 16:15, Petr Viktorin <encukou@gmail.com> wrote:
On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore <p.f.moore@gmail.com> wrote:
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python)
On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default.
But that will fail for users who only have Python 2 installed. And it won't pick up an activated virtualenv (assuming virtualenvs on Unix don't include a python3 command, which I believe is true - it certainly won't on Windows). As a Windows user, I believe that the command "python" should run the version of Python that you choose to be your default. I know it's not quite that simple on Unix, and the system "python" command is typically Python 2, with "python3" for the system Python 3 version, but I also know that tools like pyenv work like that. Equally, I've seen lots of scripts with "#!/usr/bin/env python" as the shebang line, and none with "#!/usr/bin/env python3". That may just be my limited experience with Unix though. I don't really want "#!/usr/bin/env python3" as the default shebang on Windows (at a minimum, it would do the wrong thing for my current environment). I'm open to advice from the Unix users as to how to set things up better on Unix, but there's a whole new set of problems that will arise as soon as we have different defaults on Unix and Windows, so IMO there would need to be pretty significant benefits to justify doing that. And of course it *is* only a default - users can set whatever they want. (But getting defaults right is important, so that's not to dismiss the point). Paul
On Sun, Feb 15, 2015 at 5:43 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 15 February 2015 at 16:15, Petr Viktorin <encukou@gmail.com> wrote:
On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore <p.f.moore@gmail.com> wrote:
So the usage would be something like
python -m zipapp [options] dir_to_zip
Options: -p <interpreter> The interpreter to use on the shebang line (defaulting to /usr/bin/env python)
On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default.
But that will fail for users who only have Python 2 installed. And it won't pick up an activated virtualenv (assuming virtualenvs on Unix don't include a python3 command, which I believe is true - it certainly won't on Windows).
Yes, it would fail with only Python 2 installed. I don't think that's a problem. On POSIXy systems the "python3" symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle "python3". If the app was single-source compatible with both 2 and 3, then `/usr/bin/env python` would be a good choice. But I don't think apps (as opposed to libraries) have much incentive to keep the backwards compatibility.
As a Windows user, I believe that the command "python" should run the version of Python that you choose to be your default. I know it's not quite that simple on Unix, and the system "python" command is typically Python 2, with "python3" for the system Python 3 version, but I also know that tools like pyenv work like that. Equally, I've seen lots of scripts with "#!/usr/bin/env python" as the shebang line, and none with "#!/usr/bin/env python3". That may just be my limited experience with Unix though.
On Linux, what version "python" is depends on the distro. Currently the recommendation from Python upstream is for it to be Python 2, though, and I know of just one distro that does not do this (Arch, because it switched before the recommendation was put in place). If you're in a venv, then of "python" is that venv's Python, of course. But the default mode of operation for Python-unaware users is to not be in a venv.
I don't really want "#!/usr/bin/env python3" as the default shebang on Windows (at a minimum, it would do the wrong thing for my current environment). I'm open to advice from the Unix users as to how to set things up better on Unix, but there's a whole new set of problems that will arise as soon as we have different defaults on Unix and Windows, so IMO there would need to be pretty significant benefits to justify doing that.
I'm not familiar with how the shebang works on Windows, but if "/usr/bin/env python3" breaks things, I find it highly unfortunate.
And of course it *is* only a default - users can set whatever they want. (But getting defaults right is important, so that's not to dismiss the point).
I'm afraid the point is whether it's even possible to have a shebang that does the right thing on both platforms.
On 15 February 2015 at 18:25, Petr Viktorin <encukou@gmail.com> wrote:
On POSIXy systems the "python3" symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle "python3".
Windows doesn't have "python2" or "python3" commands, just "python". To choose which version you use, set PATH or use an explicit path to the executable. The launcher offers shebang handling for scripts, and special-cases the following values for Unix compatibility: #!/usr/bin/env python #!/usr/bin/python #!/usr/local/bin/python They default to the "default Python" which is normally the latest Python 2 version available, but which can be configured by the user. They also support adding an explicit version (python2, python3, python2.7, python3.4, ...) As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change). I see no reason for the zipapp module to buck the trend that the default Python is "the latest 2.X version unless the user configures it differently themselves". Hence the default of "#!/usr/bin/env python". Certainly that means that users who use the default have to be prepared for the fact that they are subject to user configuration changes, and yes that includes running under either of Python 2 or 3. If you don't like that, force a particular version with -p, and be prepared for the *other* issues this may cause (user doesn't have that version, for example). The alternative, I guess, is to have *no* default and write no shebang unless -p is specified. That matches how .py files work (you have to type the shebang line in if you want it) and works the same way (on Unix, a file without a shebang must be run with an explicit "python" command, and on Windows it uses the default Python via the file type association). Honestly, I'd be happy with this - the Windows experience is identical, and it saves choosing something inappropriate ("in the face of ambiguity, refuse the temptation to guess" :-)) So, the options I see: 1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better. I don't see changing the Windows behaviour as an option. If nobody comes up with anything for (3) I'll take the consensus between (1) or (2). If there's no consensus, I'll pick something. Probably 1, as why change if nobody has a better idea? One final thought. This issue is no different from shebang lines in .py scripts - the only real difference is that it's easy to see and edit the shebang in a .py script. It may be that what's really needed here is an option for the user to display and update the shebang line in a pyz archive. I'll look at adding that to the PEP, regardless of the conclusion to this debate. Paul
Paul Moore wrote:
On 15 February 2015 at 18:25, Petr Viktorin <encukou@gmail.com> wrote:
On POSIXy systems the "python3" symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle "python3".
Windows doesn't have "python2" or "python3" commands, just "python". To choose which version you use, set PATH or use an explicit path to the executable.
The launcher offers shebang handling for scripts, and special-cases the following values for Unix compatibility:
#!/usr/bin/env python #!/usr/bin/python #!/usr/local/bin/python
They default to the "default Python" which is normally the latest Python 2 version available, but which can be configured by the user. They also support adding an explicit version (python2, python3, python2.7, python3.4, ...)
As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change).
We could also add special-cases for "#!/usr/...python3" in the launcher on Windows. Cheers, Steve
On 16 February 2015 at 16:34, Steve Dower <Steve.Dower@microsoft.com> wrote:
As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change).
We could also add special-cases for "#!/usr/...python3" in the launcher on Windows.
The launcher handles that. It runs the same thing as "py -3" runs. Which may not be the same as what "py" runs (in my case it runs 3.5a1 where py runs 3.4). Arguably that's an odd choice, but it's simply that I only use "py" in the normal course of events so I only set the "python" default, not the "python3" default. My point is that on Windows, users typically don't change the executable name they use[1], but rather configure the "python" (or "py") command to do what they want. So I think that on Windows we should follow that convention and execute whatever "python"/"py" execute. Paul [1] As usual with anything like this, it's hard to get a sense of what's "typical" so if hordes of Windows users suddenly post saying they routinely use "python2" and "python3" commands, I'll happily concede I'm not the norm here and ask someone to step up and document recommended practices on Windows better, and I'll update the PEP to follow them :-)
I don't know what anyone else does, but in cases where I have both on my windows box, I do use python2(.x) and python3(.y) . If I only have one version on the box, I use the generic name of course. (I don't often have only one version on my boxes though. 2.x inevitably gets drug in in for some reason or another and I hardly ever uninstall old versions of 3.x) I don't use the launcher though, so I might be out-of-scope entirely. (in which case, sorry for the noise)
On 16 February 2015 at 17:00, Mark Young <marky1991@gmail.com> wrote:
I don't know what anyone else does, but in cases where I have both on my windows box, I do use python2(.x) and python3(.y) . If I only have one version on the box, I use the generic name of course. (I don't often have only one version on my boxes though. 2.x inevitably gets drug in in for some reason or another and I hardly ever uninstall old versions of 3.x) I don't use the launcher though, so I might be out-of-scope entirely. (in which case, sorry for the noise)
No, that's good to know. One clarification, though - if you only have Python 3, I get the impression you use "python" and not "python3" then...? Paul
If I only have one version on my box, yes, I only use "python". But like I said, for me personally, that situation doesn't last very long.
On 16 February 2015 at 16:42, Paul Moore <p.f.moore@gmail.com> wrote:
My point is that on Windows, users typically don't change the executable name they use[1], but rather configure the "python" (or "py") command to do what they want. So I think that on Windows we should follow that convention and execute whatever "python"/"py" execute.
One other thought. We could add "short form" options for -p to the zipapp command: -p X[.Y] Use "#!/usr/bin/env pythonX[.Y]" as the shebang. I'd definitely want the default to be "python" in that case, though, as it's the one you can't specify in a short form otherwise. Paul
On 02/16/2015 08:16 AM, Paul Moore wrote:
The alternative, I guess, is to have *no* default and write no shebang unless -p is specified.
This seems to make sense.
So, the options I see:
1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better.
+1 on 2 +0.5 on 1 -- ~Ethan~
On Feb 16, 2015, at 04:16 PM, Paul Moore wrote:
So, the options I see:
1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better.
#2 seems to me to be the most reasonable alternative. The resulting pyz files (built w/o -p) would still be explicitly executable, just like .py files. However, -p must be able to accept any number of strings, including "/usr/bin/env python3" if the user wants that. Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think that will give all the reasonable use cases. Cheers, -Barry
On 17 February 2015 at 18:52, Barry Warsaw <barry@python.org> wrote:
So, the options I see:
1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better.
#2 seems to me to be the most reasonable alternative. The resulting pyz files (built w/o -p) would still be explicitly executable, just like .py files.
I'm pretty sure that's the way the general feeling is going.
However, -p must be able to accept any number of strings, including "/usr/bin/env python3" if the user wants that.
The code simply writes '#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so you can put whatever you want in. Given that it isn't the name of the Python executable, maybe the option should be --interpreter instead? Oh, and am I right that the shebang line should be encoded using the filesystem encoding on Unix? I know 99.999% of use cases will be ascii, but someone could have a Python interpreter in /home/léon/.local/bin/python...
Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly.
I'm not quite sure what you mean here, but maybe you're thinking that the -p option is the executable name rather than what gets put in the #! line directly? Let me know if it's not covered by what I've already said. Paul
On Feb 17, 2015, at 08:52 PM, Paul Moore wrote:
I'm pretty sure that's the way the general feeling is going.
Though the more I think about it, the more I like sys.executable. :)
However, -p must be able to accept any number of strings, including "/usr/bin/env python3" if the user wants that.
The code simply writes '#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so you can put whatever you want in. Given that it isn't the name of the Python executable, maybe the option should be --interpreter instead?
(Quoting out of order.)
I'm not quite sure what you mean here, but maybe you're thinking that the -p option is the executable name rather than what gets put in the #! line directly? Let me know if it's not covered by what I've already said.
I was thinking it would be useful to mimic virtualenv's -p/--python option, but I think that doesn't actually do the $PATH parsing, so maybe taking -p verbatim is fine.
Oh, and am I right that the shebang line should be encoded using the filesystem encoding on Unix? I know 99.999% of use cases will be ascii, but someone could have a Python interpreter in /home/léon/.local/bin/python...
Well, actually probably utf-8 in most cases, at least for Python 3 on *nix. I'm not sure sys.getfilesystemencoding() is the right encoding, rather than sys.getdefaultencoding(), if you're talking about the encoding of the shebang line rather than the encoding of the resulting pyz filename. Cheers, -Barry
On 17.02.15 23:25, Barry Warsaw wrote:
I'm not sure sys.getfilesystemencoding() is the right encoding, rather than sys.getdefaultencoding(), if you're talking about the encoding of the shebang line rather than the encoding of the resulting pyz filename.
On POSIX sys.getfilesystemencoding() is the right encoding because the shebang is read by system loader which doesn't encode/decode, but uses a file name as raw bytes string. On Mac OS always is UTF-8, but sys.getdefaultencoding() can be ASCII.
On 2/17/2015 10:52 AM, Barry Warsaw wrote:
Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think searching would be inappropriate. Just use what is given, as a string. If the user wants a search, they can use `which`. Otherwise, setting to a relative path reliably would be hard (and sometimes desirable).
On Feb 15, 2015, at 05:15 PM, Petr Viktorin wrote:
So I'd suggest `/usr/bin/env python3` for the default.
I really don't want `/usr/bin/env anything`. In general, using /usr/bin/env is a fine tool for developers, but a terrible tool for deployment. It's just too easy for end users to break an application by mucking with their $PATH. In Debian/Ubuntu, we don't allow these shebangs in system scripts, and in fact the package build tools will rewrite shebangs to avoid them. Cheers, -Barry
I'm just now catching up on this thread, so hopefully these thoughts are still relevant. IIRC, the PEP has been scaled back to mostly a convenience around packing an existing directory into a .pyz file, and (on Windows) adding an association for those as executable Python zip file. To the extent that I care about Windows, +1. I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python. Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore. I happen to like pex, but what I really want is something like: $ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in. Cheers, -Barry
On 17 February 2015 at 18:44, Barry Warsaw <barry@python.org> wrote:
The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python.
Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore.
I happen to like pex, but what I really want is something like:
$ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz
This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in.
Barry, thanks for your comments. I'll pick up on the ones directly related to the PEP separately, but I'd like to add some thoughts on this. (I'm not at Pycon so dumping stuff in an email is the best I can do :-)) I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code). The executable zip format is that solution, IMO. It's been around for ages, and has core support. There are certain things that are needed to make a compelling story, though. And it's definitely not a well known feature, which I hope is something PEP 441 will address, at least a bit. Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one. It would be nice to see something like this, but I would much rather it be an update to zipimport than be bolted on in whichever application does it. To my mind, having the core support the mechanisms is a much more robust approach (and avoids the possibility of the solution not being implemented cross-platform - look at py2exe which solves the extensions in a zipfile issue, but only for Windows). Also, modules still fail to be zip-safe for other reasons, such as needing data files to be real filesystem files (for example requests' certificate bundle). Adding more support for packages to be zip-safe would be fantastic, and would help in a lot more places than just the executable zip format. (Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this. As far as issues specific to the executable zip format are concerned, the main one is tool support. The zipapp module is intended as nothing more than the bare minimum. I think that more complex tools should live on PyPI, not in the core, for all the same reasons that pip was developed outside of the core - the ideal tool needs time to develop, needs a faster release cycle than core Python, etc, etc. At the moment, the only substantial tools I know of are pyzzer and pex. pyzzer is pretty good, IMO, but doesn't do much beyond the basic zip building. I tried pex, but it didn't work on Windows at the time - the problem I found is fixed, but I haven't really had the inclination to try again (there's only so many projects I can cope with being the Windows early adopter for :-)) One thing I would like to see, in the light of PEP 441, would be for pex to switch to using ".pyz" as its file extension, because that's what pex files are. (The documentation also refers to the support of __main__.py as "a quirk in the Python importer" which I find particularly irritating, as it's not a quirk, it's a deliberate feature implemented for precisely this sort of use case!!!) I agree that something that grabs all of the non-stdlib dependencies of a package and bundles them up automatically would be brilliant. I've intended to write something like that for a while now, but haven't had the time. But I certainly think your "what I really want" scenario above is entirely achievable. At least to the extent that the package code and its dependencies support running from a non-filesystem importer... A final thought - I wonder how easy it would be to write an "unpacking zipimporter" that worked like zipimport, but unpacked the zipfile to a temporary directory at the start and cleaned up on program termination? It would give the single-file benefits of zipimport, but would be guaranteed 100% compatible with filesystem imports, at a cost of worse startup/teardown performance, and extra disk space. It might make a nice fallback importer for troublesome packages. Paul
On Feb 17, 2015, at 08:44 PM, Paul Moore wrote:
I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code).
I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables.
Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one.
Agreed.
(Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this.
Yep, which I support. :) pkg_resources.resource_filename() is the way to do that today and it abstracts away all the nastiness. Mostly agree with your other comments, though I haven't played with pyzzer and pex is quite nice, for the pared down use case, although it's not quite integrated with virtualenv (I've had some discussions with pex's author about that). OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions. Cheers, -Barry
On 17 February 2015 at 21:40, Barry Warsaw <barry@python.org> wrote:
OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions.
I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP. As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe) Paul
On Feb 17, 2015, at 09:52 PM, Paul Moore wrote:
I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP.
As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe)
Agreed! The only difference between executable zips and the full-featured self-contained app is the goop you put at the front of the file, right? :) Cheers, -Barry
On 2/17/2015 1:40 PM, Barry Warsaw wrote:
I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users.
That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables.
Thinking ahead? Novel idea! My dream scheme... A scheme for putting all the pure- Python modules (mostly user code) into a .pyz, and a concurrent scheme for putting all the non-pure- Python modules (maybe even Python itself, for platforms that are missing the necessary version of Python 3), into some other platform-dependent bundle. User instructions then become... try the .pyz, if it doesn't work, install the platform-dependent bundle and then it will. So one could build up a platform-dependent bundle that contains all stuff used in development of various projects, and have easy user instructions for deployment of various .pyz apps. Longer instructions for people that think they know what they are doing would list the version of Python and the versions of the various dependencies in the platform-dependent bundle, if they wish to install them manually, etc. I suppose licensing might, in some cases, conflict with making a single bundle for some modules and platforms. I'm already doing the .pyz apps, using .py extensions, but don't have the other piece in place.... it's harder, and I haven't acquired the skill set for putting the binary chunks together... other than unzipping to the filesystem. Glenn
On 17 February 2015 at 18:44, Barry Warsaw <barry@python.org> wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible.
I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. But absolutely, yes, the intention is that ".pyz" is officially the standard extension for Python zip applications. (And for that matter, I'd like the PEP to promote the specific term "Python Zip Application" as the correct way to refer to these things - without a concrete term to use, it's very hard for the idea to get traction). Paul
Barry Warsaw wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible.
Paul Moore wrote:
I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately.
Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
On 18 February 2015 at 20:48, Jim J. Jewett <jimjjewett@gmail.com> wrote:
Barry Warsaw wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible.
Paul Moore wrote:
I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately.
Are you just looking for
http://www.iana.org/assignments/media-types/media-types.xhtml
and its references, including the registration procedures
http://tools.ietf.org/html/rfc6838#section-4.2.5
and the application form at
http://www.iana.org/form/media-types
?
That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. Actually, I guess the first question is whether ".py" is reserved for Python scripts. If it is, then certainly /pyz should be reserved in a similar way for Python Zip Applications. If there's no formal registration of .py, then it seems a bit unreasonable to bother for .pyz... Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Paul
On Feb 18, 2015, at 09:16 PM, Paul Moore wrote:
That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about.
I was actually thinking about both. :) The Wikipedia page has several external links to collections of file extensions, but I don't know that there's *an* official registry. http://en.wikipedia.org/wiki/File_extension Picking FileInfo.com at random, and searching for a few .py* extensions, you find the usual suspects, plus a few others, but all are Python related, and there's no .pyz extension currently registered.
Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address...
Probably so. Seems like they could pick one or more of these registries and register pyz for this purpose. Cheers, -Barry
On Wed, Feb 18, 2015 at 10:16 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 18 February 2015 at 20:48, Jim J. Jewett <jimjjewett@gmail.com> wrote:
Barry Warsaw wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible.
Paul Moore wrote:
I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately.
Are you just looking for
http://www.iana.org/assignments/media-types/media-types.xhtml
and its references, including the registration procedures
http://tools.ietf.org/html/rfc6838#section-4.2.5
and the application form at
http://www.iana.org/form/media-types
?
That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about.
Actually, I guess the first question is whether ".py" is reserved for Python scripts. If it is, then certainly /pyz should be reserved in a similar way for Python Zip Applications. If there's no formal registration of .py, then it seems a bit unreasonable to bother for .pyz...
Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address...
There is no extension registry for Unixy systems, because extensions traditionally don't matter. File types are identified by their contents; there is a "magic"* database that maps file contents to types (MIME types, nowadays). The `file` command is the CLI interface for that. Some programs will fall back on the extension if that doesn't give a result, but that would depend on the program. For a pyz file to run when you "open" it (for most values of "open"), it needs to have the executable bit set, and have a shebang line. * /usr/share/misc/magic – it originally contained just magic numbers, such as those you might find at the beginning of a .pyc file.
On Wed, Feb 18, 2015 at 4:16 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 18 February 2015 at 20:48, Jim J. Jewett <jimjjewett@gmail.com> wrote:
Barry Warsaw wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types,
...
Are you just looking for
http://www.iana.org/assignments/media-types/media-types.xhtml and ...
That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about.
Question 13 at http://www.iana.org/form/media-types asks for additional information, and specifically calls out Magic Number and File Extension, among others. I doubt there is any more official repository for file extension meaning within MIME or unix.
Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address...
Ideally, it would be a long-lasting organizational address, such as pep-editor@python.org. But often, it isn't. -jJ
On 19 February 2015 at 18:11, Jim J. Jewett <jimjjewett@gmail.com> wrote:
Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address...
Ideally, it would be a long-lasting organizational address, such as pep-editor@python.org. But often, it isn't.
Given that .py isn't on that list in any form I could locate, I don't think it's reasonable to add .pyz. If someone wants to propose adding both, I'd be fine with that, but it can be done independently of PEP 441. I'll add a note to the PEP to that effect. On that note, I intend to post the latest version of the PEP, and the reference implementation, tomorrow. I'll get the official version of the PEP on python.org updated at the same time. Paul
On 19 February 2015 at 07:16, Paul Moore <p.f.moore@gmail.com> wrote:
On 18 February 2015 at 20:48, Jim J. Jewett <jimjjewett@gmail.com> wrote:
Barry Warsaw wrote:
I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible.
Paul Moore wrote:
I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately.
Are you just looking for
http://www.iana.org/assignments/media-types/media-types.xhtml
and its references, including the registration procedures
http://tools.ietf.org/html/rfc6838#section-4.2.5
and the application form at
http://www.iana.org/form/media-types
?
That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about.
FWIW, registering the vnd.python MIME prefix with IANA has been vaguely kicking around on my todo list since we switched the PyPA metadata PEPs over to using JSON. If anyone did decide to follow up on that idea, the PSF Infrastructure working group mailing list would likely be a suitable contact address to use (Infra already handles relationships with other internet registries for DNS et al, and I can't think of another group better suited to the task). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (14)
-
Barry Warsaw -
Daniel Holth -
Ethan Furman -
Glenn Linderman -
Greg Ewing -
Jim J. Jewett -
Mark Lawrence -
Mark Young -
Nick Coghlan -
Paul Moore -
Petr Viktorin -
Serhiy Storchaka -
Steve Dower -
Thomas Wouters