[Patches] [ python-Patches-1739468 ] Add a -z interpreter flag to execute a zip file

Sun Jul 8 10:15:58 CEST 2007

Patches item #1739468, was opened at 2007-06-19 13:40
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1739468&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: andy-chu (andy-chu)
Assigned to: Nick Coghlan (ncoghlan)
Summary: Add a -z interpreter flag to execute a zip file

Initial Comment:
The motivation for this is that distributing a zip file is a very light and easy way to distribute a python program with multiple packages/modules.  I have done this on Linux, Mac and Windows and it works very nicely -- except that you need a few extra files to bootstrap it: set PYTHONPATH to the zip file and run the main function.

With this small patch, you get rid of the need for extra files.  At the bottom is a demo on Linux.

On Windows, you can do a similar thing by making a file that is both a zip file and a batch file.  The batch file will pass %~f0 (itself) to the -z flag of the Python interpreter.

I ran this by Guido and he seemed to think it was a fine idea.  At Google, we have numerous platform-specific hacks in a program called "autopar" to solve this problem.

I have included the basic patch, but if you guys agree with this, I will add some tests and documentation.  And I think it might be useful to include something in the Tools/ directory to do what update_zip.sh does below (add a __zipmain__ module and a shebang/batch file header to a zip file, to make it executable)?

I think this may also help to fix a bug with eggs:

http://peak.telecommunity.com/DevCenter/setuptools#eggsecutable-scripts

IMPORTANT NOTE: Eggs with an "eggsecutable" header cannot be renamed, or invoked via symlinks. They must be invoked using their original filename, in order to ensure that, once running, pkg_resources will know what project and version is in use. The header script will check this and exit with an error if the .egg file has been renamed or is invoked via a symlink that changes its base name.

andychu testdata$ ls
__zipmain__.py  foo.py  foo.pyc  foo.zip  foo_exe.zip  header.txt  update_zip.sh

# The main program you're going to run in "development mode"

andychu testdata$ ./foo.py foo bar
argv: ['./foo.py', 'foo', 'bar']

# Same program, packaged into a zip file

andychu testdata$ ./foo_exe.zip foo bar
argv: ['-c', 'foo', 'bar']

# Contents of the zip file

andychu testdata$ unzip -l foo_exe.zip
Archive:  foo_exe.zip
warning [foo_exe.zip]:  51 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  Length     Date   Time    Name
 --------    ----   ----    ----
      243  06-18-07 20:01   __zipmain__.py
      301  06-18-07 19:58   foo.py
 --------                   -------
      544                   2 files

# Demo script to build an executable zip file.

andychu testdata$ cat header.txt
#!/usr/local/google/clients/python/trunk/python -z

andychu testdata$ cat update_zip.sh
#!/bin/bash

# Make a regular zip file.
zip foo.zip foo.py __zipmain__.py

# Add a shebang line to it.
cat header.txt foo.zip > foo_exe.zip

# Make it executable.
chmod 750 foo_exe.zip

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2007-07-08 18:15

Message:
Logged In: YES 
user_id=1038590
Originator: NO

The new patch looks much better - the only thing is that run_zip needs to
do sys.path.pop(0) to correctly remove the zipfile from the front of the
path.

However, I do see your point about whether or not including the current
directory on sys.path is the right thing to do for this case - it may be
better to set <zipfile_name>/__zipmain__.py as argv[0] before invoking
PySys_SetArgv, and then use __zipmain__ as the module to be executed on the
same code path as the -m switch normally uses.

Rather than continuing this discussion here on SF, it may be best to post
your proposal to python-dev. I personally like the idea, but a new idiom
for running Python scripts will need broader support than just me. Getting
input from the py2exe and py2app folks that can be found on python-dev
would also be good. 

----------------------------------------------------------------------

Comment By: andy-chu (andy-chu)
Date: 2007-07-08 04:33

Message:
Logged In: YES 
user_id=1821575
Originator: YES

Nick, have you had a chance to look this over again?  I mainly care about
the -z flag support.  The makepyz.py script is just a demo, though I think
it is useful as documentation as well.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2007-06-30 00:40

Message:
Logged In: YES 
user_id=1038590
Originator: NO

I'm going to be off the net for a few days - I'll have a look at the
updated patch when I get back late next week,

----------------------------------------------------------------------

Comment By: andy-chu (andy-chu)
Date: 2007-06-28 15:17

Message:
Logged In: YES 
user_id=1821575
Originator: YES

Here is a script that documents how to make such files.  I think the
important part is just documenting the format.  Then people can write
whatever tools they need around it.  Many people could get by with this
simple tool, but others might want something more elaborate.

Demo:

andychu testprog$ find
.
./__init__.py
./package1
./package1/__init__.py
./package1/foo.py
./package1/lib.py
./package1/__init__.pyc
./package1/lib.pyc
./package1/foo.pyc

andychu testprog$ find -name "*.py" | xargs ../Tools/scripts/makepyz.py -a
zip,pyz,unix -z foo.zip -p package1 -m foo -y
/usr/local/google/clients/python/trunk/python
Added ./__init__.py to foo.zip
Added ./package1/__init__.py to foo.zip
Added ./package1/foo.py to foo.zip
Added ./package1/lib.py to foo.zip
Added __zipmain__.py to foo.zip
Prepended #!/usr/local/google/clients/python/trunk/python -z to foo.zip.
chmod foo.zip 0700

andychu testprog$ ./foo.zip
lib module here
argv: ['-c']
andychu testprog$ echo $?
3

File Added: makepyz.py

----------------------------------------------------------------------

Comment By: andy-chu (andy-chu)
Date: 2007-06-27 15:35

Message:
Logged In: YES 
user_id=1821575
Originator: YES

Nick, I've updated the code to use a new runpy.run_zip function, which
calls run_module.  This does make it a bit cleaner.

Let me know what you think.  If the code is good I'll write some tests and
documentation.

Also, I'm not sure if the '-c' is really appropriate in sys.argv, but that
seems to be what the -m flag uses.  It seems like it might make sense to
have sys.argv[0] be the zip file, if it is really a first class
executable.

And I think a script to build one of these files would be appropriate,
which I can add.  You could pass it the main module and main function, and
it would generate a __zipmain__ stub and add it to the zip file.  And it is
a good idea if the file is cross platform, so a .pyz extension would work.

Sorry the delayed response, I was a bit busy at work this week... but I'll
respond sooner this time. : )

Example:

andychu trunk$ testdata/foo_exe.zip foo bar
__name__: __main__
argv: ['-c', 'foo', 'bar']
andychu trunk$ echo $?
3

File Added: runzip7.diff

----------------------------------------------------------------------

Comment By: andy-chu (andy-chu)
Date: 2007-06-21 10:54

Message:
Logged In: YES 
user_id=1821575
Originator: YES

Nick, you're right, I think it can use run_module and be in the runpy
module.  Let me make those changes and send you another patch.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2007-06-19 21:21

Message:
Logged In: YES 
user_id=1038590
Originator: NO

I like the general idea, but it should be possible to use runpy.run_module
to get __name__ set correctly (as that is what happens when you execute a
module from a zipfile with -m). Another advantage of using run_module is
that it would allow runzip() to take a second argument (possibly defaulting
to "__zipmain__") which would specify the module to be executed from the
zipfile (the remaining 3 run_module arguments could also be passed in, and
set appropriately from main.c).

Adding the new function as runpy.run_zip() (instead of adding a new
module) would also be good.

For Windows, an alternative to making the zip file both a batch and a zip
file would be to adopt a .pyz extension convention for these files - the
file associations can then be set up to invoke the script appropriately
with python -z (similar to the way that .pyw files are associated with
pythonw instead of the standard python executable). That way the same file
could be executed on both Linux (via an embedded shebang line) and on
Windows (via filename association), as is the case with standard .py Python
scripts.

My final question is whether the change to sys.path should be reverted
once the module execution is complete - my suspicion is that it should, but
I need to look into it a bit more before giving a definite answer (for the
command line flag case, this behaviour obviously doesn't matter - it is
only significant if the Python method is invoked directly in the context of
a larger program).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1739468&group_id=5470