On 7/11/07, "Martin v. Löwis" martin@v.loewis.de wrote:

...

Nick Coghlan has reviewed the patch and seems to think it's a good idea. Thomas Wouters also said he likes it, and I ran it by Guido earlier and he seemed to think the idea is good, although I don't think he has seen the implementation.

See my comment: I must be missing the point of the patch, since I can do the same thing (make a single executable zip file on Linux) through a /bin/sh header just fine.

Right, but it's supposed to be cross platform, as mentioned in the patch. This will work on Windows. The main problem I see is that a shell script in front of a zip file seems like a relatively common idiom that people use and have different variants on, each of which have their own idiosyncrasies. So it would nice to consolidate them and make it standard and robust.

For example, it looks like eggs have an executable format that is similar to this. And see the bug I mentioned where those executable eggs can't be invoked through a symlink (which to me is a relatively severe problem). I think this has to do with some introspection on $0, but you won't run into that with this implementation.

Also, I mentioned the program called autopar we use at Google that does the same thing, and it also have a significant number of weird hacks in the shell header. I think Thomas Wouters has also worked on another program to make an executable zip file.

Another example is that the behavior of the zip in your example depends on what else is in the current directory [1], which isn't desirable. Nick pointed out this issue and I addressed it in the patch by removing "" from sys.path, since the -c flag adds that. If lots of people reinvent this wheel (and they have), there are going to be other subtleties like this that will be missed.

The -z flag also eliminates starting an extra process -- you invoke the Python interpreter directly instead of starting a shell which in turn invokes the Python interpreter.

As mentioned, it's also a very tiny amount of code, and I don't see much potential for bad interactions with other things, the way I've written it.

Andy

1) andychu test2$ ./foo_exe.zip Traceback (most recent call last): File "<string>", line 1, in ? File "foo.py", line 16, in ? import outside ImportError: No module named outside

andychu test2$ touch outside.py

andychu test2$ ./foo_exe.zip main

andychu test2$

On 7/12/07, "Martin v. Löwis" martin@v.loewis.de wrote:

...

Right, but it's supposed to be cross platform, as mentioned in the patch. This will work on Windows.

But in the description, you said that you do the same on Windows by making a file that is both a zip file and a batch file. So my approach is also cross-platform, no?

The approach is cross-platform, in that you can use the approach on different platforms. The result of the approach, however, is not cross-platform. You can't distribute your single zip-as-executable to both Windows and bourne-shell-using platforms. The -z argument does allow that.

Why is the shell script not robust?

There are a lot of subtleties in figuring out which python to execute, environment variables that you may or may not want to tweak (admittedly Google's solution that Andy referenced is more vulnerable to that, but it's not unique to Google by any means.) If you want any kind of flexibility in the packaged-up program, you need a bunch of logic in the shell script, and environment-tricks to pass information to the python process, start the python process and provide a bunch more logic in Python to boot. For instance, you need to set PYTHONPATH to include the zipfile before you can import from it, but you don't want that PYTHONPATH to be passed to subprocesses by accident.

The -z argument makes it extremely simple: the user decides which python to run, and the program is run directly just like it would if it was unpacked and run that way. It makes it extremely easy to create 'single executables' out of multiple Python files, in the form that single .py files already are. It leaves building a more complex system (such as eggs) ontop of it entirely open. The change is a good thing, IMHO. And I say this not because we use a similar solution at Google -- we already solved it, and we won't be using the -z argument anytime soon anyway. I say this because I've had many requests from non-googlers for something exactly like this :)

"" should not be removed from sys.path. It is *not* meant to be

the current directory, but the directory where the main script lives.

Yes. "" should either be interpreted as the zipfile, or be replaced by the zipfile. In the case of executing the zipfile, the main script lives *in the zipfile*.

It's baggage that is rarely needed, and the feature can be readily

implemented in a different way for people who need it.

I disagree with both statements. The bagage is much less than zipimport itself, which has proven to be quite useful. Nevertheless, zipimport built into the interpreter was by no means necessary; current users of it could have readily implemented it themselves, with no changes to Python. (In fact, Google's 'autopar' tool does exactly that to support Python 2.2, which lacks zipimport.) This is a very small, logical and useful extension to zipimport, and I believe you will find more uses for it than you expect (although I do believe you yourself don't have a need for it. I just don't think you're a typical Python programmer in this case :)

On 08:41 am, guido@python.org wrote:

On 7/12/07, Thomas Wouters thomas@python.org wrote:

...

I disagree with both statements. The bagage is much less than zipimport itself, which has proven to be quite useful. Nevertheless, zipimport built into the interpreter was by no means necessary; current users of it could have readily implemented it themselves, with no changes to Python.

I wonder, is it even necessary to say anything, after:

+1.

?

But, since I so often object to new features, and there is a heavy Google bias in the existing survey sample, I would like to say that I had a problem several months ago in a _radically_ different environment (Twisted running on an embedded system, Zipfile of PYCs used to shave off as much disk space and startup time as possible) where having the subtleties of a "-z" flag figured out already would have saved me a _ton_ of work. I was already aware of the shell-header trick, but discovering all the environment-setup details was tedious and distracting enough to make me give up and switch to writing a bunch of hard-to-test /bin/sh code.

It wasn't a bad project by any means, and Python worked out even better than expected (we weren't even sure if it would be able to load into the memory available, but it turns out that being able to do everything in a single process helped a lot) but a -z option would have been that much more impressive :).

In fact, I distinctly remember thinking "You know, if Python had an equivalent to Java's '-jar' option, this would be a whole lot easier."

(Even better, on this _particular_ project, would have been a generic "run this thing-which-looks-like-a-sys.path-entry" standard format, which could have been switched for different deployments to a directory, a zipfile, or the result of freezing. Perhaps that's starting to get too obscure, though.)

The approach is cross-platform, in that you can use the approach on different platforms. The result of the approach, however, is not cross-platform. You can't distribute your single zip-as-executable to both Windows and bourne-shell-using platforms. The -z argument does allow that.

I still don't understand how so. How will this work on Windows?

For instance, you need to set PYTHONPATH to include the zipfile before you can import from it, but you don't want that PYTHONPATH to be passed to subprocesses by accident.

That's why I say a best-practice solution should be established. In this case, if the actual main function is invoked through -c, the -c script could get a sys.path.append statement as well.

The -z argument makes it extremely simple: the user decides which python to run, and the program is run directly just like it would if it was unpacked and run that way.

Really? I think there a still a number of subtleties, like what sys.argv[0] will be, and how sys.path will look like. It's definitely *not* the same as if you unzipped it, and ran the unzipped one.

I disagree with both statements. The bagage is much less than zipimport itself, which has proven to be quite useful. Nevertheless, zipimport built into the interpreter was by no means necessary; current users of it could have readily implemented it themselves, with no changes to Python. (In fact, Google's 'autopar' tool does exactly that to support Python 2.2, which lacks zipimport.) This is a very small, logical and useful extension to zipimport, and I believe you will find more uses for it than you expect (although I do believe you yourself don't have a need for it. I just don't think you're a typical Python programmer in this case :)

Ok. I'll shut up, just hoping that this won't cause too much trouble in the long run.

Regards, Martin

On 12/07/07, "Martin v. Löwis" martin@v.loewis.de wrote:

...

Right, but it's supposed to be cross platform, as mentioned in the patch. This will work on Windows.

But in the description, you said that you do the same on Windows by making a file that is both a zip file and a batch file. So my approach is also cross-platform, no?

Getting the details of such a batch file header right on Windows is not easy, not least because there is no "exec" equivalent on Windows. The following works, but (a) uses 2 processes, and (b) doesn't preserve the exit code. The first issue is minor, but the second is a big problem (and one I don't know how to fix).

Also, on Windows, zip-packaged GUI programs could be useful - these would be executed using "pythonw -z"

How do you get the -z option to work on Windows? What extension do you use, and how is the zipfile created?

The patch suggests using .pyz and adding a default association to the installer (much like .py and .pyw have). It also offers a script for building the zipfiles - either as sample code, or to be included with Python (it's not clear to me).

It's arguable that .pyz files should use pythonw -z, not python -z, as file extensions are more often useful for clickable programs in the GUI. You could have two extensions (.pyz and .pzw, maybe) but I'm not sure it's worth it.

The point here is that the fiddly part (setting sys.path, locating the main module, etc) is covered by the -z option, deployment considerations are easier to handle (and hence the exact defaults supplied are less crucial) once -z is available.

Paul.

On 7/12/07, "Martin v. Löwis" martin@v.loewis.de wrote:

...

The patch suggests using .pyz and adding a default association to the installer (much like .py and .pyw have).

Ok. It would be good if the patch actually added that extension, rather than merely suggesting that it should be added.

So does everyone agree that there should be a new extension called .pyz? And that the definition of this is a .zip file with a __zipmain__.py module at its root? If so, I can make the change... I haven't looked around the codebase yet but it sounds easy enough.

This makes it seem like a bigger change than it is, but I think it's the right thing to do to support Windows properly.

Other points:

* I think it's true that the shebang line should only have one argument.

* Does anyone else want to change the -z flag to make more sense for directories (and possibly change __zipmain__.py to __main__.py)? In thinking about this again, I am not sure I can come up with a real use case. I think it's sufficient to treat it as a documented "trick" that you can substitute a whole directory for a zip file with the -z flag. If there is a concrete suggestion, I'd like to discuss it, but otherwise it seems like we'll get bogged down in expanding use cases.

* Magically looking at the first argument to see if it's a zip file seems problematic to me. I'd rather be explicit with the -z flag. Likewise, I'd rather be explicit and call it __zipmain__ rather than __main__.

thanks, Andy

On 7/12/07, Phillip J. Eby pje@telecommunity.com wrote:

At 10:09 AM 7/12/2007 +0200, Martin v. Löwis wrote:

...

"" should not be removed from sys.path. It is *not* meant to be the current directory, but the directory where the main script lives.

Right; it should be replaced with the zipfile path instead.

That's indeed what the current implementation does, replacing "" with the zip file.

I would personally rather see this option defined as simply placing a directory at the front of sys.path, and perhaps defining a default -m value of __main__, unless overrridden. Being able to use the option

Actually, that's a good idea, and it does work with my current implementation [1], although we'd have to change the name __zipmain__. Is __main__ a good idea considering that is used for something similar but implemented completely differently (the module name)? I thought about using __main__, but decided on __zipmain__ since seemed to be more explicit and reduce potential conflicts.

To be clear to other readers, the convention would be that if a __main__.py file exists at the root of a directory, then the whole directory is considered an executable python program.

more than once would be nice, too. On Windows, you can't set an environment variable on the same line as a command, so this would give you a one-liner way of setting sys.path and running an application.

I do not see a reason to make this option zipfile-specific in any way, though; it's just as useful (and sometimes more so) to be able to distribute an application as a directory, since that lets you use .pyd, .so, .dll etc. without needing the egg cache system for using those.

Yes, the dynamic library importing is nice.

thanks, Andy

1) andychu testprog$ find . ./__init__.py ./package1 ./package1/__init__.py ./package1/foo.py ./package1/lib.py ./__zipmain__.py

andychu testprog$ ../python -z . lib module here argv: ['.'] andychu testprog$

At 01:46 AM 7/13/2007 +1000, Nick Coghlan wrote:

The current version of runpy doesn't accept any options, but it would be pretty easy to make the following changes:

1. Accept a -p option that prepends path entries. These path entries

would be combined into a single list from left to right on the command line, then the whole list prepended to sys.path. If at least one -p option is given, the default '' entry would be removed from sys.path (the current directory could be added back in explicitly via -p '.').

2. Attempt to run the module __main__ if no module is otherwise specified

Startup would be fractionally slower than it would be with the C-level option, but the code would be much simpler, and the new feature would be readily available on any Python implementation which can execute the runpy module.

The relevant shebang line to be prepended to a zip file would then look something like:

#!/usr/bin/env python -m runpy -p

I don't have any particular objection to using runpy for this, but I believe that this shebang line won't actually work on certain non-BSD OSes, such as most Linux versions, which allow you to have at most *one* argument to a #! line, and will combine anything after the executable portion into a single argument. This means that the only workable form of this line for cross-platform use is:

#!/usr/bin/python2.6 -z

And of course that won't work if Python is somewhere else. You can't both use env to invoke Python, *and* expect arguments to work. env will receive a single argument of "python -m runpy -p", which it will then try to invoke. On Mac OS and various other BSDs, your example will work correctly, but it won't work most anywhere else, as few OSes actually support passing individual arguments from a #! line. See:

http://www.in-ulm.de/%7Emascheck/various/shebang/

The relevant shebang line to be prepended to a zip file would then look something like:

#!/usr/bin/env python -m runpy -p

I might be confusing things, but I think some systems only allow a single argument in the shebang line.

Regards, Martin

On 7/12/07, "Martin v. Löwis" martin@v.loewis.de wrote:

But in the description, you said that you do the same on Windows by making a file that is both a zip file and a batch file. So my approach is also cross-platform, no?

How do you get the -z option to work on Windows? What extension do you use, and how is the zipfile created?

Nick suggested using .pyz, and others seem to like that solution (possibly using pythonw) and that seems logical enough to me. If it's agreed that that's the right solution on Windows, I can put in the work for that.

Couldn't that also be achieved by documenting best practice in the documentation? Why is the shell script not robust?

I think it's pretty clear that it's not robust, and there have been even more anecdotal examples on this thread. Everyone does it slightly differently -- not for any particular reason, but just because the right thing isn't trivial.

As I pointed out, the example you came up with (which many others would come up with too) has a fairly serious problem, in that it will import things from outside the .zip file. I could build my .zip file on my system, test it out, and then deploy it to another machine and it will break. Ironically, this happened to *me* while developing the patch!

Why that? Why do eggs fail to process $0 correctly, whereas the -z option gets it correct? That just sounds like a bug in eggs to me, that could be fixed - or, if not, I'd expect that -z cannot fix it, either.

My understanding of this note is that pkg_resources uses sys.argv[0] to determine the version number of the egg; IIUC, -z won't help at all here because sys.argv[0] will still be the name of the symlink.

OK, I could be mistaken here, I haven't actually repro'd this bug.

What are those weird hacks, why are they necessary, and how does the -z option overcome the need for these hacks?

That people fail to make it work with /bin/sh doesn't automatically mean they succeed with -z. Either they are too unexperienced to make the shell header correct (in which case documenting best practice would help), or they have deeper problems with that approach, in which case it isn't at all obvious that the proposed change improves anything.

I don't think this is true at all. I have provided the sample code to make one of these files, and so you basically have to run a command line, rather than write a shell header -- and the shell header is currently not documented anywhere. As mentioned, this approach also prevents you from having to start the shell, and makes it more portable, since people might use #!/bin/myfavoriteshell or use #!/bin/sh and not realize they are using system-specific features of the shell.

...

Another example is that the behavior of the zip in your example depends on what else is in the current directory [1], which isn't desirable. Nick pointed out this issue and I addressed it in the patch by removing "" from sys.path, since the -c flag adds that.

"" should not be removed from sys.path. It is *not* meant to be the current directory, but the directory where the main script lives.

Regardless of what "" *should* be interpretreted as, the example you gave has the problem mentioned (with current versions of Python) -- that "" *is* the current directory and thus things get imported outside of the zip file when they are not found in the zip file.

Right now "" is replaced with the zip file. If there's a better implementation I'm willing to change it.

...

As mentioned, it's also a very tiny amount of code, and I don't see much potential for bad interactions with other things, the way I've written it.

It's baggage that is rarely needed, and the feature can be readily implemented in a different way for people who need it.

I also disagree with both statements. : ) I think others have said basically the exact same thing as I am saying: that it is *commonly* needed, it's not a lot of baggage in Python since it's so little code, and it's easy to get wrong.

Andy

On 7/12/07, Daniel Stutzbach daniel@stutzbachenterprises.com wrote:

On 7/11/07, Andy C andychup@gmail.com wrote:

...

The good thing about this is that it's extremely simple -- basically 20 lines of C code to add a -z flag that calls a 3-line Python function in the runpy module.

Instead of requiring a -z flag, why not have the interpreter peak at the file to see if it starts with one of the ZIP magic numbers?

That way it Just Works.

I guess you wouldn't recognize a zip file if it hits you in the face. Literally. :-)

Zip files don't start with a magic number.

On 7/23/07, Crutcher Dunnavant crutcher@gmail.com wrote:

Crutcher Dunnavant

On Jul 23, 2007, at 9:55 AM, "Guido van Rossum" guido@python.org wrote:

...

On 7/12/07, Daniel Stutzbach daniel@stutzbachenterprises.com wrote:

...

On 7/11/07, Andy C andychup@gmail.com wrote:

...

The good thing about this is that it's extremely simple -- basically 20 lines of C code to add a -z flag that calls a 3-line Python function in the runpy module.

Instead of requiring a -z flag, why not have the interpreter peak at the file to see if it starts with one of the ZIP magic numbers?

That way it Just Works.

I guess you wouldn't recognize a zip file if it hits you in the face. Literally. :-)

Zip files don't start with a magic number.

Don't they end with a sentinel? If so, what would be the difference?

There's an ambiguity -- a Zip file could start with a Python (or shell, or Perl) script that bootstraps execution. This is used regularly. Changing the semantics just because the file *ends* with something funny sounds like asking for trouble.

On 7/24/07, David Gowers 00ai99@gmail.com wrote:

On 7/24/07, Guido van Rossum guido@python.org wrote:

...

On 7/12/07, Daniel Stutzbach daniel@stutzbachenterprises.com wrote:

...

On 7/11/07, Andy C andychup@gmail.com wrote:

...

The good thing about this is that it's extremely simple -- basically 20 lines of C code to add a -z flag that calls a 3-line Python function in the runpy module.

Instead of requiring a -z flag, why not have the interpreter peak at the file to see if it starts with one of the ZIP magic numbers?

That way it Just Works.

I guess you wouldn't recognize a zip file if it hits you in the face. Literally. :-)

Zip files don't start with a magic number.

ZIP files *do* start with a magic number; either PK\03\04 (non-empty archive) or PK\05\06 (empty archive). This is rather easy to notice, as I did in the bad old days of DOS, and i recently doubly verified it ('zip'+'khexedit', and http://en.wikipedia.org/wiki/ZIP_%28file_format%29; I tried the infozip website too, but it seems to be down.)

You can believe that, but it's not the whole story. You can *prepend* arbitrary data and the zip tools can still read the archive.