It needed a name. I hate the word "Installer", but it expresses in one word the most common use of my stuff.

I'll be releasing a beta for Linux real soon. Only some of the tricks are Windows only (such as self-extracting executables, which is only culturally appropriate on Windows, anyway).

But more importantly it's not just for installing. The Python I use (interactively) on my wife's machine is 1 directory with about 6 files in it. On my Linux box I've been using the std lib in a .pyz for about a month now. Someone distributing a pure Python package could instead ship 3 files (imputil.py, archive.py and <package>.pyz) with the "install" consisting of adding one line to site.py in the user's perfectly normal Python installation.

And yeah, I solved the "manifest" problem, too. Mine predates Distutils, so don't accuse me of duplicate effort, (I pointed them to it a couple times). It uses ConfigParser and a config file, so it allows finer control.

While .pyz's are completely cross-platform, I have yet to work out endianness issues in the other archive I use (which should probably be zip format - it can hold anything). And at the "Installer" end, I have yet to work out how things should work on non-ELF/COFF platforms (where I can't append the archive to the executable). But there aren't any technical issues involved; just lack of time.

So no, it's not just for Windows; and no, it's not just for creating standalones (though that's what almost everyone uses it for).

Gordon, I'm sorry, but from this description I still have no idea what your stuff is (and I forgot the URL so I can't look it up). For example, if it's not (just) for installing, what *is* it for? What is the ``"manifest" problem'' and how did you solve it? Also, note that editing site.py is a no-no! You can create/edit sitecustomize.py, but you should leave site.py alone! --Guido van Rossum (home page: http://www.python.org/~guido/)

[me]

...

Also, note that editing site.py is a no-no! You can create/edit sitecustomize.py, but you should leave site.py alone!

[Gordon]

That would work fine. One of the standalone configurations will write a site.py, but that's for a completely self-contained installation (ie, one which will have no conflicts with another Python installation).

I'd also note that, for Windows at least, the path-expanding mechanism created by site.py has not caught on. I've got lots installed, and no site-python, site-packages or sitecustomize.

You shouldn't see site-python or site-packages, they only exist on Unix. On Windows, everything is installed in the top Python directory. However you should see .pth files there, which is what site.py looks for. I believe NumPy and PIL use those. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]

...

You shouldn't see site-python or site-packages, they only exist on Unix.

[Gordon]

You mean "they only exist _for_ Unix", (site.py looks for them on Windows).

No it doesn't. The code in site.py only adds site-packages and site-python when os.sep is '/'. RTSL.

I don't like that. For one thing, modulo a few platform differences, the same mechanism should work for multi-user Unix and Windows LAN installations. And single- user Windows (I know, redundant, even on NT) should be a degenerate case of the above.

What do you mean by "the same mechanism should work"? The same mechanism for what? Are you talking about sharing the installed files somehow?

...

On Windows, everything is installed in the top Python directory. However you should see .pth files there, which is what site.py looks for. I believe NumPy and PIL use those.

No NumPy, no PIL, no .pth files. 99% of everything out there just says "unzip this somewhere on your Python path".

Fair enough. Of course I know about .pth files so I unzipped them elsewhere and added a .pth file pointing there...

In this case, Jim Ahlstrom may be right - there are too many options, or at least an insufficiently emphasized "proper" method. Until I worked out my own way of installing stuff, I used to lose a large number of packages whenever I upgraded my Windows Python.

The .pth files are designed for this. Maybe they haven't been explained as well as they should.

Much as I love Mark's stuff (and hesitate to criticize crazy Aussies), I wish there weren't so much special casing here for Windows.

It's not Mark's fault, it's Microsoft's fault. If you don't do things the way MS wants you to, experienced Windows users will gripe, misunderstand what you do, etc.

And no, I don't have any solutions to this, I'm just griping...

Ditto. Understanding the problems is half of the solution though. The problems seem pretty complex! --Guido van Rossum (home page: http://www.python.org/~guido/)

I finally got around to reading the current Linux Journal (which just keeps getting better and better) and lo! there was a picture of a familiar face I just couldn't quite.... Oh no! Could it be true? I heard rumors but I refused to believe them until now. The glasses are gone! Guido now looks like an investment banker! The sky is falling! Next will probably be a Python 1.6 as a 27 Meg DLL, and a Python IPO. Well, maybe not. Now that I look more closely, he is wearing a black and white and mustard (??MUSTARD) T-shirt which says "You Need Python". At least we ought to make him wear a name tag at IPC8. JimA

...

...

...

...

"JCA" == James C Ahlstrom writes:

JCA> Oh no! Could it be true? I heard rumors but I refused to JCA> believe them until now. The glasses are gone! Guido now JCA> looks like an investment banker! The sky is falling! He's not the only one who's, like, "gone corporate", but I won't mention any names, so as to protect the guilty.

[Guido]

[Great analysis, Tim!]

I beg to differ: it's internally inconsistent and should have identified at least 3 axes and hence at least 8 cases. Still, you got more than you paid for <wink>.

...

4) The audience is Python end-users "in general", and the product is pure Python. I think this is the most important one for Distutils to address, and compilation isn't a part of it. So far, though, what Gordon is doing seems more appropriate than what Distutils has been up to. I hope his work gets folded into this.

I'm not sure what stuff by which Gordon you're referring to.

You guessed right!

I am only familiar with his installer, which I thought is win32 only (but I may be mistaken) and is an installer for a whole application, not just a bunch of modules. Please correct me if I'm wrong.

If it can install a whole app, what makes you suspect it couldn't install just a bunch of modules <0.5 wink>? It started life as Windows-only, and I believe it's been virtually ignored by non-Windows folk because of that. Bad blind spot. It supplies already-working approaches to many of the issues that are still being *talked* about on Distutils (at least archive formats, code to manipulate same, manifest files (how do you tell the tool which files to package?), and transparently bundling a Python interpreter when needed).

But this reminds me of a different issue, which Jim Ahlstrom has been hammering about before: there's a completely separate set of cases where what you are distributing is a stand-alone application, and the target consists of end users who are entirely uninterested in whether it's written in Python, C or Elvish.

I include part of that in my case #4 above, where the app happens to be written in Pure Python -- but the user doesn't have to know that. Gordon is addressing at least that part of it. AFAIK he can't deal with transparently compiling C or exorcising Elvish on the target platform, but if you're just distributing the binaries I expect his work is directly usable already.

(And then there's still the distinction between Win32, Unix or both.)

I vote "both". The world really doesn't need another Win32-only (or Unix-only) installer, archive format, compression format, or distribution model. Jim seems mostly interested in Win32-only to me, and his concerns haven't been about the mechanics of distribution but about how-- regardless of tool --to create a bulletproof Python installation by hook or by crook. Last time we went thru this, it was concluded that one couldn't without patching the Python Windows binary with a resource editor (to point to its own infernal <0.5 wink> registry entries). Distutils hasn't talked about that at all (that I've seen, anyway); if there were a less radical approach to that, I suspect Jim would be delighted to use one of the commercial Win32 installation pkgs (and if that's what his customers expect, delighted or not that's what he'll do).

The current distutil dools don't deal with this at all.

That's why I said I thought what Gordon is doing seems more appropriate to case #4 than what Distutils has been doing.

I think it should though,

Ditto.

and I think its framework is powerful enough to be able to add this, e.g. as a new "appdist" command.

I cordially invite (since Gordon will uncordially browbeat <wink>) people to look seriously at what he's done. Best I can tell, for apps that don't need compilation "on the other end", it's mostly "there" already! give-the-man-a-hand-ly y'rs - tim

[Gordon]

...

Much as I love Mark's stuff (and hesitate to criticize crazy Aussies), I wish there weren't so much special casing here for Windows.

[Guido]

It's not Mark's fault, it's Microsoft's fault. If you don't do things the way MS wants you to, experienced Windows users will gripe, misunderstand what you do, etc.

Something just occurred to me: MS's guidelines aren't arbitrary, they actually have very good reasons. In the case of putting all an app's crucial info in the Registry, it's the only way to allow a site administrator to set policy and site options remotely (an admin can fiddle other machines' registries remotely). This works very well indeed when there's only "one copy" of an app on a machine (or at most one copy "per user"). What just occurred to me is that JimA is concerned with *not* letting any info from a previously-installed Python affect the app he's installing. Similarly, Gordon's Win32 "standalone installer" modifies python.exe and pythonw.exe to use a PYTHONPATH he forces, leaving the registry out of it. Similarly, the woes I've had in trying to sell Python as a general Win32 scripting tool at work mostly boil down to that there's no effortless way to do it that doesn't risk picking up info from-- or forcing info onto --pre-existing or future distinct Python installations (in contrast, Perl "just works" in this respect). IOW, the three of us find getting path info out of the registry intolerable because we are in fact trying to do the opposite of what the registry mechanism was *designed* for: we want perfect isolation, not perfect sharing. This has come up on Python-Help a few times too, in the guise of someone installing a product that in turn installs an older version of Python, which in turn confuses another product that relies on features in a newer version of Python. So while the traditional Windows .ini file (like Unix this-or-that.rc file) model was replaced by the registry for excellent reasons, those reasons don't apply to the way we're using Python! The .ini file model was exactly right for what most of us seem to want to do, and the registry model is exactly wrong. just-thought-i'd-cheer-you-up<wink>-ly y'rs - tim

[Guido]

...

It's not Mark's fault, it's Microsoft's fault. If you don't do things the way MS wants you to, experienced Windows users will gripe, misunderstand what you do, etc. [Tim] Something just occurred to me: MS's guidelines aren't arbitrary, they actually have very good reasons. In the case of putting all an app's crucial info in the Registry, it's the only way to allow a site administrator to set policy and site options remotely (an admin can fiddle other machines' registries remotely). This works very well indeed when there's only "one copy" of an app on a machine (or at most one copy "per user").

And actually, the business about separate subtrees for the machine's configuration and the user's configuration is pretty clever. MS doesn't explain it well, and it gets misused, but when done right, it's a lot simpler than the maze of .xxxrc files you sometimes find in other OSes.

What just occurred to me is that JimA is concerned with *not* letting any info from a previously-installed Python affect the app he's installing. Similarly, Gordon's Win32 "standalone installer" modifies python.exe and pythonw.exe to use a PYTHONPATH he forces, leaving the registry out of it. Similarly, the woes I've had in trying to sell Python as a general Win32 scripting tool at work mostly boil down to that there's no effortless way to do it that doesn't risk picking up info from-- or forcing info onto --pre-existing or future distinct Python installations (in contrast, Perl "just works" in this respect).

In my Linux version, I went to the heart of the matter - getpath.c. It occurs to me that getpath.c might do better to follow a normal bootstrap process - ie, create the absolute minimal sys.path required to go to the next step. Then the rest of what goes on in getpath.c could be written in Python. Maybe that Python code needs to get frozen in (to prevent bozos from destroying an installation by stepping on getpath.py), but it would make it a lot easier to create independent installations, and also reduce the variations between platforms at the C level. (Then again, I've never heard of anyone stepping on exceptions.py.) If some registry manipulation primitives were exposed (say, through ntpath) that would mean that Windows developers could (if they wanted) play by the MS rules with at least the option of not stepping on each other. - Gordon

[Guido]

...

...

It's not Mark's fault, it's Microsoft's fault. If you don't do things the way MS wants you to, experienced Windows users will gripe, misunderstand what you do, etc. [Tim] Something just occurred to me: MS's guidelines aren't arbitrary, they actually have very good reasons. In the case of putting all an app's crucial info in the Registry, it's the only way to allow a site administrator to set policy and site options remotely (an admin can fiddle other machines' registries remotely). This works very well indeed when there's only "one copy" of an app on a machine (or at most one copy "per user"). [Gordon] And actually, the business about separate subtrees for the machine's configuration and the user's configuration is pretty clever. MS doesn't explain it well, and it gets misused, but when done right, it's a lot simpler than the maze of .xxxrc files you sometimes find in other OSes.

I agree. And I am guilty of not even try to find MS' explanation -- I just looked in the registry at what other apps did and tried to mimic that (plus what Mark had already done), without really knowing what I was doing. I now know a little better -- see the end of this message.

In my Linux version, I went to the heart of the matter - getpath.c. It occurs to me that getpath.c might do better to follow a normal bootstrap process - ie, create the absolute minimal sys.path required to go to the next step. Then the rest of what goes on in getpath.c could be written in Python. Maybe that Python code needs to get frozen in (to prevent bozos from destroying an installation by stepping on getpath.py), but it would make it a lot easier to create independent installations, and also reduce the variations between platforms at the C level. (Then again, I've never heard of anyone stepping on exceptions.py.)

Yes, this is exactly what was proposed in the thread on the Big Import Rewrite.

If some registry manipulation primitives were exposed (say, through ntpath) that would mean that Windows developers could (if they wanted) play by the MS rules with at least the option of not stepping on each other.

That's a good idea. These functions are already available through Mark's win32api extension -- much of which will eventually (I hope before 1.6 is out!) become part of the core distribution. In the mean time, I've been thinking a bit more about how Python should be using the Windows registry. (It's clear to me that Python should use the registry -- those who disagree can go build their own Python distribution.) The basic ideas of Python's current registry usage are sound: there's a resource built into the DLL which is part of the key into the registry used for all information. The problem lies in which key is used. All versions of Python 1.5.x (1.5, 1.5.1, 1.5.2) use the same key! This is a main cause of trouble, because it means that different versions cannot peacefully live together even if the user installs them into different directories -- they will all use the registry keys of the last version installed. This, in turn, means that someone who writes a Python application that has a dependency on a particular Python version (and which application worth distributing doesn't :-) cannot trust that if a Python installation is present, it is the right one. But they also cannot simply bundle the standard installer for the correct Python version with their program, because its installation would overwrite an existing Python application, thus breaking some *other* Python apps that the user might already have installed. (There's a solution for app builders who are willing to do a lot of work -- you can change the registry key resource in the DLL. For example, Alice comes with its own version of Python 1.5.1 and it uses "1.5.1-alice" as its registry key. The Alice installer installs Python in a subdirectory of the Alice installation directory and points the 1.5.1-alice registry entries there. The problem is that this is a lot of work for the average app builder.) I thought a bit about how VB solves this. I think that when you wrap up a VB app in, all the support code (mostly a big DLL) is wrapped with it. When the user runs the installer, the DLL is installed (probably in the WINDOWS directory). If a user installs several VB apps built with the same VB version, they all attempt to install the exact same DLL; of course the installers notice this and optimize it away, keeping a reference count. (Ignoring for now the fact that those reference counts don't always work!) If an app builty with a different VB version is installed, it has a DLL with a different name, and that is installed separately. Other support files, I presume, are dealt with in much the same way. Voila, there's the theory. How can we do something similar for Python? A app written in Python should need to install only three or four files: - a driver EXE to start the app - a copy of the Python DLL - the Python library in an archive - the app code in an archive The latter two could be combined into a single archive, but I propose that we use two archives so that the DLL and the Python library archive can be shared between installations of independent Python apps as long as they use the exact same Python version and don't need additional 3rd party packages. (I believe that Jim A's proposal combines the archives with the EXE and the DLL, reducing the number of files to two. That's fine too.) Is there a use for the registry here at all? Maybe not. (I notice that VB seems to have a single registry entry, pointing to a DLL; all other VB files also seem to live there.) Complications: - Some apps may need a custom extension module, which has to be installed as a PYD file. So it seems that there needs to be a directory per app, and perhaps per version of the app (if the app distributor cares). - Some apps need other, non-pyc files (e.g. data tables or help files); it would be handy if these could be stored in the archives as well. - Some standard extension modules are in their own PYD files; these also need to be installed. They aren't typically marked with a version, so perhaps a path directory per version of Python (if not per installed app) is wise. - How to distribute an app that needs 3rd party stuff, e.g. Tcl/Tk, or PIL, or NumPy? Their Python code can easily be wrapped up in another archive with a standard name incorporating a version number; but the required PYD and DLL files are a separate story. (E.g. for Tkinter, you need _tkinter.pyd which links against tcl80.dll.) Basically the same solution as for standard PYD files can work; the needed DLL files can be installed either systemwide (if they have a reliable version number in their name, like tcl80.dll) or in the per-app or per-package directory (like NumPy). - Presumably, the archives will contain PYC files only. This means that tracebacks will not show source code, only line numbers. For Jim A, this is probably exactly what he wants (if the user gets a traceback, his "robust app" has miserably failed, and he takes it in pride that this doesn't happen). But for some others, access to the sources could be essential. For example, I might want to distribute IDLE using this mechanism; users of IDLE who are curious about the standard library (or about IDLE itself) should be able to open the source for an arbitrary module (and maybe even edit it, although that's not a priority and perhaps should even be discouraged). Library source access is an important feature of the IDLE debugger as well. A way out for IDLE is to install a classic distribution of the Python library sources, into the filesystem at an IDLE specific location. Other apps, with only the need for source code in tracebacks, might choose to to have the PY files in the archives sitting next to the PYC files, and somehow the traceback mechanism should be accessing the archive to get a hold of the source. And yes, I realize that Jim A's latest offering solves most of these problems to a large extent -- well done. (Jim, would you care to comment on the issues that you don't address? Will you address them in a future version?) Final notes: There are two different problems here. One is how to distribute Python apps robustly to end users who don't particular care about Python. This is Jim A's problem (and he has a solution that works for him). In general the solutions here try to isolate the installed app from other Python installations. I'm proposing that at least the DLL and the Python library archive can probably be shared between apps without reducing robustness if we keep track more carefully of version numbers. The other problem is how to distribute packages of Python and extension modules for use by Python users. These typically need to drop into some existing Python installation. This is Paul Dubois' problem with NumPy (amongst others) and is the current focus of the distutil SIG. However I believe that there could be a lot of common infrastructure that would help us create better solutions for both problems. For package distribution, common infrastructure (a.k.a. standards) is essential. For app distribution, common infrastructure isn't so important (since the solutions strive for total isolation, there's no problem if different apps use solutions). However, this changes when app creators want to distribute robust self-sufficient apps that use 3rd party packages -- then the 3rd party packages must allow being packaged up using the app distribution creator of choice. Solving this compound problem (creating package distributions that can be redistributed easily as part of robust Python app distributions) should be an important goal for the infrastructure we're building here. The Big Import Rewrite ought to add this to its list of objectives if it isn't already on it. My guess is that the solution for this compound problem will increase the dependency of app distribution tools on the package distribution infrastructure; which to me seems like a Good Thing because it would lead to more code sharing. --Guido van Rossum (home page: http://www.python.org/~guido/)

"James C. Ahlstrom" wrote: [...]

ftp://ftp.interet.com/pub/pylib.html

Ouch - what's wrong with zip archives? There are utilities to convert to/from zip, to re-pack, to mount zip transparently so it's entries look like regular files, FTP servers, etc. Both Java (jar) and Tcl (Jan Nijtman's "Wrap") have adopted this format. Zips would seem natural with JPython. And suppose that scripting ever starts to consolidate to a common scripting kernel (yah, well), do you really want a system which is closing all doors to cross-fertilization? Zip has an advantage over .tar.gz in that its table of contents is available without having to decompress the whole kaboodle. Your format has no checksum, which for deployment and long-term storage can be important. If you want a marshalled TOC, then why not add a manifest entry for it, sort of like what ranlib does with ar? You designed the format so archives can be concatenated without any tool (other than "cat"), but this works just as well with zip files, as the Tcl Wrap approach demonstrates. Allow me to very, very loosely paraphrase Guido here: sure, everyone can design an archive format, but they are likely to make the same mistakes all over again - so why not adopt a format which is tried and tested? With all due respect - I sincerely hope you will reconsider and alter your code to work with zip files. It's probably a small adjustment? Unless your *intent* is to create a diverging standard, of course... -- Jean-Claude

"James C. Ahlstrom" wrote:

[...]

...

ftp://ftp.interet.com/pub/pylib.html

Jean-Claude Wippler replied:

Ouch - what's wrong with zip archives?

There are utilities to convert to/from zip, to re-pack, to mount zip transparently so it's entries look like regular files, FTP servers, etc.

Both Java (jar) and Tcl (Jan Nijtman's "Wrap") have adopted this format.

Zips would seem natural with JPython. And suppose that scripting ever starts to consolidate to a common scripting kernel (yah, well), do you really want a system which is closing all doors to cross-fertilization?

Zip has an advantage over .tar.gz in that its table of contents is available without having to decompress the whole kaboodle.

Your format has no checksum, which for deployment and long-term storage can be important.

If you want a marshalled TOC, then why not add a manifest entry for it, sort of like what ranlib does with ar?

You designed the format so archives can be concatenated without any tool (other than "cat"), but this works just as well with zip files, as the Tcl Wrap approach demonstrates.

Allow me to very, very loosely paraphrase Guido here: sure, everyone can design an archive format, but they are likely to make the same mistakes all over again - so why not adopt a format which is tried and tested?

With all due respect - I sincerely hope you will reconsider and alter your code to work with zip files. It's probably a small adjustment?

Unless your *intent* is to create a diverging standard, of course...

Exactly my sentiments. We have rough Python code to deal with zip files; it's very rough because we got kind of carried away adding features and ended up with spaghetti code :-( But it's working code nevertheless and we're offering it up for anyone in this group to clean up (we could do that ourselves but it's not high on our current priority list). I don't know anything about Tcl Wrap. I do know a great deal about the ZIP format, but apparently I missed the concatenation feature. How does this work? Does that work for all zip tools, or just for the ZIP reader in Wrap? (I looked up how Jim A does it -- his central directory at the end of the file contains the total size of the data covered by that directory, so he seeks back to the beginning of it and sees if another magic number precedes it; and so on. Very simple.) I quickly looked at the Wrap page; it shows how to access data files stored in the archive. Question: does the wrap::open code go out to the regular filesystem if it finds there's no wrap archive? That would be handy so you can test the code in its unwrapped form without change. Python needs this too. --Guido van Rossum (home page: http://www.python.org/~guido/)

Jean-Claude Wippler wrote:

Ouch - what's wrong with zip archives?

Thanks very much for looking over the format. In general Zip archives store whole branches of a file system. A Python ./Lib zip archive would contain: N:/python/Python-1.5.2/Lib/string.pyc N:/python/Python-1.5.2/Lib/os.pyc N:/python/Python-1.5.2/Lib/copy.pyc N:/python/Python-1.5.2/Lib/test/testall.pyc Zip archives are isomorphic to branches of a file system. That means there must be a sys.path for each zip archive file. How would this be specified? The archive format stores modules as dotted names, just as they appear in the import statement. The search path is "." in every archive file by definition. The import statement "import foo" just results in a dictionary lookup for key "foo", not a search through a zip directory along a local search path for "foo.something" where "something" can be pyc, pyo, py, etc. The intent was to link the archives to the import statement, not re-create a directory tree. It borrowed this feature from the archive formats of Greg and Gordon.

There are utilities to convert to/from zip, to re-pack, to mount zip transparently so it's entries look like regular files, FTP servers, etc.

Basic operations (to, from, repack) are easy in Python.

Both Java (jar) and Tcl (Jan Nijtman's "Wrap") have adopted this format.

Hmmm....

Your format has no checksum, which for deployment and long-term storage can be important.

Actually the pylib.py "dir()" method reads all *.pyc with marshal, and I am depending on marshal to object to bad data and also out-of-date magic numbers. But this is a good point.

If you want a marshalled TOC, then why not add a manifest entry for it, sort of like what ranlib does with ar?

Sorry, I don't understand. Please explain.

You designed the format so archives can be concatenated without any tool (other than "cat"), but this works just as well with zip files, as the Tcl Wrap approach demonstrates.

Are you saying that cat zip1.zip zip2.zip > myzip.zip works? An important feature is the ability to concatenate to a binary: cat python.exe zip1.zip > myapp.exe Searching for this isn't fast unless magic numbers are at the end. Are zip files recognizable from the end (I don't know)?

Allow me to very, very loosely paraphrase Guido here: sure, everyone can design an archive format, but they are likely to make the same mistakes all over again - so why not adopt a format which is tried and tested?

With all due respect - I sincerely hope you will reconsider and alter your code to work with zip files. It's probably a small adjustment?

Unless your *intent* is to create a diverging standard, of course...

The intent is to create a standard but not a diverging standard. Are there any zip experts out there? Can zip files satisfy all the design requirements I listed in pylib.html? Is there zip code available? All my code is in Python. JimA

...

...

...

...

"JW" == Jean-Claude Wippler writes:

JW> Same for Wrap. Standard tools would not see the preceding ZIP JW> groups. JW> In terms of maintenance, I'd avoid this trick. I merely JW> wanted to point out that zip archives can be stacked, if the JW> reader is set up to it. I agree. I can't recall the details now, but I had a lot of problems with zip concatenation in JPython. I think at least some of the older Java tools for groking zips don't work with contatenation. -Barry

James C. Ahlstrom wrote:

Jean-Claude Wippler wrote:

...

Ouch - what's wrong with zip archives?

In general Zip archives store whole branches of a file system.

The archive format stores modules as dotted names, just as they appear in the import statement. The search path is "." in every archive file by definition. The import statement "import foo" just results in a dictionary lookup for key "foo", not a search through a zip directory along a local search path for "foo.something" where "something" can be pyc, pyo, py, etc.

The intent was to link the archives to the import statement, not re-create a directory tree. It borrowed this feature from the archive formats of Greg and Gordon.

As I've stated before, I have 2 archive formats. This may seem a needless complication, but my suspicion is that sooner or later, people will want 2 different kinds. One is a .pyz format, which corresponds closely to Jim's .pyl format (with a number of minor differences: it's compressed, the archive as a whole has the Python magic number, instead of each entry, and it's not designed for concatenation). The other is like a zip, and probably should be zip format. It's designed to hold _anything_, and can be manipulated from C and from Python. It can be concatenated and / or embedded (and the innner one opened without extraction). It's table of contents is more file-system like. Importing from one is slower, but that's not really what it's for. It's for packaging up arbitrary resources. Like .pyz's, or Tcl/Tk for Tkinter apps, or configuration files. Jim is correct that a good importer (which can say "No, it's not mine" as quickly as possible) is better satisfied by a simple dictionary lookup than fooling with file extensions and directories (virtual or real).

...

If you want a marshalled TOC, then why not add a manifest entry for it, sort of like what ranlib does with ar?

Sorry, I don't understand. Please explain.

The table of contents is just another entry.

An important feature is the ability to concatenate to a binary: cat python.exe zip1.zip > myapp.exe Searching for this isn't fast unless magic numbers are at the end. Are zip files recognizable from the end (I don't know)?

Where do you think we got this idea?

Are there any zip experts out there? Can zip files satisfy all the design requirements I listed in pylib.html? Is there zip code available? All my code is in Python.

Hmm. My bookmark appears to be dead (I was there not long ago): http://www.cubic.org/source/archive/fileform/packers/appnote.t xt There have been several references on this list to Guido et al having some Python / zip code. - Gordon

In general Zip archives store whole branches of a file system. A Python ./Lib zip archive would contain:

N:/python/Python-1.5.2/Lib/string.pyc N:/python/Python-1.5.2/Lib/os.pyc N:/python/Python-1.5.2/Lib/copy.pyc N:/python/Python-1.5.2/Lib/test/testall.pyc

Zip archives are isomorphic to branches of a file system. That means there must be a sys.path for each zip archive file. How would this be specified?

Not true. It's easy (using the proper Zip tools) to creat an archive containing this instead: string.pyc os.pyc copy.pyc testall.pyc Thus the entire archive is considered the directory. The Java "jar" tool uses this approach. It's also easy to have packages in there (again this is what Java does): test/ test/__init__.pyc test/pystone.pyc test_support.pyc (etc.)

The archive format stores modules as dotted names, just as they appear in the import statement. The search path is "." in every archive file by definition. The import statement "import foo" just results in a dictionary lookup for key "foo", not a search through a zip directory along a local search path for "foo.something" where "something" can be pyc, pyo, py, etc.

The intent was to link the archives to the import statement, not re-create a directory tree. It borrowed this feature from the archive formats of Greg and Gordon.

Maybe you've gone overboard. The time it takes to translate the dots into slashes really isn't the big deal.

Are there any zip experts out there? Can zip files satisfy all the design requirements I listed in pylib.html? Is there zip code available? All my code is in Python.

Yes (all of us here at CNRI), yes, yes (we have the spaghetti code). While zip files support compression, they support uncompressed files as well and we could go either way. Their most popular compression format is gzip compatible and can be read and written with the zlib module, which is in the standard Python distribution (even on Windows) -- though to build it you need the zlib C library which is of course external (but solid open source). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

[... horrors of cross-OS mounts and ":\/" separators ...] I agree, this has some very hairy sides to it. But VFS is really more about mounting non-FS things in a "root" FS (presumably the real one).

On the other hand the VFS concept could be used as a totally different solution to the sys.importers vs. sys.path

Heck, I'll be the "enfant terrible" once more: yes, and this stuff could well be implemented generically across scripting languages. Of course the act of "importing" is a very Pythonic issue - but FS/VFS traversal and the actual shared library load need not be. Anyway, enough of that.

Take for example the Windows registry -- looks a lot like a filesystem, doesn't it? Yet it has one fundamental property that a typical FS doesn't: directory nodes can have data *and* children...

What you're saying is that dir = set-of-subdirs + set-of-files, and that this is a more general requirement than plain FS's. Doesn't that simply mean that the more general model is needed as basis to handle both?

Trees are a universal concept, but code sharing is still elusive...

Ah, but think of the implications: archives, networks, XML, the world! -- Jean-Claude

Tim Peters wrote:

[Skip Montanaro]

...

Is there some way that people writing applications in Python can set

Yes, but they can't get Python to look at those before it's too late. I spent a whole evening a month or two ago just trying to figure out where all the cruft in my Windows sys.path *came* from. This is out-of-the-box; I .....

Excellent discussion Tim!

I suspect it still wouldn't help with the problem I was facing, though. That is, I wanted to be able to tell people to run

\\dragres01\mrec\reduce\python \\dragres01\mrec\reduce\reduce.py

which is just a Windows way of saying "run a Python executable from a shared network location". When they tried that, though, the network Python looked in *their* individual registries for its Python path info, and some of the hackers with mondo customized Python setups on their own machines watched things go down in flames.

I think a sensible way to run little apps is to put everything in an archive file including the main.py. On Windows you concattenate that to python.exe, and it Just Works.

Windows registry serves. Is there anyone on Windows who *doesn't* have their Python Lib/ etc as direct subdirectories of the directory containing python.exe? Not that I've seen.

Point on the curve. We don't. We freeze everything except the main.py. JimA

Jean-Claude Wippler wrote:

...

Ouch - what's wrong with zip archives?

...

With all due respect - I sincerely hope you will reconsider and alter your code to work with zip files. It's probably a small adjustment?

OK, I now have a new module "zipfile" which reads and writes ZIP files. It is written in Python and has been tested on Windows and Linux. I tested it with WinZip and found that the files it creates are read OK with WinZip, and WinZip files are read OK with zipfile. So I am withdrawing my Python archive file format, and re-writing all my stuff using zipfile. It should all be done in a week. Basically everything works fine. But there are some problems. Python seems to lack a CRC-32 function, so I wrote one in Python. It is slow. We need to add a CRC-32 function to some Python built-in module that it always present, like md5 or binascci. The zlib module is not necessarily present. I can't seem to get WinZip to record a partial path. That is, I want the ./Lib/test package to have these ZIP paths: test/__init__.pyc test/testall.pyc ... but WinZip creates files with either no path at all or the fully specified path. Am I missing something? Do all other ZIP tools do this too? JimA Return-Path: Delivered-To: python-dev@dinsdale.python.org Received: from python.org (parrot.python.org [132.151.1.90]) by dinsdale.python.org (Postfix) with ESMTP id EFDA11CDB9 for ; Mon, 13 Dec 1999 10:21:56 -0500 (EST) Received: from cnri.reston.va.us (ns.CNRI.Reston.VA.US [132.151.1.1] (may be forged)) by python.org (8.9.1a/8.9.1) with ESMTP id KAA06423 for ; Mon, 13 Dec 1999 10:21:55 -0500 (EST) Received: from kaluha.cnri.reston.va.us (kaluha.cnri.reston.va.us [132.151.7.31]) by cnri.reston.va.us (8.9.1a/8.9.1) with ESMTP id KAA04774 for ; Mon, 13 Dec 1999 10:21:56 -0500 (EST) Received: from eric.cnri.reston.va.us (eric.cnri.reston.va.us [10.27.10.23]) by kaluha.cnri.reston.va.us (8.9.1b+Sun/8.9.1) with ESMTP id KAA04556 for ; Mon, 13 Dec 1999 10:22:34 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by eric.cnri.reston.va.us (8.9.3+Sun/8.9.1) with ESMTP id KAA18858 for ; Mon, 13 Dec 1999 10:22:34 -0500 (EST) Resent-Message-Id: <199912131522.KAA18858@eric.cnri.reston.va.us> Message-Id: <199912131522.KAA18858@eric.cnri.reston.va.us> To: "James C. Ahlstrom" Subject: Re: [Python-Dev] Re: [Distutils] Questions about distutils strategy In-reply-to: Your message of "Mon, 13 Dec 1999 09:50:11 EST." <385507A3.9F6AAF0F@interet.com> References: <000301bf4206$b39e5b80$36a2143f@tim> <384FC47A.BB4DA517@interet.com> <384FDAF5.C25C447C@equi4.com> <38510254.ED15D32B@interet.com> <385507A3.9F6AAF0F@interet.com> Date: Mon, 13 Dec 1999 10:22:12 -0500 From: Guido van Rossum Resent-Cc: python-dev@python.org Resent-Date: Mon, 13 Dec 1999 10:22:34 -0500 Resent-From: Guido van Rossum Sender: python-dev-admin@python.org Errors-To: python-dev-admin@python.org X-BeenThere: python-dev@python.org X-Mailman-Version: 1.2 (experimental) Precedence: bulk List-Id: Python core developers

OK, I now have a new module "zipfile" which reads and writes ZIP files. It is written in Python and has been tested on Windows and Linux. I tested it with WinZip and found that the files it creates are read OK with WinZip, and WinZip files are read OK with zipfile. So I am withdrawing my Python archive file format, and re-writing all my stuff using zipfile. It should all be done in a week.

Ah, good! (This saves me the trouble of cleaning up our own zip code :-)

Basically everything works fine. But there are some problems.

Python seems to lack a CRC-32 function, so I wrote one in Python. It is slow. We need to add a CRC-32 function to some Python built-in module that it always present, like md5 or binascci. The zlib module is not necessarily present.

I can't seem to get WinZip to record a partial path. That is, I want the ./Lib/test package to have these ZIP paths: test/__init__.pyc test/testall.pyc ... but WinZip creates files with either no path at all or the fully specified path. Am I missing something? Do all other ZIP tools do this too?

Unclick the "Save Extra Folder Info" and then drag the *parent* folder into the archive. --Guido van Rossum (home page: http://www.python.org/~guido/)

[James C. Ahlstrom]

... Python seems to lack a CRC-32 function, so I wrote one in Python. It is slow. We need to add a CRC-32 function to some Python built-in module that it always present, like md5 or binascci. The zlib module is not necessarily present.

Unfortunately, there are many different CRC functions in common use. None belong in md5; if the intent is to support just zip's version, adding a (say) zipcrc32 function to binascii would be ok; if we expect to support others as well, a new parameterized crc module would be in order.

I can't seem to get WinZip to record a partial path. That is, I want the ./Lib/test package to have these ZIP paths: test/__init__.pyc test/testall.pyc ... but WinZip creates files with either no path at all or the fully specified path. Am I missing something? Do all other ZIP tools do this too?

No, it's a clumsiness unique to WinZip (damn GUIs <0.9 wink>). In the Add dialog box, you need to cd to the *Lib* directory, check the "Save extra folder info" box, and then, e.g., 1. Put test\*.pyc in the Add Files line, and click Add With Wildcards. Then all test\*.pyc files will be added, with paths test/__init__.pyc etc. or 2. Put "test\__init__.pyc" "test\testall.pyc" (including the quotes!) in the Add Files line, and click Add. Since #2 can be unbearable, other useful strategies include: 3. Use #1 (e.g. with dir\*.*) then delete the files you didn't really want. 4. Use #1 repeatedly, cleverly using a number of wildcard patterns that cover the files of interest. 5. Mixtures of #3 and #4. 6. Use a comand-line zip tool instead (e.g., pkzip; I think WinZip has an "experimental" cmdline add-on too, but haven't tried it).

Tim Peters wrote:

[James C. Ahlstrom]

...

... Python seems to lack a CRC-32 function, so I wrote one

Unfortunately, there are many different CRC functions in common use. None belong in md5; if the intent is to support just zip's version, adding a (say) zipcrc32 function to binascii would be ok; if we expect to support others as well, a new parameterized crc module would be in order.

OK, a CRC-32 in binascii it is. The CRC-32 I have comes with these comments which seem to indicate it is a more "official standard" CRC-32 than average: # * Crc - 32 BIT ANSI X3.66 CRC checksum files #*********************************************************************\ #* *| #* Demonstration program to compute the 32-bit CRC used as the frame *| #* check sequence in ADCCP (ANSI X3.66, also known as FIPS PUB 71 *| #* and FED-STD-1003, the U.S. versions of CCITT's X.25 link-level *| #* protocol). The 32-bit FCS was added via the Federal Register, *| #* 1 June 1982, p.23798. I presume but don't know for certain that *| #* this polynomial is or will be included in CCITT V.41, which *| #* defines the 16-bit CRC (often called CRC-CCITT) polynomial. FIPS *| #* PUB 78 says that the 32-bit FCS reduces otherwise undetected *| #* errors by a factor of 10^-5 over 16-bit FCS. *| #* *| #********************************************************************* #* Copyright (C) 1986 Gary S. Brown. You may use this program, or #* code or tables extracted from it, as desired without restriction. I can submit this as a patch to binascii, or if the Copyright bothers anyone, maybe it is better for Guido to use his CRC-32 from his ZIP code. Preference?

...

I can't seem to get WinZip to record a partial path. That is,

dialog box, you need to cd to the *Lib* directory, check the "Save extra folder info" box, and then, e.g.,

Thanks. I knew there had to be some magic incantation to do it.

6. Use a comand-line zip tool instead (e.g., pkzip; I think WinZip has an "experimental" cmdline add-on too, but haven't tried it).

Actually pkzip 2.04g doesn't work because it writes names in upper case and is limited to 8.3 names (I think). My zipfile.py can be used as a basis for a command line tool. Actually I use makefiles with imbedded Python programs and find this easier than command line tools. JimA

Guido van Rossum wrote:

I propose to use Gary Brown's code. I'll defend this to CNRI's lawyers if need be.

Jim, have you checked that this is the right CRC to use for zip's CRC? (This in the light of Tim's assertion that there are many CRCs around.)

The CRC it calculates agrees with the CRC of WinZip for all files I have tried. The original Gary Brown code was much longer and included file reading. Here is the shortened version: JimA # * Crc - 32 BIT ANSI X3.66 CRC checksum files #*********************************************************************\ #* *| #* Demonstration program to compute the 32-bit CRC used as the frame *| #* check sequence in ADCCP (ANSI X3.66, also known as FIPS PUB 71 *| #* and FED-STD-1003, the U.S. versions of CCITT's X.25 link-level *| #* protocol). The 32-bit FCS was added via the Federal Register, *| #* 1 June 1982, p.23798. I presume but don't know for certain that *| #* this polynomial is or will be included in CCITT V.41, which *| #* defines the 16-bit CRC (often called CRC-CCITT) polynomial. FIPS *| #* PUB 78 says that the 32-bit FCS reduces otherwise undetected *| #* errors by a factor of 10^-5 over 16-bit FCS. *| #* *| #********************************************************************* # #* Copyright (C) 1986 Gary S. Brown. You may use this program, or #* code or tables extracted from it, as desired without restriction. # First, the polynomial itself and its table of feedback terms. The # polynomial is # X^32+X^26+X^23+X^22+X^16+X^12+X^11+X^10+X^8+X^7+X^5+X^4+X^2+X^1+X^0 # Note that we take it "backwards" and put the highest-order term in # the lowest-order bit. The X^32 term is "implied"; the LSB is the # X^31 term, etc. The X^0 term (usually shown as "+1") results in # the MSB being 1. # Note that the usual hardware shift register implementation, which # is what we're using (we're merely optimizing it by doing eight-bit # chunks at a time) shifts bits into the lowest-order term. In our # implementation, that means shifting towards the right. Why do we # do it this way? Because the calculated CRC must be transmitted in # order from highest-order term to lowest-order term. UARTs transmit # characters in order from LSB to MSB. By storing the CRC this way, # we hand it to the UART in the order low-byte to high-byte; the UART # sends each low-bit to hight-bit; and the result is transmission bit # by bit from highest- to lowest-order term without requiring any bit # shuffling on our part. Reception works similarly. # The feedback terms table consists of 256, 32-bit entries. Notes: # # 1. The table can be generated at runtime if desired; code to do so # is shown later. It might not be obvious, but the feedback # terms simply represent the results of eight shift/xor opera- # tions for all combinations of data and CRC register values. # # 2. The CRC accumulation logic is the same for all CRC polynomials, # be they sixteen or thirty-two bits wide. You simply choose the # appropriate table. Alternatively, because the table can be # generated at runtime, you can start by generating the table for # the polynomial in question and use exactly the same "updcrc", # if your application needn't simultaneously handle two CRC # polynomials. (Note, however, that XMODEM is strange.) # # 3. For 16-bit CRCs, the table entries need be only 16 bits wide; # of course, 32-bit entries work OK if the high 16 bits are zero. # # 4. The values must be right-shifted by eight bits by the "updcrc" # logic; the shift must be unsigned (bring in zeroes). On some # hardware you could probably optimize the shift in assembler by # using byte-swap instructions. # Converted to Python by James C. Ahlstrom crc_32_tab = [ # CRC polynomial 0xedb88320 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, 0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5, 0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, 0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59, 0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433, 0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01, 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65, 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f, 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, 0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, 0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b, 0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d, 0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713, 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777, 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9, 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d ] def crc32(string): crc = 0xFFFFFFFF for ch in string: crc = crc_32_tab[((crc) ^ ord(ch)) & 0xff] ^ (((crc) >> 8) & 0xFFFFFF) return ~crc

Tim Peters wrote:

If WinZip unzips Jim's files without griping, the odds that he's got the wrong CRC are about 1 in 2**36 <wink>.

You mean 2**32, right? Oh, sorry, you must be using a DEC-10 <wink again>. JimA

[Tim]

If WinZip unzips Jim's files without griping, the odds that he's got the wrong CRC are about 1 in 2**36 <wink>.

[JimA]

You mean 2**32, right?

Nope! For each of the 2**32 polynomials you may have pulled out of thin air, there are about a dozen common variations in the details of CRC algorithms. For example, a CRC used for hashing usually initializes "the register" to 0, but a CRC used to protect against transmission errors usually initializes to a block of 1 bits (since leading zeroes don't affect the result, and a common transmission error is dropping a prefix of the msg). Similarly, algorithms vary in the order they scan the data; in whether they use the raw data or its complement; and in whether they return the actual remainder, the complement of the remainder, or a checksum cleverly computed so that "the other end" always sees a fixed remainder other than 0 (or ~0).

Oh, sorry, you must be using a DEC-10 <wink again>.

I used a Univac 1108 in college, back when ASCII was in its infancy. They couldn't decide on the natural size for a character, so the 36-bit 1108 could be configured to treat each word as either 6 6-bit bytes or 4 9-bit ones. If they had been thinking ahead, they would have defined it as two Unicode characters plus a 4-bit tag field for the Python implementation to play with <wink>. now-they-make-their-living-suing-.gif-bandits-ly y'rs - tim

[Guido]

...

I propose to use Gary Brown's code. I'll defend this to CNRI's lawyers if need be.

If there's a hassle, I can do a clean-room implementation easily enough -- although I'd rather not.

or you can grab the code from PIL, which already comes with a Python compatible license... (it's based on ISO 3307, but judging from the table James posted, it's the same thing...) </F>

Greg Stein wrote:

Release early. Release often. :-)

You are right of course. OK, the zipfile.py code and docs are at: ftp://ftp.interet.com/pub/pylib.html Despite the ftp URL, clicking on it should display the html. Please don't panic if is seems to be slow. It uses a Python CRC-32 which is slow. You may want to hack it to use zlib.crc32() if you have it. I am testing with WinZip. If you have another zip tool, it would be interesting to see how compatible it is. JimA

Did anyone look at this yet? ftp://ftp.interet.com/pub/pylib.html ftp://ftp.interet.com/pub/zipfile.py JimA

"James C. Ahlstrom" wrote:

ftp://ftp.interet.com/pub/pylib.html

I just changed zipfile.py so that regular zip compression works. And if zlib is available, its crc32() is used instead of the Python version. I should mention that the current code rejects zip files which have an archive comment added to the end. Accepting them would require a search, and I am not sure it is worth it. JimA

"James C. Ahlstrom" wrote:

"James C. Ahlstrom" wrote:

...

ftp://ftp.interet.com/pub/pylib.html

I just changed zipfile.py so that regular zip compression works. And if zlib is available, its crc32() is used instead of the Python version.

I should mention that the current code rejects zip files which have an archive comment added to the end. Accepting them would require a search, and I am not sure it is worth it.

I don't think it is needed for our purposes, but maybe a subclass could provide it ? FYI, I've tested the module against mxStack-0.3.0.zip which you can find on my Python Pages. It was created using Info-ZIP's zip 2.2 on Linux. Unfortunately, I always get the following traceback when trying to print the directory:

...

...

z.open('../projects/distribution/mxStack-0.3.0.zip','rb') z.dir() File Name Modified Size Stack/mxStack/mxStack.h 1999-04-16 10:50:06 4368 Stack/mxStack/mxstdlib.h 1999-04-13 15:37:52 5433 Traceback (innermost last): File "<stdin>", line 1, in ? File "/home/lemburg/lib/zipfile.py", line 120, in dir bytes = self.read(name) # Just to check CRC-32 File "/home/lemburg/lib/zipfile.py", line 133, in read bytes = zlib.decompress(bytes, -15) zlib.error: Error -5 while decompressing data

Some notes on the API: ---------------------- * I would find it more convenient if the filename and mode would be constructor parameters, e.g. zfile = zipfile('myfile.zip','rb') with compression defaulting to 8 rather than 0 (most zip files will be deflated since this is the ZIP default). * Also, I would like a method much like the os.listdir() which returns a list of filenames rather than print it to stdout. * .is_zipfile() should probably be a separate function: it doesn't use any of the class' features. More wishes to come ;-) So far: Great Work ! Aside: I found that you are using undocumented arguments to zlib.compressobj() ... are these extra arguments left out of the documentation on purpose or by simple oversight ? I couldn't find them in the HTML docs and neither in the docstrings. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 15 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg writes:

Aside: I found that you are using undocumented arguments to zlib.compressobj() ... are these extra arguments left out of the documentation on purpose or by simple oversight ? I couldn't find them in the HTML docs and neither in the docstrings.

The documentation is way out of date and Jeremy Hylton and Andrew Kuchling haven't updated it. I'm not sure which of them changed the signatures for that module, but I've pestered Jeremy about it a few times. If anyone would like to update the documentation, I'd certainly appreciate it. I don't know the details of those interfaces, and this is somewhere where the details are pretty critical. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

Greg Stein wrote:

On Thu, 16 Dec 1999, James C. Ahlstrom wrote:

...

Did anyone look at this yet?

ftp://ftp.interet.com/pub/pylib.html

ftp://ftp.interet.com/pub/zipfile.py

Looking at it now... The writepy() as a method is questionable, I think. I think it should open the file at instantiation time. I don't see a reason to allow that to be deferred. Especially given that some of the methods fail if open() hasn't been called.

I eliminated open and added its args to the constructor.

It would be good to have symbolic names for the 0 and 8 compression constants, and to fail if 8 is passed and zlib is not available (otherwise, it doesn't fail until read/write time, and with a NameError). There should probably be a __del__ that calls close(). Oh, and a "closed" attribute that can be checked and an error raised if an operation is done after the file has been closed.

All done.

I think dir() should return the contents, rather than print them.

I added listdir() and documented self.TOC. I kept printdir() as example code.

read() and write() ought to fail if the mode is incorrect. Oh, some symbolic constants for things like "PK\005\006" would be nice.

All done. JimA

Fredrik Lundh wrote:

James C. Ahlstrom wrote:

...

...

ftp://ftp.interet.com/pub/pylib.html

-- it would be great if "open" could take an open file object as well as a file name.

I put these arguments into the constructor now.

(in this case, you also need to document what you expect from the underlying file object: read, write, seek, tell should be enough, right? haven't looked at the code -- assuming it works, I'm only interested in the interface)

OK, docs updated.

-- I assume "open" adds "b" to the given mode argument.

Correct. The mode can be either "w" or "wb" etc., and it works.

-- "dir" looks a bit strange. and hey, there's no "listdir" in there. I'd prefer a recursive "listdir" method, which takes an optional "depth" argument (e.g. 0=this dir, 1=this dir and first subdir, None=infinity, i.e. the full tree).

I added a plain listdir() and changed dir() to printdir(). I also documented self.TOC which gets you the values too. JimA

"James C. Ahlstrom" wrote:

"M.-A. Lemburg" wrote:

...

"James C. Ahlstrom" wrote:

...

...

ftp://ftp.interet.com/pub/pylib.html

...

Unfortunately, I always get the following traceback when trying to print the directory:

Yes, compression isn't there yet. I am looking into it.

Great :-)

...

Some notes on the API: ---------------------- * I would find it more convenient if the filename and mode would be constructor parameters, e.g.

zfile = zipfile('myfile.zip','rb')

OK, done.

...

with compression defaulting to 8 rather than 0 (most zip files will be deflated since this is the ZIP default).

Until compression works, and zlib ships with Python I would rather default to no compression (method 0). Otherwise this is not useful as a Python import archive.

Point taken. Perhaps it would be even better to not have a default at all: that way people will have to think about the issue *before* implementing it, rather than debug code that produces tracebacks.

...

* Also, I would like a method much like the os.listdir() which returns a list of filenames rather than print it to stdout.

OK, done.

...

* .is_zipfile() should probably be a separate function: it doesn't use any of the class' features.

OK, done.

Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 13 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

...

...

...

...

"JCA" == James C Ahlstrom writes:

...

Aside: I found that you are using undocumented arguments to zlib.compressobj() ... are these extra arguments left out of the documentation on purpose or by simple oversight ? I couldn't find them in the HTML docs and neither in the docstrings.

JCA> I am following the CNRI code blindly here. I don't have docs JCA> either. The docs for the zlib module are quite out of date, although I think the docstrings may be better (not necessarily completely up-to-date thought :-). The specific parameters to pass to zlib don't seem to be documented anywhere either; IIRC I dug them out of some example C code somewhere that used zlib to read Zip files. Jeremy

"M.-A. Lemburg" wrote:

One thing I'd suugest is to include some way to delete and update contents, e.g. the write() method should overwrite any existing entry in the archive (if it not already does -- I haven't tested it, just read the code and it seems to raise an exception), plus maybe a .remove() method which deletes an entry.

Currently, adding a file requires the "a" append mode, while the "w" mode re-writes the file. Adding a duplicate file name produces an error message. I can change this, but removing a file would either waste space, or else the file contents must be copied over the old file and all the offsets updated. I don't like this because it is complicated, and I think it is fast enough to just re-write the archive. But it could be added if people want.

True. How about making the compression argument mandatory for file opened in 'wb' mode only ?

The default of zero provides a little guidance that you should use zero. I added a warning message if 8 is used which should discourage people from using 8. Or I could disallow 8. Is that OK? JimA

"M.-A. Lemburg" wrote:

I guess it would be ok to waste space. You could provide a .cleanup() or .rewrite() method that takes care of reorganizing the file to fill up the gaps.

OK, adding a duplicate name replaces the old file.

Well the module seems to work just fine with compression on, so disallowing it or issuing a warning would reduce its value, IMHO.

Yes compression works, but 90% of Python installations don't have zlib, so it is an ERROR to create archives with compression when these archives are distributed to other sites.

How about making compression a boolean value and then converting any true value to 8 ?

It would close the door to future or other compression methods. Currently the method must be 0 or 8 or a traceback will result. JimA

Greg Stein wrote:

On Mon, 20 Dec 1999, James C. Ahlstrom wrote:

...

"M.-A. Lemburg" wrote: But it shouldn't print a warning(!). If an application wants to replace a file, then stuff shouldn't appear on stdout as a result.

OK, no warning.

The warning that __init__ prints shouldn't be there.

OK, it is gone.

Really: there should not be a single "print" in the library (well,

No print unless _debug > 0 JimA

"M.-A. Lemburg" wrote:

Sure, for the sake of creating Python code archives, but your module is much more versatile: e.g. I could automatically create ZIP archives of log files or sets of other files and

OK, zipfile.py no longer complains about compression != 0 JimA

Finn Bock wrote:

James C. Ahlstrom wrote:

...

ftp://ftp.interet.com/pub/pylib.html

I feel that it smell a bit too much like a tool and too little like an general programming api.

It was meant to be an API except for writepy(), which is clearly a tool.

- It can only add disk files. The ability to write data to a zip entry through a file-like object or from a string would make it more like an API, IMHO

I could add a method writestr(self, string, year, month, day, hour, minute, second, ...) There are a lot of fields required which usually come from the file.

- Some kind of access to the TOC entry fields (date, size, compressed size etc) also seems like a nice feature.

This access is provided directly by self.TOC, and the fields are documented.

- The data for an entry must be available in memory. Could be a problem for huge files, but most like not in practical use.

I agree, but adding loops will make it slower. What do others think?

I admit that I am fond of the api from java.util.zip.ZipFile and java.util.zip.ZipOutputStream.

I don't know this API. If writestr() is not sufficient, what API would you like? JimA

Briefly backtracking to an old thread: [Guido]

... The problem lies in which key is used. All versions of Python 1.5.x (1.5, 1.5.1, 1.5.2) use the same key! This is a main cause of trouble, because it means that different versions cannot peacefully live together even if the user installs them into different directories -- they will all use the registry keys of the last version installed. This, in turn, means that someone who writes a Python application that has a dependency on a particular Python version (and which application worth distributing doesn't :-) cannot trust that if a Python installation is present, it is the right one. But they also cannot simply bundle the standard installer for the correct Python version with their program, because its installation would overwrite an existing Python application, thus breaking some *other* Python apps that the user might already have installed.

Right, that's one class of intractable problem under Windows. *Inside* my workplace, another kind of problem is caused when people try to make a Python app available over the Windows network. They stick the Python they want and its libraries out on the network, with python.exe in the same directory as the app. Now some people have highly customized Python setups, and the network Python picks up "the wrong" site.py etc. That sucks, and there appears no sane way to stop it. Telling internal app distributors they need to invent a unique registry key and fiddle their python.exe's resources is a non-starter. Ditto telling people with highly customized Pythons "don't do that". Ditto telling anyone they have to run any sort of installation script just to use a network app (sometimes they don't even know they're running it! e.g., when it's a subsystem invoked by another app). So while everyone is thinking about the hardest possible scenarios, please give a thought to the dirt simple one too <0.5 wink>: an app distributor who knows exactly what they're doing, and for whom *any* magical inference is simply a barrier to overcome. The latter can be satisfied by any number of means, from an envar that says "please don't try to be helpful, *this* is the directory you look in, and if you don't find stuff there give up" to a cmdline switch that says the same. Nothing Windows-specific there -- any OS with an envar or a cmdline will play along <wink>.

... I thought a bit about how VB solves this. I think that when you wrap up a VB app in, all the support code (mostly a big DLL) is wrapped with it. When the user runs the installer, the DLL is installed (probably in the WINDOWS directory). If a user installs several VB apps built with the same VB version, they all attempt to install the exact same DLL; of course the installers notice this and optimize it away, keeping a reference count.

This is the way most *MS* DLLs work; stuff like the C runtime libraries and MS database drivers work exactly the same way. It's rare for pkgs other than MS's to attempt to use this mechanism, though (the reason is given below).

(Ignoring for now the fact that those reference counts don't always work!)

? They work very well, in my experience. Where they fail is when installers & uninstallers break the rules. MS publishes the list of MS DLLs that are to be treated this way: an installer "must" use refcounting on the DLLs in the list. Alas, some (especially older) installation pkgs don't. Then the refcounts get screwed up. That's what makes the mechanism brittle: "the system" doesn't enforce it, it relies on universal & intelligent cooperation. It's very likely that someone distributing a Python app will neglect (out of ignorance) to bump the refcount on their Python components, so the refcount will be artificially low, and a later uninstall of some unrelated pkg that *did* follow the rules will merrily delete Python. Gordon and I will repeat this until it sinks in <wink>: almost everyone with a successful Windows product ships the non-MS DLLs they rely on and copies them into their own app directory. It's simple and it works; alternatives are complicated and don't work. Many even ship & copy MS DLLs (e.g., Scriptics copies its own msvcrt.dll (the MS C runtime) into Tcl's directories). Worrying about space consumed by redundant Python components is a bad case of premature optimization <0.3 wink>.

... How can we do something similar for Python?

Seriously, short of getting MS to distribute Python and put the Python DLLs on The List of refcounted resources, we should pursue this line reluctantly if at all. MS may have a better scheme in the future, but for now better safe than sorry. a-couple-mb-on-a-modern-pc-isn't-worth-the-time-it-took- to-read-this<wink>-ly y'rs - tim