[Distutils] resource_filename() breaks cxfreeze (py2exe?)

[Phillip J. Eby]

At 07:14 PM 10/12/2005 -0400, Robby Dermody wrote:
>Hi guys,
>
>The subject line no doubt tells you all something you already know. This
>morning, I didn't even know that setuputils existed, so excuse my
>ignorance on the topic.
>
>Basically, I have an application that makes use of a python web
>application development library called nevow. The current version of
>nevow (0.5.0) makes use of resource_filename() to get the paths of
>certain static files that it uses. Things work fine when my app is not
>frozen, but after I freeze it with cx_freeze resource_filename() turns
>out to be a breakage point. I would think the same kind of thing would
>happen with py2exe. See the traceback at the end of this message for
>more info.
>
>Once I freeze the application and run it, my sys.path is set to
>['/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1/inquest-director',
>'/home/robbyd/dev/inquest/dist/inquest-director-0.7.0-1']
>(Meaning the frozen zip file executable, 'inquest-director', and its
>parent path.)
>
>Normally this works fine, due to zipimporter. However, with the static
>files nevow is attempting to locate with resource_filename(), it seems
>that the problem is that, resource_filename() wants to look into the zip
>file but it doesn't know how to do so. (I have not done too much
>research into how this function works, so I may be missing the point
>here.) What I'd like to know is why this is the case, and what may
>happen when resource_filename() gets used in more libraries whose users
>freeze their applications with cxfreeze/py2exe/macmillian/etc.  Is this
>a move by setuptools to take over the world (of python packaging
>utilities)? ;)
>
>Thanks in advance for any feedback that is offered. At this point I'm
>just unsure if this is more of a setuptools issue, or more of a
>cx_Freeze issue. I did take a look at setuptools and I was impressed
>with it, but my app has certain requirements that make cx_freeze a good
>choice.

You could say that it's a collision between the program's desire to use the 
more convenient resource_filename() function in place of the more-portable 
resource_stream() and resource_string() functions.  The latter two 
functions will work just fine with arbitrary importable zip files, but 
resource_filename() requires either an extracted zipfile or a zipped egg.

The issue here is that resource_filename() requires the static file to be 
extracted to the filesystem.  To be extracted to the filesystem, there 
needs to be a place to extract.  For eggs, the extract location is partly 
determined by the egg's project name and version number, which are assumed 
to be "relatively unique", allowing the resource to be extracted to a 
unique location.  The runtime complains in this case because it doesn't 
find any data that tells it what the name of the egg is that it's doing the 
extraction on behalf of, so it doesn't know where it should extract to.

There are two possible solutions:

1. Lobby the nevow author(s) to use resource_stream() or resource_string() 
instead
2. Package the .egg in the zipfile as a subdirectory

The first one is self-explanatory; it will make the package work even if 
it's not packaged as an egg.  I suggest submitting a patch, if you can.

The second one is straightforward in principle, but perhaps not in 
practice, depending on the constraints of your tools.  What you want to do 
is have a subdirectory *in your zipfile* that is named 
'nevow-0.5.0-py2.4.egg', or whatever the correct name is.  Inside that 
subdirectory, you include the original contents of the egg.  Then, 
pkg_resources will know what egg contains the package, and thus do the 
extraction correctly.  A zipfile that contains multiple eggs is sometimes 
called a "basket", as in "putting all your eggs in one basket".  :)

In general, pkg_resources tries to treat zipfiles and directories inside 
zipfiles as equivalents to "real" directories, which means that if you have 
a .zip on sys.path (or even an executable with a zipfile appended to it), 
then pkg_resources will happily search the zipfile's root for .egg 
subdirectories when it's trying to resolve dependencies.

As for your world domination question, I suspect that eventually most 
freezing tools will just dump all the necessary eggs into a basket, since 
this is simpler and less likely to break than scouting out individual 
module dependencies.  There's also an opportunity for installation-oriented 
tools to create "download only the bits I need" web installers, etc.