[Pythonmac-SIG] [Numpy-discussion] Packaging numpy with py2app

[Josh Marshall]

Thanks for the info on how the various recipes work, Bob. Very helpful.

On 19/07/2006, at 9:28 AM, Bob Ippolito wrote:
> The recipe mechanism doesn't allow for it because it doesn't  
> generally make sense. There are very few packages that can find  
> their resources in an alternative manner. I'm not totally opposed  
> to adding another feature to support that use case, but throwing it  
> directly in the resources dir seems like a bad idea to do  
> automatically from a recipe. The files should sit under some kind  
> of matplotlib hierarchy. It would be nicer to see matplotlib just  
> figure out how to make their app work from a zip... pygame does,  
> and it has fonts and icons.

I think matplotlib can access files inside the zip, since it searches  
inside the following directories upon startup and is frozen-aware.

MatplotlibEmbedSample.app/Contents/Resources/lib/python2.4/site- 
packages.zip/matplotlib/mpl-data
MatplotlibEmbedSample.app/Contents/mpl-data
MatplotlibEmbedSample.app/Contents/Resources/mpl-data

So I have been attempting to put the data files in the zip.

> loader_files puts data files *in the zip*

Indeed it does. When I use the following recipe: (noting that the  
recipe's filename can't be "matplotlib.py", I've been using  
"matplotlib_recipe.py"):

def check(cmd, mf):
     m = mf.findNode('matplotlib')
     if m is None or m.filename is None:
         return None

     import matplotlib, glob, os
     mpl_datapath = matplotlib.get_data_path()
     mpl_datafilelist = glob.glob(mpl_datapath + r'/*')

     # only include individual files from the mpl-data directory
     # this should exclude directories, which should only be  
Matplotlib.nib
     mpl_datafilelist = [mpl_file for mpl_file in mpl_datafilelist
                         if os.path.isfile(mpl_file)]

     mf.import_hook('pytz.zoneinfo', m, ['UTC'])
     mf.import_hook('matplotlib.numerix', m, ['random_array'])
     return dict(
         loader_files = [
             ('matplotlib', mpl_datafilelist),
         ],
     )

The data files then get put in the matplotlib directory in the zip.  
However, matplotlib searches in the mpl-data directory *inside* the  
matplotlib directory, as shown in the search paths above.

I have not been able to find a way to include a directory within the  
matplotlib directory. If I use
         loader_files = [
             ('matplotlib/mpl-data', mpl_datafilelist),
         ],
py2app barfs when trying to copy the files since the directory  
doesn't exist. Any ideas Bob?

Chris, can you please copy my email when replying to the thread? I'm  
getting the digest, so I don't get your messages for quite some time  
otherwise.

Christopher Barker wrote:
> I wonder what the plan should be for recipes for MPL and the like.  
> As Bob mentioned, perhaps it's best for the developer to find a way  
> to manually include what they want, as there is a LOT of stuff in  
> MPL (and certainly SciPy!) and much of will not be used by a given  
> application.
If we can get numpy and scipy to work without a recipe that just  
includes the packages we can definitely do this. The problem here is  
not with py2app, but rather with numpy/scipy themselves and their  
crazy custom importer which doesn't work with a zipped site-packages.

I am close to getting matplotlib working with a decent non-naive  
recipe. This means it'll only include what is being used by the  
application.

> It seems the only way to make a recipe that is useful would be for  
> it to include EVERYTHING. That may not be a bad default, but there  
> should certainly be a way to turn it off if need be.
No, a useful and "proper" recipe just adds the files that are not  
referenced directly through imports, ie., data files. This is what I  
am doing with matplotlib.

> Also, I got a numpy app to work yesterday by using:

> packages=['numpy']

The recipe I sent out in the zip yesterday did exactly that. Bob has  
added it to py2app trunk. There is a similar recipe for scipy and  
matplotlib as well. These will do until we get proper ones working.

The packages option does include *everything*, by doing a full copy  
of the package. This means that source, documentation, example files,  
etc gets included along with the bytecode files.

> That then includes the entire numpy package. That works, but it  
> puts in a lot of stuff I don't need (dft, linear_algebra, etc).  
> Maybe the recipe should include only the core stuff, and let the  
> user add the extra packages, if need be.
AFAICT, this can't really be done. From looking at the numpy  
structure, there are subpackages  
'testing','core','lib','linalg','dft','random','f2py', which are all  
loaded in the numpy __init__.py via the following:

     pkgload = PackageLoader()
     pkgload('testing','core','lib','linalg','dft','random','f2py',
             verbose=NUMPY_IMPORT_VERBOSE,postpone=False)

The doc string for PackageLoader.__call__() says:

>        This function is intended to shorten the need to import many of
>        subpackages, say of scipy, constantly with statements such as
>
>        import scipy.linalg, scipy.fftpack, scipy.etc...
>
>        Instead, you can say:
>
>          import scipy
>          scipy.pkgload('linalg','fftpack',...)
>
>        or
>
>          scipy.pkgload()

so it seems like the entire purpose of PackageLoader is to make life  
difficult for me, just to save a few lines of typing. :) Seriously,  
can a numpy developer tell me why PackageLoader is necessary?

> Or should numpy be fixed so that it doesn't do weird imports?
The weird imports are okay, but they need to be able to work from a  
site-packages.zip. That's the next job. Hopefully I'll get time to  
look at it before numpy goes too far into beta. Scipy is a long way off.

Cheers all,
Josh