Problem with .pth files under linux

[Fernando Pérez]

> I'm sure this is simple and stupid, but I still don't know what's going
> on. I have some modules without a package structure (no __init__.py) but
> which have a .pth file. Specifically, Numeric. It comes in its own
> directory: site-packages/Numeric, and in site-packages I have a file
> Numeric.pth whose contents is simply 'Numeric' (no quotes). Yet import
> Numeric doesn't work! I solved it by hand-coding Numeric into sys.path,
> but it annoys me and I have the same problem with other modules I'll need
> to use soon.

I'm replying to myself b/c some people made suggestions but none quite
worked. However they got me on track and eventually I found a solution, this
might help others.

Basically the path search mechanism in python is a bit primitive for
non-packages (modules without an __init__.py). Packages are fine.

For non-packages, from: http://www.python.org/doc/current/lib/module-site.html
here's the relevant part:

<quote>
It starts by constructing up to four directories from a head and a tail part.
For the head part, it uses sys.prefix and sys.exec_prefix; empty heads are
skipped. For the tail part, it uses the empty string (on Macintosh or Windows)
or it uses first lib/python2.1/site-packages and then lib/site-python (on
Unix). For each of the distinct head-tail combinations, it sees if it refers
to an existing directory, and if so, adds to sys.path, and also inspects the
path for configuration files.

A path configuration file is a file whose name has the form package.pth; its
contents are additional items (one per line) to be added to sys.path.
Non-existing items are never added to sys.path, but no check is made that the
item refers to a directory (rather than a file). No item is added to sys.path
more than once. Blank lines and lines beginning with # are skipped. Lines
starting with import are executed.
</quote>

So the point is, your .pth files have to be in your normal site-packages
hierarchy. But my problem is that my python is installed in /usr, but I also
have a /usr/local/lib/python2.1/site-packages directory where I put things I
get off the web. I like it that way because when I upgrade linux, I can just
let the installer wipe /usr and all the things I have in /usr/local remain.

Since the searcher will only expand on sys.prefix and sys.exec_prefix (and
it's a *bad* idea changing these to anything that's not your real python
directory), things that are not under that will not be easily found (as .pth
files).

I found  a workaround: since I don't have a /usr/lib/site-python directory
(I'm guessing this is for old compatibility), I made that be a symlink to
/usr/local/lib/site-packages. Now  all the .pth's in
/usr/local/lib/site-packages are found! However, this is a kludgy solution.

I understand that modules should be moving to a proper package structure, but
until everyone in the world does it, it wouldn't hurt if Python loaded .pth
files found *anywhere* in sys.path, not just under sys.prefix and
sys.exec_prefix + those special 'tails' which are hard-coded.

Anyway. I posted this long story so at least there's a documented solution to
this problem *somewhere*, since nobody seemed quite sure what the right
approach was and it took me a while to piece things together.

Cheers,

f.