module import, but different

[Tim Peters]

[Andrew Dalke]
>   I would like to add a test suite to our software so at the top
> level of the distribution I can do a "make check" and run the tests.
> (We are planning to do something based to Python's regrtest code.)
> I'm running into a problem because I don't know how to deal with
> packages; and we do almost everything as packages.  It comes down
> to needing a way to import a package when you can't include the
> package's parent directory in the PYTHONPATH.
>
>   I'll describe the problem with some detail first.
>
>   Here's an example package layout:
>
> MCS/
>   Makefile
>   __init__.py
>   Graph.py
>   ...
>   doc/
>
> to which I plan to add
>   test/
>     br_regrtest.py     ("br_" for "Bioreason's version of")
>     test_graph.py
>     ...
>
> And in the Makefile add something like
>
> check:
>   cd test && $(PYTHON) br_regrtest.py
>
> Inside the test directory will be programs like "test_graph.py".
> They will have code like "import MCS.Graph".  This will fail
> because MCS isn't a module or package on the PYTHONPATH.  (Setting
> PYTHONPATH to ".." will fail for modules which have non-trivial
> __init__.py files; and we have a couple of that form.)
>
> [and much more]

I don't understand this stuff, so let me ask a dumb question:  why can't the
"check" target simply be

check:
    cd .. && $(PYTHON) MCS/test/br_regrtest.py

The directory from which Python is invoked is always the first thing on
sys.path, so MCS will be found.  IOW, make test into a normal MCS package
too.

> One solution would be to include "../.." but that wouldn't be
> appropriate because that may include files which get imported
> by accident.  For example, I keep all of my development code
> under src/, so src/MCS, src/daylight, etc.  But when I'm doing
> unit testing I want to import the globally installed "daylight"
> and not the ../../daylight imported by adding ../.. to the path.

Ah.  Trying to get something done with directory trees *is* a lot like
trying to get something done with regexps <0.7 wink>.  This is much harder
than the problem you claimed to have at the start:  the problem isn't that
you don't have the package's parent on PYTHONPATH, the problem is that your
intended package structure differs from your actual directory structure.
That means your only hope is to write a custom importer that knows about
your secret desires -- all Python can grok on its own is the directory
structure that actually exists.

> ...
> Besides, we distribute the packages with version numbers, as in
> MCS-1.0.  Including ../.. means we would have to convert the name
> somehow.

OK, your intended package names have *nothing* to do with your actual
directory names!

> The Python distribution solves the problem, I think, because in
> getpath.c it adds `Lib' to the PYTHONPATH if python was was run
> in the build tree.  Seems sneaky to me!

Yes, but not half as sneaky as what you're trying to do <wink>.

> Another possibility is to make a symbolic link in the test/
> directory, as in
>   ln -s ../ MCS
>
> This will work under unix, but isn't very portable, and I expect
> we'll have to support Windows someday.  (Or do the other OSes have
> something equivalent to this?)

Windows doesn't have links, symbolic or hard.  This was a very successful
strategy to increase sales of hard drives and OS upgrades <wink>.

> A third solution ...

Ack, enough already.  If your packages don't match your directories, the
builtin importer simply doesn't do what you want.  Look in the Library
manual's Built-in Functions section for __import__:  that's the builtin
function an "import" stmt invokes.  You can replace it with whatever you
like.  The std imp module packages many functions that will be helpful when
doing this.  As I said at the start, I don't understand this stuff (never
had reason to use it!), but Greg Stein is widely reputed to have developed a
much more flexible import mechanism; maybe DejaNews will turn up something
useful.

the-distutils-sig-may-also-be-your-friend-ly y'rs  - tim