module import, but different

[Andrew Dalke]

Hello,

  I would like to add a test suite to our software so at the top
level of the distribution I can do a "make check" and run the tests.
(We are planning to do something based to Python's regrtest code.)
I'm running into a problem because I don't know how to deal with
packages; and we do almost everything as packages.  It comes down
to needing a way to import a package when you can't include the
package's parent directory in the PYTHONPATH.

  I'll describe the problem with some detail first.

  Here's an example package layout:

MCS/
  Makefile
  __init__.py
  Graph.py
  ...
  doc/

to which I plan to add
  test/
    br_regrtest.py     ("br_" for "Bioreason's version of")
    test_graph.py
    ...

And in the Makefile add something like

check:
  cd test && $(PYTHON) br_regrtest.py

Inside the test directory will be programs like "test_graph.py".
They will have code like "import MCS.Graph".  This will fail
because MCS isn't a module or package on the PYTHONPATH.  (Setting
PYTHONPATH to ".." will fail for modules which have non-trivial
__init__.py files; and we have a couple of that form.)

One solution would be to include "../.." but that wouldn't be
appropriate because that may include files which get imported
by accident.  For example, I keep all of my development code
under src/, so src/MCS, src/daylight, etc.  But when I'm doing
unit testing I want to import the globally installed "daylight"
and not the ../../daylight imported by adding ../.. to the path.

Of course, I could arbitrarily decide to enforce that ../..
not contain importable code, but that require me to change my
development style, and for everyone using one of these
packages to understand the problem and obey the solution.  Ain't
gonna happen.

Besides, we distribute the packages with version numbers, as in
MCS-1.0.  Including ../.. means we would have to convert the name
somehow.


The Python distribution solves the problem, I think, because in
getpath.c it adds `Lib' to the PYTHONPATH if python was was run
in the build tree.  Seems sneaky to me!

Another possibility is to make a symbolic link in the test/
directory, as in
  ln -s ../ MCS

This will work under unix, but isn't very portable, and I expect
we'll have to support Windows someday.  (Or do the other OSes have
something equivalent to this?)

A third solution is to move the code tree to its own subdirectory
-- what automake calls a "deep" package -- as in:

MCS/
  Makefile
  MCS/
    __init__.py
    Graph.py
    ...
  doc/
  test/

In that way I can do
  cd test && PYTHONPATH=.. $(PYTHON) br_regrtest.py

This is probably the best solution, except that
  1) we already have about 20 modules with code in the top level,
  2) the patches I made for automake assume the code will be of
this form
  3) I will be contributing the patches back to automake, and I
can guarantee that others won't be writing packages of this form.
  4) other reasons I can't currently come up with, other than
"it doesn't feel right/is too complicated" (eg, because there will
always have to be two Makefiles, one in the top directory and one
for the source subdir)

The last solution, and the one I want, is to have some way to
fake the import system so that "../" is treated as a module, say,
by calling
  special_import("MCS", "../")
which fakes out sys.modules.

Here is a unix specific implementation:
 = = = =
import os, sys
# One way to remove the links with Python exits...
class CleanUp:
    def __init__(self, name):
       self.name = name
       self.unlink = os.unlink  # os gets deleted too early!
    def __del__(self):
       self.unlink(self.name)

clean_up = []
def special_import(name, location):
  os.symlink(location, name)
  sys.modules[name] = __import__(name)
  clean_up.append(CleanUp(name))
 = = = =

Suggestions?

						Andrew Dalke
						dalke at bioreason.com