[Distutils] distutils and the development process

Greg Ward gward@python.net
Thu, 24 Feb 2000 20:21:31 -0500

On 24 February 2000, est@hyperreal.org said:
> OK.  A recipient of my MIDI stream parser package (called `MIDI')
> installs via distutils.  They're still in the distribution directory
> and want to run a demo script to see that things are working.  The
> script does `import MIDI'..
> Traceback (innermost last):
>   File "./test.py", line 9, in ?
>     import MIDI
>   File "./MIDI/__init__.py", line 1, in ?
>     In = __import__('MIDI.In').In.MidiIn
>   File "./MIDI/In.py", line 27, in ?
>     _midi = __import__('MIDI._In')
> ImportError: No module named _In
> The script went to the source directory (MIDI) and, of course, didn't
> find the extension module.


I see the problem.  It's because Python always puts the directory of the
script being executed *first* in sys.path.  Allow me to demonstrate: I'm
doing this in the Distutils source directory, because it's handy for me.
You can do it in any Distutilized package laid out in the canonical
fashion that has been built, i.e. module "foo.bar" is in both
"foo/bar.py" *and* "build/lib/foo/bar.py", and you want the built
version -- "build/lib/foo/bar.py".  (This is irrelevant with pure Python
modules -- in fact, you usually want the unbuilt one, because you don't
want to go through the pointless exercise of "building" -- copying files
around -- just because you changed one line of Python.  But with
extensions, it's essential, since Distutils builds them in the build
tree rather than in the source tree!)

Anyways.  Here's my test setup:

  $ ls -lF       # partial display!
  drwxr-xr-x   3 greg     users        1024 Feb 16 22:17 build/
  drwxr-xr-x   3 greg     users        1024 Feb 17 19:25 distutils/
  -rwxr-xr-x   1 greg     users         748 Feb 17 19:01 setup.py*
  -rw-r--r--   1 greg     users          71 Feb 24 20:05 which.py

"distutils" is where my development copy is, "build/lib/distutils" is
the built version of that.  Again, the distinction only matters for
extensions, but just play along.

My test script is which.py:

  $ cat which.py
  import sys, pprint, distutils
  pprint.pprint (sys.path)
  print distutils

Let's try to force Python to find the "build/lib/distutils" version of
the code:

  $ export PYTHONPATH=build/lib

(I didn't bother to specify build/platlib here, but it's a good habit to
supply both just in case there are extensions present.)

Let's run it twice, two different ways:

  $ python which.py
  <module 'distutils' from 'distutils/__init__.pyc'>

  $ python ./which.py
  <module 'distutils' from './distutils/__init__.pyc'>

Spot the difference?  Yep, that's right: Python uses dirname(scriptname)
as the very first element of sys.path.  Apart from sys.path[0] and the
first element of the path to the distutils __init__.py, everything else
is the same.  Most importantly, my PYTHONPATH comes *after*
dirname(scriptname) in sys.path.

In other words, the problem suggests the solution: put your test script
in another directory!  I humbly recommend "test" for test scripts,
"example" for example scripts, "demo" for demo scripts, etc.  If I move
my which.py and run it again:

  $ mv which.py test/ 
  $ python test/which.py 
  <module 'distutils' from 'build/lib/distutils/__init__.pyc'>

...well, isn't that interesting -- It Just Works!  Python finds my
"build/lib" version of Distutils as expected.  Problem solved, hooray.

Looks like this little anectode ought to be written up for the mythical
"Distributing Python Modules" manual.

Greg Ward - all-purpose geek                            gward@python.net
Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer.