distutils and the development process

Greetings, disutils people! I'd like to be able to use distutil's build process while I'm developing and debugging code. This has proven difficult so far because of the use of the build directory. I've ended up just using Makefile-driven in-place builds during the development cycle. Does anyone have any tips or tricks in this area? I'm also curious how people like to run test scripts. One special case is post-install hello-world-type example programs for the installer to play with. To keep mine from being confused by the package source directory, I end up adding the following to them: # Ugh..make sure we look only in system directories for the package. # This is only important when executing a script in the package # distribution directory. import sys sys.path = sys.path[1:] There's got to be a better way. In fact, I often end up commenting out that assignment during development so I can test my in-place Makefile-driven builds! Anyhow, I feel like I must be doing things wrong and I'd appreciate opinions on these topics. :) Eric

On 23 February 2000, est@hyperreal.org said:
As they say in the trade, "That's not a bug -- that's a feature!" But seriously: what *about* the build directory is causing you problems? My current thinking is to put *everything* somewhere in the build directory, leaving the source tree pristine. I.e., .o or .obj files would go in build/tmp.<plat>, along with the surprise compiler turds that Visual C++ leaves behind. (When you think about it, .o and .obj files are really just compiler turds too -- but we all expect them from years of experience.) The idea of the build directory is that you just do this: export PYTHONPATH=build/lib:build/platlib while you're in the develop-test-debug cycle for extension modules. Just make sure you stay in the root of the source tree. You can work this way too for pure Python modules if you *want*, but there's no good reason to. If your source tree is laid out in a sensible way (like Distutils, of course, or PyXML), then you don't have to do anything special -- eg. when I'm hacking on Distutils, I just let Python find distutils.core as distutils/core.py under the Distutils source root. No problem.
That sounds like a "demo script" to me, but that's just quibbling over semantics. I simply don't understand what you mean by "confused by the package source directory". Please explain.
Ugh, barf, blechh indeed. Why *ever* do you need to do this? Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ The NSA. We care: we listen to our customers.

Greg Ward discourseth:
Sounds good. :)
OK. A recipient of my MIDI stream parser package (called `MIDI') installs via distutils. They're still in the distribution directory and want to run a demo script to see that things are working. The script does `import MIDI'.. Traceback (innermost last): File "./test.py", line 9, in ? import MIDI File "./MIDI/__init__.py", line 1, in ? In = __import__('MIDI.In').In.MidiIn File "./MIDI/In.py", line 27, in ? _midi = __import__('MIDI._In') ImportError: No module named _In The script went to the source directory (MIDI) and, of course, didn't find the extension module.
Well, it makes the script work. As I said, I'm sure there must be a better way. :) Eric

On 24 February 2000, est@hyperreal.org said:
Ah-ha! I see the problem. It's because Python always puts the directory of the script being executed *first* in sys.path. Allow me to demonstrate: I'm doing this in the Distutils source directory, because it's handy for me. You can do it in any Distutilized package laid out in the canonical fashion that has been built, i.e. module "foo.bar" is in both "foo/bar.py" *and* "build/lib/foo/bar.py", and you want the built version -- "build/lib/foo/bar.py". (This is irrelevant with pure Python modules -- in fact, you usually want the unbuilt one, because you don't want to go through the pointless exercise of "building" -- copying files around -- just because you changed one line of Python. But with extensions, it's essential, since Distutils builds them in the build tree rather than in the source tree!) Anyways. Here's my test setup: $ ls -lF # partial display! drwxr-xr-x 3 greg users 1024 Feb 16 22:17 build/ drwxr-xr-x 3 greg users 1024 Feb 17 19:25 distutils/ -rwxr-xr-x 1 greg users 748 Feb 17 19:01 setup.py* -rw-r--r-- 1 greg users 71 Feb 24 20:05 which.py "distutils" is where my development copy is, "build/lib/distutils" is the built version of that. Again, the distinction only matters for extensions, but just play along. My test script is which.py: $ cat which.py import sys, pprint, distutils pprint.pprint (sys.path) print distutils Let's try to force Python to find the "build/lib/distutils" version of the code: $ export PYTHONPATH=build/lib (I didn't bother to specify build/platlib here, but it's a good habit to supply both just in case there are extensions present.) Let's run it twice, two different ways: $ python which.py ['', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from 'distutils/__init__.pyc'> $ python ./which.py ['.', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from './distutils/__init__.pyc'> Spot the difference? Yep, that's right: Python uses dirname(scriptname) as the very first element of sys.path. Apart from sys.path[0] and the first element of the path to the distutils __init__.py, everything else is the same. Most importantly, my PYTHONPATH comes *after* dirname(scriptname) in sys.path. In other words, the problem suggests the solution: put your test script in another directory! I humbly recommend "test" for test scripts, "example" for example scripts, "demo" for demo scripts, etc. If I move my which.py and run it again: $ mv which.py test/ $ python test/which.py ['test', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from 'build/lib/distutils/__init__.pyc'> ...well, isn't that interesting -- It Just Works! Python finds my "build/lib" version of Distutils as expected. Problem solved, hooray. Looks like this little anectode ought to be written up for the mythical "Distributing Python Modules" manual. Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer.

est@hyperreal.org discourseth:
Yes, this solves my problems. I feel stupid. :)
I take that back. I have two major continued problems even when I do, "export PYTHONPATH=build/lib:build/platlib". 1) When developing in the top-level directory of a distribution, the current directory still comes first in sys.path, so python still thinks the package source directory (with its __init__.py) is where the package lives. 2) Even when executing an example script in a sub-directory, python gets the idea that the package is in build/lib and doesn't find it's extension submodules. If I copy those over to the build/lib subdirectory they are found. I see the PyXML distribution does not pretty things to control where the extension modules are built, probably for these reasons. Does everyone who has extension modules in their packages have to do that? A final note, even if these problems didn't exist, I'd really prefer not to have to do "python setup.py build" before I test any change to a .py file. I don't want the `compile' back in the edit/compile/debug cycle for .py files. If I think of good strategies for dealing with these problems, I'll post them. In the meantime, it's back to Makefiles for me. :) Eric

On 25 February 2000, est@hyperreal.org said:
Yeah, I see the problem -- putting test scripts in a test directory is not a full solution; what about interactive sessions? And while it's good practice to put test scripts in a test directory, I don't want to *force* people to do that.
Oops. This is handled properly on installation, since build/lib and build/platlib are merged at that point. I've been wondering for a while why I decided always to build pure Python modules to build/lib, and extensions to build/platlib. You have discovered a very good reason *not* to do so: it breaks the idea of the build tree as a pseudo-installation for testing purposes.
If I think of good strategies for dealing with these problems, I'll post them. In the meantime, it's back to Makefiles for me. :)
I see a couple of solutions: first, put all modules in either build/lib or build/platlib, i.e. do the merge at build-time rather than install-time. Second, have some sort of "build in-place" option that lets developers, well, build in-place. That is, .py files in the source tree wouldn't be touched (they're already in-place!), and extension modules would be put into the source tree. (I still favour putting compiler turds in build/temp.plat -- anyone foresee problems with that?) I'm still struggling with the Right Way to install modules, so I won't be able to work on fixing the build stuff right away. Keep bugging me though! Greg -- Greg Ward - Linux bigot gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package.

On 23 February 2000, est@hyperreal.org said:
As they say in the trade, "That's not a bug -- that's a feature!" But seriously: what *about* the build directory is causing you problems? My current thinking is to put *everything* somewhere in the build directory, leaving the source tree pristine. I.e., .o or .obj files would go in build/tmp.<plat>, along with the surprise compiler turds that Visual C++ leaves behind. (When you think about it, .o and .obj files are really just compiler turds too -- but we all expect them from years of experience.) The idea of the build directory is that you just do this: export PYTHONPATH=build/lib:build/platlib while you're in the develop-test-debug cycle for extension modules. Just make sure you stay in the root of the source tree. You can work this way too for pure Python modules if you *want*, but there's no good reason to. If your source tree is laid out in a sensible way (like Distutils, of course, or PyXML), then you don't have to do anything special -- eg. when I'm hacking on Distutils, I just let Python find distutils.core as distutils/core.py under the Distutils source root. No problem.
That sounds like a "demo script" to me, but that's just quibbling over semantics. I simply don't understand what you mean by "confused by the package source directory". Please explain.
Ugh, barf, blechh indeed. Why *ever* do you need to do this? Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ The NSA. We care: we listen to our customers.

Greg Ward discourseth:
Sounds good. :)
OK. A recipient of my MIDI stream parser package (called `MIDI') installs via distutils. They're still in the distribution directory and want to run a demo script to see that things are working. The script does `import MIDI'.. Traceback (innermost last): File "./test.py", line 9, in ? import MIDI File "./MIDI/__init__.py", line 1, in ? In = __import__('MIDI.In').In.MidiIn File "./MIDI/In.py", line 27, in ? _midi = __import__('MIDI._In') ImportError: No module named _In The script went to the source directory (MIDI) and, of course, didn't find the extension module.
Well, it makes the script work. As I said, I'm sure there must be a better way. :) Eric

On 24 February 2000, est@hyperreal.org said:
Ah-ha! I see the problem. It's because Python always puts the directory of the script being executed *first* in sys.path. Allow me to demonstrate: I'm doing this in the Distutils source directory, because it's handy for me. You can do it in any Distutilized package laid out in the canonical fashion that has been built, i.e. module "foo.bar" is in both "foo/bar.py" *and* "build/lib/foo/bar.py", and you want the built version -- "build/lib/foo/bar.py". (This is irrelevant with pure Python modules -- in fact, you usually want the unbuilt one, because you don't want to go through the pointless exercise of "building" -- copying files around -- just because you changed one line of Python. But with extensions, it's essential, since Distutils builds them in the build tree rather than in the source tree!) Anyways. Here's my test setup: $ ls -lF # partial display! drwxr-xr-x 3 greg users 1024 Feb 16 22:17 build/ drwxr-xr-x 3 greg users 1024 Feb 17 19:25 distutils/ -rwxr-xr-x 1 greg users 748 Feb 17 19:01 setup.py* -rw-r--r-- 1 greg users 71 Feb 24 20:05 which.py "distutils" is where my development copy is, "build/lib/distutils" is the built version of that. Again, the distinction only matters for extensions, but just play along. My test script is which.py: $ cat which.py import sys, pprint, distutils pprint.pprint (sys.path) print distutils Let's try to force Python to find the "build/lib/distutils" version of the code: $ export PYTHONPATH=build/lib (I didn't bother to specify build/platlib here, but it's a good habit to supply both just in case there are extensions present.) Let's run it twice, two different ways: $ python which.py ['', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from 'distutils/__init__.pyc'> $ python ./which.py ['.', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from './distutils/__init__.pyc'> Spot the difference? Yep, that's right: Python uses dirname(scriptname) as the very first element of sys.path. Apart from sys.path[0] and the first element of the path to the distutils __init__.py, everything else is the same. Most importantly, my PYTHONPATH comes *after* dirname(scriptname) in sys.path. In other words, the problem suggests the solution: put your test script in another directory! I humbly recommend "test" for test scripts, "example" for example scripts, "demo" for demo scripts, etc. If I move my which.py and run it again: $ mv which.py test/ $ python test/which.py ['test', 'build/lib', '/usr/local/python/lib/python1.5/', '/usr/local/python/lib/python1.5/plat-linux2', '/usr/local/python/lib/python1.5/lib-tk', '/usr/local/bin/../python.i86-linux/lib/python1.5/lib-dynload'] <module 'distutils' from 'build/lib/distutils/__init__.pyc'> ...well, isn't that interesting -- It Just Works! Python finds my "build/lib" version of Distutils as expected. Problem solved, hooray. Looks like this little anectode ought to be written up for the mythical "Distributing Python Modules" manual. Greg -- Greg Ward - all-purpose geek gward@python.net http://starship.python.net/~gward/ Disclaimer: All rights reserved. Void where prohibited. Limit 1 per customer.

est@hyperreal.org discourseth:
Yes, this solves my problems. I feel stupid. :)
I take that back. I have two major continued problems even when I do, "export PYTHONPATH=build/lib:build/platlib". 1) When developing in the top-level directory of a distribution, the current directory still comes first in sys.path, so python still thinks the package source directory (with its __init__.py) is where the package lives. 2) Even when executing an example script in a sub-directory, python gets the idea that the package is in build/lib and doesn't find it's extension submodules. If I copy those over to the build/lib subdirectory they are found. I see the PyXML distribution does not pretty things to control where the extension modules are built, probably for these reasons. Does everyone who has extension modules in their packages have to do that? A final note, even if these problems didn't exist, I'd really prefer not to have to do "python setup.py build" before I test any change to a .py file. I don't want the `compile' back in the edit/compile/debug cycle for .py files. If I think of good strategies for dealing with these problems, I'll post them. In the meantime, it's back to Makefiles for me. :) Eric

On 25 February 2000, est@hyperreal.org said:
Yeah, I see the problem -- putting test scripts in a test directory is not a full solution; what about interactive sessions? And while it's good practice to put test scripts in a test directory, I don't want to *force* people to do that.
Oops. This is handled properly on installation, since build/lib and build/platlib are merged at that point. I've been wondering for a while why I decided always to build pure Python modules to build/lib, and extensions to build/platlib. You have discovered a very good reason *not* to do so: it breaks the idea of the build tree as a pseudo-installation for testing purposes.
If I think of good strategies for dealing with these problems, I'll post them. In the meantime, it's back to Makefiles for me. :)
I see a couple of solutions: first, put all modules in either build/lib or build/platlib, i.e. do the merge at build-time rather than install-time. Second, have some sort of "build in-place" option that lets developers, well, build in-place. That is, .py files in the source tree wouldn't be touched (they're already in-place!), and extension modules would be put into the source tree. (I still favour putting compiler turds in build/temp.plat -- anyone foresee problems with that?) I'm still struggling with the Right Way to install modules, so I won't be able to work on fixing the build stuff right away. Keep bugging me though! Greg -- Greg Ward - Linux bigot gward@python.net http://starship.python.net/~gward/ A man wrapped up in himself makes a very small package.
participants (2)
-
est@hyperreal.org
-
Greg Ward