[Distutils] Can now build & install real distributions
Greg Ward
gward@cnri.reston.va.us
Tue, 21 Sep 1999 16:49:32 -0400
Well, the last bit of discussion seems to have died down again here, so
I guess it's time I kicked off another one.
This time, it's good news: the Distutils are now able to build and
install three non-trivial module distributions (Numeric Python,
mxDateTime, and PIL). Four, if you count Distutils itself (easier
because it's pure Python, but I still had to worry about packages and
stuff to get it right).
WHAT HAS CHANGED AND HOW IT WORKS
---------------------------------
[this is long -- if you're not interested, just skip down to "TRYING IT OUT"]
I should explain what's different, since the last time I made an
announcement here it was about the ability to build Numeric Python.
That was with the example setup.py that Perry Stoll contributed; at that
time, building only worked properly on Windows, because my
UnixCCcompiler class didn't deal well with source files in another
directory. That's been fixed. Also, Perry's setup.py didn't deal with
the pure Python modules in Numeric -- now fixed (tiny change to the
setup.py, big changes behind the scenes in the Distutils build_py
command).
Dealing with pure Python modules actually got a lot more complex inside
Distutils, but a lot simpler for module developers: in particular, the
setup.py for Distutils itself no longer has to explicitly list all the
modules to install. Rather, it just lists the packages to install
from/to. To explain, let me dissect the four existing setup.py files.
First, Distutils' own -- it's easiest because I laid out the source tree
based on someone else's wisdom and experience (thanks Fred!). Here's
the relevant bit from the Distutils setup script:
setup ([...meta-data omitted...]
packages = ['distutils', 'distutils.command'],
)
Yep, that's all it takes: one line to say, "Please build and install all
pure Python modules in the 'distutils' and 'distutils.command'
packages." So how does it find those modules? Easy, it does the most
sensible thing: package 'distutils' is assumed to be in the directory
'distutils' under the distribution directory (ie. where setup.py lives).
So, saying "packages = ['foo']" means "Please install foo/*.py, oh and
while you're at it make sure I included foo/__init__.py."
But what if you don't follow that layout? What if you install to a
package, but the source files for that package are right in the
distribution directory alongside setup.py? No problem: just use the
'package_dir' option. For example, mxDateTime works this way, so in the
example setup.py file for it (examples/mxdatetime_setup.py in the
Distutils distribution), you'll see:
setup (# ...
packages = ['DateTime', 'DateTime.Examples', 'DateTime.mxDateTime'],
package_dir = {'DateTime': ''},
# ...
)
Together, those two options mean: "please install ./*.py,
./Examples/*.py, and ./mxDateTime/*.py -- oh, and by the way, make sure
I included ./__init__.py, ./Examples/__init__.py, and
./mxDateTime/__init__.py."
(Oops! I just realized that this means Distutils will install setup.py,
which it probably shouldn't do. D'oh! Well, that's another argument
against organizing your source this way. ;-)
BTW, if you're really pathological, you can supply an entry in
'package_dir' for each package you install. But note that normally
Distutils will do the Right Thing: when the package_dir for "DateTime"
is '' (the current directory), it looks for sub-packages of "DateTime"
in subdirectories of the current directory. Thus, more complicated
directory layouts are certainly possible, but involve a bit of effort in
your setup.py. (My subtle way of discouraging such complexity. ;-)
Another way to lay things out that is essentially the same is used by
Numeric Python: the .py files live in "Lib". The difference is that
Numeric is not package-ized, so we have to deal with giving it a
directory and a .pth file. For example, my first pass at supporting
Numeric's pure Python modules looked like this
setup (# ...
packages = [''],
package_dir = {'': 'Lib'},
# ...
)
which of course dumped a whole bunch of stuff (Numeric's Lib/*.py) right
in my "site-packages" directory when I installed it. Oops. The fix is
to add an "install_path" option:
setup (# ...
packages = [''],
package_dir = {'': 'Lib'},
install_path = 'Numeric',
# ...
)
this then interposes "Numeric" between 'install_site_platlib' (err, I
think that's what the installation directory for module distributions
that include extensions is called... anyways, it's just the
"site-packages" dir under exec-prefix) and where files get installed,
*and* creates Numeric.pth in 'install_site_platlib'.
If for some reason you want your .pth file to have a different name from
the directory it refers to, no problem:
install_path = ('foo','bar/baz/qux')
creates foo.pth which contains "bar/baz/qux", as does
install_path = "foo,bar/baz/qux"
(The string/tuple duality is needed because 'install_path' can also be
specified on the command line, eg.
./setup.py install --install-path=foo,bar/baz/qux
has the same effect as the two previous examples.)
Finally, PIL uses the same layout as Distutils: it provides a package
"PIL" in the directory "./PIL" under the distribution directory. Thus:
setup (# ...
packages = ['PIL'],
# ...
)
Err, this assumes you're installing PIL in package form, which is what
my current setup.py for PIL does. Unfortunately, it installs PIL's
extension module at top-level. I think Fredrik will have to deal with
this problem -- it boils down to PIL not being 100% packagized yet. To
install PIL as a collection of top-level modules in their own directory
with a .pth file, the way Numeric is done:
setup (# ...
packages = [''],
package_dir = { '': 'PIL' },
install_path = 'PIL'
# ...
)
(Another oops: this will install PIL/__init__.py, which means having an
__init__.py on a directory in sys.path. Not sure what the consequences
of this are, but it just sounds like a bad idea. Again, it boils down
to PIL being not-quite-fully packagized.)
HOW EXTENSION MODULES ARE DEALT WITH
------------------------------------
Basically the same. Extension building and installing now knows about
packages, but it still uses the same moderately clunky "you provide the
low-level data structure that drives Distutils" mode. For example, the
Numeric setup.py contains:
ext_modules = [('_numpy',
{ 'sources' : ['Src/_numpymodule.c',
'Src/arrayobject.c',
'Src/ufuncobject.c'],
'def_file' : 'Src/numpy.def' }
)
# ...
I would dearly love to get rid of that "def_file" thing by requiring
that .def files be named after their respective modules,
ie. "Src/_numpy.def" here instead of "Src/numpy.def". Or just
auto-generate them if that'll work.
Another example: here's how mxDateTime's extension is defined:
ext_modules = [('DateTime.mxDateTime.mxDateTime',
{ 'sources': ['mxDateTime/mxDateTime.c'],
'include_dirs': ['mxDateTime'],
'macros': [('HAVE_STRFTIME', None),
('HAVE_STRPTIME', None),
('HAVE_TIMEGM', None)],
'def_file': 'mxDateTime/mxDateTime.def' }
)]
Note that currently we'll require users to edit setup.py to comment out
the HAVE_* macros that don't apply to their platform. Yechh! This
*will* have to change, I'm just not sure how yet. The all-singing
all-dancing solution is to reimplement (bits of) Autoconf in Python.
On the bright side, note that the name of the extension is fully
packagized: this tells Distutils where to put the .so (or .pyd) file.
PIL's setup.py has a similar problem:
ext_modules = \
[('_imaging',
{ 'sources': ['_imaging.c', 'decode.c', 'encode.c',
'map.c', 'display.c', 'outline.c', 'path.c'],
# This must include the directories with the JPEG,
# zlib, and Tcl/Tk header files (if installed)
'include_dirs': ['libImaging', '/usr/local/include],
# Keep this for Tcl/Tk support
'macros': [('WITH_TKINTER', None)],
# This must include the directories with the JPEG, zlib,
# and Tcl/Tk libraries (whatever's available)
'library_dirs': ['libImaging', '/usr/local/lib'],
# And this, of course, lists which of those external
# libraries to link against (plus libImaging, which *must*
# be included!)
'libraries': ['Imaging', 'jpeg', 'z', 'tcl8.0', 'tk8.0']
}
)]
At least here, I've commented what users would have to change. ;-)
Still, this is not viable for the long term, and again this could be
dealt with in an Autoconf-ish way -- but it would have to be a
cross-platform Autoconf!
TRYING IT OUT
-------------
Intrigued? Interested? Want to play with it? First, download the
current snapshot:
http://www.python.org/sigs/distutils-sig/distutils-19990921.tar.gz
Unpack it and dive in. Note there is no documentation, not so much as a
README -- that's what this message is for. ;-)
First, you want to install the Distutils proper:
python setup.py -v install
If *this* doesn't work, then we have real problems. ;-)
-v (verbose) is optional but *highly* recommended while developing/
debugging. (If you're doing this, you're developing/debugging -- thanks
for the help!)
If you wish to peruse the source, distutils/core.py is the place to
start. distutils/command/build_py.py implements the build_py command
for building pure Python modules; figuring out the rest of the commands
is left as an exercise for the reader.
If you want to try out one of the example setup.py scripts, they're in
the examples subdirectory. For instance, if you have unpacked Distutils
to /tmp/Distutils, and the LLNL distribution to /tmp/LLNLDistribution11,
then:
cd /tmp/LLNLDistribution11/Numerical
ln -s /tmp/distutils/examples/numpy_setup.py setup.py
(or local equivalent, for non-Unix geeks) and then build and install
Numeric:
python ./setup.py -v install
and ba-da-boom: you have one brand-spanking new Numeric Python
installation. (If you have one already, you should rename it out of
the way first.) Well, it takes a while with those enormous C source
files in Numeric, but you get the idea.
Similar shenanigans will get you an mxDateTime installation, or a
partially package-ized PIL installation. (The problem with PIL is that
its extension module isn't in a package, so it winds up in your
site-packages directory. Yuck. I blame Fredrik, I'm sure he'll blame
me. ;-)
WHAT DOESN'T WORK YET
---------------------
Arg, I'll save this for later. I need to get back to my day job. I'm
sure if you try it out, you'll find plenty o' problems. (Especially
Windows people -- I have made no attempt to keep MSVCCompiler up-to-date
with some minor changes in UnixCCompiler, so I'm sure things will
break. Send those patches in, please!)
Oh, as soon as this *does* work on both Unix and Windows, I'm going to
write up a spot of documentation and throw it out the door as
Distutils 0.1. So let's get those patches rolling in...
Enjoy --
Greg
--
Greg Ward - software developer gward@cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913