[Distutils] Improving distutils vs redesigning it (was people want CPAN)

David Cournapeau david at ar.media.kyoto-u.ac.jp
Thu Nov 12 11:25:29 CET 2009

Tarek Ziadé wrote:
> The build and instalation scheme should be part of the API. If you
> look in sysconfig,
> you get most of the install schemes via the API. If anything is
> missing, tell us.

(I am reading the code there:

I have not tried running the code, but here is what I see missing:
    - anything related to build (where are libraries put in the build
directory, where are the extensions put - necessary if you are
interested in inter-operating with make)
    - I don't see the equivalent of
distutils.command.install.install_libbase and similar prefixes
    - How can we retrieve the full customized install path ? I have not
found how I could use the code in sysconfig to solve the use case I have
written about before (related to retrieving install prefix).

> Now for the build I am not sure what you are trying to perform here.

This is always the same use-case: assuming you push the build to a 3rd
party tool like make, how can you make it work within distutils: build
directories, compiler options, etc... If you are familiar with
autootols, the idea could be similar in the sense that you would have a
Makefile.in, and a distutils command build_make would generate a
makefile, filling up the variables in makefile.in. You could also
imagine to put those variables in a separate file, which could be
queried from an API (using a makefile.in-like process for scons is not
very practical for example).

So the list of variables needed:
    - everything in sysconfig (but available for any compiler, including
MS ones)
    - any distutils command option. What is needed here is a way to
query the options given to command globally, that is the possibility to
access at any point in your setup.py every option of every command,
without running them as first as we have to do currently (like running
install_cmd within build_clib).

To be even more down to earth, here is how a scons call looks like on
linux for numscons:

-f numpy/core/SConstruct -I. scons_tool_path="" src_dir="numpy/core"
pkg_path="numpy/core" pkg_name="numpy.core" log_level=50
cc_opt=gcc cc_opt_path="/usr/bin" debug=0 f77_opt=gfortran
f77_opt_path="/usr/bin" cxx_opt=g++ cxx_opt_path="/usr/bin"
include_bootstrap=../../../../numpy/core/include bypass=0 import_env=0
silent=0 bootstrapping=1

To understand the weird relative paths, you need to be aware than scons
(and waf) consider all the paths relatively to a build directory. Scons
copy the sources files in the build directory (this is an
oversimplification, but should be enough for this discussion). For
example, here, we build the package numpy.core, so it looks like
(relatively to the top setup.py file):

   /build/lib.linux-i686-2.6/ # where distutils would put all the
compiled code if build_ext were used
   /build/scons/   # any code generated by scons is put here
               numpy/core # the build directory for the current package

We pass the following informations:
 * distutils_libdir: where distutils put/expects python extensions
(which is replaced by current directory + the src_dir in case of
in-place build)
 * distutils_install_prefix: install prefix  + subpackage path. Where
the extensions would be installed
 * distutils_clibdir: where disstutils put/expects pure C libraries (the
ones built through build_clib)
 * numscons also encode compilation options in .ini-like files
(regularly auto-generated from the info at sysconfig). The idea is that
if someone wants to modify the flags, there is no need to touch any
python code.

Right now, I have to reimplement the distutils code to find those, as
the logic is not available through API. Generally, the build directories
are only known through the corresponding build_* commands, but they
become available only after running the commands, which we want to avoid.

> And I made a proposal for this last one too in this thread.  But, like
> the Extensions proposal I have made,
> you seem not to hear them.

Because I genuinely don't understand why it is required not to replace
distutils but it is ok to change its behavior in a fundamental way. For
some reasons (which may well just be my own fault), I fail to see any
reason  to start from existing code, publicly and widely used, if it is
ok to break it and change its behavior, especially if it will be done
several times. I was not at Pycon, so there may have been some other
requirements I have missed.

> Sorry, but that's pretty vague. Are you mentioning the changes in
> build_ext, or something else ?

For 2.6.3, yes. For python 2.6, the mingw support was broken because of
VS manifests; our fortran tools were broken because the msvc compiler
was change (to msvc9compiler). There were also some other changes
related to install directories and the --local option in 2.6 which
caused issues. Several setuptools have also broken various parts of our
own extensions in the past. If you really need to know the details for
the last ones, I can look back in our svn log.



More information about the Distutils-SIG mailing list