[Numpy-discussion] Splitting multiarray and umath into smaller files: ready for review

[David Cournapeau]

Hi,

    Following the "linker script, smaller source files and symbols
visibility" discussion, I have implemented something which seems to work
reasonably well. The code is available on github for review:

http://github.com/cournape/numpy/tree/split_comp

This branch implements the following:
    - Both umath and multiarray sources are split into smaller files, in
separate directories.
    - By default, they are still all included into one big .c file, in
which case nothing much is changed.
    - The more interesting stuff happens when
ENABLE_SEPARATE_COMPILATION is set to True in setup.py/setupscons.py
file, in which case the files are actually compiled separately and link
together at the end.

This separate compilation mode uses some toolchain specific feature to
avoid exporting symbols; I have tested it on mac os x, linux and windows
(windows needs nothing as by default, no symbols are exported in
dynamically loaded libraries). At some point, it should be enabled on
platforms where it is known to work, I guess. One big advantage of
separate compilation is the compilation/debug cycle which is much
shorter (compilation time after modifying one file of multiarray goes
from half a minute to a couple of seconds on my configuration). I split
the file in multiarray following the C reference guide, as well as some
logical units (one file for the mapping protocol, one file for
iterators, one file for 'array constructors', etc....). There are still
some things which do not belong where they should, but this can be
improved later.

There is a second branch which implements symbol export maps on a few
platforms (darwin and linux for now), to specify exactly the symbols we
want to export:

http://github.com/cournape/numpy/tree/version_script

This branch is much smaller, as it just implements a couple of functions
for distutils. With this work, multiarray and umath now only export one
function, and this can be easily extended to other extensions as well,

cheers,

David