[Numpy-discussion] Should we drop support for "one file" compilation mode?

David Cournapeau cournape at gmail.com
Tue Oct 6 06:00:30 EDT 2015

On Mon, Oct 5, 2015 at 11:26 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Hi all,
> For a long time, NumPy has supported two different ways of being compiled:
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
> Supporting these two different build modes creates a drag on
> development progress; in particular Stefan recently ran into this in
> this experiments with porting parts of the NumPy internals to Cython:
>   https://github.com/numpy/numpy/pull/6408
> (I suspect the particular problem he's running into can be fixed b/c
> so far he only has one .pyx file, but I also suspect that it will be
> impossible to support "one file" mode once we have multiple .pyx
> files.)
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors. If all we can
> get are rumors, then eventually we have to risk breaking things just
> to force anyone who cares to actually show up and explain what they
> need so we can support it properly :-).

Assuming one of the rumour is related to some comments I made some time
(years ?) earlier, the context was the ability to hide exported symbols. As
you know, the issue is not to build extensions w/ multiple compilation
units, but sharing functionalities between them without sharing them
outside the extension. I am just reiterating that point so that we all
discuss under the right context :)

I also agree the current situation is not sustainable -- as we discussed
privately before, cythonizing numpy.core is made quite more complicated by
this. I have myself quite a few issues w/ cythonizing the other parts of
umath. I would also like to support the static link better than we do now
(do we know some static link users we can contact to validate our approach

Currently, what we have in numpy core is the following:

numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ +
statically link npymath
numpy.core.umath -> compilation units in numpy/core/src/umath + statically
link npymath/npysort + some shenanigans to use things in

I would suggest to have a more layered approach, to enable both 'normal'
build and static build, without polluting the public namespace too much.
This is an approach followed by most large libraries (e.g. MKL), and is
fairly flexible.

Concretely, we could start by putting more common functionalities (aka the
'core' library) into its own static library. The API would be considered
private to numpy (no stability guaranteed outside numpy), and every
exported symbol from that library would be decorated appropriately to avoid
potential clashes (e.g. '_npy_internal_').

FWIW, that has always been my intention to go toward this when I split up
multiarray/umath into multiple .c files and extracted out npymath.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20151006/039ec10e/attachment.html>

More information about the NumPy-Discussion mailing list