[Numpy-discussion] Combined versus separate build

Nathaniel Smith njs at pobox.com
Wed Jun 27 15:53:50 EDT 2012

On Wed, Jun 27, 2012 at 8:29 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Wed, Jun 27, 2012 at 8:07 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Wed, Jun 27, 2012 at 7:50 PM, David Cournapeau <cournape at gmail.com> wrote:
>>> On Wed, Jun 27, 2012 at 7:17 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>> Currently the numpy build system(s) support two ways of building
>>>> numpy: either by compiling a giant concatenated C file, or by the more
>>>> conventional route of first compiling each .c file to a .o file, and
>>>> then linking those together. I gather from comments in the source code
>>>> that the former is the traditional method, and the latter is the newer
>>>> "experimental" approach.
>>>> It's easy to break one of these builds without breaking the other (I
>>>> just did this with the NA branch, and David had to clean up after me),
>>>> and I don't see what value we really get from having both options --
>>>> it seems to just double the size of the test matrix without adding
>>>> value.
>>> There is unfortunately a big value in it: there is no standard way in
>>> C to share symbols within a library without polluting the whole
>>> process namespace, except on windows where the default is to export
>>> nothing.
>>> Most compilers support it (I actually know of none that does not
>>> support it in some way or the others), but that's platform-specific.
>> IIRC this isn't too tricky to arrange for with gcc
> No, which is why this is supported for gcc and windows :)
>>, but why is this an
>> issue in the first place for a Python extension module? Extension
>> modules are opened without RTLD_GLOBAL, which means that they *never*
>> export any symbols. At least, that's how it should work on Linux and
>> most Unix-alikes; I don't know much about OS X's linker, except that
>> it's unusual in other ways.
> The pragmatic answer is that if it were not an issue, python itself
> would not bother with it. Every single extension module in python
> itself is built from a single compilation unit. This is also why we
> have this awful system to export the numpy C API with array of
> function pointers instead of simply exporting things in a standard
> way.

The array-of-function-pointers is solving the opposite problem, of
exporting functions *without* having global symbols.

> See this: http://docs.python.org/release/2.5.3/ext/using-cobjects.html
> Looking quickly at the 2.7.3 sources, the more detailed answer is that
> python actually does not use RTLD_LOCAL (nor RTLD_GLOBAL), and what
> happens when neither of them is used is implementation-dependent. It
> seems to be RTLD_LOCAL on linux, and RTLD_GLOBAL on mac os x. There
> also may be consequences on the use of RTLD_LOCAL in embedded mode (I
> have ancient and bad memories with matlab related to this, but I
> forgot the details).

See, I knew OS X was quirky :-). That's what I get for trusting dlopen(3).

But seriously, what compilers do we support that don't have
-fvisibility=hidden? ...Is there even a list of compilers we support
available anywhere?


More information about the NumPy-Discussion mailing list