[Numpy-discussion] Combined versus separate build

[Dag Sverre Seljebotn]

On 06/27/2012 09:53 PM, Nathaniel Smith wrote:
> On Wed, Jun 27, 2012 at 8:29 PM, David Cournapeau<cournape at gmail.com>  wrote:
>> On Wed, Jun 27, 2012 at 8:07 PM, Nathaniel Smith<njs at pobox.com>  wrote:
>>> On Wed, Jun 27, 2012 at 7:50 PM, David Cournapeau<cournape at gmail.com>  wrote:
>>>> On Wed, Jun 27, 2012 at 7:17 PM, Nathaniel Smith<njs at pobox.com>  wrote:
>>>>> Currently the numpy build system(s) support two ways of building
>>>>> numpy: either by compiling a giant concatenated C file, or by the more
>>>>> conventional route of first compiling each .c file to a .o file, and
>>>>> then linking those together. I gather from comments in the source code
>>>>> that the former is the traditional method, and the latter is the newer
>>>>> "experimental" approach.
>>>>>
>>>>> It's easy to break one of these builds without breaking the other (I
>>>>> just did this with the NA branch, and David had to clean up after me),
>>>>> and I don't see what value we really get from having both options --
>>>>> it seems to just double the size of the test matrix without adding
>>>>> value.
>>>>
>>>> There is unfortunately a big value in it: there is no standard way in
>>>> C to share symbols within a library without polluting the whole
>>>> process namespace, except on windows where the default is to export
>>>> nothing.
>>>>
>>>> Most compilers support it (I actually know of none that does not
>>>> support it in some way or the others), but that's platform-specific.
>>>
>>> IIRC this isn't too tricky to arrange for with gcc
>>
>> No, which is why this is supported for gcc and windows :)
>>
>>> , but why is this an
>>> issue in the first place for a Python extension module? Extension
>>> modules are opened without RTLD_GLOBAL, which means that they *never*
>>> export any symbols. At least, that's how it should work on Linux and
>>> most Unix-alikes; I don't know much about OS X's linker, except that
>>> it's unusual in other ways.
>>
>> The pragmatic answer is that if it were not an issue, python itself
>> would not bother with it. Every single extension module in python
>> itself is built from a single compilation unit. This is also why we
>> have this awful system to export the numpy C API with array of
>> function pointers instead of simply exporting things in a standard
>> way.
>
> The array-of-function-pointers is solving the opposite problem, of
> exporting functions *without* having global symbols.
>
>> See this: http://docs.python.org/release/2.5.3/ext/using-cobjects.html
>>
>> Looking quickly at the 2.7.3 sources, the more detailed answer is that
>> python actually does not use RTLD_LOCAL (nor RTLD_GLOBAL), and what
>> happens when neither of them is used is implementation-dependent. It
>> seems to be RTLD_LOCAL on linux, and RTLD_GLOBAL on mac os x. There
>> also may be consequences on the use of RTLD_LOCAL in embedded mode (I
>> have ancient and bad memories with matlab related to this, but I
>> forgot the details).
>
> See, I knew OS X was quirky :-). That's what I get for trusting dlopen(3).
>
> But seriously, what compilers do we support that don't have
> -fvisibility=hidden? ...Is there even a list of compilers we support
> available anywhere?

You could at the very least switch the default for a couple of releases, 
introducing a new flag with a "please email numpy-discussion if you use 
this" note, and see if anybody complains?

Dag