[Numpy-discussion] Combined versus separate build

David Cournapeau cournape at gmail.com
Sun Jul 1 14:36:49 EDT 2012


On Sun, Jul 1, 2012 at 6:36 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Jun 27, 2012 at 9:05 PM, David Cournapeau <cournape at gmail.com> wrote:
>> On Wed, Jun 27, 2012 at 8:53 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>> But seriously, what compilers do we support that don't have
>>> -fvisibility=hidden? ...Is there even a list of compilers we support
>>> available anywhere?
>>
>> Well, I am not sure how all this is handled on the big guys (bluegen
>> and co), for once.
>>
>> There is also the issue of the consequence on statically linking numpy
>> to python: I don't what they are (I would actually like to make
>> statically linked numpy into python easier, not harder).
>
> All the docs I can find in a quick google seem to say that bluegene
> doesn't do shared libraries at all, though those may be out of date.
>
> Also, it looks like our current approach is not doing a great job of
> avoiding symbol table pollution... despite all the NPY_NO_EXPORTS all
> over the source, I still count ~170 exported symbols on Linux with
> numpy 1.6, many of them with non-namespaced names
> ("_n_to_n_data_copy", "_next", "npy_tan", etc.) Of course this is
> fixable, but it's interesting that no-one has noticed. (Current master
> brings this up to ~300 exported symbols.)
>
> It sounds like as far as our "officially supported" platforms go
> (linux/windows/osx with gcc/msvc), then the ideal approach would be to
> use -fvisibility=hidden or --retain-symbols-file to convince gcc to
> hide symbols by default, like msvc does. That would let us remove
> cruft from the source code, produce a more reliable result, and let us
> use the more convenient separate build, with no real downsides.

What cruft would it allow us to remove ? Whatever method we use, we
need a whitelist of symbols to export.

On the exported list I see on mac, most of them are either from
npymath (npy prefix) or npysort (no prefix, I think this should be
added). Once those are ignored as they should be, there are < 30
symbols exported.

> (Static linking is trickier because no-one uses it anymore so the docs
> aren't great, but I think on Linux at least you could accomplish the
> equivalent by building the static library with 'ld -r ... -o
> tmp-multiarray.a; objcopy --keep-global-symbol=initmultiarray
> tmp-multiarray.a multiarray.a'.)

I am not sure why you say that static linking is not used anymore: I
have met some people who do statically link numpy into python.

>
> Of course there are presumably other platforms that we don't support
> or test on, but where we have users anyway. Building on such a
> platform sort of intrinsically requires build system hacks, and some
> equivalent to the above may well be available (e.g. I know icc
> supports -fvisibility). So I while I'm not going to do anything about
> this myself in the near future, I'd argue that it would be a good idea
> to:
>  - Switch the build-system to export nothing by default when using
> gcc, using -fvisibility=hidden
>  - Switch the default build to "separate"
>  - Leave in the single-file build, but not "officially supported",
> i.e., we're happy to get patches but it's not used on any systems that
> we can actually test ourselves. (I suspect it's less fragile than the
> separate build anyway, since name clashes are less common than
> forgotten include files.)

I am fine with making the separate build the default (I have a patch
somewhere that does that on supported platforms), but not with using
-fvisibility=hidden. When I implemented the initial support around
this, fvisibility was buggy on some platforms, including mingw 3.x

I don't think changing what our implementation does here is worthwhile
given that it works, and fsibility=hidden has no big advantages (you
would still need to mark the functions to be exported).

David



More information about the NumPy-Discussion mailing list