[Numpy-discussion] Combined versus separate build

[Nathaniel Smith]

On Wed, Jun 27, 2012 at 9:05 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Wed, Jun 27, 2012 at 8:53 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> But seriously, what compilers do we support that don't have
>> -fvisibility=hidden? ...Is there even a list of compilers we support
>> available anywhere?
>
> Well, I am not sure how all this is handled on the big guys (bluegen
> and co), for once.
>
> There is also the issue of the consequence on statically linking numpy
> to python: I don't what they are (I would actually like to make
> statically linked numpy into python easier, not harder).

All the docs I can find in a quick google seem to say that bluegene
doesn't do shared libraries at all, though those may be out of date.

Also, it looks like our current approach is not doing a great job of
avoiding symbol table pollution... despite all the NPY_NO_EXPORTS all
over the source, I still count ~170 exported symbols on Linux with
numpy 1.6, many of them with non-namespaced names
("_n_to_n_data_copy", "_next", "npy_tan", etc.) Of course this is
fixable, but it's interesting that no-one has noticed. (Current master
brings this up to ~300 exported symbols.)

It sounds like as far as our "officially supported" platforms go
(linux/windows/osx with gcc/msvc), then the ideal approach would be to
use -fvisibility=hidden or --retain-symbols-file to convince gcc to
hide symbols by default, like msvc does. That would let us remove
cruft from the source code, produce a more reliable result, and let us
use the more convenient separate build, with no real downsides.

(Static linking is trickier because no-one uses it anymore so the docs
aren't great, but I think on Linux at least you could accomplish the
equivalent by building the static library with 'ld -r ... -o
tmp-multiarray.a; objcopy --keep-global-symbol=initmultiarray
tmp-multiarray.a multiarray.a'.)

Of course there are presumably other platforms that we don't support
or test on, but where we have users anyway. Building on such a
platform sort of intrinsically requires build system hacks, and some
equivalent to the above may well be available (e.g. I know icc
supports -fvisibility). So I while I'm not going to do anything about
this myself in the near future, I'd argue that it would be a good idea
to:
 - Switch the build-system to export nothing by default when using
gcc, using -fvisibility=hidden
 - Switch the default build to "separate"
 - Leave in the single-file build, but not "officially supported",
i.e., we're happy to get patches but it's not used on any systems that
we can actually test ourselves. (I suspect it's less fragile than the
separate build anyway, since name clashes are less common than
forgotten include files.)

-N