[Numpy-discussion] Should we drop support for "one file" compilation mode?

[Nathaniel Smith]

On Tue, Oct 6, 2015 at 4:07 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 6 Oct 2015 11:00:30 +0100
> David Cournapeau <cournape at gmail.com> wrote:
>>
>> Assuming one of the rumour is related to some comments I made some time
>> (years ?) earlier, the context was the ability to hide exported symbols. As
>> you know, the issue is not to build extensions w/ multiple compilation
>> units, but sharing functionalities between them without sharing them
>> outside the extension.
>
> Can't you use the visibility attribute with gcc for this?
> Other Unix compilers probably provide something similar. The issue
> doesn't exist on Windows by construction.
>
> https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes

This is what we do normally when building a shared extension, but in
the exceptional case where people want to statically link numpy into
cpython, it doesn't work -- the visibility attribute and related
machinery only works on shared libraries, not static libraries.
(Recall that traditionally, doing 'a.o + b.o -> static.a; static.a +
c.o -> final' is just a shorthand for doing 'a.o + b.o + c.o ->
final'.)

But this is still a solved problem, you just have to use the static
linking version instead of the shared library version :-). E.g. with
GNU tools the magic incantation is
  objcopy --keep-symbol-name PyInit_multiarray multiarray.a

It's possible that there's some system somewhere that both needs
static linking *and* doesn't have access to objcopy-or-equivalent, but
then we're back with the thing where it's not a great plan to keep
spending energy on supporting purely theoretical platforms.

> By the way, external packages may reuse the npy_* functions, so I would
> like them not the be hidden :-)

This is a separate issue from the one file/multi-file compilation mode, but:

I'd really prefer that we stick to just one way of exporting stuff to
external packages, and that that be the (admittedly somewhat
cumbersome) public API / import_array() mechanism.

Trying to manage numpy's API/ABI exposure is a huge challenge in
general, so having two mechanisms is not really sustainable. And
trying to use the linker directly creates huge cross-platform
headaches -- no-one can agree on what's exported by default, or how
you find shared libraries, and the numpy extensions will certainly not
be on the default library search path, so you need some
platform-specific hack to find them... OTOH the "public API" mechanism
takes some ugly stuff on numpy's side to set things up, but then the
result is a uniform mechanism that uses Python's familiar package
lookup rules to find the relevant symbols.

If you need some npy_* function it'd be much better to let us know
what it is and let us export it in an intentional way, instead of just
relying on whatever stuff we accidentally exposed?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org