Should we drop support for "one file" compilation mode?
Hi all, For a long time, NumPy has supported two different ways of being compiled: "Separate compilation" mode: like most C projects, each .c file gets compiled to a .o file, and then the .o files get linked together to make a shared library. (This has been the default since 1.8.0.) "One file" mode: first concatenate all the .c files together to make one monster .c file, and then compile that .c file to make a shared library. (This was the default before 1.8.0.) Supporting these two different build modes creates a drag on development progress; in particular Stefan recently ran into this in this experiments with porting parts of the NumPy internals to Cython: https://github.com/numpy/numpy/pull/6408 (I suspect the particular problem he's running into can be fixed b/c so far he only has one .pyx file, but I also suspect that it will be impossible to support "one file" mode once we have multiple .pyx files.) There are some rumors that "one file" mode might be needed on some obscure platform somewhere, or that it might be necessary for statically linking numpy into the CPython executable, but we can't continue supporting things forever based only on rumors. If all we can get are rumors, then eventually we have to risk breaking things just to force anyone who cares to actually show up and explain what they need so we can support it properly :-). Would anyone object if we dropped support for the "one file" mode, making "separate compilation" mandatory, e.g. in 1.11? -n -- Nathaniel J. Smith -- http://vorpus.org
On Mon, 5 Oct 2015 15:26:17 -0700 Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
For a long time, NumPy has supported two different ways of being compiled:
"Separate compilation" mode: like most C projects, each .c file gets compiled to a .o file, and then the .o files get linked together to make a shared library. (This has been the default since 1.8.0.)
"One file" mode: first concatenate all the .c files together to make one monster .c file, and then compile that .c file to make a shared library. (This was the default before 1.8.0.)
[...]
There are some rumors that "one file" mode might be needed on some obscure platform somewhere, or that it might be necessary for statically linking numpy into the CPython executable, but we can't continue supporting things forever based only on rumors.
If those rumors were true, CPython would not even be able to build (the _io module in 3.x is linked from several C object files, for example). Regards Antoine.
On Mon, Oct 5, 2015 at 11:26 PM, Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
For a long time, NumPy has supported two different ways of being compiled:
"Separate compilation" mode: like most C projects, each .c file gets compiled to a .o file, and then the .o files get linked together to make a shared library. (This has been the default since 1.8.0.)
"One file" mode: first concatenate all the .c files together to make one monster .c file, and then compile that .c file to make a shared library. (This was the default before 1.8.0.)
Supporting these two different build modes creates a drag on development progress; in particular Stefan recently ran into this in this experiments with porting parts of the NumPy internals to Cython: https://github.com/numpy/numpy/pull/6408 (I suspect the particular problem he's running into can be fixed b/c so far he only has one .pyx file, but I also suspect that it will be impossible to support "one file" mode once we have multiple .pyx files.)
There are some rumors that "one file" mode might be needed on some obscure platform somewhere, or that it might be necessary for statically linking numpy into the CPython executable, but we can't continue supporting things forever based only on rumors. If all we can get are rumors, then eventually we have to risk breaking things just to force anyone who cares to actually show up and explain what they need so we can support it properly :-).
Assuming one of the rumour is related to some comments I made some time (years ?) earlier, the context was the ability to hide exported symbols. As you know, the issue is not to build extensions w/ multiple compilation units, but sharing functionalities between them without sharing them outside the extension. I am just reiterating that point so that we all discuss under the right context :) I also agree the current situation is not sustainable -- as we discussed privately before, cythonizing numpy.core is made quite more complicated by this. I have myself quite a few issues w/ cythonizing the other parts of umath. I would also like to support the static link better than we do now (do we know some static link users we can contact to validate our approach ?) Currently, what we have in numpy core is the following: numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ + statically link npymath numpy.core.umath -> compilation units in numpy/core/src/umath + statically link npymath/npysort + some shenanigans to use things in numpy.core.multiarray I would suggest to have a more layered approach, to enable both 'normal' build and static build, without polluting the public namespace too much. This is an approach followed by most large libraries (e.g. MKL), and is fairly flexible. Concretely, we could start by putting more common functionalities (aka the 'core' library) into its own static library. The API would be considered private to numpy (no stability guaranteed outside numpy), and every exported symbol from that library would be decorated appropriately to avoid potential clashes (e.g. '_npy_internal_'). FWIW, that has always been my intention to go toward this when I split up multiarray/umath into multiple .c files and extracted out npymath. cheers, David
On Tue, 6 Oct 2015 11:00:30 +0100 David Cournapeau <cournape@gmail.com> wrote:
Assuming one of the rumour is related to some comments I made some time (years ?) earlier, the context was the ability to hide exported symbols. As you know, the issue is not to build extensions w/ multiple compilation units, but sharing functionalities between them without sharing them outside the extension.
Can't you use the visibility attribute with gcc for this? Other Unix compilers probably provide something similar. The issue doesn't exist on Windows by construction. https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Functi... By the way, external packages may reuse the npy_* functions, so I would like them not the be hidden :-) Regards Antoine.
On Tue, Oct 6, 2015 at 12:07 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 11:00:30 +0100 David Cournapeau <cournape@gmail.com> wrote:
Assuming one of the rumour is related to some comments I made some time (years ?) earlier, the context was the ability to hide exported symbols.
As
you know, the issue is not to build extensions w/ multiple compilation units, but sharing functionalities between them without sharing them outside the extension.
Can't you use the visibility attribute with gcc for this?
We do that already for gcc, I think the question was whether every platform supported this or not (and whether we should care).
Other Unix compilers probably provide something similar. The issue doesn't exist on Windows by construction.
https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Functi...
By the way, external packages may reuse the npy_* functions, so I would like them not the be hidden :-)
The npy_ functions in npymath were designed to be exported. Those would stay that way. David
Regards
Antoine.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those would stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.) -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those would stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.)
Unless something changed, that's more or less how it works already (npymath is used in scipy, for example, which was one of the rationale for writing it in the first place !). You access the compilation/linking issues through the numpy distutils get_info function. David
-n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 5:51 PM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those would stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.)
Unless something changed, that's more or less how it works already (npymath is used in scipy, for example, which was one of the rationale for writing it in the first place !).
You access the compilation/linking issues through the numpy distutils get_info function.
And my suggestion is to use a similar mechanism for multiarray and umath, so that in the end the exported Python C API is just a thin layer on top of the underlying static library. That would make cython and I suspect static linking quite a bit easier. The API between the low layer and python C API of multiarray/umath would be considered private and outside any API/ABI stability. IOW, it would be an internal change, and should not cause visible changes to the users, except that some _npy_private_ symbols would be exported (but you would be crazy to use them, and the prototype declarations would not be available when you install numpy anyway). Think of those as the internal driver API/ABI of Linux or similar. David
David
-n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those would stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.)
Unless something changed, that's more or less how it works already (npymath is used in scipy, for example, which was one of the rationale for writing it in the first place !).
Okay... in fact multiarray.so right now *does* export tons and tons of random junk into the global symbol namespace (on systems like Linux that do have a global symbol namespace), so it isn't obvious whether people are asking for that to continue :-). I'm just specifically saying that we should try to get this back down to the 1 exported symbol. (Try: objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)') This *should* print 1 line... I currently get ~700. numpy.core.umath is similar.) -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 5:58 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those
would
stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.)
Unless something changed, that's more or less how it works already (npymath is used in scipy, for example, which was one of the rationale for writing it in the first place !).
Okay... in fact multiarray.so right now *does* export tons and tons of random junk into the global symbol namespace (on systems like Linux that do have a global symbol namespace), so it isn't obvious whether people are asking for that to continue :-). I'm just specifically saying that we should try to get this back down to the 1 exported symbol.
(Try: objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)') This *should* print 1 line... I currently get ~700. numpy.core.umath is similar.)
I think this overestimates the amount by quite a bit, since you see GLIBC symbols, etc... I am using nm -Dg --defined-only $(python -c 'import numpy; print(numpy.core.multiarray.__file__)') instead. I see around 290 symboles: the npy_ from npymath don't bother me, but the ones from npysort do. We should at least prefix those with npy_ (I don't think npysort API has ever been publisher in our header like npymath was ?) David -n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <cournape@gmail.com> wrote:
The npy_ functions in npymath were designed to be exported. Those would stay that way.
If we want to export these then I vote that we either: - use the usual API export mechanism, or else - provide a static library for people to link to, instead of trying to do runtime binding. (I.e. drop it in some known place, and then provide some functions for extension modules to find it at build time -- similar to how np.get_include() works.)
Unless something changed, that's more or less how it works already (npymath is used in scipy, for example, which was one of the rationale for writing it in the first place !).
Okay... in fact multiarray.so right now *does* export tons and tons of random junk into the global symbol namespace (on systems like Linux that do have a global symbol namespace), so it isn't obvious whether people are asking for that to continue :-). I'm just specifically saying that we should try to get this back down to the 1 exported symbol. (Try: objdump -T $(python -c 'import numpy; print(numpy.core.multiarray.__file__)') This *should* print 1 line... I currently get ~700. numpy.core.umath is similar.) -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 4:07 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 11:00:30 +0100 David Cournapeau <cournape@gmail.com> wrote:
Assuming one of the rumour is related to some comments I made some time (years ?) earlier, the context was the ability to hide exported symbols. As you know, the issue is not to build extensions w/ multiple compilation units, but sharing functionalities between them without sharing them outside the extension.
Can't you use the visibility attribute with gcc for this? Other Unix compilers probably provide something similar. The issue doesn't exist on Windows by construction.
https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Functi...
This is what we do normally when building a shared extension, but in the exceptional case where people want to statically link numpy into cpython, it doesn't work -- the visibility attribute and related machinery only works on shared libraries, not static libraries. (Recall that traditionally, doing 'a.o + b.o -> static.a; static.a + c.o -> final' is just a shorthand for doing 'a.o + b.o + c.o -> final'.) But this is still a solved problem, you just have to use the static linking version instead of the shared library version :-). E.g. with GNU tools the magic incantation is objcopy --keep-symbol-name PyInit_multiarray multiarray.a It's possible that there's some system somewhere that both needs static linking *and* doesn't have access to objcopy-or-equivalent, but then we're back with the thing where it's not a great plan to keep spending energy on supporting purely theoretical platforms.
By the way, external packages may reuse the npy_* functions, so I would like them not the be hidden :-)
This is a separate issue from the one file/multi-file compilation mode, but: I'd really prefer that we stick to just one way of exporting stuff to external packages, and that that be the (admittedly somewhat cumbersome) public API / import_array() mechanism. Trying to manage numpy's API/ABI exposure is a huge challenge in general, so having two mechanisms is not really sustainable. And trying to use the linker directly creates huge cross-platform headaches -- no-one can agree on what's exported by default, or how you find shared libraries, and the numpy extensions will certainly not be on the default library search path, so you need some platform-specific hack to find them... OTOH the "public API" mechanism takes some ugly stuff on numpy's side to set things up, but then the result is a uniform mechanism that uses Python's familiar package lookup rules to find the relevant symbols. If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of just relying on whatever stuff we accidentally exposed? -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of just relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required). And, of course, we would also benefit from the CBLAS functions (or any kind of C wrappers around them) :-) https://github.com/numpy/numpy/issues/6324 Regards Antoine.
On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of just relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required).
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
And, of course, we would also benefit from the CBLAS functions (or any kind of C wrappers around them) :-) https://github.com/numpy/numpy/issues/6324
This is difficult to do from NumPy itself -- we don't necessarily have access to a full BLAS or LAPACK API -- in some configurations we fall back on our minimal internal implementations that just have what we need. There was an interesting idea that came up in some discussions here a few weeks ago -- we already know that we want to package up BLAS inside a Python package that (numpy / scipy / scikit-learn / ...) can depend on and assume is there to link against. Maybe this new package would also be a good place for exposing these wrappers? -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of just relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required).
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
It is possible (and documented) to use any of the npy_ symbols from npymath from outside numpy: http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-ag... The design is not perfect (I was young and foolish :) ), but it has worked fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC (including windows). David
And, of course, we would also benefit from the CBLAS functions (or any kind of C wrappers around them) :-) https://github.com/numpy/numpy/issues/6324
This is difficult to do from NumPy itself -- we don't necessarily have access to a full BLAS or LAPACK API -- in some configurations we fall back on our minimal internal implementations that just have what we need.
There was an interesting idea that came up in some discussions here a few weeks ago -- we already know that we want to package up BLAS inside a Python package that (numpy / scipy / scikit-learn / ...) can depend on and assume is there to link against.
Maybe this new package would also be a good place for exposing these wrappers?
-n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of just relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required).
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
It is possible (and documented) to use any of the npy_ symbols from npymath from outside numpy: http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-ag...
The design is not perfect (I was young and foolish :) ), but it has worked fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC (including windows).
Okay, so just to confirm, it looks like this does indeed implement the static linking thing I just suggested (so perhaps I am also young and foolish ;-)) -- from looking at the output of get_info("npymath"), it seems to add -I.../numpy/core/include to the compiler flags, add -lnpymath -L.../numpy/core/lib to the linker flags, and then .../numpy/core/lib contains only libnpymath.a, so it's static linking. -n -- Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of
just
relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required).
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
It is possible (and documented) to use any of the npy_ symbols from npymath from outside numpy:
http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-ag...
The design is not perfect (I was young and foolish :) ), but it has
worked
fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC (including windows).
Okay, so just to confirm, it looks like this does indeed implement the static linking thing I just suggested (so perhaps I am also young and foolish ;-)) -- from looking at the output of get_info("npymath"), it seems to add -I.../numpy/core/include to the compiler flags, add -lnpymath -L.../numpy/core/lib to the linker flags, and then .../numpy/core/lib contains only libnpymath.a, so it's static linking.
Yes, I was not trying to argue otherwise. If you thought I was, blame it on my poor English (which sadly does not get better as I get less young...). My proposal is to extend this technique for *internal* API, but with the following differences: * the declarations are not put in any public header * we don't offer any way to link to this library, and name it something scary enough that people would have to be foolish (young or not) to use it. David
-n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 6, 2015 at 6:18 PM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 6 Oct 2015 09:40:43 -0700 Nathaniel Smith <njs@pobox.com> wrote:
If you need some npy_* function it'd be much better to let us know what it is and let us export it in an intentional way, instead of
just
relying on whatever stuff we accidentally exposed?
Ok, we seem to be using only the complex math functions (npy_cpow and friends, I could make a complete list if required).
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
It is possible (and documented) to use any of the npy_ symbols from npymath from outside numpy:
http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-ag...
The design is not perfect (I was young and foolish :) ), but it has
worked
fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC (including windows).
Okay, so just to confirm, it looks like this does indeed implement the static linking thing I just suggested (so perhaps I am also young and foolish ;-)) -- from looking at the output of get_info("npymath"), it seems to add -I.../numpy/core/include to the compiler flags, add -lnpymath -L.../numpy/core/lib to the linker flags, and then .../numpy/core/lib contains only libnpymath.a, so it's static linking.
Yes, I was not trying to argue otherwise. If you thought I was, blame it on my poor English (which sadly does not get better as I get less young...).
My proposal is to extend this technique for *internal* API, but with the following differences: * the declarations are not put in any public header * we don't offer any way to link to this library, and name it something scary enough that people would have to be foolish (young or not) to use it.
I am stupid: we of course do not even ship that internal library, it would just be linked into multiarray/umath and never installed or part of binary packages. David
David
-n
-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
PR #6429 is a preliminary cut at removing single file build support. A bit of cleanup remains, mostly rearranging some defines for style. <snip> Chuck
On Tue, 6 Oct 2015 10:07:13 -0700 Nathaniel Smith <njs@pobox.com> wrote:
And how are you getting at them? Are you just relying the way that on ELF systems, if two libraries are loaded into the same address space then they automatically get access to each other's symbols, even if they aren't linked to each other? What do you do on Windows?
Well it seems to work on Windows too, thanks to numpy.distutils.misc_util.get_info('npymath'). Under Windows, I seem to have a "<Python root>\site-packages\numpy\core\lib\npymath.lib" static library, and there's also a "npy-pkg-config" subdirectory there with some INI files in it. Hopefully you know better than me what this all is :-)
And, of course, we would also benefit from the CBLAS functions (or any kind of C wrappers around them) :-) https://github.com/numpy/numpy/issues/6324
This is difficult to do from NumPy itself -- we don't necessarily have access to a full BLAS or LAPACK API -- in some configurations we fall back on our minimal internal implementations that just have what we need.
I'm thinking about the functions exposed in "numpy/core/src/private/npy_cblas.h". My knowledge of the Numpy build system doesn't allow me to tell if it's always available or not :-)
There was an interesting idea that came up in some discussions here a few weeks ago -- we already know that we want to package up BLAS inside a Python package that (numpy / scipy / scikit-learn / ...) can depend on and assume is there to link against.
Maybe this new package would also be a good place for exposing these wrappers?
Yeah, why not - as long as there's something well-known and well-supported to depend on. But given Numpy is a hard dependency for all the other packages you mentioned, it may make sense (and simplify dependency management) to bundle it with Numpy. Regards Antoine.
participants (4)
-
Antoine Pitrou -
Charles R Harris -
David Cournapeau -
Nathaniel Smith