[Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

Nathaniel Smith njs at pobox.com
Fri Oct 9 02:14:55 EDT 2015

On Thu, Oct 8, 2015 at 1:07 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Thu, Oct 8, 2015 at 8:47 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Oct 8, 2015 06:30, "David Cournapeau" <cournape at gmail.com> wrote:
>> >
>> [...]
>> >
>> > Separating the pure C code into static lib is the simple way of
>> > achieving the same goal. Essentially, you write:
>> >
>> > # implemented in npyinternal.a
>> > _npy_internal_foo(....)
>> >
>> > # implemented in merged_multiarray_umath.pyx
>> > cdef PyArray_Foo(...):
>> >     # use _npy_internal_foo()
>> >
>> > then our merged_multiarray_umath.so is built by linking the .pyx and the
>> > npyinternal.a together. IOW, the static link is internal.
>> >
>> > Going through npyinternal.a instead of just linking .o from pure C and
>> > Cython together gives us the following:
>> >
>> >  1. the .a can just use normal linking strategies instead of the awkward
>> > capsule thing. Those are easy to get wrong when using cython as you may end
>> > up with multiple internal copies of the wrapped object inside capsule,
>> > causing hard to track bugs (this is what we wasted most of the time on w/
>> > Stefan and Kurt during ds4ds)
>> Check out Stéfan's branch -- it just uses regular linking to mix cython
>> and C.
> I know, we worked on this together after all ;)
> My suggested organisation is certainly not mandatory, I was not trying to
> claim otherwise, sorry if that was unclear.
> At that point, I guess the consensus is that I have to prove my suggestion
> is useful. I will take a few more hours to submit a PR with the umath
> conversion (maybe merging w/ the work from Stéfan).

Okay! Still not sure what capsules have to do with anything, but I
guess the PR will either make it clear or else make it clear that it
doesn't matter :-).

> I discovered on my
> flight back that you can call PyModule_Init multiple times for a given
> module, which is useful while we do the transition C->Cython for the module
> initialization (it is not documented as possible, so I would not to rely on
> it for long either).

Oh, right, Stefan mentioned something about this... PyModule_Init is a
python-2-only thing, so whatever it does now is what it will do
forever and ever amen. But I can't think of any good reason to call it
twice -- if your goal is just to get a reference to the new module,
then once PyModule_Init has run once, you can just run
PyImport_ImportModule (assuming you know your fully-qualified module


Nathaniel J. Smith -- http://vorpus.org

More information about the NumPy-Discussion mailing list