[Numpy-discussion] Long-standing issue with using numpy in embedded CPython

[Robert Kern]

On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam at gmail.com> wrote:
> On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam at gmail.com> wrote:
>> On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince at gmail.com> wrote:
>>> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam at gmail.com> wrote:
>>>> It turns out that there's a long-standing problem in numpy that
>>>> prevents it from being used in embedded CPython environments:
>>>
>>> Just wanted to make the point for reference that in general Numpy does
>>> work fine in (non-threaded) embedded CPython situations, see for
>>> example pymex [1] which embeds Python + Numpy in a Matlab mex file and
>>> works really well.
>>>
>>> This seems to a be a problem specific to Jepp.
>>>
>>> Just wanted to mention it in case it puts someone off trying something
>>> unnecessarily in the future.
>>
>> My (second-hand) understanding is that this is a problem with having
>> multiple CPython interpreters, which both Jepp and numpy utilize,
>> incompatibly - is that right?  I.e., if either one were restricted to
>> using a single CPython interpreter, we wouldn't see this problem?
>>
>> I'm curious how to disable threads in numpy (not an ideal solution).
>> Googling seems to point me to setting NPY_ALLOW_THREADS to
>> 0....somewhere.
>
> Anyone?

numpy does not use multiple interpreters. The threading options have
nothing to do with multiple interpreters, and will not let you use
multiple CPython interpreters in your application. The problem is that
Python does not have good isolation between multiple interpreters for
extension modules. Many extension modules happen to work in this
environment, but numpy is not one of them. We have some global state
that we need to keep, and this gets interfered with in a multiple
interpreter environment.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco