Longstanding issue with using numpy in embedded CPython
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
http://stackoverflow.com/questions/7592565/whenembeddingcpythoninjavawh... http://mail.scipy.org/pipermail/numpydiscussion/2009July/044046.html Is there any fix or workaround for this? Thanks.
As far as a workaround in concerned, that scipy archive post says you can disable threads in numpy. Sorry can't help more, I don't know much about how to bypass such GIL issues.
= Olivier
2011/10/3 Yang Zhang yanghatespam@gmail.com
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
http://stackoverflow.com/questions/7592565/whenembeddingcpythoninjavawh... http://mail.scipy.org/pipermail/numpydiscussion/2009July/044046.html Is there any fix or workaround for this? Thanks.  Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang yanghatespam@gmail.com wrote:
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (nonthreaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
Cheers
Robin
[1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/whenembeddingcpythoninjavawh... http://mail.scipy.org/pipermail/numpydiscussion/2009July/044046.html Is there any fix or workaround for this? Thanks.  Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
On Tue, Oct 4, 2011 at 1:28 AM, Robin robince@gmail.com wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang yanghatespam@gmail.com wrote:
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (nonthreaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (secondhand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly  is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Cheers
Robin
[1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/whenembeddingcpythoninjavawh... http://mail.scipy.org/pipermail/numpydiscussion/2009July/044046.html Is there any fix or workaround for this? Thanks.  Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang yanghatespam@gmail.com wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin robince@gmail.com wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang yanghatespam@gmail.com wrote:
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (nonthreaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (secondhand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly  is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
Cheers
Robin
[1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/whenembeddingcpythoninjavawh... http://mail.scipy.org/pipermail/numpydiscussion/2009July/044046.html Is there any fix or workaround for this? Thanks.  Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
 Yang Zhang http://yz.mit.edu/
On Thu, Dec 8, 2011 at 23:15, Yang Zhang yanghatespam@gmail.com wrote:
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang yanghatespam@gmail.com wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin robince@gmail.com wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang yanghatespam@gmail.com wrote:
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (nonthreaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (secondhand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly  is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment.
On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern robert.kern@gmail.com wrote:
On Thu, Dec 8, 2011 at 23:15, Yang Zhang yanghatespam@gmail.com wrote:
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang yanghatespam@gmail.com wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin robince@gmail.com wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang yanghatespam@gmail.com wrote:
It turns out that there's a longstanding problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (nonthreaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (secondhand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly  is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment.
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
 Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth."  Umberto Eco _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
On Fri, Dec 9, 2011 at 11:00, Yang Zhang yanghatespam@gmail.com wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
Quoting Robert Kern robert.kern@gmail.com:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang yanghatespam@gmail.com wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython.
http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8....
Armando
Hi Armando,
No comment on the Java thing ;)
However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more uptodate and we are on github too: https://github.com/DiamondLightSource
Peter
On 9 December 2011 13:05, Vicente Sole sole@esrf.fr wrote:
Quoting Robert Kern robert.kern@gmail.com:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang yanghatespam@gmail.com wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython.
http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8....
Armando _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
Hi Peter,
The obsolete link was not deliberate. It was the first reference I found via google.
Best regards,
Armando
Quoting Peter CYC pcyc.uk@gmail.com:
Hi Armando,
No comment on the Java thing ;)
However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more uptodate and we are on github too: https://github.com/DiamondLightSource
Peter
On 9 December 2011 13:05, Vicente Sole sole@esrf.fr wrote:
Quoting Robert Kern robert.kern@gmail.com:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang yanghatespam@gmail.com wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython.
http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8....
Armando _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
Le 09/12/2011 09:31, Robert Kern a écrit :
We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment.
I recently got interested in multiprocessing computation with numpy and now I get scare by your statement ! Please don't tell me it is unsafe to launch multiple jobs (for instance with multiprocressing's Pool.map) just doing some ndarray arithmetic !
That's why I'd like to understand better the issue raised by Yang. For instance, what does exactly "multiple CPython interpreters" stands for ?
Best, Pierre
On Fri, Dec 9, 2011 at 13:18, Pierre Haessig pierre.haessig@crans.org wrote:
Le 09/12/2011 09:31, Robert Kern a écrit :
We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment.
I recently got interested in multiprocessing computation with numpy and now I get scare by your statement ! Please don't tell me it is unsafe to launch multiple jobs (for instance with multiprocressing's Pool.map) just doing some ndarray arithmetic !
That's why I'd like to understand better the issue raised by Yang. For instance, what does exactly "multiple CPython interpreters" stands for ?
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a nonPython program that embeds the CPython interpreter and starts up multiple copies of it in the same process.
Le 09/12/2011 15:00, Robert Kern a écrit :
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a nonPython program that embeds the CPython interpreter and starts up multiple copies of it in the same process.
Ok, now I think I understand. I was not aware it was possible to embed multiple CPython instances into one process. So I guess IPython's multiprocessing infrastructure I once briefly considered is also safe since it runs multiple kernels. I'm relieved to hear that !
Thank you very much for the explanation.
Pierre
Yang Zhang <yanghatespam <at> gmail.com> writes:
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213.
I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application.
Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork.
Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ?
Thanks. Raphael.
Can you expand a bit? Are you trying to disable threads at compiletime or at runtime? Which threaded functionality are you trying to disable? Are you using numpy as a computational library with multiple threads making calls into its functions? I think NPY_ALLOW_THREADS is for interacting with the GIL, but I have not played with it much.
A
On Mon, Sep 24, 2012 at 6:54 PM, Raphael de Feraudy feraudy@phimeca.com wrote:
Yang Zhang <yanghatespam <at> gmail.com> writes:
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213.
I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application.
Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork.
Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ?
Thanks. Raphael.
NumPyDiscussion mailing list NumPyDiscussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpydiscussion
participants (9)

Aron Ahmadia

Olivier Delalleau

Peter CYC

Pierre Haessig

Raphael de Feraudy

Robert Kern

Robin

Vicente Sole

Yang Zhang