Long-standing issue with using numpy in embedded CPython
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments: http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/
As far as a workaround in concerned, that scipy archive post says you can disable threads in numpy. Sorry can't help more, I don't know much about how to bypass such GIL issues. -=- Olivier 2011/10/3 Yang Zhang <yanghatespam@gmail.com>
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well. This seems to a be a problem specific to Jepp. Just wanted to mention it in case it puts someone off trying something unnecessarily in the future. Cheers Robin [1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem? I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Cheers
Robin
[1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Yang Zhang http://yz.mit.edu/
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
Cheers
Robin
[1] https://github.com/kw/pymex
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Yang Zhang http://yz.mit.edu/
-- Yang Zhang http://yz.mit.edu/
On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam@gmail.com> wrote:
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern <robert.kern@gmail.com> wrote:
On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam@gmail.com> wrote:
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well.
This seems to a be a problem specific to Jepp.
Just wanted to mention it in case it puts someone off trying something unnecessarily in the future.
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem?
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment.
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Yang Zhang http://yz.mit.edu/
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Quoting Robert Kern <robert.kern@gmail.com>:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython. http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... Armando
Hi Armando, No comment on the Java thing ;-) However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more up-to-date and we are on github too: https://github.com/DiamondLightSource Peter On 9 December 2011 13:05, Vicente Sole <sole@esrf.fr> wrote:
Quoting Robert Kern <robert.kern@gmail.com>:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython.
http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8....
Armando _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi Peter, The obsolete link was not deliberate. It was the first reference I found via google. Best regards, Armando Quoting Peter CYC <pcyc.uk@gmail.com>:
Hi Armando,
No comment on the Java thing ;-)
However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more up-to-date and we are on github too: https://github.com/DiamondLightSource
Peter
On 9 December 2011 13:05, Vicente Sole <sole@esrf.fr> wrote:
Quoting Robert Kern <robert.kern@gmail.com>:
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no.
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython.
http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8....
Armando _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Le 09/12/2011 09:31, Robert Kern a écrit :
We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. I recently got interested in multiprocessing computation with numpy and now I get scare by your statement ! Please don't tell me it is unsafe to launch multiple jobs (for instance with multiprocressing's Pool.map) just doing some ndarray arithmetic !
That's why I'd like to understand better the issue raised by Yang. For instance, what does exactly "multiple CPython interpreters" stands for ? Best, Pierre
On Fri, Dec 9, 2011 at 13:18, Pierre Haessig <pierre.haessig@crans.org> wrote:
Le 09/12/2011 09:31, Robert Kern a écrit :
We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. I recently got interested in multiprocessing computation with numpy and now I get scare by your statement ! Please don't tell me it is unsafe to launch multiple jobs (for instance with multiprocressing's Pool.map) just doing some ndarray arithmetic !
That's why I'd like to understand better the issue raised by Yang. For instance, what does exactly "multiple CPython interpreters" stands for ?
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Le 09/12/2011 15:00, Robert Kern a écrit :
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. Ok, now I think I understand. I was not aware it was possible to embed multiple CPython instances into one process. So I guess IPython's multiprocessing infrastructure I once briefly considered is also safe since it runs multiple kernels. I'm relieved to hear that !
Thank you very much for the explanation. Pierre
Yang Zhang <yanghatespam <at> gmail.com> writes:
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213. I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application. Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork. Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ? Thanks. Raphael.
Can you expand a bit? Are you trying to disable threads at compile-time or at run-time? Which threaded functionality are you trying to disable? Are you using numpy as a computational library with multiple threads making calls into its functions? I think NPY_ALLOW_THREADS is for interacting with the GIL, but I have not played with it much. A On Mon, Sep 24, 2012 at 6:54 PM, Raphael de Feraudy <feraudy@phimeca.com> wrote:
Yang Zhang <yanghatespam <at> gmail.com> writes:
I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
Anyone?
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213.
I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application.
Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork.
Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ?
Thanks. Raphael.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (9)
-
Aron Ahmadia -
Olivier Delalleau -
Peter CYC -
Pierre Haessig -
Raphael de Feraudy -
Robert Kern -
Robin -
Vicente Sole -
Yang Zhang