Long-standing issue with using numpy in embedded CPython
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments: http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/a5cbd07efd53a3e16124de3829abec0e.jpg?s=120&d=mm&r=g)
As far as a workaround in concerned, that scipy archive post says you can disable threads in numpy. Sorry can't help more, I don't know much about how to bypass such GIL issues. -=- Olivier 2011/10/3 Yang Zhang <yanghatespam@gmail.com>
![](https://secure.gravatar.com/avatar/ac4e2152839c56c5326cd95ae54296e9.jpg?s=120&d=mm&r=g)
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well. This seems to a be a problem specific to Jepp. Just wanted to mention it in case it puts someone off trying something unnecessarily in the future. Cheers Robin [1] https://github.com/kw/pymex
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem? I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
Anyone?
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam@gmail.com> wrote:
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern <robert.kern@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/864a330f9f2791b501d1cba63aaa0165.jpg?s=120&d=mm&r=g)
Quoting Robert Kern <robert.kern@gmail.com>:
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython. http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... Armando
![](https://secure.gravatar.com/avatar/1dea3585eb16c5d7a3d868f7b896e98c.jpg?s=120&d=mm&r=g)
Hi Armando, No comment on the Java thing ;-) However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more up-to-date and we are on github too: https://github.com/DiamondLightSource Peter On 9 December 2011 13:05, Vicente Sole <sole@esrf.fr> wrote:
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 13:18, Pierre Haessig <pierre.haessig@crans.org> wrote:
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/206300f003271ac0e15490d304cf2da4.jpg?s=120&d=mm&r=g)
Yang Zhang <yanghatespam <at> gmail.com> writes:
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213. I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application. Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork. Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ? Thanks. Raphael.
![](https://secure.gravatar.com/avatar/821b8bf222958f83097ac40befc9edd7.jpg?s=120&d=mm&r=g)
Can you expand a bit? Are you trying to disable threads at compile-time or at run-time? Which threaded functionality are you trying to disable? Are you using numpy as a computational library with multiple threads making calls into its functions? I think NPY_ALLOW_THREADS is for interacting with the GIL, but I have not played with it much. A On Mon, Sep 24, 2012 at 6:54 PM, Raphael de Feraudy <feraudy@phimeca.com> wrote:
![](https://secure.gravatar.com/avatar/a5cbd07efd53a3e16124de3829abec0e.jpg?s=120&d=mm&r=g)
As far as a workaround in concerned, that scipy archive post says you can disable threads in numpy. Sorry can't help more, I don't know much about how to bypass such GIL issues. -=- Olivier 2011/10/3 Yang Zhang <yanghatespam@gmail.com>
![](https://secure.gravatar.com/avatar/ac4e2152839c56c5326cd95ae54296e9.jpg?s=120&d=mm&r=g)
On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments:
Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well. This seems to a be a problem specific to Jepp. Just wanted to mention it in case it puts someone off trying something unnecessarily in the future. Cheers Robin [1] https://github.com/kw/pymex
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince@gmail.com> wrote:
My (second-hand) understanding is that this is a problem with having multiple CPython interpreters, which both Jepp and numpy utilize, incompatibly - is that right? I.e., if either one were restricted to using a single CPython interpreter, we wouldn't see this problem? I'm curious how to disable threads in numpy (not an ideal solution). Googling seems to point me to setting NPY_ALLOW_THREADS to 0....somewhere.
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam@gmail.com> wrote:
Anyone?
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam@gmail.com> wrote:
numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/cedc87cb539ef417bb32576b9d5c18e1.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern <robert.kern@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
-- Yang Zhang http://yz.mit.edu/
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam@gmail.com> wrote:
Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp?
I don't think so, no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/864a330f9f2791b501d1cba63aaa0165.jpg?s=120&d=mm&r=g)
Quoting Robert Kern <robert.kern@gmail.com>:
It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython. http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... Armando
![](https://secure.gravatar.com/avatar/1dea3585eb16c5d7a3d868f7b896e98c.jpg?s=120&d=mm&r=g)
Hi Armando, No comment on the Java thing ;-) However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.... is more up-to-date and we are on github too: https://github.com/DiamondLightSource Peter On 9 December 2011 13:05, Vicente Sole <sole@esrf.fr> wrote:
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Dec 9, 2011 at 13:18, Pierre Haessig <pierre.haessig@crans.org> wrote:
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
![](https://secure.gravatar.com/avatar/206300f003271ac0e15490d304cf2da4.jpg?s=120&d=mm&r=g)
Yang Zhang <yanghatespam <at> gmail.com> writes:
It's appearing to me I had to face this very issue, which I reported @Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213. I just tried your suggestion : set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h. It allowed my atomic example to run without stalling, and also fixed the issue in my application. Though i'm not entirely satisfied by this workaround, which might slow down heavy computations. I also find it too intrusive in numpy source code and don't wish to maintain a powerless numpy fork. Has anyone else settled with this fix ? Or may anybody have any other suggestion / comments ? Thanks. Raphael.
![](https://secure.gravatar.com/avatar/821b8bf222958f83097ac40befc9edd7.jpg?s=120&d=mm&r=g)
Can you expand a bit? Are you trying to disable threads at compile-time or at run-time? Which threaded functionality are you trying to disable? Are you using numpy as a computational library with multiple threads making calls into its functions? I think NPY_ALLOW_THREADS is for interacting with the GIL, but I have not played with it much. A On Mon, Sep 24, 2012 at 6:54 PM, Raphael de Feraudy <feraudy@phimeca.com> wrote:
participants (9)
-
Aron Ahmadia
-
Olivier Delalleau
-
Peter CYC
-
Pierre Haessig
-
Raphael de Feraudy
-
Robert Kern
-
Robin
-
Vicente Sole
-
Yang Zhang