I am working on a project where we have a very peculiar requirements for an embedded interpreter. In my opinion these requirements cannot be extremely rare. I am posting this message to get some feedback from people who have worked on similar stuff or are interested in such a solution. If such a solution would be useful to a lot people if readily available as open source. We need to run multiple interpreters in-process. Which means we need to create completely independent environments for each interpreter using some very dirty hack. Since Python inherently does not support this. Python threads( created using python's threading module ) are not of much help because of the GIL. For this purpose the solution we have adopted is moving all the mutable python globals to heap. Nokia did a similar change to Python while porting it to S60 platform. Reference <http://research.nokia.com/files/PythonOnAPhone.ppt> It seems Nokia has stopped using this solution but I am not aware of how are they going about now since I cannot think of any other solution to run multiple in-process interpreters. Jukka, can you throw some light ? Lua already supports this. I would expect other projects like mod_python would also have to face similar problem but I am not aware how they are handling it. People who have ported Python on flat-memory model OSes such as VxWorks surely face this problem (Since there are no processes) unless they don't have requirement of running multiple interpreters. The second major change we did is to reinitialize python after executing of each script. This is because of the limitation of running it on embedded platform hardware and limited memory constraints. Again Python inherently does not provide support for this. (Calling Py_Initialize and Py_Finalize repetitively is not enough. It does not completely reinitialize python and is also known to leak memory) Would it be worthwhile (benefit a lot of people ) if these changes be made to standard python (eg. http://bugs.python.org/issue3329 ). Possibly a new branch of Python which supports multiple in-process interpreters?
On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar <swapnil.st@gmail.com> wrote:
It used* "sub-interpreters" apparently: http://modpython.org/live/current/doc-html/pyapi-interps.html http://docs.python.org/c-api/init.html#Py_NewInterpreter Cheers, Chris *mod_python stopped being actively developed in June, and was apparently moribund since 2007. -- http://blog.rebertia.com
On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert <pyideas@rebertia.com> wrote:
Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.
This is one of the things I'd have liked to see as part of py3k: "fixing" python for embedding in applications and defices. It would have meant getting rid of all globals and statics and tracking them in some sort of "context" structure. Since you'd have to change the C API, you could take the opportunity to fix some of its ugliest warts, such as inconsistent reference counting semantics between methods (making it difficult to write good smart pointer wrappers for it, as in COM. COM has very simple, clear and universal reference counting rules). K From: python-ideas-bounces+kristjan=ccpgames.com@python.org [mailto:python-ideas-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Swapnil Talekar Sent: Wednesday, November 17, 2010 22:46 To: Chris Rebert; python-ideas@python.org Subject: [Python-ideas] In-process interpreters On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert <pyideas@rebertia.com<mailto:pyideas@rebertia.com>> wrote: On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar <swapnil.st<http://swapnil.st>@gmail.com<http://gmail.com>> wrote:
Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.
2010/11/18 Kristján Valur Jónsson <kristjan@ccpgames.com>:
This is one of the things I’d have liked to see as part of py3k: “fixing” python for embedding in applications and defices.
My current work project has similar issues (also an embedded Python on embedded hardware). We've examined Py_Initialize()/Py_Finalize() pairing, subinterpreters, and out-of-proc interpreters, and are in the same predicament. I believe one of the main difficulties will be doing it in a way that doesn't penalize the 99% use case that most of the Python world experiences (one interpreter/one process, named python). That's been, for example, the fatal issue with most of the efforts to eliminate the global interpreter lock. It doesn't matter how nice it makes life for us if it causes performance or other headaches for the vast majority of Python users. -- Tim Lesher <tlesher@gmail.com>
On 11/18/2010 4:08 AM, Tim Lesher wrote:
I keep wondering whether it would be possible to rearrange the current code in any way that a) would not hinder performance significantly but b) would make it easier to maintain patches for compiler variations for the minority use cases. -- Terry Jan Reedy
On 19.11.2010 05:02, Terry Reedy wrote:
Restating problem this way leads to this idea: struct PythonStaticState { ... }; #ifdef EMBED #define GLOBAL_STATE_MACRO PythoStaticState globalPythonState, #else #define GLOBAL_STATE_MACRO PythonStaticState globalPythonState; #endif PyObject * PyAnyApi(GLOBAL_STATE_MACRO, a, b, c, d) { globalPythonState.var1 = 0; } and using good compiler and linker for nonembeded case ;^) Niki
Em Wed, 17 Nov 2010 13:56:43 +0530 Swapnil Talekar <swapnil.st@gmail.com> escreveu:
What about using multiple processes? You could use pipes to exchange information between them. http://docs.python.org/library/subprocess.html http://docs.python.org/library/os.html#os.fork (Unix only) Regards, -- .:''''':. .:' ` Sérgio Surkamp | Gerente de Rede :: ........ sergio@gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florianópolis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br
On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar <swapnil.st@gmail.com> wrote:
It used* "sub-interpreters" apparently: http://modpython.org/live/current/doc-html/pyapi-interps.html http://docs.python.org/c-api/init.html#Py_NewInterpreter Cheers, Chris *mod_python stopped being actively developed in June, and was apparently moribund since 2007. -- http://blog.rebertia.com
On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert <pyideas@rebertia.com> wrote:
Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.
This is one of the things I'd have liked to see as part of py3k: "fixing" python for embedding in applications and defices. It would have meant getting rid of all globals and statics and tracking them in some sort of "context" structure. Since you'd have to change the C API, you could take the opportunity to fix some of its ugliest warts, such as inconsistent reference counting semantics between methods (making it difficult to write good smart pointer wrappers for it, as in COM. COM has very simple, clear and universal reference counting rules). K From: python-ideas-bounces+kristjan=ccpgames.com@python.org [mailto:python-ideas-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Swapnil Talekar Sent: Wednesday, November 17, 2010 22:46 To: Chris Rebert; python-ideas@python.org Subject: [Python-ideas] In-process interpreters On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert <pyideas@rebertia.com<mailto:pyideas@rebertia.com>> wrote: On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar <swapnil.st<http://swapnil.st>@gmail.com<http://gmail.com>> wrote:
Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.
2010/11/18 Kristján Valur Jónsson <kristjan@ccpgames.com>:
This is one of the things I’d have liked to see as part of py3k: “fixing” python for embedding in applications and defices.
My current work project has similar issues (also an embedded Python on embedded hardware). We've examined Py_Initialize()/Py_Finalize() pairing, subinterpreters, and out-of-proc interpreters, and are in the same predicament. I believe one of the main difficulties will be doing it in a way that doesn't penalize the 99% use case that most of the Python world experiences (one interpreter/one process, named python). That's been, for example, the fatal issue with most of the efforts to eliminate the global interpreter lock. It doesn't matter how nice it makes life for us if it causes performance or other headaches for the vast majority of Python users. -- Tim Lesher <tlesher@gmail.com>
On 11/18/2010 4:08 AM, Tim Lesher wrote:
I keep wondering whether it would be possible to rearrange the current code in any way that a) would not hinder performance significantly but b) would make it easier to maintain patches for compiler variations for the minority use cases. -- Terry Jan Reedy
On 19.11.2010 05:02, Terry Reedy wrote:
Restating problem this way leads to this idea: struct PythonStaticState { ... }; #ifdef EMBED #define GLOBAL_STATE_MACRO PythoStaticState globalPythonState, #else #define GLOBAL_STATE_MACRO PythonStaticState globalPythonState; #endif PyObject * PyAnyApi(GLOBAL_STATE_MACRO, a, b, c, d) { globalPythonState.var1 = 0; } and using good compiler and linker for nonembeded case ;^) Niki
Em Wed, 17 Nov 2010 13:56:43 +0530 Swapnil Talekar <swapnil.st@gmail.com> escreveu:
What about using multiple processes? You could use pipes to exchange information between them. http://docs.python.org/library/subprocess.html http://docs.python.org/library/os.html#os.fork (Unix only) Regards, -- .:''''':. .:' ` Sérgio Surkamp | Gerente de Rede :: ........ sergio@gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florianópolis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br
participants (7)
-
Chris Rebert -
Kristján Valur Jónsson -
Niki Spahiev -
Swapnil Talekar -
Sérgio Surkamp -
Terry Reedy -
Tim Lesher