This is one of the things I'd have liked to see as part of py3k: "fixing" python for embedding in applications and defices. It would have meant getting rid of all globals and statics and tracking them in some sort of "context" structure. Since you'd have to change the C API, you could take the opportunity to fix some of its ugliest warts, such as inconsistent reference counting semantics between methods (making it difficult to write good smart pointer wrappers for it, as in COM. COM has very simple, clear and universal reference counting rules). K From: python-ideas-bounces+kristjan=ccpgames.com@python.org [mailto:python-ideas-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Swapnil Talekar Sent: Wednesday, November 17, 2010 22:46 To: Chris Rebert; python-ideas@python.org Subject: [Python-ideas] In-process interpreters On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert mailto:pyideas@rebertia.com> wrote: On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar http://swapnil.st@gmail.comhttp://gmail.com> wrote:

I am working on a project where we have a very peculiar requirements for an embedded interpreter. In my opinion these requirements cannot be extremely rare. I am posting this message to get some feedback from people who have worked on similar stuff or are interested in such a solution. If such a solution would be useful to a lot people if readily available as open source. We need to run multiple interpreters in-process. Which means we need to create completely independent environments for each interpreter using some very dirty hack. Since Python inherently does not support this. Python threads( created using python's threading module ) are not of much help because of the GIL. For this purpose the solution we have adopted is moving all the mutable python globals to heap. Nokia did a similar change to Python while porting it to S60 platform. Reference It seems Nokia has stopped using this solution but I am not aware of how are they going about now since I cannot think of any other solution to run multiple in-process interpreters. Jukka, can you throw some light ? Lua already supports this. I would expect other projects like mod_python would also have to face similar problem but I am not aware how they are handling it. It used* "sub-interpreters" apparently: http://modpython.org/live/current/doc-html/pyapi-interps.html http://docs.python.org/c-api/init.html#Py_NewInterpreter

Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.

On 11/18/2010 4:08 AM, Tim Lesher wrote:

2010/11/18 Kristján Valur Jónsson:

...

This is one of the things I’d have liked to see as part of py3k: “fixing” python for embedding in applications and defices.

My current work project has similar issues (also an embedded Python on embedded hardware). We've examined Py_Initialize()/Py_Finalize() pairing, subinterpreters, and out-of-proc interpreters, and are in the same predicament.

I believe one of the main difficulties will be doing it in a way that doesn't penalize the 99% use case that most of the Python world experiences (one interpreter/one process, named python). That's been, for example, the fatal issue with most of the efforts to eliminate the global interpreter lock.

It doesn't matter how nice it makes life for us if it causes performance or other headaches for the vast majority of Python users.

I keep wondering whether it would be possible to rearrange the current code in any way that a) would not hinder performance significantly but b) would make it easier to maintain patches for compiler variations for the minority use cases. -- Terry Jan Reedy