[Python-ideas] In-process interpreters

[Kristján Valur Jónsson]

This is one of the things I'd have liked to see as part of py3k:  "fixing" python for embedding in applications and defices.
It would have meant getting rid of all globals and statics and tracking them in some sort of "context" structure.  Since you'd have to change the C API, you could take the opportunity to fix some of its ugliest warts, such as inconsistent reference counting semantics between methods (making it difficult to write good smart pointer wrappers for it, as in COM.  COM has very simple, clear and universal reference counting rules).
K


From: python-ideas-bounces+kristjan=ccpgames.com at python.org [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Swapnil Talekar
Sent: Wednesday, November 17, 2010 22:46
To: Chris Rebert; python-ideas at python.org
Subject: [Python-ideas] In-process interpreters


On Wed, Nov 17, 2010 at 2:11 PM, Chris Rebert <pyideas at rebertia.com<mailto:pyideas at rebertia.com>> wrote:
On Wed, Nov 17, 2010 at 12:26 AM, Swapnil Talekar <swapnil.st<http://swapnil.st>@gmail.com<http://gmail.com>> wrote:
> I am working on a project where we have a very peculiar requirements for an
> embedded interpreter. In my opinion these requirements cannot be extremely
> rare. I am posting this message to get some feedback from people who have
> worked on similar stuff or are interested in such a solution. If such a
> solution would be useful to a lot people if readily available as open
> source.
> We need to run multiple interpreters in-process. Which means we need to
> create completely independent environments for each interpreter using some
> very dirty hack. Since Python inherently does not support this. Python
> threads( created using python's threading module ) are not of much help
> because of the GIL.
> For this purpose the solution we have adopted is moving all the mutable
> python globals to heap.
> Nokia did a similar change to Python while porting it to S60 platform.
> Reference
> It seems Nokia has stopped using this solution but I am not aware of how are
> they going about now since I cannot think of any other solution to run
> multiple in-process interpreters. Jukka, can you throw some light ?
> Lua already supports this.
> I would expect other projects like mod_python would also have to face
> similar problem but I am not aware how they are handling it.
It used* "sub-interpreters" apparently:
http://modpython.org/live/current/doc-html/pyapi-interps.html
http://docs.python.org/c-api/init.html#Py_NewInterpreter

Sub-interpreters aren't attractive for the very reasons given in the Bugs and caveats section of the above link. Moreover Sub-interpreters do not release memory (as far as I have tested) even after calling Py_EndInterpreter. This makes them unusable on platforms with high memory constraint.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20101118/17722e22/attachment.html>