Multiple independent Python interpreters within one process (or sth

[Titus Brown]

In article <39D1F198.F3BFC7FF at softax.com.pl>,
Marcin Kasperski  <Marcin.Kasperski at softax.com.pl> wrote:
>1) Is it possible to embed multiple, independent Python interpreters
>within single process? In case it is possible, will they work in
>multithreaded environment?

Yes, sort of.  Yes, sort of.

I'm planning to write a longer article about this at some point, but
interpreters in Python aren't, strictly speaking, isolated completely.

This can be divided into two types of root causes: lack of thread isolation,
and design flaws within Python.

Lack of thread isolation causes problems like:

* threads share the same environment (not os.environ, but real environment);
* memory leaks & bugs affect the whole process;
* running os.system is a chancy thing at best.

However, any language that supports threads will have these "problems" on a
UNIX system.

Design flaws within CPython, however, are more problematic.  (I don't know
if JPython has the same problems.)

For example:

* per-interpreter memory is not automatically freed upon destruction of
	interpreter;
* modules are not automatically reloaded on 'import', and moreover some
	module state is shared between interpreters in an ill-documented,
	if not ill-defined, way;
* Python is not free-threading: there's a global interpreter lock, which
	means that in CPU-bound computational situations you won't take full
	advantage of multiple CPUs.

>2) How would you solve the problem of using Python as scripting language
>in multithreaded process using many independent scripts to handle simple
>tasks?

I'm not sure what you mean by "solve" -- I would certainly use Python to
do it, though ;).

>We are currently debating whether Python can be language of choice to
>use as a scripting language for some project. The main engine (written
>in C++) will accept some type of requests (we interface some hardware,
>let me skip the details) and pass them to working scripts - which should
>work in parallel, without influencing one another.

Sounds very similar to the work we've done with AOLserver/PyWX; we've
integrated a Python interpreter into a multithreaded Web server written
in C.  You might also check out SWIG (swig.sourceforge.net) to automate
some of the C++ <--> Python nitty-gritty.

AOLserver: http://www.aolserver.com/
PyWX:      http://pywx.idyll.org/
SWIG:      http://swig.sourceforge.net/

cheers,
--titus