[BangPypers] Ideas for Python concurrency...

[Anand Chitipothu]

2011/2/8 Vishal <vsapre80 at gmail.com>:
> On Tue, Feb 8, 2011 at 2:02 PM, Anand Chitipothu <anandology at gmail.com>wrote:
>
>> 2011/2/8 Vishal <vsapre80 at gmail.com>:
>> > Hello,
>> >
>> > This might sound crazy..and dont know if its even possible, but...
>> >
>> > Is it possible that the Python process, creates copies of the interpreter
>> > for each thread that is launched, and some how the thread is bound to its
>> > own interpreter ?
>> >
>> > This will increase the python process size...for sure, however data
>> sharing
>> > will remain just like it is in threads.
>> >
>> > and it "may" also allow the two threads to run in parallel, assuming the
>> > processors of today can send independent instructions from the same
>> process
>> > to multiple cores?
>> >
>> > Comments, suggestions, brush offs  are welcome :))
>>
>> Why not use multiprocessing?
>>
>> http://docs.python.org/library/multiprocessing.html
>>
>> Anand
>>
>
> Mutliprocessing means, data copying, talking to each other through PIPES,
> also it has its issues with running on Windows (all function calls should be
> pickelable)
>
> Threads seems pretty stable on most platforms where Python runs.

There are two options. Shared data with locks or message passing.
Threads use shared data model and multiprocessing uses message passing
model.

If you want to avoid message passing (copying data form one process to
another process), then you have to pay the price of synchronization
with locks, which is GIL in case of Python.

Python, CPython esp. is designed be run as a single instance and uses
global state for managing the imported modules and other state. So it
is not possible to have another instance of interpreter running in the
same process.

However, it might be possible to do that with Jython, the
implementation of Python in Java. IIRC, It allows you to create
multiple instances of Python Interpreter and start executing jobs in
parallel.

Anand