[Python-Dev] Making python C-API thread safe (try 2)

Alan Kennedy alanmk at hotmail.com
Fri Sep 12 12:27:51 EDT 2003


Harri Pesonen <fuerte at sci.fi> writes:
>> The simple solution is,
>> that each thread created in Python gets its own independent
>> interpreter state as well. And there could be a separate
>> thread-global interpreter state for shared memory access. Access
>> to this global state would always be
>> synchronized. 

"Christopher A. Craig" wrote:
> Couldn't you do this now with multiple processes and the shm module?

Yes, you can store representations of python objects in shared memory,
e.g. pickles, etc: an in-memory database, with indices, etc. 

So when you want to access a "shared" object, you go to the shared
store, retrieve the pickle, unpickle it and use it, making sure to
repickle any changes and store them back in the shared memory. Which
might be cumbersome and inefficient, particularly if your goal is to
maximise efficiency in a multi-processor situation.

But if you want to store actual python objects in shared memory, and
avoid all the pickling etc,then you have to change the interpreter so
that it obtains the memory for new python objects from the shared
memory pool instead of "local" memory.

Which leads to problems with reference counting and garbage
collection. These would have to take multiple processes into account:
what happens when a process goes down? Should the objects allocated by
it be destroyed? Or remain persistent? Until when? If a process died
in an unclean fashion, it might not delete its references to objects
cleanly.

And it gets more complex again when you're dealing with users and
permissions.

I think that there are so many questions associated with the approach
of sharing python objects through shared memory that it will probably
remain an "application specific" technique for some time to come.

Though doubtless some more knowledgable person than I will now
contradict me. Please. :-)

-- 
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan:              http://xhaus.com/mailto/alan




More information about the Python-list mailing list