[Twisted-Python] How can I keep data integrity in multiple threads environment
Hello, I am new to twisted and I have a few questions: I have implemented a TCPServer and the associated Factory, which manages data and dispatches treatments (According to the tutorial it's not the best solution but actually this is not the point ;) ) All of the treatments run in threads and thus use shared data. The allData dictionnary is critical since it can be used by multiple threads. How can I keep the allData integrity ? Does Twisted core provide such facilites or do I have to use semaphore or other stuff ? class GestionnaireXXX(protocol.Protocol) def dataReceived(self, data): self.factory.dispatch(data) class GestionnaireXXXFactory(protocol.ServerFactory): def __init__(self): self.allData = {} self.initData() self._dispatcher = { 1: self.alert, 4: self.create, } def initData(self): if os.path.isfile(TEMP) self.allData = aot.unjellyFromSource(file(TEMP, 'r')) def dispatch(self, data): splitedData = data.split('\n') method = self._dispatcher.get(len(splitedData), self.doNothing) reactor.runInThread(method, splitedData) def create(self, *data): # Some treatments on allData ... aot.jellyToSource(file(TEMP, 'w')) def alert(self, *unused): # XXX sem = DeferredSemaphore() sem.acquire() allDataCopy = self.allData.copy() self.allData.clear() sem.release() # End XXX # some long treatments on allDataCopy ... Thank's, -- Olivier Guilloux
Kaladan wrote:
Hello,
I am new to twisted and I have a few questions: I have implemented a TCPServer and the associated Factory, which manages data and dispatches treatments (According to the tutorial it's not the best solution but actually this is not the point ;) )
All of the treatments run in threads and thus use shared data. The allData dictionnary is critical since it can be used by multiple threads. How can I keep the allData integrity ? Does Twisted core provide such facilites or do I have to use semaphore or other stuff ?
You have to do it yourself. Twisted is not thread-safe and does not provide any thread support beyond the two most basic methods to talk to threads: deferToThread/callInThread and callFromThread
On Mon, 2005-12-12 at 16:04 +0100, Kaladan wrote:
All of the treatments run in threads and thus use shared data. The allData dictionnary is critical since it can be used by multiple threads. How can I keep the allData integrity ? Does Twisted core provide such facilites or do I have to use semaphore or other stuff ?
"Don't use threads" is a good way to start; unless you're calling into blocking C libraries using threads won't help you much. Even if you are (e.g. using Numeric, or DB-API or something), you can still avoid shared data structures in most cases: Get data from shared structure in the main (i.e. Twisted's) thread, hand it to thread that does self-contained operation on data. Thread then does callFromThread(self.putBackInSharedDataStructure, result), which will run in the Twisted thread.
participants (3)
-
Itamar Shtull-Trauring
-
Kaladan
-
Phil Mayers