[Twisted-Python] Shared resource manioulation example
Hi, I'm looking for a Twisted based example how it should be properly done update/delete of global dictionary's SAME key:value pair from different functions all together. Or by another words, two functions are trying to update the same key:value pair of the same dictionary at the same time, than how to do it properly with Twisted? Thanks a lot.
On Fri, May 28, 2010 at 8:46 AM, <vitaly@synapticvision.com> wrote:
I'm looking for a Twisted based example how it should be properly done update/delete of global dictionary's SAME key:value pair from different functions all together. Or by another words, two functions are trying to update the same key:value pair of the same dictionary at the same time, than how to do it properly with Twisted?
Twisted doesn't change anything about Python in this regard. One update to the dictionary is going to happen before the other. If these updates are being done from different threads, you need some kind of synchronization to make them happen in the order you want. However, if you aren't explicitly using threads in your code, remember that Twisted doesn't run your code outside the main thread unless you ask it to. So nothing happens at "the same time" -- one function runs, then the other. What problems are you running into?
Thank you for reply. I'm calling each update/delete function as a separate Deferred, no explicit threads on those functions. OK than, let's complicated it a bit with "dictionary of lists of dictionaries" instead of simple dictionary and let's put it all under reactor(). There are 2 incoming requests: one to "update aa1=aval100" and another to "delete 'a1'": a={ 'a1':[{'aa1':'aval1'},{'bb1':'bval1'}], 'a2':[{'aa2':'aval1'},{'bb2':'bval2'}] } So "delete 'a1'" will exec always only after "update aa1=aval100" is done? Quoting "Allen Short" <washort@twistedmatrix.com>:
On Fri, May 28, 2010 at 8:46 AM, <vitaly@synapticvision.com> wrote:
I'm looking for a Twisted based example how it should be properly done update/delete of global dictionary's SAME key:value pair from different functions all together. Or by another words, two functions are trying to update the same key:value pair of the same dictionary at the same time, than how to do it properly with Twisted?
Twisted doesn't change anything about Python in this regard. One update to the dictionary is going to happen before the other. If these updates are being done from different threads, you need some kind of synchronization to make them happen in the order you want.
However, if you aren't explicitly using threads in your code, remember that Twisted doesn't run your code outside the main thread unless you ask it to. So nothing happens at "the same time" -- one function runs, then the other.
What problems are you running into?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Fri, 28 May 2010 vitaly@synapticvision.com wrote:
Hi,
I'm looking for a Twisted based example how it should be properly done update/delete of global dictionary's SAME key:value pair from different functions all together. Or by another words, two functions are trying to update the same key:value pair of the same dictionary at the same time, than how to do it properly with Twisted?
Thanks a lot.
Vitaly, As a relative newbie to both Twisted and Python (I wish I had more time to play with it, as there are lots of things I don't really understand yet, but other work intervenes), I *think* the whole point of the Twisted reactor is to serialize the callbacks. So two functions *can't* be trying to update the same Python object (dictionary or anything else, global or local) at the same time. While one thread has control, it can do anything it wants to the global dictionary, and no other function can see anything but the final result. So "just do it"! This assumes you aren't using threads, which are *not* compatible with Twisted anyway. The only more complicated scenario I can see is if some desired process uses the current value in a dictionary to initiate a request to something else (for example, a TCP/IP messages to another system) which returns its result in another callback which then stores the result (or some value that depends on the result) in the same dictionary element. If two such sequences can overlap at the same time, but using the initial value of dictionary element in the 2nd sequence would produce an incorrect final result, but instead it should be defered until the 1st sequence completes, then you would need to use some sort of locking mechanism. A very simple mechanism would be to add a boolean to each object (i.e. each dictionary value.) When the first callback executes, it checks the boolean. If set, it waits by calling reactor.callLater, passing itself as the callback. If the boolean is clear, it sets it and initiates the processing. When the final callback executes, it sets the new value in the dictionary and clears the boolean. More sophisticated mechanisms could use an exponential waiting period for the reactor.callLater time; add a timeout (producing an error if the whole process takes too long, perhaps saving the initial value so the process could be rolled back if it fails); replace the callLater's with a different deferal mechanism that could be called immediately when the lock gets released; check for deadlocks amoung multiple locks, etc. There may very well already exist a Twisted package for doing all this already (complete with debugging hooks, statistics, etc.) Someone with more Twisted experience could probably point you right at it. If no such package exists, I think it would be a very nice thing to have. If people wouldn't mind waiting for a very long (possibly infinite) time for it, I could probably be coerced into taking a stab at producing one. -- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539
hi, Callbacks serialization is clear to me. I'll try to rephrase the question: two requests are incoming for reactor() to handle. Each request has its own chain of callbacks. At some point one of the requests need to update the same global data structure, and another request need to delete from the same global data structure. So theoretically, correct me if I wrong, will be a data collision. If it is, than how can I prevent it? Example please. thank you. Quoting "John Santos" <JOHN@egh.com>:
On Fri, 28 May 2010 vitaly@synapticvision.com wrote:
Hi,
I'm looking for a Twisted based example how it should be properly done update/delete of global dictionary's SAME key:value pair from different functions all together. Or by another words, two functions are trying to update the same key:value pair of the same dictionary at the same time, than how to do it properly with Twisted?
Thanks a lot.
Vitaly,
As a relative newbie to both Twisted and Python (I wish I had more time to play with it, as there are lots of things I don't really understand yet, but other work intervenes), I *think* the whole point of the Twisted reactor is to serialize the callbacks.
So two functions *can't* be trying to update the same Python object (dictionary or anything else, global or local) at the same time. While one thread has control, it can do anything it wants to the global dictionary, and no other function can see anything but the final result.
So "just do it"!
This assumes you aren't using threads, which are *not* compatible with Twisted anyway.
The only more complicated scenario I can see is if some desired process uses the current value in a dictionary to initiate a request to something else (for example, a TCP/IP messages to another system) which returns its result in another callback which then stores the result (or some value that depends on the result) in the same dictionary element. If two such sequences can overlap at the same time, but using the initial value of dictionary element in the 2nd sequence would produce an incorrect final result, but instead it should be defered until the 1st sequence completes, then you would need to use some sort of locking mechanism.
A very simple mechanism would be to add a boolean to each object (i.e. each dictionary value.) When the first callback executes, it checks the boolean. If set, it waits by calling reactor.callLater, passing itself as the callback. If the boolean is clear, it sets it and initiates the processing. When the final callback executes, it sets the new value in the dictionary and clears the boolean.
More sophisticated mechanisms could use an exponential waiting period for the reactor.callLater time; add a timeout (producing an error if the whole process takes too long, perhaps saving the initial value so the process could be rolled back if it fails); replace the callLater's with a different deferal mechanism that could be called immediately when the lock gets released; check for deadlocks amoung multiple locks, etc. There may very well already exist a Twisted package for doing all this already (complete with debugging hooks, statistics, etc.) Someone with more Twisted experience could probably point you right at it. If no such package exists, I think it would be a very nice thing to have. If people wouldn't mind waiting for a very long (possibly infinite) time for it, I could probably be coerced into taking a stab at producing one.
-- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Am 28.05.2010, 20:34 Uhr, schrieb <vitaly@synapticvision.com>:
hi,
Callbacks serialization is clear to me.
I'll try to rephrase the question: two requests are incoming for reactor() to handle. Each request has its own chain of callbacks. At some point one of the requests need to update the same global data structure, and another request need to delete from the same global data structure. So theoretically, correct me if I wrong, will be a data collision. If it is, than how can I prevent it? Example please.
It will go like this: reactor loop: 1) receive Event A - fire deferred callback - deferred callback does data[key] = value 2) receive Event B - fire deferred callback - del data[key] So one event is processed after another. Determining if it's a data collision is up to you. E.g. you could use something like a revision number. Each time you change the dictionary, increase it by one. Then if you try to change the dictionary and the client gave a different revision id then the last one, you know there is a collision. What you do sounds very much like database transactions. Those are not easy to do. One example is the ZODB which can do what you want. For example via BTrees. But I guess there are more lightweight systems which do what you want, too. It all depends where you want to go with your app. That's up to you though and not really a question in regards to twisted or networking. -Matthias
Of course its up to me to determine if it's a data collision or not, but I was thinking may be Twisted can provide me with some already existing mechanism of its own. After all its already a pretty massive and complicated framework with tons of capabilities ... But I'd like to get back to the point of the question: if at some point EventA callbacks and EventB callbacks need to work on the same global variable, than what would be a Twisted way (if exist) to prevent data collision ? Thank you. Quoting Nitro <nitro@dr-code.org>:
Am 28.05.2010, 20:34 Uhr, schrieb <vitaly@synapticvision.com>:
hi,
Callbacks serialization is clear to me.
I'll try to rephrase the question: two requests are incoming for reactor() to handle. Each request has its own chain of callbacks. At some point one of the requests need to update the same global data structure, and another request need to delete from the same global data structure. So theoretically, correct me if I wrong, will be a data collision. If it is, than how can I prevent it? Example please.
It will go like this:
reactor loop:
1) receive Event A - fire deferred callback - deferred callback does data[key] = value 2) receive Event B - fire deferred callback - del data[key]
So one event is processed after another. Determining if it's a data collision is up to you. E.g. you could use something like a revision number. Each time you change the dictionary, increase it by one. Then if you try to change the dictionary and the client gave a different revision id then the last one, you know there is a collision. What you do sounds very much like database transactions. Those are not easy to do. One example is the ZODB which can do what you want. For example via BTrees. But I guess there are more lightweight systems which do what you want, too. It all depends where you want to go with your app. That's up to you though and not really a question in regards to twisted or networking.
-Matthias
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Fri, May 28, 2010 at 12:10 PM, <vitaly@synapticvision.com> wrote:
Of course its up to me to determine if it's a data collision or not, but I was thinking may be Twisted can provide me with some already existing mechanism of its own. After all its already a pretty massive and complicated framework with tons of capabilities ...
But I'd like to get back to the point of the question: if at some point EventA callbacks and EventB callbacks need to work on the same global variable, than what would be a Twisted way (if exist) to prevent data collision ?
In the way that you have described it, that's a question for your code. In the general case, you don't know what order callbacks on two chains will be called in. The advantage of async is that you can use a standard Python type as a mutex, and relinquish control if it's not ready. Set a variable when one chain is working, and if another chain encounters that, have it try later by creating and retuning a new Deferred with its work on that callback, or with reactor.callLater().
Is it acceptable to post job openings to this list?
TelTech Systems is hiring! We are looking for candidates with a strong background in network programming using Twisted. Telecommunications experience (particularly VoIP) is also important. We use SER+SEMS for the most part. Pluses include knowledge of PHP, JavaScript, JQuery, and other front- end technologies. To learn more about our company and products, visit teltechcorp.com, trapcall.com , and spoofcard.com. This is a full-time position, but contractors with the right skill set may be considered. Please feel free to post to the list or reply to me personally with any questions. Matthew Williams TelTech Systems Inc.
participants (7)
-
Allen Short -
Itamar Turner-Trauring -
John Santos -
Karl Anderson -
Matthew Williams -
Nitro -
vitaly＠synapticvision.com