Seeking advice on locking iterators

[Gonçalo Rodrigues]

On Thu, 14 Nov 2002 19:29:49 +0100, Ype Kingma <ykingma at accessforall.nl>
wrote:

>Gonçalo Rodrigues schreef:
>
>> Hi,
>> 
>> My problem is the following: I have a top object Application, roughly
>> corresponding to the main thread, that spawns some child threads -
>> objects on their own sake, deriving from the Thread class in the
>> threading module. They communicate when they need to via queues in the
>> usual way. So far so nice.
>> 
>> The problem is that those same child objects expose some public
>> attributes (or properties, or methods, does not matter) that the
>> Application object may need to change/call and doing the change via the
>> queue mechanism is conceptually wrong (and a real PITA). The solution is
>> obvious: wrap the attributes with locks around it. What is not so
>> obvious is when these attributes are complex objects themselves, in
>> particular they are iterables. In order to call iter() on these objects
>> safely I coded the following helper class:
>> 
>> #WARNING: Brittle - use this at your own risk.
>> class TIter(object):
>>     """The TIter helper class, wrapping an iterator for thread safe
>> acess."""
>> 
>>     def __init__(self, lock, iterator):
>>         super(TIter, self).__init__(lock, iterator)
>>         self.__lock = lock
>>         self.__iterator = iter(iterator)
>>         #Acquire lock => You cannot change the underlying structure
>>         #while iter not exhausted.
>>         self.__lock.acquire()
>>         self.__acquired = 1
>> 
>>     #Iterator protocol.
>>     def __iter__(self):
>>         return self
>> 
>>     def next(self):
>>         try:
>>             ret = self.__iterator.next()
>>         except StopIteration:
>>             self.__lock.release()
>>             self.__acquired = 0
>>             raise StopIteration
>>         else:
>>             return ret
>> 
>>     def __del__(self):
>>         if self.__acquired:
>>             self.__lock.release()
>> 
>> The idea is that for the wrapping of a given attribute one has an
>> associated lock (a reentrant lock, usually) that is passed to the ITer
>> constructor. I have to ensure two things, though:
>> 
>> * The lock is acquired while the iterator is alive.
>> * The lock is released when the iterator is disposed of.
>> 
>> So, finally, my question is: Is the above enough to ensure this? Can I
>
>In general, no. Some reasons:
>One can never be sure from the code shown when the __del__ method will be 
>called, since this depends on when the last reference to the iterator goes 
>out of scope.
>Although CPython more or less guarantees that the __del__ method will be 
>called quite soon after the last reference goes out of scope, the language 
>by itself does not make such a guarantee. In Jython, you are at the mercy
>of a separate a garbage collecting thread in the JVM to call the __del__ 
>method, ie. it might take more than a few million CPU cycles before the 
>__del__ method is called.
>
>> be sure that __del__ is called if the iterator is garbage collected
>> before being exhausted? Or is there a better (e.g. safer) mechanism for
>> this?
>
>
>One safe way is:
>
>yourObject.lock()
>try:
>    # use an iterator on yourObject.
>finally:
>    yourObject.unlock()
>    # and don't use the iterator again.
>
>Ie. I don't see the point of locking the iterator, as you seem to want to
>lock an object, ie. one the attributes you mentioned above.
>

I want to lock the iterator because I want the wrapped object to have
the same interface as the object it wraps.

>The lock() and unlock() methods need to be added to the class of yourObject 
>for this to work.

Anyway I think I have found a solution: The problem lies in the fact
that the object I am iterating over and the iterator itself can be
different objects => while the iterator acquires the lock it may get
garbage collected before it releases it. Therefore I just wrap the
object in a way that the wrapper is its own iterator and the problem
disappears.
>
>Have fun,

I am. Python *is* fun.

>Ype

With my best regards,
G. Rodrigues