Seeking advice on locking iterators

[Ype Kingma]

Gonçalo Rodrigues schreef:

> Hi,
> 
> My problem is the following: I have a top object Application, roughly
> corresponding to the main thread, that spawns some child threads -
> objects on their own sake, deriving from the Thread class in the
> threading module. They communicate when they need to via queues in the
> usual way. So far so nice.
> 
> The problem is that those same child objects expose some public
> attributes (or properties, or methods, does not matter) that the
> Application object may need to change/call and doing the change via the
> queue mechanism is conceptually wrong (and a real PITA). The solution is
> obvious: wrap the attributes with locks around it. What is not so
> obvious is when these attributes are complex objects themselves, in
> particular they are iterables. In order to call iter() on these objects
> safely I coded the following helper class:
> 
> #WARNING: Brittle - use this at your own risk.
> class TIter(object):
>     """The TIter helper class, wrapping an iterator for thread safe
> acess."""
> 
>     def __init__(self, lock, iterator):
>         super(TIter, self).__init__(lock, iterator)
>         self.__lock = lock
>         self.__iterator = iter(iterator)
>         #Acquire lock => You cannot change the underlying structure
>         #while iter not exhausted.
>         self.__lock.acquire()
>         self.__acquired = 1
> 
>     #Iterator protocol.
>     def __iter__(self):
>         return self
> 
>     def next(self):
>         try:
>             ret = self.__iterator.next()
>         except StopIteration:
>             self.__lock.release()
>             self.__acquired = 0
>             raise StopIteration
>         else:
>             return ret
> 
>     def __del__(self):
>         if self.__acquired:
>             self.__lock.release()
> 
> The idea is that for the wrapping of a given attribute one has an
> associated lock (a reentrant lock, usually) that is passed to the ITer
> constructor. I have to ensure two things, though:
> 
> * The lock is acquired while the iterator is alive.
> * The lock is released when the iterator is disposed of.
> 
> So, finally, my question is: Is the above enough to ensure this? Can I

In general, no. Some reasons:
One can never be sure from the code shown when the __del__ method will be 
called, since this depends on when the last reference to the iterator goes 
out of scope.
Although CPython more or less guarantees that the __del__ method will be 
called quite soon after the last reference goes out of scope, the language 
by itself does not make such a guarantee. In Jython, you are at the mercy
of a separate a garbage collecting thread in the JVM to call the __del__ 
method, ie. it might take more than a few million CPU cycles before the 
__del__ method is called.

> be sure that __del__ is called if the iterator is garbage collected
> before being exhausted? Or is there a better (e.g. safer) mechanism for
> this?


One safe way is:

yourObject.lock()
try:
    # use an iterator on yourObject.
finally:
    yourObject.unlock()
    # and don't use the iterator again.

Ie. I don't see the point of locking the iterator, as you seem to want to
lock an object, ie. one the attributes you mentioned above.

The lock() and unlock() methods need to be added to the class of yourObject 
for this to work.

Have fun,
Ype


-- 

email at xs4all.nl