[Python-ideas] Thread-safe generators

Serhiy Storchaka storchaka at gmail.com
Fri Apr 14 12:47:24 EDT 2017

When use a generator from different threads you can get a ValueError 
"generator already executing". Getting this exception with the single 
thread is a programming error, it in case of different threads it could 
be possible to wait until other thread finish executing the generator. 
The generator can be made thread-safe after wrapping it in a class that 
acquire a lock before calling the generator's __next__ method (for 
example see [1]). But this is not very efficient of course.

I wondering if it is worth to add support of thread-safe generators in 
the stdlib. Either by providing standard decorator (written in C for 
efficiency), or adding threading support just in the generator object. 
The latter may need increasing the size of the generator object for a 
lock and thread identifier (but may be GIL is enough), but should not 
affect performance since locking is used only when you faced with a 
generator running in other thread.

This topic already was raised on Python-Dev [2] but didn't moved too 
much. There are a number of StackOverflow questions about threads and 
generators. We have already encountered this issue in the stdlib. Once 
in regrtest with the -j option ([3], [4]), other time after 
reimplementing tempfile._RandomNameSequence as a generator [5].

[1] http://anandology.com/blog/using-iterators-and-generators/
[2] https://mail.python.org/pipermail/python-dev/2004-February/042390.html
[3] https://bugs.python.org/issue7996
[4] https://bugs.python.org/issue15320
[5] https://bugs.python.org/issue30030

More information about the Python-ideas mailing list