[Python-ideas] Thread-safe generators
storchaka at gmail.com
Fri Apr 14 12:47:24 EDT 2017
When use a generator from different threads you can get a ValueError
"generator already executing". Getting this exception with the single
thread is a programming error, it in case of different threads it could
be possible to wait until other thread finish executing the generator.
The generator can be made thread-safe after wrapping it in a class that
acquire a lock before calling the generator's __next__ method (for
example see ). But this is not very efficient of course.
I wondering if it is worth to add support of thread-safe generators in
the stdlib. Either by providing standard decorator (written in C for
efficiency), or adding threading support just in the generator object.
The latter may need increasing the size of the generator object for a
lock and thread identifier (but may be GIL is enough), but should not
affect performance since locking is used only when you faced with a
generator running in other thread.
This topic already was raised on Python-Dev  but didn't moved too
much. There are a number of StackOverflow questions about threads and
generators. We have already encountered this issue in the stdlib. Once
in regrtest with the -j option (, ), other time after
reimplementing tempfile._RandomNameSequence as a generator .
More information about the Python-ideas