[Python-ideas] Thread-safe generators

[Victor Stinner]

Thread safety is very complex and has an impact on performance. I dislike
the idea of providing such property to generators which can have a complex
next method.

IMHO it's better to put a generator in wrapper which adds thread safety.

What do you think?

Victor

Le 14 avr. 2017 18:48, "Serhiy Storchaka" <storchaka at gmail.com> a écrit :

> When use a generator from different threads you can get a ValueError
> "generator already executing". Getting this exception with the single
> thread is a programming error, it in case of different threads it could be
> possible to wait until other thread finish executing the generator. The
> generator can be made thread-safe after wrapping it in a class that acquire
> a lock before calling the generator's __next__ method (for example see
> [1]). But this is not very efficient of course.
>
> I wondering if it is worth to add support of thread-safe generators in the
> stdlib. Either by providing standard decorator (written in C for
> efficiency), or adding threading support just in the generator object. The
> latter may need increasing the size of the generator object for a lock and
> thread identifier (but may be GIL is enough), but should not affect
> performance since locking is used only when you faced with a generator
> running in other thread.
>
> This topic already was raised on Python-Dev [2] but didn't moved too much.
> There are a number of StackOverflow questions about threads and generators.
> We have already encountered this issue in the stdlib. Once in regrtest with
> the -j option ([3], [4]), other time after reimplementing
> tempfile._RandomNameSequence as a generator [5].
>
> [1] http://anandology.com/blog/using-iterators-and-generators/
> [2] https://mail.python.org/pipermail/python-dev/2004-February/042390.html
> [3] https://bugs.python.org/issue7996
> [4] https://bugs.python.org/issue15320
> [5] https://bugs.python.org/issue30030
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170416/c5298ebc/attachment.html>