Adding a `wait_in_order` function to concurrent.futures
Hello list, it's my first email here, so let me know if this is in any way out of order. I've been working a lot with concurrent.futures lately. I find both `wait` and `as_completed` very useful, but something I'm usually lacking is a `wait` version where the order of the futures is preserved. I quickly hacked a proof of concept to discuss with you and see if anybody else would find it useful: https://gist.github.com/santiagobasulto/10b689ba5fcadf307ffc5cd4f4ae00ec I also realized that the implementation of `ThreadPoolExecutor.map` could be greatly simplified with this function as part of the stdlib. What do you think? Thanks!
This seems useful to me. But I think it would be better if made more
eager. Let's say you have tasks A, B, C, D, E to process.
If they complete in order E, D, C, B, A then of course you can't do any
in-order processing until they've all completed.
But what if they finish in order A, B, E, D, C. I'd like to be able to
start working on the A and B results right away. I only skimmed the code,
but I think the proposed implementation doesn't support that.
On Sat, Jun 26, 2021, 10:28 AM
Hello list, it's my first email here, so let me know if this is in any way out of order.
I've been working a lot with concurrent.futures lately. I find both `wait` and `as_completed` very useful, but something I'm usually lacking is a `wait` version where the order of the futures is preserved. I quickly hacked a proof of concept to discuss with you and see if anybody else would find it useful: https://gist.github.com/santiagobasulto/10b689ba5fcadf307ffc5cd4f4ae00ec
I also realized that the implementation of `ThreadPoolExecutor.map` could be greatly simplified with this function as part of the stdlib.
What do you think? Thanks! _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VCXVVT... Code of Conduct: http://python.org/psf/codeofconduct/
Hey David, great to find you here!! I really like what you're proposing. I could make it a generator instead of returning a list. I returned a list because wait returns sets and I thought I would keep returning collections. I'll play around and send a new proposal.
Nice to see you on the list also, Santiago.
The generator is my impression of what is most useful. But someone else
might opine differently, of course.
On Sat, Jun 26, 2021, 1:50 PM
Hey David, great to find you here!!
I really like what you're proposing. I could make it a generator instead of returning a list. I returned a list because wait returns sets and I thought I would keep returning collections. I'll play around and send a new proposal. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/I42UHE... Code of Conduct: http://python.org/psf/codeofconduct/
participants (2)
-
David Mertz -
santiago.basulto＠gmail.com