Adding a `wait_in_order` function to concurrent.futures
Hello list, it's my first email here, so let me know if this is in any way out of order. I've been working a lot with concurrent.futures lately. I find both `wait` and `as_completed` very useful, but something I'm usually lacking is a `wait` version where the order of the futures is preserved. I quickly hacked a proof of concept to discuss with you and see if anybody else would find it useful: https://gist.github.com/santiagobasulto/10b689ba5fcadf307ffc5cd4f4ae00ec I also realized that the implementation of `ThreadPoolExecutor.map` could be greatly simplified with this function as part of the stdlib. What do you think? Thanks!
This seems useful to me. But I think it would be better if made more eager. Let's say you have tasks A, B, C, D, E to process. If they complete in order E, D, C, B, A then of course you can't do any in-order processing until they've all completed. But what if they finish in order A, B, E, D, C. I'd like to be able to start working on the A and B results right away. I only skimmed the code, but I think the proposed implementation doesn't support that. On Sat, Jun 26, 2021, 10:28 AM <santiago.basulto@gmail.com> wrote:
Hey David, great to find you here!! I really like what you're proposing. I could make it a generator instead of returning a list. I returned a list because wait returns sets and I thought I would keep returning collections. I'll play around and send a new proposal.
This seems useful to me. But I think it would be better if made more eager. Let's say you have tasks A, B, C, D, E to process. If they complete in order E, D, C, B, A then of course you can't do any in-order processing until they've all completed. But what if they finish in order A, B, E, D, C. I'd like to be able to start working on the A and B results right away. I only skimmed the code, but I think the proposed implementation doesn't support that. On Sat, Jun 26, 2021, 10:28 AM <santiago.basulto@gmail.com> wrote:
Hey David, great to find you here!! I really like what you're proposing. I could make it a generator instead of returning a list. I returned a list because wait returns sets and I thought I would keep returning collections. I'll play around and send a new proposal.
participants (2)
-
David Mertz -
santiago.basulto＠gmail.com