FWIW, I agree with Andrew here. Being able to swap a ThreadPoolExecutor or ProcessPoolExecutor with a serial version using the same API can have benefits in various situations. One is easier debugging (in case the problem you have to debug isn't a race condition, of course :-)). Another is writing a library a command-line tool or library where the final decision of whether to parallelize execution (e.g. through a command-line option for a CLI tool) is up to the user, not the library developer.
After Andrew explained his own use case for it with isolating bugs to ensure that the issue wasn't occurring as a result of parallelism, threads, processes, etc; I certainly can see how it would be useful. I could also see a use case in a CLI tool for a conveniently similar parallel and non-parallel version, although I'd likely prefer just having an entirely separate implementation. Particularly if the parallel version includes diving a large, computationally intensive task into many sub-tasks (more common for PPE), that seems like it could result in significant additional unneeded overhead for the non-parallel version. I think at this point, it's potential usefulness is clear though. But, IMO, the main question is now the following: would it be better *initially* placed in the standard library or on PyPI (which could eventually transition into stdlib if it sees widespread usage)?
It seems there are two possible design decisions for a serial executor: - one is to execute the task immediately on `submit()` - another is to execute the task lazily on `result()`
To me, it seems like the latter would be more useful for debugging purposes, since that would be more similar to how the submitted task/function would actually be executed. ``submit()`` could potentially "fake" the process of scheduling the execution of the function, but without directly executing it; perhaps with something like this: ``executor.submit()`` => create a pending item => add pending item to dict => add callable to call queue => fut.result() => check if in pending items => get from top of call queue => run work item => pop from pending items => set result/exception => return result (skip last three if fut is not in/associated with a pending item). IMO, that would be similar enough to the general workflow followed in the executors without any of the parallelization. On Sun, Feb 16, 2020 at 6:29 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
On Feb 15, 2020, at 13:36, Jonathan Crall <erotemic@gmail.com> wrote:
Also, there is no duck-typed class that behaves like an executor, but does its processing in serial. Often times a develop will want to run a task in parallel, but depending on the environment they may want to disable
On Sat, 15 Feb 2020 14:16:39 -0800 Andrew Barnert via Python-ideas <python-ideas@python.org> wrote: threading or process execution. To address this I use a utility called a `SerialExecutor` which shares an API with ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially in the same python thread:
This makes sense. I think most futures-and-executors frameworks in other
languages have a serial/synchronous/immediate/blocking executor just like this. (And the ones that don’t, it’s usually because they have a different way to specify the same functionality—e.g., in C++, you only use executors via the std::async function, and you can just pass a launch option instead of an executor to run synchronously.)
FWIW, I agree with Andrew here. Being able to swap a ThreadPoolExecutor or ProcessPoolExecutor with a serial version using the same API can have benefits in various situations. One is easier debugging (in case the problem you have to debug isn't a race condition, of course :-)). Another is writing a library a command-line tool or library where the final decision of whether to parallelize execution (e.g. through a command-line option for a CLI tool) is up to the user, not the library developer.
It seems there are two possible design decisions for a serial executor: - one is to execute the task immediately on `submit()` - another is to execute the task lazily on `result()`
This could for example be controlled by a constructor argument to SerialExecutor.
Regards
Antoine.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PCDN4J... Code of Conduct: http://python.org/psf/codeofconduct/