Starting a new thread
Hello all. It occurs to me that creating threads in Python is more awkward than it needs to be. Every time I want to start a new thread, I end up writing the same thing over and over again: def target(*args, **kwds): ... t = threading.Thread(target = target, args = <something>, kwargs= <something>) t.start() This becomes especially repetitive when calling a target function that only makes sense when run in a new thread, such as a timer. In my own code, I’ve taken to decorating functions that always run in new threads. Then I can call the function using the usual function call syntax, and have the new thread returned to me. With the decorator, the code reads instead: @threaded def target(*args, **kwds): … t = target(*args, **kwds) This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I can concentrate more on what my code does and less in how it does it. It feels like the right place for this decorator is the standard library, so I’ve created PR #91878 for it. @rhettinger suggests that this is a bit premature, and that we should discuss it here first. Thoughts? Cheers, Barney.
On Tue, 10 May 2022 at 16:31, Barney Stratford <barney_stratford@fastmail.fm> wrote:
I like this. I'd probably use it if it existed. Your example isn't particularly compelling, though - calling the function "target" makes sense when it's the target of the thread, but not so much when it's the thread itself. Could you give a motivating example based on real code, rather than just the "this is what the syntax would look like" as you did above? Ideally showing what it would look like with and without the decorator? Things I *think* i get, but I'd like to see clearly in an example, include: 1. Does t = target(...) start the thread? I think it does. 2. Is it possible to create daemon threads? @threaded(daemon=True) seems plausible. I suspect your PR doesn't have this, but it wouldn't be difficult to add. But I don't actually know if it's worth adding, TBH. 3. Can you create multiple threads for the same function? I assume t1, t2, t3 = target(arg1), target(arg2), target(arg3) would work. I'm probably being dense here - I'm pretty sure your proposal actually is as simple as it looks (which is a good thing!) and I'm just over-complicating things in my head. Paul
On Tue, May 10, 2022 at 1:20 PM Paul Moore <p.f.moore@gmail.com> wrote: > On Tue, 10 May 2022 at 16:31, Barney Stratford > <barney_stratford@fastmail.fm> wrote: > > > > Hello all. > > > > It occurs to me that creating threads in Python is more awkward than it > needs to be. Every time I want to start a new thread, I end up writing the > same thing over and over again: > > > > def target(*args, **kwds): > > ... > > t = threading.Thread(target = target, args = <something>, kwargs= > <something>) > > t.start() > > > > This becomes especially repetitive when calling a target function that > only makes sense when run in a new thread, such as a timer. > > > > In my own code, I’ve taken to decorating functions that always run in > new threads. Then I can call the function using the usual function call > syntax, and have the new thread returned to me. With the decorator, the > code reads instead: > > > > @threaded > > def target(*args, **kwds): > > … > > t = target(*args, **kwds) > > > > This has a couple of advantages. I don’t have to import the threading > module all over my code. I can use the neater syntax of function calls. The > function’s definition makes it clear it’s returning a new thread since it’s > decorated. It gets the plumbing out of the way so I can concentrate more on > what my code does and less in how it does it. > > > > It feels like the right place for this decorator is the standard > library, so I’ve created PR #91878 for it. @rhettinger suggests that this > is a bit premature, and that we should discuss it here first. Thoughts? > > I like this. I'd probably use it if it existed. Your example isn't > particularly compelling, though - calling the function "target" makes > sense when it's the target of the thread, but not so much when it's > the thread itself. Could you give a motivating example based on real > code, rather than just the "this is what the syntax would look like" > as you did above? Ideally showing what it would look like with and > without the decorator? > > Things I *think* i get, but I'd like to see clearly in an example, include: > > 1. Does t = target(...) start the thread? I think it does. > 2. Is it possible to create daemon threads? @threaded(daemon=True) > seems plausible. I suspect your PR doesn't have this, but it wouldn't > be difficult to add. But I don't actually know if it's worth adding, > TBH. > 3. Can you create multiple threads for the same function? I assume t1, > t2, t3 = target(arg1), target(arg2), target(arg3) would work. > > I'm probably being dense here - I'm pretty sure your proposal actually > is as simple as it looks (which is a good thing!) and I'm just > over-complicating things in my head. > > Things that come to mind that should be considered here: 1) not every two liner function should be added to the stdlib and, on the other hand, 2) asyncio features `loop.run_in_executor` which does essentially the same, with more care for what happens in "real world" code, but is only usable in async code. all in all it can be useful for quick and dirty code - and maybe something more elaborate allowing control on whether to create daemon threads, or maybe something that will deal with concurrent.future.executors as well, and allow access to the function call results All in all, I find it a nice snippet, but one that is not aligned with the trend Python has been taking over the last years, facilitating the writing of "quick and dirty" scripts in contrast with application code that will take care of all possible corner-cases. As a good example of what I am talking about, we've seen the contrary movement take place, long ago, when "os.popen" was superseded by "subprocess.*": I think up to today people had not catched up with the far more complicated API of the latter. All in all, I'd eventually use this "@threaded" if it was there. > Paul > > js -><-
1. Does t = target(...) start the thread? I think it does. I think it does too. In the commonest use case, immediately after creating a thread, you start it. And if you want to delay the thread but still use the decorator, then you can do that explicitly with some locks. In fact, it’s probably better to explicitly delay execution than have hidden requirements concerning the timing of thread creation and startup.
2. Is it possible to create daemon threads? Not at the moment. I did think about this, but felt that simpler is better. Like you say, it’d be easy to add. In fact, I might just go ahead and add it to the PR in a bit. The simplest way to do it is probably to define a second decorator for daemonic threads.
There isn’t a single use case where the decorator is particularly compelling; rather, it’s syntactic sugar to hide the mechanism of thread creation so that code reads better. I could give some examples of where I used it recently if you want, but I don’t think it would be terribly illuminating. More useful might be to look at the problem from the opposite perspective: would anyone like to write result = call_function(target = foo, args = (42,), kwargs = {“bar”: 60}) in preference to result = foo(42, bar = 60) Cheers, Barney.
On Tue, May 10, 2022 at 10:34 AM Barney Stratford < barney_stratford@fastmail.fm> wrote:
If this is even to be added (i personally lean -1 on it), I suggest intentionally not supporting daemon threads. We should not encourage them to be used, they were a misfeature that in hindsight we should never have created. Daemon threads can lead to very bad surprises upon interpreter finalization - an unfixable problem given how daemon threads are defined to behave.
This is my take as well. I don't like calling code to hide the fact that a thread is being spawned. Use this decorator and if you fail to give the callable a name communicating that it spawns and returns a thread, you will have surprised readers of the calling code. A nicer design pattern is to explicitly manage threads. Use concurrent.futures.ThreadPoolExecutor. Or use the async stuff that Joao mentioned or similar libraries. I think we already provide decent batteries with the threading APIs. -gps
On Tue, 10 May 2022 16:12:13 +0100 Barney Stratford <barney_stratford@fastmail.fm> wrote:
This seems like an attractive nuisance. Creating threads comes with its own constraints and subtleties. I don't think it really helps users to hide it behind a "regular" function call. Like Greg I'm leaning towards -1 on this. Regards Antoine.
It seems like the consensus is that this is a good idea, but it’s the wrong good idea. Should I cancel the PR or should we try to make it into a better good idea? Cheers, Barney.
It’s definitely too early for a PR, so if you already have one (I didn’t see one linked to this thread) please close it. Then once we’ve bikeshedded the right good idea you can start a new PR. On Thu, May 12, 2022 at 12:21 Barney Stratford <barney_stratford@fastmail.fm> wrote:
-- --Guido (mobile)
On 12May2022 20:17, Barney Stratford <barney_stratford@fastmail.fm> wrote:
Why not shift slightly? As remarked, having a function automatically spawn threads can be confusing, because spawning a thread has implications for both the thread code itself and for the person calling the function. The caller might find that confusing or complex. Personally, my approach has been a tiny shim function named "bg" in my personal kit, to make it easy to spawn a regular function in a thread: T = bg(func, args....) # returns a running Thread An approach I've also seen is Celery's one of decorating a function with attributes which can dispatch it as a task, roughly: @task def regular_function(.....) ... and it can be dispatched by saying regular_function.defer(......) and variations. Some downsides to decorators include: - only decorated functions can be kicked off "automatically as threads"; that could be a good thing too - the decorator wires in the dispatch mechanism: your decorator spawns a thread, the Celery @task queues to a Celery task queue, and so forth So my personal inclination is to provide an easy to use shim for the caller, not the function. Eg: from cs.threads import bg ..... T = bg(func, args.....) or: from cs.threads import bg as bg_thread from cs.later import bg as bg_later ..... T = bg_thread(func, args......) # run in a Thread ..... R = bg_later(func, args.......) # hand off to a Later, get a Result for collection ...... bg = bg_later # specify the queuing system ...... ... do stuff via bg(), the chosen queuing system ... or whatever other queuing system you might be using. The idea here is to make it easy to submit a function to any of several things rather than decorating the function itself to submit to a now-hardwired thing. Just things to consider.... Cheers, Cameron Simpson <cs@cskk.id.au>
10.05.22 18:12, Barney Stratford пише:
This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I an concentrate more on what my code does and less in how it does it.
I do not see advantages. You definitely need to import the threading module or other module depending on the threading module in which you define the decorator. And you need to decorate the function. It will not save you a line of code. If you need to run a lot of functions in threads, note that a thread has some starting cost. Consider using concurrent.futures.ThreadPoolExecutor. If you only run few long living threads, the syntax of starting them does not matter, in comparison with the rest of your code. Also, it looks wrong to me to mix two different things: what code to execute and how to run it. If we need a neater syntax, I would propose: t = threading.start_thread(func, *args, **kwargs) But I am not sure that it is worth to add such three-line function in the stdlib.
Thinking about it, it's actually a fine design to have a decorator that turns a regular function into one that starts a thread -- from the caller's POV it's no different than having a function that explicitly starts a thread, and it could be a nice shorthand if you do this all the time. (You have to document it in either case.) That said, I don't think we need this 3-line decorator in the stdlib. --Guido On Thu, May 12, 2022 at 10:40 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Tue, 10 May 2022 at 16:31, Barney Stratford <barney_stratford@fastmail.fm> wrote:
I like this. I'd probably use it if it existed. Your example isn't particularly compelling, though - calling the function "target" makes sense when it's the target of the thread, but not so much when it's the thread itself. Could you give a motivating example based on real code, rather than just the "this is what the syntax would look like" as you did above? Ideally showing what it would look like with and without the decorator? Things I *think* i get, but I'd like to see clearly in an example, include: 1. Does t = target(...) start the thread? I think it does. 2. Is it possible to create daemon threads? @threaded(daemon=True) seems plausible. I suspect your PR doesn't have this, but it wouldn't be difficult to add. But I don't actually know if it's worth adding, TBH. 3. Can you create multiple threads for the same function? I assume t1, t2, t3 = target(arg1), target(arg2), target(arg3) would work. I'm probably being dense here - I'm pretty sure your proposal actually is as simple as it looks (which is a good thing!) and I'm just over-complicating things in my head. Paul
On Tue, May 10, 2022 at 1:20 PM Paul Moore <p.f.moore@gmail.com> wrote: > On Tue, 10 May 2022 at 16:31, Barney Stratford > <barney_stratford@fastmail.fm> wrote: > > > > Hello all. > > > > It occurs to me that creating threads in Python is more awkward than it > needs to be. Every time I want to start a new thread, I end up writing the > same thing over and over again: > > > > def target(*args, **kwds): > > ... > > t = threading.Thread(target = target, args = <something>, kwargs= > <something>) > > t.start() > > > > This becomes especially repetitive when calling a target function that > only makes sense when run in a new thread, such as a timer. > > > > In my own code, I’ve taken to decorating functions that always run in > new threads. Then I can call the function using the usual function call > syntax, and have the new thread returned to me. With the decorator, the > code reads instead: > > > > @threaded > > def target(*args, **kwds): > > … > > t = target(*args, **kwds) > > > > This has a couple of advantages. I don’t have to import the threading > module all over my code. I can use the neater syntax of function calls. The > function’s definition makes it clear it’s returning a new thread since it’s > decorated. It gets the plumbing out of the way so I can concentrate more on > what my code does and less in how it does it. > > > > It feels like the right place for this decorator is the standard > library, so I’ve created PR #91878 for it. @rhettinger suggests that this > is a bit premature, and that we should discuss it here first. Thoughts? > > I like this. I'd probably use it if it existed. Your example isn't > particularly compelling, though - calling the function "target" makes > sense when it's the target of the thread, but not so much when it's > the thread itself. Could you give a motivating example based on real > code, rather than just the "this is what the syntax would look like" > as you did above? Ideally showing what it would look like with and > without the decorator? > > Things I *think* i get, but I'd like to see clearly in an example, include: > > 1. Does t = target(...) start the thread? I think it does. > 2. Is it possible to create daemon threads? @threaded(daemon=True) > seems plausible. I suspect your PR doesn't have this, but it wouldn't > be difficult to add. But I don't actually know if it's worth adding, > TBH. > 3. Can you create multiple threads for the same function? I assume t1, > t2, t3 = target(arg1), target(arg2), target(arg3) would work. > > I'm probably being dense here - I'm pretty sure your proposal actually > is as simple as it looks (which is a good thing!) and I'm just > over-complicating things in my head. > > Things that come to mind that should be considered here: 1) not every two liner function should be added to the stdlib and, on the other hand, 2) asyncio features `loop.run_in_executor` which does essentially the same, with more care for what happens in "real world" code, but is only usable in async code. all in all it can be useful for quick and dirty code - and maybe something more elaborate allowing control on whether to create daemon threads, or maybe something that will deal with concurrent.future.executors as well, and allow access to the function call results All in all, I find it a nice snippet, but one that is not aligned with the trend Python has been taking over the last years, facilitating the writing of "quick and dirty" scripts in contrast with application code that will take care of all possible corner-cases. As a good example of what I am talking about, we've seen the contrary movement take place, long ago, when "os.popen" was superseded by "subprocess.*": I think up to today people had not catched up with the far more complicated API of the latter. All in all, I'd eventually use this "@threaded" if it was there. > Paul > > js -><-
1. Does t = target(...) start the thread? I think it does. I think it does too. In the commonest use case, immediately after creating a thread, you start it. And if you want to delay the thread but still use the decorator, then you can do that explicitly with some locks. In fact, it’s probably better to explicitly delay execution than have hidden requirements concerning the timing of thread creation and startup.
2. Is it possible to create daemon threads? Not at the moment. I did think about this, but felt that simpler is better. Like you say, it’d be easy to add. In fact, I might just go ahead and add it to the PR in a bit. The simplest way to do it is probably to define a second decorator for daemonic threads.
There isn’t a single use case where the decorator is particularly compelling; rather, it’s syntactic sugar to hide the mechanism of thread creation so that code reads better. I could give some examples of where I used it recently if you want, but I don’t think it would be terribly illuminating. More useful might be to look at the problem from the opposite perspective: would anyone like to write result = call_function(target = foo, args = (42,), kwargs = {“bar”: 60}) in preference to result = foo(42, bar = 60) Cheers, Barney.
On Tue, May 10, 2022 at 10:34 AM Barney Stratford < barney_stratford@fastmail.fm> wrote:
If this is even to be added (i personally lean -1 on it), I suggest intentionally not supporting daemon threads. We should not encourage them to be used, they were a misfeature that in hindsight we should never have created. Daemon threads can lead to very bad surprises upon interpreter finalization - an unfixable problem given how daemon threads are defined to behave.
This is my take as well. I don't like calling code to hide the fact that a thread is being spawned. Use this decorator and if you fail to give the callable a name communicating that it spawns and returns a thread, you will have surprised readers of the calling code. A nicer design pattern is to explicitly manage threads. Use concurrent.futures.ThreadPoolExecutor. Or use the async stuff that Joao mentioned or similar libraries. I think we already provide decent batteries with the threading APIs. -gps
On Tue, 10 May 2022 16:12:13 +0100 Barney Stratford <barney_stratford@fastmail.fm> wrote:
This seems like an attractive nuisance. Creating threads comes with its own constraints and subtleties. I don't think it really helps users to hide it behind a "regular" function call. Like Greg I'm leaning towards -1 on this. Regards Antoine.
It seems like the consensus is that this is a good idea, but it’s the wrong good idea. Should I cancel the PR or should we try to make it into a better good idea? Cheers, Barney.
It’s definitely too early for a PR, so if you already have one (I didn’t see one linked to this thread) please close it. Then once we’ve bikeshedded the right good idea you can start a new PR. On Thu, May 12, 2022 at 12:21 Barney Stratford <barney_stratford@fastmail.fm> wrote:
-- --Guido (mobile)
On 12May2022 20:17, Barney Stratford <barney_stratford@fastmail.fm> wrote:
Why not shift slightly? As remarked, having a function automatically spawn threads can be confusing, because spawning a thread has implications for both the thread code itself and for the person calling the function. The caller might find that confusing or complex. Personally, my approach has been a tiny shim function named "bg" in my personal kit, to make it easy to spawn a regular function in a thread: T = bg(func, args....) # returns a running Thread An approach I've also seen is Celery's one of decorating a function with attributes which can dispatch it as a task, roughly: @task def regular_function(.....) ... and it can be dispatched by saying regular_function.defer(......) and variations. Some downsides to decorators include: - only decorated functions can be kicked off "automatically as threads"; that could be a good thing too - the decorator wires in the dispatch mechanism: your decorator spawns a thread, the Celery @task queues to a Celery task queue, and so forth So my personal inclination is to provide an easy to use shim for the caller, not the function. Eg: from cs.threads import bg ..... T = bg(func, args.....) or: from cs.threads import bg as bg_thread from cs.later import bg as bg_later ..... T = bg_thread(func, args......) # run in a Thread ..... R = bg_later(func, args.......) # hand off to a Later, get a Result for collection ...... bg = bg_later # specify the queuing system ...... ... do stuff via bg(), the chosen queuing system ... or whatever other queuing system you might be using. The idea here is to make it easy to submit a function to any of several things rather than decorating the function itself to submit to a now-hardwired thing. Just things to consider.... Cheers, Cameron Simpson <cs@cskk.id.au>
10.05.22 18:12, Barney Stratford пише:
This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I an concentrate more on what my code does and less in how it does it.
I do not see advantages. You definitely need to import the threading module or other module depending on the threading module in which you define the decorator. And you need to decorate the function. It will not save you a line of code. If you need to run a lot of functions in threads, note that a thread has some starting cost. Consider using concurrent.futures.ThreadPoolExecutor. If you only run few long living threads, the syntax of starting them does not matter, in comparison with the rest of your code. Also, it looks wrong to me to mix two different things: what code to execute and how to run it. If we need a neater syntax, I would propose: t = threading.start_thread(func, *args, **kwargs) But I am not sure that it is worth to add such three-line function in the stdlib.
Thinking about it, it's actually a fine design to have a decorator that turns a regular function into one that starts a thread -- from the caller's POV it's no different than having a function that explicitly starts a thread, and it could be a nice shorthand if you do this all the time. (You have to document it in either case.) That said, I don't think we need this 3-line decorator in the stdlib. --Guido On Thu, May 12, 2022 at 10:40 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (8)
-
Antoine Pitrou -
Barney Stratford -
Cameron Simpson -
Gregory P. Smith -
Guido van Rossum -
Joao S. O. Bueno -
Paul Moore -
Serhiy Storchaka