@run_as_thread decorator
The same thing could be done for multiprocessing module. Would this be acceptable for inclusion? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
On Sat, Mar 5, 2011 at 11:21 PM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
So basically: def run_as_thread(f): @functools.wraps(f): def wrapped(*args, **kwds): t = threading.Thread(target=f, args=args, kwds=kwds) t.start() return t return wrapped Something like that would make defining worker threads *really* easy. A similar idea may make sense as an addition to the concurrent.futures.Executor ABC. For example: def autosubmit(self): def decorator(f): @functools.wraps(f): def wrapped(*args, **kwds): return self.submit(f, *args, **kwds) return wrapped return decorator Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 6 Mar 2011 00:10:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't really agree. First, as you guess, there's already a rather obvious one-liner: threading.Thread(target=f).start() Second, any decorator that implicitly spawns a thread is a very bad idea (especially when used at module level...). I'm rather opposed to this, it's a useless addition to the API with no real point. Calling the Thread() constructor works basically ok. Regards Antoine.
On 05.03.2011 15:50, Antoine Pitrou wrote:
It doesn't spawn the thread on definition, but on function call time. That's not as bad, but I agree that it is too magical for the stdlib.
I'm rather opposed to this, it's a useless addition to the API with no real point. Calling the Thread() constructor works basically ok.
Problem is, the one-liner doesn't give you a reference to the Thread object. Georg
On 2011-03-05, at 14:21 , Giampaolo Rodolà wrote: >>>> import time, threading >>>> >>>> @threading.run_as_thread > ... def foo(): > ... time.sleep(100) > ... return 1 > ... >>>> t = foo() >>>> t.isAlive() > True >>>> t.join() >>>> t.isAlive() > False >>>> > > The same thing could be done for multiprocessing module. > Would this be acceptable for inclusion? That looks good, though though I think run_as_thread needs to take arguments: * daemonifying needs to be performed *before* the thread is started, so it needs at least one argument `daemon=False` (which runs a daemonified thread if set to true) * maybe run_as_thread could take a second argument `start=True` to know whether the function should generate a started thread or a thread to start? Not sure about that one, are there situations where you'd *need* a yet-to-be-started thread apart from daemonification? * threads can take names, not sure if this is often used, should it be handled by run_as_thread? This is not as important as daemon because I think thread names can be set after start() * what are the semantics of the function's return value? None and it's basically ignored (as with regular target semantics)?
I agree it should be possible to pass the same arguments of the
original constructor, in fact this is the code I use across the
various "private" projects I work on:
def run_as_thread(group=None, name=None, verbose=None, daemon=None):
"""Decorator to run a callable in a thread returning a
thread instance.
>>> @run_as_thread
... def foo():
... time.sleep(100)
... return "done"
...
>>> t = foo()
>>> t.is_alive()
True
>>> t.join()
>>> t.is_alive()
False
"""
def outer(fun):
def inner(*args, **kwargs):
t = threading.Thread(target=fun, args=args, kwargs=kwargs,
group=group, name=name, verbose=verbose)
t.start()
return t
return inner
if hasattr(group, '__call__'):
# defined as @run_as_thread rather than @run_as_thread(arg=...)
fun = group
group = None
outer = outer(fun)
return outer
I don't know whether it is a good idea to provide such a thing
natively, but I can't even figure out what exactly is wrong/weird with
this exactly.
This is why I opened this discussion, basically. =)
--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
2011/3/5 Masklinn <masklinn@masklinn.net>:
> On 2011-03-05, at 14:21 , Giampaolo Rodolà wrote:
>>>>> import time, threading
>>>>>
>>>>> @threading.run_as_thread
>> ... def foo():
>> ... time.sleep(100)
>> ... return 1
>> ...
>>>>> t = foo()
>>>>> t.isAlive()
>> True
>>>>> t.join()
>>>>> t.isAlive()
>> False
>>>>>
>>
>> The same thing could be done for multiprocessing module.
>> Would this be acceptable for inclusion?
> That looks good, though though I think run_as_thread needs to take arguments:
> * daemonifying needs to be performed *before* the thread is started, so it needs at least one argument `daemon=False` (which runs a daemonified thread if set to true)
> * maybe run_as_thread could take a second argument `start=True` to know whether the function should generate a started thread or a thread to start? Not sure about that one, are there situations where you'd *need* a yet-to-be-started thread apart from daemonification?
> * threads can take names, not sure if this is often used, should it be handled by run_as_thread? This is not as important as daemon because I think thread names can be set after start()
> * what are the semantics of the function's return value? None and it's basically ignored (as with regular target semantics)?
>
>
On Sat, 5 Mar 2011 19:00:52 +0100 Giampaolo Rodolà <g.rodola@gmail.com> wrote:
I don't understand what this two line variant brings over the other two-line variant: t = threading.Thread(target=func) t.start() Basically you are proposing to complicate the API for no real benefit except that it "feels good". It also makes things harder to learn for beginners since there are two abstractions stacked one over the other. It doesn't sound like a sensible addition. Regards Antoine.
It is probably a bad idea, I don't know. I'm the first one being skeptical about it. Maybe a real world example can bring some contribution to this discussion since using this kind of approach at module-level doesn't bring real benefits over using threading.Thread(target=func).start(). In my experience, I've found this extremely elegant when I was developing a web application in which I wanted to start a serie of long running tasks involving the db, and didn't care about the function return value, nor I wanted to return anything relevant to the user other than a simple "task started - this may take some time" message. The code looked like this: from utils import run_as_thread class Admin: @run_as_thread def refresh_persons(self): ... @run_as_thread def refresh_voyages(self): ... @run_as_thread def refresh_addresses(self): ...
Basically you are proposing to complicate the API for no real benefit except that it "feels good".
I'd say there's a benefit in terms of elegance if this is used in a certain way. On the other hand I understand your complaints, and this probably fits better in an "util" module rather than threading. I like the function return value example proposed by Bruce. Is there a reason why this is not provided by base Thread class? http://code.activestate.com/recipes/84317/ ...and: http://www.google.com/#sclient=psy&hl=en&q=python+thread+return+value&aq=f&aqi=g1&aql=&oq=&pbx=1&bav=on.2,or.&fp=369c8973645261b8 ...suggest that users tend to require this feature. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/3/5 Antoine Pitrou <solipsis@pitrou.net>:
On Sat, Mar 5, 2011 at 11:31 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I just don't really find this that convincing; you're calling the function, you should probably know what it does. That's doubly true in this case, since you have to code around the fact that it's working in a separate thread if you want to use the return value. Geremy Condra
Le samedi 05 mars 2011 à 11:36 -0800, geremy condra a écrit :
Sure... My point is that the stdlib shouldn't really encourage this, especially when it doesn't subtantially reduce typing. This proposal seems similar in principle to the proposal of having "implicit" asynchronous function calls without writing "yield", which has also been shot down several times. Regards Antoine.
On Sat, Mar 5, 2011 at 11:31 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
You can always call functions that don't do what you think. There's a common pattern for this: def foo(bar): return x(y(z(bar))) @threading.run_as_thread def foo_async(bar): return foo(bar) --- Bruce New Puzzazz newsletter: http://j.mp/puzzazz-news-2011-02 Make your web app more secure: http://j.mp/gruyere-security
On Sat, Mar 5, 2011 at 8:21 AM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
I've long wanted to put something into the stdlib like this, but as others in the thread have pointed out - there's some semantics that remain to be hashed out and the behavior is dangerous (imo), and magical to have in the stdlib right now. In this case, I would recommend building out a library that contains these decorators (both threads and processes) building from the futures (concurrent.futures.Executor ABC) library as possible, and let's see how it pans out. I've struggled with really liking/wanting this and the fact that it's dangerous, and surprising. jesse
On Sat, Mar 5, 2011 at 9:16 AM, Jesse Noller <jnoller@gmail.com> wrote:
I've personally written this about five times. Having said that, what I'd really like would be a context manager that executed the contained block of code in a new thread or process, and I haven't gotten that to work the way I'd like so far. Geremy Condra
2011/3/5 geremy condra <debatem1@gmail.com>:
I've been thingking about that as well but... would that even be possible? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
On Sun, Mar 6, 2011 at 3:16 AM, Jesse Noller <jnoller@gmail.com> wrote:
Well said, especially the last line :) However, I suspect this is one of those things where: - rewriting it yourself is easier than finding a library for it, so a PyPI module would gather little interest or feedback - doing it "right" in the stdlib would eliminate the temptation to develop custom not-quite-right implementations (e.g. ones where the decorator actually *creates* and starts the thread) It would make a good topic for a PEP, IMO. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Mar 5, 2011 at 7:55 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Well, why stop at one-function in a thread - why not have: @threading.pool(10): def func(me): .... runs func in a threadpool of 10 Or with threading.pool(10) as pool: pool.in.put(blah) pool.out.get(blah) And so on - I can totally buy that it's pep fodder to do it "right" once, for the stdlib, I've just struggled and debated with a lot of people about how helpful this really is and the right way of doing it. The original proposal seems OK at first - just run this function in a thread - at first glance it seems like it's a harmless decorator to throw a function into a thread or process. But then what about the "surprise" that a function call forked a new thread - or new process? Maybe you're right - maybe this is good PEP territory. If nothing more we could all hash out the various way of biting yourself in the butt with these :) jesse ps: i love controlling thread/process pools with context managers. I'm ill.
On Sat, Mar 5, 2011 at 7:25 PM, Jesse Noller <jnoller@gmail.com> wrote:
I'm thinking that since all of this is all pretty simple it's something that applications or 3rd party libraries can easily have in a "utilities" module of their own. The bar is a lot lower that way.
jesse
ps: i love controlling thread/process pools with context managers. I'm ill.
Sick, you mean. :-) -- --Guido van Rossum (python.org/~guido)
On Sat, Mar 5, 2011 at 10:31 PM, Guido van Rossum <guido@python.org> wrote:
True, they're trivial to implement, and Nick may be right that there's no way for a package to be able to gain traction that's solely based on this idea. But something that builds on decorator and context manager usage of threads and pools / futures might be able to gain a fair amount of it.
ps: i love controlling thread/process pools with context managers. I'm ill.
Sick, you mean. :-)
Yes, this is true as well.
On Sat, Mar 5, 2011 at 11:21 PM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
So basically: def run_as_thread(f): @functools.wraps(f): def wrapped(*args, **kwds): t = threading.Thread(target=f, args=args, kwds=kwds) t.start() return t return wrapped Something like that would make defining worker threads *really* easy. A similar idea may make sense as an addition to the concurrent.futures.Executor ABC. For example: def autosubmit(self): def decorator(f): @functools.wraps(f): def wrapped(*args, **kwds): return self.submit(f, *args, **kwds) return wrapped return decorator Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 6 Mar 2011 00:10:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't really agree. First, as you guess, there's already a rather obvious one-liner: threading.Thread(target=f).start() Second, any decorator that implicitly spawns a thread is a very bad idea (especially when used at module level...). I'm rather opposed to this, it's a useless addition to the API with no real point. Calling the Thread() constructor works basically ok. Regards Antoine.
On 05.03.2011 15:50, Antoine Pitrou wrote:
It doesn't spawn the thread on definition, but on function call time. That's not as bad, but I agree that it is too magical for the stdlib.
I'm rather opposed to this, it's a useless addition to the API with no real point. Calling the Thread() constructor works basically ok.
Problem is, the one-liner doesn't give you a reference to the Thread object. Georg
On 2011-03-05, at 14:21 , Giampaolo Rodolà wrote: >>>> import time, threading >>>> >>>> @threading.run_as_thread > ... def foo(): > ... time.sleep(100) > ... return 1 > ... >>>> t = foo() >>>> t.isAlive() > True >>>> t.join() >>>> t.isAlive() > False >>>> > > The same thing could be done for multiprocessing module. > Would this be acceptable for inclusion? That looks good, though though I think run_as_thread needs to take arguments: * daemonifying needs to be performed *before* the thread is started, so it needs at least one argument `daemon=False` (which runs a daemonified thread if set to true) * maybe run_as_thread could take a second argument `start=True` to know whether the function should generate a started thread or a thread to start? Not sure about that one, are there situations where you'd *need* a yet-to-be-started thread apart from daemonification? * threads can take names, not sure if this is often used, should it be handled by run_as_thread? This is not as important as daemon because I think thread names can be set after start() * what are the semantics of the function's return value? None and it's basically ignored (as with regular target semantics)?
I agree it should be possible to pass the same arguments of the
original constructor, in fact this is the code I use across the
various "private" projects I work on:
def run_as_thread(group=None, name=None, verbose=None, daemon=None):
"""Decorator to run a callable in a thread returning a
thread instance.
>>> @run_as_thread
... def foo():
... time.sleep(100)
... return "done"
...
>>> t = foo()
>>> t.is_alive()
True
>>> t.join()
>>> t.is_alive()
False
"""
def outer(fun):
def inner(*args, **kwargs):
t = threading.Thread(target=fun, args=args, kwargs=kwargs,
group=group, name=name, verbose=verbose)
t.start()
return t
return inner
if hasattr(group, '__call__'):
# defined as @run_as_thread rather than @run_as_thread(arg=...)
fun = group
group = None
outer = outer(fun)
return outer
I don't know whether it is a good idea to provide such a thing
natively, but I can't even figure out what exactly is wrong/weird with
this exactly.
This is why I opened this discussion, basically. =)
--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
2011/3/5 Masklinn <masklinn@masklinn.net>:
> On 2011-03-05, at 14:21 , Giampaolo Rodolà wrote:
>>>>> import time, threading
>>>>>
>>>>> @threading.run_as_thread
>> ... def foo():
>> ... time.sleep(100)
>> ... return 1
>> ...
>>>>> t = foo()
>>>>> t.isAlive()
>> True
>>>>> t.join()
>>>>> t.isAlive()
>> False
>>>>>
>>
>> The same thing could be done for multiprocessing module.
>> Would this be acceptable for inclusion?
> That looks good, though though I think run_as_thread needs to take arguments:
> * daemonifying needs to be performed *before* the thread is started, so it needs at least one argument `daemon=False` (which runs a daemonified thread if set to true)
> * maybe run_as_thread could take a second argument `start=True` to know whether the function should generate a started thread or a thread to start? Not sure about that one, are there situations where you'd *need* a yet-to-be-started thread apart from daemonification?
> * threads can take names, not sure if this is often used, should it be handled by run_as_thread? This is not as important as daemon because I think thread names can be set after start()
> * what are the semantics of the function's return value? None and it's basically ignored (as with regular target semantics)?
>
>
On Sat, 5 Mar 2011 19:00:52 +0100 Giampaolo Rodolà <g.rodola@gmail.com> wrote:
I don't understand what this two line variant brings over the other two-line variant: t = threading.Thread(target=func) t.start() Basically you are proposing to complicate the API for no real benefit except that it "feels good". It also makes things harder to learn for beginners since there are two abstractions stacked one over the other. It doesn't sound like a sensible addition. Regards Antoine.
It is probably a bad idea, I don't know. I'm the first one being skeptical about it. Maybe a real world example can bring some contribution to this discussion since using this kind of approach at module-level doesn't bring real benefits over using threading.Thread(target=func).start(). In my experience, I've found this extremely elegant when I was developing a web application in which I wanted to start a serie of long running tasks involving the db, and didn't care about the function return value, nor I wanted to return anything relevant to the user other than a simple "task started - this may take some time" message. The code looked like this: from utils import run_as_thread class Admin: @run_as_thread def refresh_persons(self): ... @run_as_thread def refresh_voyages(self): ... @run_as_thread def refresh_addresses(self): ...
Basically you are proposing to complicate the API for no real benefit except that it "feels good".
I'd say there's a benefit in terms of elegance if this is used in a certain way. On the other hand I understand your complaints, and this probably fits better in an "util" module rather than threading. I like the function return value example proposed by Bruce. Is there a reason why this is not provided by base Thread class? http://code.activestate.com/recipes/84317/ ...and: http://www.google.com/#sclient=psy&hl=en&q=python+thread+return+value&aq=f&aqi=g1&aql=&oq=&pbx=1&bav=on.2,or.&fp=369c8973645261b8 ...suggest that users tend to require this feature. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/3/5 Antoine Pitrou <solipsis@pitrou.net>:
On Sat, Mar 5, 2011 at 11:31 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I just don't really find this that convincing; you're calling the function, you should probably know what it does. That's doubly true in this case, since you have to code around the fact that it's working in a separate thread if you want to use the return value. Geremy Condra
Le samedi 05 mars 2011 à 11:36 -0800, geremy condra a écrit :
Sure... My point is that the stdlib shouldn't really encourage this, especially when it doesn't subtantially reduce typing. This proposal seems similar in principle to the proposal of having "implicit" asynchronous function calls without writing "yield", which has also been shot down several times. Regards Antoine.
On Sat, Mar 5, 2011 at 11:31 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
You can always call functions that don't do what you think. There's a common pattern for this: def foo(bar): return x(y(z(bar))) @threading.run_as_thread def foo_async(bar): return foo(bar) --- Bruce New Puzzazz newsletter: http://j.mp/puzzazz-news-2011-02 Make your web app more secure: http://j.mp/gruyere-security
On Sat, Mar 5, 2011 at 8:21 AM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
I've long wanted to put something into the stdlib like this, but as others in the thread have pointed out - there's some semantics that remain to be hashed out and the behavior is dangerous (imo), and magical to have in the stdlib right now. In this case, I would recommend building out a library that contains these decorators (both threads and processes) building from the futures (concurrent.futures.Executor ABC) library as possible, and let's see how it pans out. I've struggled with really liking/wanting this and the fact that it's dangerous, and surprising. jesse
On Sat, Mar 5, 2011 at 9:16 AM, Jesse Noller <jnoller@gmail.com> wrote:
I've personally written this about five times. Having said that, what I'd really like would be a context manager that executed the contained block of code in a new thread or process, and I haven't gotten that to work the way I'd like so far. Geremy Condra
2011/3/5 geremy condra <debatem1@gmail.com>:
I've been thingking about that as well but... would that even be possible? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
On Sun, Mar 6, 2011 at 3:16 AM, Jesse Noller <jnoller@gmail.com> wrote:
Well said, especially the last line :) However, I suspect this is one of those things where: - rewriting it yourself is easier than finding a library for it, so a PyPI module would gather little interest or feedback - doing it "right" in the stdlib would eliminate the temptation to develop custom not-quite-right implementations (e.g. ones where the decorator actually *creates* and starts the thread) It would make a good topic for a PEP, IMO. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Mar 5, 2011 at 7:55 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Well, why stop at one-function in a thread - why not have: @threading.pool(10): def func(me): .... runs func in a threadpool of 10 Or with threading.pool(10) as pool: pool.in.put(blah) pool.out.get(blah) And so on - I can totally buy that it's pep fodder to do it "right" once, for the stdlib, I've just struggled and debated with a lot of people about how helpful this really is and the right way of doing it. The original proposal seems OK at first - just run this function in a thread - at first glance it seems like it's a harmless decorator to throw a function into a thread or process. But then what about the "surprise" that a function call forked a new thread - or new process? Maybe you're right - maybe this is good PEP territory. If nothing more we could all hash out the various way of biting yourself in the butt with these :) jesse ps: i love controlling thread/process pools with context managers. I'm ill.
On Sat, Mar 5, 2011 at 7:25 PM, Jesse Noller <jnoller@gmail.com> wrote:
I'm thinking that since all of this is all pretty simple it's something that applications or 3rd party libraries can easily have in a "utilities" module of their own. The bar is a lot lower that way.
jesse
ps: i love controlling thread/process pools with context managers. I'm ill.
Sick, you mean. :-) -- --Guido van Rossum (python.org/~guido)
On Sat, Mar 5, 2011 at 10:31 PM, Guido van Rossum <guido@python.org> wrote:
True, they're trivial to implement, and Nick may be right that there's no way for a package to be able to gain traction that's solely based on this idea. But something that builds on decorator and context manager usage of threads and pools / futures might be able to gain a fair amount of it.
ps: i love controlling thread/process pools with context managers. I'm ill.
Sick, you mean. :-)
Yes, this is true as well.
participants (9)
-
Antoine Pitrou -
Bruce Leban -
Georg Brandl -
geremy condra -
Giampaolo Rodolà -
Guido van Rossum -
Jesse Noller -
Masklinn -
Nick Coghlan