Re: [Twisted-Python] newbie: scheduling/queuing tasks with xmlrpc
On Thu, 17 Mar 2005 10:29:09 -0700, Dustin Lee <qhfgva@gmail.com> wrote:
So I've got a simple prototype of what I'm trying to achieve with my "queued tasks" xmlrpc server.
I'd appreciate any comments, suggestions. So far it seems to work exactly I wanted but I have a feeling I'm abusing the intended use of deferreds. And that by polling I'm deviating from the twisted workflow model. But you have to start somewhere.
Yes, much abuse is present in this code. A couple grievous wrongs: Deferreds (and almost all other Twisted APIs) are not thread-safe. You cannot invoke their methods in a thread other than that of the reactor. The called and paused Deferred attributes are not really public. You should avoid using them if at all possible (and it almost always is). Likewise, the pause() and unpause() methods are more for the convenience of Deferred's implementation and you should never need to call them. Attached is a modified version. It has a problem with shutdown, which I imagine is related to not properly closing the reactor's threadpool. Maybe someone else can suggest the correct way to deal with that. Jp
Thanks for the response. I will give it a look see. Still have much to learn about twisted, but I figure I'll learn more by trying out my ideas than just reading the tutorials over and over... dustin On Thu, 17 Mar 2005 18:17:15 GMT, Jp Calderone <exarkun@divmod.com> wrote:
On Thu, 17 Mar 2005 10:29:09 -0700, Dustin Lee <qhfgva@gmail.com> wrote:
So I've got a simple prototype of what I'm trying to achieve with my "queued tasks" xmlrpc server.
I'd appreciate any comments, suggestions. So far it seems to work exactly I wanted but I have a feeling I'm abusing the intended use of deferreds. And that by polling I'm deviating from the twisted workflow model. But you have to start somewhere.
Yes, much abuse is present in this code. A couple grievous wrongs:
Deferreds (and almost all other Twisted APIs) are not thread-safe. You cannot invoke their methods in a thread other than that of the reactor.
The called and paused Deferred attributes are not really public. You should avoid using them if at all possible (and it almost always is). Likewise, the pause() and unpause() methods are more for the convenience of Deferred's implementation and you should never need to call them.
Attached is a modified version. It has a problem with shutdown, which I imagine is related to not properly closing the reactor's threadpool. Maybe someone else can suggest the correct way to deal with that.
Jp
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Dustin Lee qhfgva=rot13(dustin)
Jp Calderone wrote:
The called and paused Deferred attributes are not really public. You should avoid using them if at all possible (and it almost always is). Likewise, the pause() and unpause() methods are more for the convenience of Deferred's implementation and you should never need to call them.
Thinking over the recent deferred timeout thread: http://marc.free.net.ph/message/20050225.195330.793c275f.en.html#twisted-pyt... it looks like methods like pause, unpause, cancel and timeout should all be handled by the source of the operation. A common operation (job) api that exposed these methods would make building generic scheduling frameworks easier. Dustin I don't know if you'd find this thread of interest: http://twistedmatrix.com/pipermail/twisted-python/2004-March/007271.html The problem with the DeferredPriorityQueue there for your case is if enough hour long jobs come to use up all available tokens, the server would be clogged up until they finish. If the queue took jobs with an agreed on way of pausing them, long running jobs low priority jobs could be put on hold when higher priority jobs turn up .. Andy.
participants (3)
-
Andy Gayton -
Dustin Lee -
Jp Calderone