[Twisted-Python] feeding-worker-loop problem
Hi! I've been thinking for some time now about the following problem, but can't find a solution. I have a queue with a fluctuant number of tasks. This queue might be empty at some time but might also become such full, that all workers have work to do and we have to wait until one becomes free again. I'd like to have a number of workers which process these tasks from the queue. What I've got: - the worker, which returns a deferred for further processing - a worker-pool class, which hands out the workers with an attached callback to re-queue them to the "available-pool" My problem: - the connection between the queue and the worker-pool and integrating this into the main reactor-loop My first thought was a twisted.internet.task.LoopingCall, trying to get a worker and on success giving it a task, but that's not very asynchronous, isn't it? Especially no when the queue is empty. I've no clue how to implement this. I've read something about producer-consumer in twisted, but the documentation here is very rare and I think it's not what I'm looking for. Can someone please give me a hint, if twisted offers some techniques to implement this, or point me in the right direction? Thanks Patrick
I don't understand that terminology. What kind of objects are these workers? Threads? Sub-processes? Python generators? Long-lived functions (these would block the reactor)
I would suggest looking at this: http://foss.eepatents.com/AsynQueue
Hi all concerning twisted.web: How do I run once-per-application stuff ? Assume, I want to put sth into the static.Registry or I want to read in some application data from a file, so that every resource can use it. My main problem can be split in two: - where do it do it (using twistd) ? Please no complicated plugin (or alike) suggestions. I'm seeing red just hearing 'zope' ... - how can resource scripts use it ? Probably this is an idiots question but I have sth. like 8 Browsertabs open with twisted "docu" and getting more and more confused by hunting pieces of information that are just splattered all over the site. Additional: Is there any docu about twistd ? How can I use all its miracles rather than just dumbly guess and try an err and try and err and ... ? Yes, I'm demotivated and pissed far over the edge. Forgive me. Thanks
On Saturday 19 April 2008, Dr Pythoniac wrote:
For our webapp, we load lots of data, so I made a "startup in progress" page which both does the loading and serves a page that shows how much is loaded so far. The loading itself is done by loading small chunks of data and then returning control back to the reactor, so it can actually serve the progress page. When loading is done, the progress page resource is replaced by the root page of the webapp. If you only want to load a small amount of data, maybe subclass Application? I haven't tried this though. Our webapp loads from a Python file (using "twistd -y"), so that first Python file could also load data. If I recall correctly, that initial file is run as the same user that started twistd (root, in our case), while the started application runs as the user selected from the twistd --uid and --gid options. So it is safer to do as little as possible at this stage of startup. Bye, Maarten
Dr Pythoniac wrote:
Hi all
When posting a new topic, please don't use reply-to on a previous different topic, it breaks threading.
Assuming you are using a .tac file, you might do something like this in "application.tac": from twisted.application import service, strports import mycode application = service.Application('myapp') services = service.IServiceCollection(application) # do startup code here mycode.doStartupStuff() site = mycode.site() web = strports.service('tcp:80', site) web.setServiceParent(services) ...then run the "tac" file with: twistd -oy application.tac ...this is such a simple solution that I guess you are having more basic misunderstandings.
- how can resource scripts use it ?
What does this mean?
Have you read the finger tutorial from start to end, and understood it?
http://twistedmatrix.com/projects/core/documentation/howto/application.html ...describes the contents of a ".tac" file, the recommended way to use twistd. If you could be more specific about *what* you don't understand, what you've tried already, why it didn't work and with what error messages, I can perhaps be more helpful.
Hi!
I would suggest looking at this:
That looks very promising. I'll take a deeper look at that tomorrow. Thanks! Patrick
I don't understand that terminology. What kind of objects are these workers? Threads? Sub-processes? Python generators? Long-lived functions (these would block the reactor)
I would suggest looking at this: http://foss.eepatents.com/AsynQueue
Hi all concerning twisted.web: How do I run once-per-application stuff ? Assume, I want to put sth into the static.Registry or I want to read in some application data from a file, so that every resource can use it. My main problem can be split in two: - where do it do it (using twistd) ? Please no complicated plugin (or alike) suggestions. I'm seeing red just hearing 'zope' ... - how can resource scripts use it ? Probably this is an idiots question but I have sth. like 8 Browsertabs open with twisted "docu" and getting more and more confused by hunting pieces of information that are just splattered all over the site. Additional: Is there any docu about twistd ? How can I use all its miracles rather than just dumbly guess and try an err and try and err and ... ? Yes, I'm demotivated and pissed far over the edge. Forgive me. Thanks
On Saturday 19 April 2008, Dr Pythoniac wrote:
For our webapp, we load lots of data, so I made a "startup in progress" page which both does the loading and serves a page that shows how much is loaded so far. The loading itself is done by loading small chunks of data and then returning control back to the reactor, so it can actually serve the progress page. When loading is done, the progress page resource is replaced by the root page of the webapp. If you only want to load a small amount of data, maybe subclass Application? I haven't tried this though. Our webapp loads from a Python file (using "twistd -y"), so that first Python file could also load data. If I recall correctly, that initial file is run as the same user that started twistd (root, in our case), while the started application runs as the user selected from the twistd --uid and --gid options. So it is safer to do as little as possible at this stage of startup. Bye, Maarten
Dr Pythoniac wrote:
Hi all
When posting a new topic, please don't use reply-to on a previous different topic, it breaks threading.
Assuming you are using a .tac file, you might do something like this in "application.tac": from twisted.application import service, strports import mycode application = service.Application('myapp') services = service.IServiceCollection(application) # do startup code here mycode.doStartupStuff() site = mycode.site() web = strports.service('tcp:80', site) web.setServiceParent(services) ...then run the "tac" file with: twistd -oy application.tac ...this is such a simple solution that I guess you are having more basic misunderstandings.
- how can resource scripts use it ?
What does this mean?
Have you read the finger tutorial from start to end, and understood it?
http://twistedmatrix.com/projects/core/documentation/howto/application.html ...describes the contents of a ".tac" file, the recommended way to use twistd. If you could be more specific about *what* you don't understand, what you've tried already, why it didn't work and with what error messages, I can perhaps be more helpful.
Hi!
I would suggest looking at this:
That looks very promising. I'll take a deeper look at that tomorrow. Thanks! Patrick
participants (4)
-
Dr Pythoniac -
Maarten ter Huurne -
Patrick Scharrenberg -
Phil Mayers