[Twisted-Python] Taking a look at pending deferToThread actions
I am using deferToThread in my application to put a number of web service queries into the background and then respond to them when they are complete. This is all working very well, but once in a while I seem to overload the web service with queries and I would like to track this down by checking to see how many queries are in flight and how many are still waiting to be processed. One I have put an action into the background using deferToThread, is there a way to check to see how many functions are currently processing and similarly how many functions are waiting to be processed? Thanks, Allen
Sorry for answering my own question, but I found a way to get this information and I thought it may be helpful for other users. threadpool = reactor.threadpool num_threads = threadpool.workers q_size = threadpool.q.qsize() num_threads is a good approximation of how many functions are currently active, and q_size is the number still pending. Now for another question: - Has anyone extended threadpool to allow code to remove queued functions from the pending queue? I need this capability so I can cull off pending queries that are not still needed. -Allen On 6/4/07, Allen Bierbaum <abierbaum@gmail.com> wrote:
I am using deferToThread in my application to put a number of web service queries into the background and then respond to them when they are complete. This is all working very well, but once in a while I seem to overload the web service with queries and I would like to track this down by checking to see how many queries are in flight and how many are still waiting to be processed.
One I have put an action into the background using deferToThread, is there a way to check to see how many functions are currently processing and similarly how many functions are waiting to be processed?
Thanks, Allen
Hi. A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart. /BG Sven-Erik Tiberg
On Tuesday 05 June 2007, Sven-Erik Tiberg wrote:
A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart.
PyReverse can analyze Python code and produce UML. I never used it myself, but PyLint is written by the same people and that's a very useful tool. However, PyReverse is not actively maintained, while PyLint is. http://www.logilab.org/2560 Probably PyReverse will only analyze the static structure, so you'll get class diagrams. For analyzing dynamic behaviour, PyPy might be able to gather a lot of useful data, but as far as I know there is not yet a tool that makes nice graphs of it. Also, I'm not sure how useful PyPy will be if your program uses Python features that are not in the RPython subset. http://codespeak.net/pypy/ Bye, Maarten
On 09:07 pm, sven-erik.tiberg@ltu.se wrote:
Hi.
Hi Sven,
A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart.
Not that I'm aware of; it would be a massive task, with no obvious benefits. Why do you want such a thing? Perhaps we can suggest an alternative means of extracting the information you want without UML diagrams.
Hi Would like to visuliaze the signal ( state ) flow between a Interactive VR-World ( using X3D ) and a Mathworks / Simulink model using twisted and UDP. It's the realtime system that we are planning to make for development and training of vehicle drivers and process plant controllers. Have tried to draw by hand and figures but would prefere to use a more formal way to describe it. Been there before made the software for a motionplatform for offroad and marine vehicles back in -79 and had to modify the JSP model with Real time functions, but it was worth it. Had to add symbol for priority, sceeduling, process start / termination and semaphores. PS. downloaded RhapsodyModeler today from telelogic ( FOC) beacuse it had UML-extensions for RT compared to MS Visio. Con: Lacking code handling like reverse engineering, so it's onky to visulaize a event driven system like the one described above. PS, it still my sparetime project but there are some thoughts about making a course out of it for design-, HMI-, system developer and mashine enginners. Wouldn't that be fun. /BG Sven-Erik ________________________________ From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of glyph@divmod.com Sent: den 5 juni 2007 23:55 To: Twisted general discussion Subject: Re: [Twisted-Python] A longshot, UML and twisted. On 09:07 pm, sven-erik.tiberg@ltu.se wrote:
Hi.
Hi Sven,
A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart.
Not that I'm aware of; it would be a massive task, with no obvious benefits. Why do you want such a thing? Perhaps we can suggest an alternative means of extracting the information you want without UML diagrams.
Sven-Erik Tiberg wrote:
A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart.
In addition to PyReverse, you may try these other two: PyNSource http://www.atug.com/andypatterns/pynsource.htm snakefood http://furius.ca/snakefood/ and let us know the results. :-) -- Nicola Larosa - http://www.tekNico.net/ Jerub: someone motivate me to write a real http client. lifeless: Jerub: write a real http client spiv: Jerub: write a real http client oubiwann: Jerub: write a real client and 1000 virgins are yours for the taking dash: oubiwann: all of #gentoo?! -- doc/fun/Twisted.Quotes, May 2006
Thanks, Looks usefull. I will put them on my project site http://tellus.arb.ltu.se/moodle > WBSP_Public, and try them during the summer. /BG Sven-Erik -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Nicola Larosa Sent: den 6 juni 2007 08:23 To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] Re: A longshot, UML and twisted. Sven-Erik Tiberg wrote:
A longshot, anyone modelled a system with twisted components in uml using f.ex. a stateschart.
In addition to PyReverse, you may try these other two: PyNSource http://www.atug.com/andypatterns/pynsource.htm snakefood http://furius.ca/snakefood/ and let us know the results. :-) -- Nicola Larosa - http://www.tekNico.net/ Jerub: someone motivate me to write a real http client. lifeless: Jerub: write a real http client spiv: Jerub: write a real http client oubiwann: Jerub: write a real client and 1000 virgins are yours for the taking dash: oubiwann: all of #gentoo?! -- doc/fun/Twisted.Quotes, May 2006 _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
An example of UML StateChart: http://tellus.arb.ltu.se/distr_3d/Distr_3D.jpg . Original MagicDraw project report: http://tellus.arb.ltu.se/distr_3d/Distr_3D.html will add descriptions and docs. This is about what I needed,,, so far. /Sven-Erik Tiberg
On Tue, 2007-06-05 at 11:02 -0500, Allen Bierbaum wrote:
- Has anyone extended threadpool to allow code to remove queued functions from the pending queue?
How about just having the functions themselves check to see if they really should run their intended task? (Being careful about thread-safety, of course). Why are you using threads, btw?
On 6/6/07, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Tue, 2007-06-05 at 11:02 -0500, Allen Bierbaum wrote:
- Has anyone extended threadpool to allow code to remove queued functions from the pending queue?
How about just having the functions themselves check to see if they really should run their intended task? (Being careful about thread-safety, of course).
Thanks for the idea. I will give it a try.
Why are you using threads, btw?
I am using threads because the calls I am making are into a non-blocking library. I may see about rewriting it in the future to be more twisted-friendly, but for now I have to live with it. -Allen
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Allen Bierbaum wrote:
- Has anyone extended threadpool to allow code to remove queued functions from the pending queue?
How about just having the functions themselves check to see if they really should run their intended task? (Being careful about thread-safety, of course).
Thanks for the idea. I will give it a try.
You also might want to take a look at my taskqueue sub-package in Twisted-Goodies, http://foss.eepatents.com/Twisted-Goodies. One feature that might come in handy is the IWorker implementors' ability to resign when they feel that they are no longer able to run jobs. Any jobs that the resigning worker has queued up individually at that point are returned to the main queue for dispatching to other workers, current or future. The whole thing revolves around a priority queue, which might be useful to you, too. Also, a ProcessWorker implementation of IWorker is pending that spawns a separate Python interpreter to run tasks. If the ProcessWorker hangs, you'll be able to just kill it and move right along, without hanging your main Python interpreter.
I am using threads because the calls I am making are into a non-blocking library. I may see about rewriting it in the future to be more twisted-friendly, but for now I have to live with it.
That was my original motivation for writing taskqueue, with its ThreadWorker implementation of IWorker. It's used in sAsync, which makes a wonderful but totally synchronous database API (SQLAlchemy) play very nicely (IMHO) with Twisted. Debian packages are in the works for both, by the way. Best regards, Ed Suominen
participants (7)
-
Allen Bierbaum -
Ed Suominen -
glyph＠divmod.com -
Itamar Shtull-Trauring -
Maarten ter Huurne -
Nicola Larosa -
Sven-Erik Tiberg