[Twisted-Python] web.static patch waiting for commit (before 1.0.1)
the patch i submitted for pre-1.0 because to fix query_string's for index.rpy's (see list archive), hasn't been commited nor rejected so far. testcase is: http://www.twistedmatrix.com/, repeatedly adding ?bla=foo strings and wondering about the redirect that discards the query_string. anyone care to look at this?: diff -u -r1.74 static.py --- static.py 12 Nov 2002 13:00:06 -0000 1.74 +++ static.py 13 Nov 2002 19:31:41 -0000 @@ -267,10 +267,12 @@ # If someone is looking for children with a PathReferenceContext, # the request won't have a prepath, and we shouldn't do this kind # of mangling anyway because it has already been done. - if hasattr(request, 'postpath') and not request.postpath and request.uri[-1] != '/': + if hasattr(request, 'postpath') and not request.postpath and \ + string.split(request.uri,'?')[0][-1] != '/': return Redirect(request) if os.path.exists(childPath): - if hasattr(request, 'postpath') and not request.postpath and not self.getIndex(request): + if hasattr(request, 'postpath') and not request.postpath and \ + not self.getIndex(request): return widgets.WidgetPage(DirectoryListing(self.path)) ##
Hi, Nobody seems to have used twisted.python.threadable.ThreadAttr recently..: --- /usr/lib/python2.2/site-packages/twisted/python/threadable.py 2002-09-23 05:30:46.000000000 +0200 +++ twisted/python/threadable.py 2002-11-13 21:15:46.000000000 +0100 @@ -297,7 +297,7 @@ return getattr(self.__get(), key) def __setattr__(self, key, val): - return setattr(self.__get(), key) + return setattr(self.__get(), key, val) def __delattr__(self, key): return delattr(self.__get(), key) Disclaimer: I found this by looking at the code -- but I didn't test it either. :-/ -- Matthias Urlichs | noris network AG | http://smurf.noris.de/
On Wed, 13 Nov 2002 21:19:59 +0100, "Matthias Urlichs" <smurf@noris.de> wrote:
Nobody seems to have used twisted.python.threadable.ThreadAttr recently..:
Wow. If it's _that_ rarely used, maybe the right thing to do is just to trash it. Would this code be useful to you if it worked? -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |
Hi, Glyph Lefkowitz:
On Wed, 13 Nov 2002 21:19:59 +0100, "Matthias Urlichs" <smurf@noris.de> wrote:
Nobody seems to have used twisted.python.threadable.ThreadAttr recently..:
Wow. If it's _that_ rarely used, maybe the right thing to do is just to trash it. Would this code be useful to you if it worked?
Personally, I woudn't need it. IMHO: if you use threads in Twisted, you do it via the thread pool. That doesn't mesh at all with ThreadAttr. State is usually carried in objects or procedure arguments, which don't need global storage anyway. I find myself more in immediate need of a queueing mechanism; the quick-and-dirty solution seems to be to fork off two threads and to use a bounded Python Queue. However, the continuous switching between the reader (which blocks on the queue, therefore needs to be in its own thread) and Twisted's main thread (which writes to the process-or-whatever that's going to eventually consume the data) seems rather sub-optimal for performance. :-/ -- Matthias Urlichs | noris network AG | http://smurf.noris.de/
On Wed, 13 Nov 2002 22:19:34 +0100 "Matthias Urlichs" <smurf@noris.de> wrote:
I find myself more in immediate need of a queueing mechanism; the quick-and-dirty solution seems to be to fork off two threads and to use a bounded Python Queue. However, the continuous switching between the reader(which blocks on the queue, therefore needs to be in its own thread) and Twisted's main thread (which writes to the process-or-whatever that's going to eventually consume the data) seems rather sub-optimal for performance. :-/
I don't see why you need a thread for queueing - a simple list would do, or a class wrapping one. Or look at transport's producer/consumer interface if this is for a protocol. Adding to queue is appending to list, dequeuing is getting item from beginning of list. If, and *only* if you need threads - i.e. you are doing either blocking operations or really long running calculations, there are some options for you in Twisted. We have a thread pool dispatcher that queues methods to be run in the thread pool, that covers some the use cases. If you want to use a queue I suggest just sticking to Python's Queue or the slighly nicer twisted.python.timeoutqueue. In general using threads in Python is slow anyway do to the global interpreter lock, which means in most cases only one thread at a time is running anyway. That being said, Queue.Queue is rather slow in and of itself, and can be implemented in a much faster way by using one lock, and use mxQueue. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python, Twisted, Zope and Java consulting ***> http://VoteNoWar.org -- vote/donate/volunteer <***
Hi, Itamar Shtull-Trauring:
I don't see why you need a thread for queueing - a simple list would do, or a class wrapping one. Or look at transport's producer/consumer interface if this is for a protocol. Adding to queue is appending to list, dequeuing is getting item from beginning of list.
Some sample code would be nice... or pointers to which modules to look at.
If, and *only* if you need threads - i.e. you are doing either blocking operations or really long running calculations, there are some options for you in Twisted.
Yes -- the source data are coming from a database (and the consumer may have to do database look-ups too). Since the consumer can be slower and I don't want in-core memory to grow indiscriminately, some sort of thread-safe bounded queuing makes sense.
We have a thread pool dispatcher that queues methods to be run in the thread pool, that covers some the use cases. If you want to use a queue I suggest just sticking to Python's Queue or the slighly nicer twisted.python.timeoutqueue.
Unrelated nit: twisted.python.timeoutqueue lacks a pop() function with timeout (the wait+pop combo is NOT thread safe: you might have multiple consumers), and, frankly, the basic idea of timeout-waiting on a queue the way this module does it is just plain ugly. TODO: Rewrite the whole thing, using mxQueue for the actual data and Twisted support for everything else -- I'd use a Deferred() for pulling one item from the queue, for instance, but a "call this function whenever you have a new item" interface might make more sense with a high-volume queue. (This function might then want to return a Deferred() to indicate that it will be ready for the next item LATER.) Same for pushing stuff into the queue. And all of that thread-safe. ;-) I think I'll do that, probably sometime next week, because I need it.
In general using threads in Python is slow anyway do to the global interpreter lock, which means in most cases only one thread at a time is running anyway.
While that is certainly true, working with blocking external interfaces more-or-less requires them.
That being said, Queue.Queue is rather slow in and of itself, and can be implemented in a much faster way by using one lock, and use mxQueue.
I'd have to reinvent all of Queue's thread-support features, though. That might be a nice exercise, but I'll spend my time on required features first. ;-) -- Matthias Urlichs | noris network AG | http://smurf.noris.de/
On Thu, 14 Nov 2002 07:02:21 +0100 "Matthias Urlichs" <smurf@noris.de> wrote:
Yes -- the source data are coming from a database (and the consumer may have to do database look-ups too). Since the consumer can be slower and I don't want in-core memory to grow indiscriminately, some sort of thread-safe bounded queuing makes sense.
Just use Queue.Queue then?
Unrelated nit: twisted.python.timeoutqueue lacks a pop() function with timeout (the wait+pop combo is NOT thread safe: you might have multiple consumers), and, frankly, the basic idea of timeout-waiting on a queue the way this module does it is just plain ugly.
It sucks, yes. I needed it though and couldn't think up a better way of doing it.
I'd have to reinvent all of Queue's thread-support features, though. That might be a nice exercise, but I'll spend my time on required features first. ;-)
I have code that does this. But if you want bounded size you may as well stick to Queue, I sped it up by removing support for this :) -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python, Twisted, Zope and Java consulting ***> http://VoteNoWar.org -- vote/donate/volunteer <***
participants (4)
-
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Matthias Urlichs -
Paul Boehm