Mailman 3 January 2010 - Twisted

[Twisted-Python] lore2sphinx themeing
by Kevin Horn 05 Jan '10

05 Jan '10

Thanks to a quick patch from Michael Thompson, the current output of the Sphinx conversion [1] now matches the color scheme and typographic style of the main twisted site. It's using the layout of the 'default' sphinx theme (previously was using the 'sphinxdoc' theme). I'd like to know what people out there want/expect the final theme of the Sphinx version of the Twisted docs to look like. The floor is open...please offer your suggestions! [1] http://twistedsphinx.funsize.net/

3 3

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones 05 Jan '10

05 Jan '10

As a final followup before bed... I chose not to have a 'cancel' method on my class because that seemed misleading. You're not really canceling anything. You're just asking that a deferred that you got earlier be fired right now with a value of your choosing. So I made 2 methods, and named them 'callback' and 'errback'. The Subject above is reasonably accurate, it's like a pseudo-Deferred class. Everyone works with regular deferreds, as usual, but if you know you might want to "cancel" a deferred by firing it early, you can call the call/errback methods on an instance of CancellableDeferred. Because the deferred you're holding is not the one that was obtained by calling self._f, it will fire immediately, which is surely part of the point of cancellation. In other words, using the word "cancel" is just a convenient short / casual description of what's actually being done. I hope all these words are making things more clear, not less... Terry

1 0

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones 05 Jan '10

05 Jan '10

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> I already lost you at the first sentence. :-) Sorry. Glyph> The class below never appears to use 'self._f' Oops, that should have been a self._f in the call method. Glyph> and ... Deferreds are things that fire, I don't see how the callable Glyph> (f) can fire. Can you rephrase your intent? Yes. I wanted a way to be able to call something (anything) that returns a deferred and then, at some later point, to be able to decide that I no longer want the deferred to fire in the way the original function would have done it. Instead I want to fire it myself, to in some sense cancel the outstanding call (by ignoring its result and instead returning a result of my choosing, right now). E.g., I call twisted.web.client.getPage and get a deferred (d) back. I add a callback to d. If sometime later (perhaps just due to too much time elapsing, but perhaps for another reason - e.g., maybe I'm shutting down a service, who knows) I want to effectively cancel that deferred, what choice do I have? The deferred was made elsewhere (in HTTPClientFactory in this case), which is presumably going to fire it at some point. Somewhere else in my code I've now got a callback function that's going to get the result, though I now don't want that to happen. I could set a flag somewhere to indicate to my own callback(s) that it(they) should ignore any incoming result and/or fail, but that's pretty messy. Instead I'd like to just manually cancel the deferred myself - where by "cancel" I mean fire it with a value of my choosing. Most convenient, probably, is to call its errback with a TimeoutError of some kind (just as defer.timeout does) which I can just absorb or log etc. I can't do this though with the deferred I got back from getPage. So the (untested) class I posted sits in the middle. If the original deferred (from getPage, in our example) fires first, it passes (via chainDeferred) the result along to the deferred it gave me. If instead I trigger the deferred myself (by calling 'callback' or 'errback' *on the CancellableDeferred instance*) then it fires the deferred it gave me back and will later ignore the result from getPage (if any). Glyph> I'm glad you're thinking about this, because it is an *extremely* Glyph> thorny issue which I would really like to address one day. Many of Glyph> the issues you're talking about were brought up, and various Glyph> solutions suggested, then found problematic, then modified ad Glyph> nauseam on <http://twistedmatrix.com/trac/ticket/990>. If you can Glyph> read that discussion and make some sense of it, perhaps you can post Glyph> a recommendation there, or at least a summary of the discussion so Glyph> far so that I don't have to read the whole discussion again to Glyph> remember what I think should happen next? :) OK, I'll take a look (in my copious spare time, etc). >> I don't know why defer.Deferred.setTimeout is deprecated, but I guess >> it's partly to do with this control issue. Glyph> This is one of the reasons, but another major reason is that Glyph> 'setTimeout' does not belong as a method of Deferred. If we did Glyph> support cancellation somehow, the way to set a timeout would be to Glyph> do 'reactor.callLater(5.0, myDeferred.cancel)'. Deferred was Glyph> originally in twisted.python and it really should have remained Glyph> there, decoupled from twisted.internet. OK, I agree with that aim. Cancelation (if you want to call it that) using my class is, IMO, decoupled from normal deferreds. Under normal circumstances you don't want to cancel a deferred you've asked for. But if you might, you can get yourself an instance of CancellableDeferred and use it instead. It provides you with a way to cancel the deferred (by which I mean: manually trigger it yourself with a value of your choosing, and thereafter ignore anything that might eventually come back from whatever it was you originally called (and which it is presumably now too late to undo)). You can do the canceling via reactor.callLater, just as you describe, but you call a method on the CancellableDeferred instance, not on the deferred. That keeps the canceling functionality out of the deferred class. Is that any clearer? Sorry the code was untested - it was meant to be so simple that it couldn't possibly be wrong :-) Terry

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 03 Jan '10

03 Jan '10

1 0

Re: [Twisted-Python] Problem fetching page with getPage
by Terry Jones 03 Jan '10

03 Jan '10

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> I'm glad that you tracked this down! Me too. Glyph> According to comments on <http://twistedmatrix.com/trac/ticket/886>, Glyph> this problem was addressed in the new HTTP client implementation. Glyph> Have you considered using the new twisted.web.client.Agent instead Glyph> of getPage? I hadn't looked at it, but now have. The _computeHostValue method looks very promising :-) We've yet to switch to 9.0.0. So httplib (apparently) changed to drop the :port part of the Host header in Python 2.6, and now the Twisted client has added it. I think you guys are right, so I wonder why httplib dropped it. Terry

1 0

Re: [Twisted-Python] Problem fetching page with getPage
by Terry Jones 03 Jan '10

03 Jan '10

Hi Glyph Thanks for the reply. I just sent another mail in the thread. >>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> Well, I know this isn't terribly helpful, but "a bug in getPage" is Glyph> really the only thing that comes to mind. Or, some Glyph> legal-but-unusual behavior in getPage which triggers a bug on the Glyph> EC2 side of things. The error arose from a combination of things (signing a string that included a host:port but then only sending a host in the Host header). Turns out you can resolve it either way - using a port in both, or omitting the port from both. BTW, in reading about the Host header, it seems like getPage (more specifically HTTPPageGetter) should be sending a port number in the header, at least when the port is not 80. I base that remark on these: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23 http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.2.2 That's a 1.1 spec as you surely know, and http.py sends an HTTP/1.0 header, so you could argue that sending the Host is therefore just a nicety and there's no need for a port. But the Host header isn't described in the HTTP 1.0 RFC, so it seems more like if you're going to send it you may as well conform to HTTP 1.1. But I guess that argument is somehow incorrect. I say that because a comment in some other code I'm looking at that uses httplib, says that prior to 2.6, httplib *used* to append a ":443" to SSL requests, but that it no longer does. I guess sending the port was dropped from httplib for good reason, and so HTTPPageGetter shouldn't add it. But I don't know. I'm very far from being an expert on HTTP headers though. Not as far as I'd like to be, though :-) Thanks again for the reply. Terry

1 0

Re: [Twisted-Python] Problem fetching page with getPage
by Terry Jones 03 Jan '10

03 Jan '10

>>>>> "Steve" == ssteinerX@gmail com <ssteinerx(a)gmail.com> writes: Steve> On Jan 2, 2010, at 9:34 AM, Terry Jones wrote: >> In any case, it looks like the problem is not in the setup of the request. >> Can anyone offer a reason why httplib might be able to fetch the page >> whereas getPage receives an error? I'm stumped. Steve> Steve> I've had to debug things like this recently and I have two suggestions: Hi Steve Thanks for the helpful reply - I can now make the call successfully. The difference turned out to be that httplib puts a Host: hostname:port header into its calls, whereas getPage uses just Host: hostname. Plus there was something else going on in some other code I'm using that made this a problem (it was calculating a signature based on host:port). Steve> 1> Recreate the headers and make it work with curl. Curl won't add Steve> anything to your headers and such and you'll be sure that you're Steve> getting the result you want with completely stripped down case. At least on my machine (curl 7.18.0 on Linux Ubuntu/Hardy) it adds a User-agent, an Accept: */*, and also the Host header. Steve> 2> Get Charles http://www.charlesproxy.com/ if you're on OS X. It Steve> rocks. Otherwise, get one of the Windows tools (sorry, no recos Steve> from me on that), and watch exactly what goes by. It's available for Linux & Windows too. I tried it, but didn't make it work fully when sending requests from the command line (with SSL, spoofing DNS, etc). So in the end I just used netcat -l -p 443 and changed to HTTP to see what was being sent. I wouldn't have thought of doing that without your suggestion, so thanks a lot for the tip. Terry

2 1

[Twisted-Python] Problem fetching page with getPage
by Terry Jones 03 Jan '10

03 Jan '10

I've run into a problem fetching an HTTP page with t.w.client.getPage. It's not simple to make standalone code showing what's going wrong, but the following summarizes where I am and why I find this puzzling. After some setup, I have some a url path, and some headers I want to send. A summary: host = 'ec2.amazon.com' port = 443 path = '/?some=params&are=here&etc=etc' method = 'GET' data = '' headers = { 'some' : 'headers', 'Content-Length' : '0' } url = 'https://%s:%d%s' % (host, port, path) the actual details don't matter right now, I don't think. When I call d = getPage(url, headers=headers) d's errback fires with a twisted.web.error.Error with a 403 status. So you'd think I had something wrong in my headers, or was trying to access a forbidden resource, etc. But.... when I drop this code in instead of the call to getPage: import httplib cx = httplib.HTTPSConnection(host, port) cx.request(method, path, data, headers) response = cx.getresponse() print 'response status:', response.status body = response.read() print 'body:', body I get a 200 status, and the body is exactly as expected. BTW, the path above does start with a slash. I've tried using HTTPClientFactory and reactor.connectSSL directly. I've tried with and without the '' postdata and Content-Length header. I've tried with Twisted 8.2.0 and 9.0.0. And of course I've checked many times that the URL and its query params requested by httplib and getPage are identical (apart from the time-sensitive signature). The reason it's not easy to provide a simple example is that the URL and headers have signed components, based in part on a timestamp, and based in part on Amazon secret keys, etc. It's not easy to separate all that, and if I did I'd be posting at least 100 lines of code that would only run if you had your Amazon AWS details provided etc. In any case, it looks like the problem is not in the setup of the request. Can anyone offer a reason why httplib might be able to fetch the page whereas getPage receives an error? I'm stumped. Terry

4 3

[Twisted-Python] Fwd: Problem fetching page with getPage
by ssteinerX＠gmail.com 02 Jan '10

02 Jan '10

Sorry... >> I had a situation where python's HTTPlib stuff was adding an Accept Encoding header >> that didn't put there, that I didn't put there,

1 0

[Twisted-Python] how manual run LoopingCall.f
by ploutosss ploutosss 01 Jan '10

01 Jan '10

I use _threadedselect reactor. My waker its one line: eventmodule.post(eventmodule.Event(TWISTEDEVENT, iterateTwisted=func)) its send event, to pygame queue. My pygame loop: class PygameController(object): _tick = Tick() def _eventIterator(self): while True: if self._ticking: yield eventmodule.poll() else: yield eventmodule.wait() def run(self): for event in self._eventIterator(): if event.type == NOEVENT: self.manager.post(self._tick) continue elif event.type == TWISTEDEVENT: event.iterateTwisted() if not reactor.running: break continue ... When I need to quickly perform some function, I set the parameter PygameController._ticking to True, and my manager call some function. Its work. When I need to perform some function with a time interval of more than one second, I use LoopingCall. Its work. When I need to perform some function with a time interval of less than one second, LoopingCall too slow. I try use my ticking, but for function is not executed too often, I used the following code: def some_function(self): looping_instance = self.looping call = looping_instance.call if call and call.time - call.seconds() <= 0: looping_instance() but I get this traceback when I stop looping (LoopingCall.stop()): File "/usr/lib64/python2.6/site-packages/twisted/internet/base.py", line 779, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/lib64/python2.6/site-packages/twisted/internet/task.py", line 196, in __call__ d.addErrback(eb) File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 199, in addErrback errbackKeywords=kw) File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 181, in addCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib64/python2.6/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/lib64/python2.6/site-packages/twisted/internet/task.py", line 191, in eb d.errback(failure) exceptions.AttributeError: 'NoneType' object has no attribute 'errback'

2 1