Mailman 3 December 2011 - Twisted-web

Test
by Philippe Lafoucrière 05 Aug '21

05 Aug '21

Hi all just a test

3 2

Porting the features of nevow.guard to twisted.web.guard
by L. Daniel Burr 05 Jan '12

05 Jan '12

Hi gang, Per a short chat with exarkun (thanks exarkun), I'm posting this to the list for discussion. A project upon which I am working requires a twisted.web service that allows a user to delegate authentication/authorization to a third party, e.g., Facebook, Twitter, et cetera. In so doing, I've found myself missing nevow.guard, since web.guard currently only has out-of-the-box support for HTTP Auth. I'd like to see web.guard expand to cover more kinds of auth, such as form-based auth or the OAuth-based solutions coming from so many social graph sites. With this in mind, I asked exarkun about this, and he mentioned that the only thing he does not consider to be garbage in nevow.guard is the feature-set, and that I should port the features to web, discarding the baggage. I'd like to solicit the list for some guidance as to what those features are, or should be, and hopefully end up with a ticket describing the features to be implemented, which I will then attempt to implement and contribute to twisted.web. Thanks, Daniel -- L. Daniel Burr ldanielburr(a)mac.com (312) 656-8387

5 5

twisted.web.template output encoding
by exarkun＠twistedmatrix.com 05 Jan '12

05 Jan '12

Hello all, I recently did some work on <http://twistedmatrix.com/trac/ticket/4896> ("twisted.web.util.formatFailure vulnerable to XSS in rare cases"). Rather than apply the patch adding extra html.escape calls, I tried porting the function to twisted.web.template (presently it builds the html string manually). Apart from various issues relating to the lack of patterns in twisted.web.template, the main difficulty is in handling non-ascii contents in the traceback. Apart from any unicode that may show up in the source code being rendered (or, perhaps, eventually, the values of variables to be rendered - though for now I do not plan to implement this) the no-break space characters which are necessary to get traceback lines indented properly mean that there is always some non-ascii to include in the output. twisted.web.template encodes its output using UTF-8, and this is not customizable. Thus, using twisted.web.template, formatFailure's result will be a str containing UTF-8 encoded text. Previously the result was a str containing only ASCII encoded text, with no-break space represented as ` �. Consequently, callers of `formatFailure� will probably mishandle the result - the caller in `twisted.web.server� does, at least, including the bytes in a page with a content type of "text/html". The solutions that come to mind are all about removing this incompatible change and making it so `formatFailure� can continue to return a str with ASCII-encoded text. One solution is to add support for named entities or numeric character references to twisted.web.template. Very likely this is a good idea regardless (Nevow supported these). Another solution is to use a different encoding in `twisted.web.template� - ASCII, with xmlcharrefreplace as the error handler. This is tempting since it avoids an obtrusive non-ASCII support API (the way Nevow supports these is via `nevow.entities�, which must be used rather than normal Python unicode objects). Perhaps another question is whether the encoding used by `twisted.web.template� should be a parameter. A related question raised might be whether `twisted.web.template� should encoded to bytes at all, or delegate the responsibility for that to code closer to a socket. As a work-around in `formatFailure� I can decode the output of the flattener using UTF-8 and then re-encode it to avoid non-ASCII, but it seems like this should be solved in `twisted.web.template� rather than over and over again in application code. Jean-Paul

5 7

Noob question about POST
by Donald Steiny 03 Jan '12

03 Jan '12

Hi, I have just been learning twisted and been Googling for an example of something that is simple to do with pycurl or even straight python but I can't figure out for the life of me how to do with twisted. I want to be able to do a POST and debug a login to a system. Ultimately, what I am trying to do (and twisted seems to be the best choice for this) is to make the login process automatic for the user so after he or she is logged in he or she will interact with the remote system as if he or she had logged in his or herself. The proxy class seems perfect for the second part and getPage should allow me to do the first, but I don't seem to understand something. What I have done is to set up a simple login/password page on my own server with login fred and password 1234. I am using Python 2.6 on Linux 2.6. I will describe what I expect to happen, then what I did, then what happened that I did not expect and if anyone could set me straight I would be eternally grateful AND because of the good Karma you will find parking spaces easily all during 2012. If I telnet to localhost port 80 and copy and paste the following: === BEGINNING OF WORKING INPUT TO SERVER === POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 ===== END OF WORKING INPUT TO SERVER ==== I get just what I would expect. On successful login it goes to the page "login_success.php"and I print out the _POST array to make sure the variables are getting passed: ==== BEGINNING OF CORRECT OUTPUT FROM SERVER === Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 HTTP/1.1 302 Found Date: Sat, 31 Dec 2011 18:42:28 GMT Server: Apache/2.2.16 (Debian) X-Powered-By: PHP/5.3.3-7+squeeze3 Set-Cookie: PHPSESSID=tueqdd71kha040d18d1qabg424; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: login_success.php Vary: Accept-Encoding Content-Length: 57 Content-Type: text/html IN CHECKLOGIN myusername: john, mypassword: 1234 Connection closed by foreign host. === END OF CORRECT OUTPUT FROM SERVER === I took a getPage example that is in the documentation: === START CODE === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) agent = Agent(reactor) d = agent.request( 'GET', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): reactor.stop() d.addBoth(cbShutdown) reactor.run() ==== END CODE === The output was just what I expected: === OUTPUT FROM RUN OF CODE == Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 18:55:17 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN myusername: , mypassword: Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ===== END OF OUTPUT FROM RUN OF CODE -- I modified the example by changing the GET to a POST and including StringProducer and changing the None to a body and the header to multipart/form-data. Here: ==== START OF CODE MODIFIED WITH POST === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers from twisted.web.iweb import IBodyProducer from zope.interface import implements, Interface from stringprod import StringProducer class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) creds = "myusername=john&mypassword=1234\r\n" body = StringProducer(creds) agent = Agent(reactor) d = agent.request( 'POST', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Mozilla 5.0'], 'Content-Type': ['multipart/form-data; charset=utf-8']}), body) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): print "shutdown called" reactor.stop() d.addBoth(cbShutdown) reactor.run() === END CODE MODIFIED WITH POST === When I run this, I do not get the output I expect, for some reason, it does not seem to be sending the body. ==== BEGINNING OF RUN OF MODIFIED CODE ==== Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 19:06:39 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN myusername: , mypassword: Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ==== END OF RUN OF MODIFIED CODE === What am I missing? The body does not seem to be being send. I would like to have a way to easily be able to debug logins to remote systems. This is NOT for spamming. I am trying to develop a system for higher security where our users would log locally (VPN) with their own favorite passwords and that would map to awful username and password that no one would ever use or remember and would be unguessable to log them into external cloud based systems. In some cases there are libraries and APIs to help me, but I want a general solution. Mechanic seems to do part of this and pycurl, but the proxy and networking part seem much better in twisted. HELP!!! Thanks, Donald

2 1

manually inject data into a site object
by gary jefferson 29 Dec '11

29 Dec '11

I have a web service written in twisted-web. I have it running behind a NAT. I have a component (based on twisted Protocol) that makes a connection out from behind the NAT to a relay. Let's call this ReverseThing. Web clients can connect to the relay to send data back over this ReverseThing. Once I receive the data, how can I inject it directly into the web service? Assume both these services run in the same twisted process. Here's what I've tried: I can pass in the Site object to the ReverseThing protocol, so that I can call p=buildProtocol() on it, then I can call dataReceived() manually on that p. But obviously this doesn't work, because there isn't really a transport object to call things like getPeer() on. Is there some other way to send data to a Site, without it actually coming over the wire? Thanks, Gary

2 1

Twisted web client digest authentication?
by Paul de Vries 05 Dec '11

05 Dec '11

Hi Made something that connects to a httpserver and parses the xml stream it sends. The webserver it has to connect to uses digest authentication. How can I implement digest authentication in this? thanks in advance the code I put together: from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers from StringIO import StringIO from elementtree import ElementTree class ParseReceived(Protocol): def __init__(self, finished): self.finished = finished self.dataStream = StringIO() def dataReceived(self, bytes): display = bytes #print 'some data received:' #print display context = ElementTree.iterparse(StringIO(bytes), events = ("start", "end")) context = iter(context) event, root = context.next() for event, elem in context: if elem.tag == "Tag": print "Tag " root.clear() def connectionLost(self, reason): print 'finished receiving', reason.getErrorMessage() agent = Agent(reactor) d = agent.request('GET', 'http:someadress', Headers({'User-Agent': ['Twisted web client']}), None) def cbResponse(response): print 'Response version: ', response.version finished = Deferred() response.deliverBody(ParseReceived(finished)) return finished d.addCallback(cbResponse) reactor.run()

2 2