Mailman 3 January 2012 - Twisted-web

Test
by Philippe Lafoucrière Aug. 5, 2021

Aug. 5, 2021

Hi all just a test

3 2

cooperate and keeping data integrity in response
by John Aherne Jan. 16, 2012

Jan. 16, 2012

I have been looking at JPCalderones example of using web.request with JSON which seems much like what I want. One thing I am not clear about is if I get a lot of queries coming in more or less simultaneously and I am using cooperate to allow other functions to run, will I need to guard against my data in a list being overwritten by subsequent requests. The way I see it the functions to read data and and store it in my list are in danger of impacting each other. The response is being built cooperatively bit by bit to permit other functions to run so it could happen that the next request overwrites my list where the database query is being stored. If this is a danger, then I need to prevent this, which seems to imply that I will need to block each request and not service another request until the previous one has completed. Have I got that right or am I way off target or have I missed the obvious What would be good is to keep on servicing requests so that the response is good but keep the data integrity. The test code I am using is JP's with some minor variations shown below. I hope it formats correctly. Thanks for any help. Regards John Aherne #Asynchronous JSON #Today in #twisted.web the topic of generating large JSON responses #in a Twisted Web server came up. The problem was that the data being #serialized into JSON was so large that the JSON serialization process #itself would block the web server, preventing other requests from being #serviced. # #The first solution that came up was to split the web server into two #pieces, so that the URLs which could have these JSON responses were #served by a different process than was serving the rest. This is a #pretty decent solution, and it also provides the benefit of using extra #CPU cores if there are any available. In this case, it complicated #things a little since it meant sharing a session across two processes. #So we went looking for another approach. # #It turns out that the json module supports incremental serialization. #When I saw the JSONEncoder.iterencode method, I thought it would be #great used in combination with cooperate to create a producer. This #would let an application serialize a large structure to JSON without #multiple processes, threads, or unreasonably blocking the reactor. # #Here's the little bit of glue necessary to make things work: import cgi from json import JSONEncoder from twisted.enterprise import adbapi from twisted.internet.task import cooperate #db = sqlite3.connect('c:\\sqlite\\test.db') #cur = db.cursor() dbpool = adbapi.ConnectionPool("pyodbc","DSN=testsql",cp_reconnect='True') class AsyncJSON(object): def __init__(self, value): self._value = value def beginProducing(self, consumer): #print 'value', self._value self._consumer = consumer self._iterable = JSONEncoder().iterencode(self._value) #print 'iterable', self._iterable self._consumer.registerProducer(self, True) self._task = cooperate(self._produce()) d = self._task.whenDone() d.addBoth(self._unregister) return d def pauseProducing(self): self._task.pause() def resumeProducing(self): self._task.resume() def stopProducing(self): self._task.stop() def _produce(self): for chunk in self._iterable: #print 'chunk', chunk self._consumer.write(chunk) yield None def _unregister(self, passthrough): self._consumer.unregisterProducer() return passthrough #By using the iterencode method, this avoids spending too much time #generating json output at once. Instead, a little bit of the input #will be serialized at a time, and each short resulting string is available #from the iterator returned by iterencode. # #By using cooperate, the _produce generator will iterated in a way that #lets it cooperate with the reactor and other event sources/handlers. #A few chunks of json data will be written to the consumer, then execution #will switch away to something else, then come back and a few more will #be written, and so on. # #And by using the producer/consumer interface, if the HTTP client which #issued the request doesn't read the results as fast as they're being #generated, the server will stop generating new output until the client #catches up. # #Altogether, this provides a very cool, efficient way to generate JSON #output. # #Here's an example to make it easier to see how one might use AsyncJSON #in a resource: # from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET from twisted.web.server import Site from twisted.internet import reactor def read_pcodes(pcode, request): """ Read postcode data and premise data for single postocde """ sql_mail = """select rcmplc01.substreet,rcmplc01.street, rcmplc01.sublocality, rcmplc01.locality, rcmplc01.town, rcmplc01.postcode, rcmplc02.data from rcmplc01 left outer join rcmplc02 on rcmplc01.postcode = rcmplc02.postcode where rcmplc01.postcode = ? """ pcode = pcode.strip().upper() def run(): return dbpool.runQuery(sql_mail,(pcode,)) d = run() d.addCallback(read_result, request) d.addErrback(read_failure,request) return d def read_failure(o, request): print 'failure', str(o) request.finish() def read_result(res, request): """ read result for postcode lookup. Build return list""" #print 'res', res print 'len res', len(res) my_list = [] for item in res: my_list.append([item[0], item[1], item[2], item[3], item[4], item[5]]) d = AsyncJSON(my_list).beginProducing(request) d.addCallback(lambda ignored: request.finish()) d.addErrback(got_error, request) root = Resource() root.putChild("json", PostcodeFinder()) factory = Site(root) reactor.listenTCP(8086, factory) reactor.run()

2 2

cancelling agent.request
by John Aherne Jan. 15, 2012

Jan. 15, 2012

I've just glanced through 7 years discussion on cancelling defers. So I think I'll take a breather before going further. John Aherne

1 0

cancelling agent.request
by John Aherne Jan. 15, 2012

Jan. 15, 2012

I have a situation where a service I am using will occasionally not respond for considerable periods - hours not minutes. I am using cooperate and yield to loop round a series of requests I want to make The actual function to send the request returns a deferred to the yield. I add the callbacks onto this and set a calllater to fire in 30 seconds. If that fires I can then cancel the deferred. But I think I ought to be cancelling the request as well else I assume I will be leaking request objects that might not get dealt with. However, I can't see where I might cancel the request. Or should I be dropping the connection associated with the request and if so where would I be doing that. Have I got that right? Thanks for any info. John Aherne

2 2

cooperate and keeping data integrity in response
by John Aherne Jan. 14, 2012

Jan. 14, 2012

Sorry about the double post. But copy/paste failed me. I left out part of the code. Now included I have been looking at JPCalderones example of using web.request with JSON which seems much like what I want. One thing I am not clear about is if I get a lot of queries coming in more or less simultaneously and I am using cooperate to allow other functions to run, will I need to guard against my data in a list being overwritten by subsequent requests. The way I see it the functions to read data and and store it in my list are in danger of impacting each other. The response is being built cooperatively bit by bit to permit other functions to run so it could happen that the next request overwrites my list where the database query is being stored. If this is a danger, then I need to prevent this, which seems to imply that I will need to block each request and not service another request until the previous one has completed. Have I got that right or am I way off target or have I missed the obvious What would be good is to keep on servicing requests so that the response is good but keep the data integrity. The test code I am using is JP's with some minor variations shown below. I hope it formats correctly. Thanks for any help. Regards John Aherne #Asynchronous JSON #Today in #twisted.web the topic of generating large JSON responses #in a Twisted Web server came up. The problem was that the data being #serialized into JSON was so large that the JSON serialization process #itself would block the web server, preventing other requests from being #serviced. # #The first solution that came up was to split the web server into two #pieces, so that the URLs which could have these JSON responses were #served by a different process than was serving the rest. This is a #pretty decent solution, and it also provides the benefit of using extra #CPU cores if there are any available. In this case, it complicated #things a little since it meant sharing a session across two processes. #So we went looking for another approach. # #It turns out that the json module supports incremental serialization. #When I saw the JSONEncoder.iterencode method, I thought it would be #great used in combination with cooperate to create a producer. This #would let an application serialize a large structure to JSON without #multiple processes, threads, or unreasonably blocking the reactor. # #Here's the little bit of glue necessary to make things work: import cgi from json import JSONEncoder from twisted.enterprise import adbapi from twisted.internet.task import cooperate #db = sqlite3.connect('c:\\sqlite\\test.db') #cur = db.cursor() dbpool = adbapi.ConnectionPool("pyodbc","DSN=testsql",cp_reconnect='True') class AsyncJSON(object): def __init__(self, value): self._value = value def beginProducing(self, consumer): #print 'value', self._value self._consumer = consumer self._iterable = JSONEncoder().iterencode(self._value) #print 'iterable', self._iterable self._consumer.registerProducer(self, True) self._task = cooperate(self._produce()) d = self._task.whenDone() d.addBoth(self._unregister) return d def pauseProducing(self): self._task.pause() def resumeProducing(self): self._task.resume() def stopProducing(self): self._task.stop() def _produce(self): for chunk in self._iterable: #print 'chunk', chunk self._consumer.write(chunk) yield None def _unregister(self, passthrough): self._consumer.unregisterProducer() return passthrough #By using the iterencode method, this avoids spending too much time #generating json output at once. Instead, a little bit of the input #will be serialized at a time, and each short resulting string is available #from the iterator returned by iterencode. # #By using cooperate, the _produce generator will iterated in a way that #lets it cooperate with the reactor and other event sources/handlers. #A few chunks of json data will be written to the consumer, then execution #will switch away to something else, then come back and a few more will #be written, and so on. # #And by using the producer/consumer interface, if the HTTP client which #issued the request doesn't read the results as fast as they're being #generated, the server will stop generating new output until the client #catches up. # #Altogether, this provides a very cool, efficient way to generate JSON #output. # #Here's an example to make it easier to see how one might use AsyncJSON #in a resource: # from twisted.web.resource import Resource from twisted.web.server import NOT_DONE_YET from twisted.web.server import Site from twisted.internet import reactor def read_pcodes(pcode, request): """ Read postcode data and premise data for single postocde """ sql_mail = """select rcmplc01.substreet,rcmplc01.street, rcmplc01.sublocality, rcmplc01.locality, rcmplc01.town, rcmplc01.postcode, rcmplc02.data from rcmplc01 left outer join rcmplc02 on rcmplc01.postcode = rcmplc02.postcode where rcmplc01.postcode = ? """ pcode = pcode.strip().upper() def run(): return dbpool.runQuery(sql_mail,(pcode,)) d = run() d.addCallback(read_result, request) d.addErrback(read_failure,request) return d def read_failure(o, request): print 'failure', str(o) request.finish() def read_result(res, request): """ read result for postcode lookup. Build return list""" #print 'res', res print 'len res', len(res) my_list = [] for item in res: my_list.append([item[0], item[1], item[2], item[3], item[4], item[5]]) d = AsyncJSON(my_list).beginProducing(request) d.addCallback(lambda ignored: request.finish()) d.addErrback(got_error, request) class PostcodeFinder(Resource): """ Handle http POST requests for postcode lookup""" def render_POST(self, request): print 'req', request.args if request.args.get('pcode', None): pcode = cgi.escape(request.args['pcode'][0]).strip().upper() if (pcode[-3:][0].isdigit() and pcode[-2:].isalpha() and len(pcode[:len(pcode)-3].strip()) < 5): postcode2 = '%s %s' % (pcode[:len(pcode)-3].strip(),pcode[-3:]) d = read_pcodes(postcode2, request) else: return 'Not a VALID REQUEST' elif request.args.get('street', None): m25 = True if request.args.get('M25', None) else False d = read_street(cgi.escape(request.args['street'][0]).strip().upper(), m25, request) elif request.args.get('orgname', None): d = read_orgname(cgi.escape(request.args['orgname'][0]).strip().upper(), request) else: return 'Not a VALID REQUEST' return NOT_DONE_YET root = Resource() root.putChild("json", PostcodeFinder()) factory = Site(root) reactor.listenTCP(8086, factory) reactor.run()

1 0

Porting the features of nevow.guard to twisted.web.guard
by L. Daniel Burr Jan. 5, 2012

Jan. 5, 2012

Hi gang, Per a short chat with exarkun (thanks exarkun), I'm posting this to the list for discussion. A project upon which I am working requires a twisted.web service that allows a user to delegate authentication/authorization to a third party, e.g., Facebook, Twitter, et cetera. In so doing, I've found myself missing nevow.guard, since web.guard currently only has out-of-the-box support for HTTP Auth. I'd like to see web.guard expand to cover more kinds of auth, such as form-based auth or the OAuth-based solutions coming from so many social graph sites. With this in mind, I asked exarkun about this, and he mentioned that the only thing he does not consider to be garbage in nevow.guard is the feature-set, and that I should port the features to web, discarding the baggage. I'd like to solicit the list for some guidance as to what those features are, or should be, and hopefully end up with a ticket describing the features to be implemented, which I will then attempt to implement and contribute to twisted.web. Thanks, Daniel -- L. Daniel Burr ldanielburr(a)mac.com (312) 656-8387

5 5

twisted.web.template output encoding
by exarkun＠twistedmatrix.com Jan. 5, 2012

Jan. 5, 2012

Hello all, I recently did some work on <http://twistedmatrix.com/trac/ticket/4896> ("twisted.web.util.formatFailure vulnerable to XSS in rare cases"). Rather than apply the patch adding extra html.escape calls, I tried porting the function to twisted.web.template (presently it builds the html string manually). Apart from various issues relating to the lack of patterns in twisted.web.template, the main difficulty is in handling non-ascii contents in the traceback. Apart from any unicode that may show up in the source code being rendered (or, perhaps, eventually, the values of variables to be rendered - though for now I do not plan to implement this) the no-break space characters which are necessary to get traceback lines indented properly mean that there is always some non-ascii to include in the output. twisted.web.template encodes its output using UTF-8, and this is not customizable. Thus, using twisted.web.template, formatFailure's result will be a str containing UTF-8 encoded text. Previously the result was a str containing only ASCII encoded text, with no-break space represented as ` �. Consequently, callers of `formatFailure� will probably mishandle the result - the caller in `twisted.web.server� does, at least, including the bytes in a page with a content type of "text/html". The solutions that come to mind are all about removing this incompatible change and making it so `formatFailure� can continue to return a str with ASCII-encoded text. One solution is to add support for named entities or numeric character references to twisted.web.template. Very likely this is a good idea regardless (Nevow supported these). Another solution is to use a different encoding in `twisted.web.template� - ASCII, with xmlcharrefreplace as the error handler. This is tempting since it avoids an obtrusive non-ASCII support API (the way Nevow supports these is via `nevow.entities�, which must be used rather than normal Python unicode objects). Perhaps another question is whether the encoding used by `twisted.web.template� should be a parameter. A related question raised might be whether `twisted.web.template� should encoded to bytes at all, or delegate the responsibility for that to code closer to a socket. As a work-around in `formatFailure� I can decode the output of the flattener using UTF-8 and then re-encode it to avoid non-ASCII, but it seems like this should be solved in `twisted.web.template� rather than over and over again in application code. Jean-Paul

5 7

Noob question about POST
by Donald Steiny Jan. 3, 2012

Jan. 3, 2012

Hi, I have just been learning twisted and been Googling for an example of something that is simple to do with pycurl or even straight python but I can't figure out for the life of me how to do with twisted. I want to be able to do a POST and debug a login to a system. Ultimately, what I am trying to do (and twisted seems to be the best choice for this) is to make the login process automatic for the user so after he or she is logged in he or she will interact with the remote system as if he or she had logged in his or herself. The proxy class seems perfect for the second part and getPage should allow me to do the first, but I don't seem to understand something. What I have done is to set up a simple login/password page on my own server with login fred and password 1234. I am using Python 2.6 on Linux 2.6. I will describe what I expect to happen, then what I did, then what happened that I did not expect and if anyone could set me straight I would be eternally grateful AND because of the good Karma you will find parking spaces easily all during 2012. If I telnet to localhost port 80 and copy and paste the following: === BEGINNING OF WORKING INPUT TO SERVER === POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 ===== END OF WORKING INPUT TO SERVER ==== I get just what I would expect. On successful login it goes to the page "login_success.php"and I print out the _POST array to make sure the variables are getting passed: ==== BEGINNING OF CORRECT OUTPUT FROM SERVER === Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 HTTP/1.1 302 Found Date: Sat, 31 Dec 2011 18:42:28 GMT Server: Apache/2.2.16 (Debian) X-Powered-By: PHP/5.3.3-7+squeeze3 Set-Cookie: PHPSESSID=tueqdd71kha040d18d1qabg424; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: login_success.php Vary: Accept-Encoding Content-Length: 57 Content-Type: text/html IN CHECKLOGIN myusername: john, mypassword: 1234 Connection closed by foreign host. === END OF CORRECT OUTPUT FROM SERVER === I took a getPage example that is in the documentation: === START CODE === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) agent = Agent(reactor) d = agent.request( 'GET', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): reactor.stop() d.addBoth(cbShutdown) reactor.run() ==== END CODE === The output was just what I expected: === OUTPUT FROM RUN OF CODE == Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 18:55:17 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN myusername: , mypassword: Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ===== END OF OUTPUT FROM RUN OF CODE -- I modified the example by changing the GET to a POST and including StringProducer and changing the None to a body and the header to multipart/form-data. Here: ==== START OF CODE MODIFIED WITH POST === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers from twisted.web.iweb import IBodyProducer from zope.interface import implements, Interface from stringprod import StringProducer class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) creds = "myusername=john&mypassword=1234\r\n" body = StringProducer(creds) agent = Agent(reactor) d = agent.request( 'POST', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Mozilla 5.0'], 'Content-Type': ['multipart/form-data; charset=utf-8']}), body) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): print "shutdown called" reactor.stop() d.addBoth(cbShutdown) reactor.run() === END CODE MODIFIED WITH POST === When I run this, I do not get the output I expect, for some reason, it does not seem to be sending the body. ==== BEGINNING OF RUN OF MODIFIED CODE ==== Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 19:06:39 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN myusername: , mypassword: Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ==== END OF RUN OF MODIFIED CODE === What am I missing? The body does not seem to be being send. I would like to have a way to easily be able to debug logins to remote systems. This is NOT for spamming. I am trying to develop a system for higher security where our users would log locally (VPN) with their own favorite passwords and that would map to awful username and password that no one would ever use or remember and would be unguessable to log them into external cloud based systems. In some cases there are libraries and APIs to help me, but I want a general solution. Mechanic seems to do part of this and pycurl, but the proxy and networking part seem much better in twisted. HELP!!! Thanks, Donald

2 1