Mailman 3 June 2005 - Twisted-web

Re: Twisted-web Digest, Vol 15, Issue 20
by Richard Meraz June 28, 2005

June 28, 2005

Thanks Dave: very clear and easy to follow answer and example code. I definitely appreciate your time. Final question. Is there a convenient way to put an upper-bound on how long twisted.web.client.getPage is allowed to complete its work. I know twisted.web.client.getPage takes a timeout parameter, but this seems more like a socket timeout which won't kill for example a getPage waiting on a low-bandwidth server (is my understanding correct?) For example, if I'm using the asyncore.py framework to mange IO i can use the channel.timestamp attribute to examine how long things have been going in order to kill long-running IO in a polling loop. Of course with asyncore I can manage my own polling loop which I can't see an easy way to do using reactor() (someone care to comment on that since I'm probably missing something). -Thanks again -Richard Meraz On 6/23/05, twisted-web-request(a)twistedmatrix.com <twisted-web-request(a)twistedmatrix.com> wrote: > Send Twisted-web mailing list submissions to > twisted-web(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web > or, via email, send a message with subject or body 'help' to > twisted-web-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-web-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-web digest..." > > > Today's Topics: > > 1. Re: Defers, the reactor, and idiomatic/proper usage -- new > user needs some advice? (Dave Gray) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 23 Jun 2005 12:18:06 -0400 > From: Dave Gray <dgray(a)omniti.com> > Subject: Re: [Twisted-web] Defers, the reactor, and idiomatic/proper > usage -- new user needs some advice? > To: Richard Meraz <rfmeraz(a)gmail.com>, "Discussion of twisted.web, > Nevow, and Woven" <twisted-web(a)twistedmatrix.com> > Message-ID: <42BAE0BE.1040209(a)omniti.com> > Content-Type: text/plain; charset="windows-1252" > > I'm not familiar with feedlib, etc, but I'll answer what I can. > > Richard Meraz wrote: > > MAXTIME = 60 # Kill crawl after this time > > TIMEOUT = 20 # Kill page retrieval after this time inactive > > MAXDEPTH = 3 # Recurse this depth when crawling page. > > > > # Question: There seem to be many idioms to aggregate information from > > different defered call-back chains in twisted.. Since everything runs > > in a single thread I just stuck my stuff in a global class and everybody > > modifies the vars there as I pass it around to the call-backs that > > should see it. Seems okay for a small script like this? > > That seems fine, yeah. I think I would pass around the StateVars > instance as a context if I were coding this. Probably the same effect. > > > class StateVars: > > '''Keep Global state for starting/stopping feedfinding and a record > > of links we have checked and their status''' > > connections = 1 > > links_checked = {} # Structure: {url: (is RSS/ATOM/RDF, page-content)} > > > > # Question: start_feed_crawl is where I set up my defers. getPage > > returns a defer and I attach my call-back process_link. > > # addCallbacks adds a callback/errback in parallel so only one or the > > other is called? so I need to add > > # the final errback to catch errors from callback process_link ? > > Correct. Well, sort of. See below. > > > def start_feed_crawl(uri,depth): > > '''Harvest feeds from a uri''' > > # Question: how to time-out this deferred chain if getPage is taking too > > long to finish its work. > > # what exactly does the argument timeout to getPage do, does it timeout > > the socket after a no-response > > # or does it put an upper-bound on how long getPage has to finish its work? > > > > getPage(uri, timeout=TIMEOUT).addCallbacks(callback=process_link, > > callbackArgs=(uri, > > depth, StateVars), > > errback = > > process_error, > > errbackArgs=(uri,StateVars) > > ).addErrback(process_error, > > uri, StateVars) > > It seems clearer to me to write this as follows, but that's personal > preference: > > d = getPage(...) > d.addCallbacks(...) > d.addErrback(...) > > But since you're setting up the call to the same errback twice, you > could simplify this to: > > d = getPage(...) > d.addCallback(process_link, uri, depth, StateVars) > d.addErrback(process_error, uri, StateVars) > > <http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto4> > has a nice visual explanation of what happens when. > > > # Question: since I'm starting up these defers in a callback they are > > # being created after I've called reactor.run() since we call start_feed_crawl > > # as we find new links that meet our criteria. Am I doing anything bad here? > > # All the examples I've seen (eg. p. 548-552 Python Cookbook, great eg by V. Volonghi > > # and P. Cogolo) have their data up-front and therefore set-up all the defers before calling > > # reactor.run(). Here I'm discovering my data as I go along and setting up deferrs while > > # the reactor is spinning. Here is my fundamental lack of understanding. While this script > > # seems to run okay, is it okay to do this? > > Yes, that's fine. I think the one you've seen the most is the odd case - > being able to set up all the Deferreds beforehand. > > > # Question: Is this how I kill the reactor -- ie. using some sort of > > state condition. Is there a better way, > > # should I try better to understand deferred-list. For example. A > > top-level deferred-list that contains > > # other deferred-lists which get created to hold all the defers > > (created by start_feed_crawl) for the > > # links on a given page. Could this deferred-list be told to stop > > the reactor when the other lists have > > # fired their callback (after the component defers have finished) ? > > (Sorry for the convoluted question here > > # I'm new at this) > > What you want to do is stop the reactor when everything is done > processing. So after you call start_feed_crawl the first time, returning > the Deferred that getPage gives you, you can add a callback to that > which stops the reactor. The trick here is that if you stuff that > deferred into a DeferredList before you add the callback that stops the > reactor then if your first operation itself returns a deferred, the > DeferredList won't call its callbacks until the other Deferred operation > completes. So you'll be stacking up a whole bunch of Deferreds inside > the first one, and the callback on the DeferredList that does the > reactor.stop won't fire until you don't return a Deferred. > > There might be an easier way to do this, but this the way I know > (example attached). Someone please let me know if there's an easier way. > To see the example, run it with 'twistd -noy fetchpage.tac' then do > 'telnet localhost 9000' and send: > > GET /?target=http://www.google.com/ HTTP/1.1 > Host: localhost > > > > > Final question: occasionally I get errors that come from the http.py > > code in twisted. This get printed to the console, but don't necessarily > > stop my program. Should my errbacks be catching these? How do I keep > > errors from getting logged to the console (beside redirecting stderr). I > > can post an example if necessary of the errors I'm getting. > > When you create the DeferredList, pass in consumeErrors=1 - this will > make debugging that much more annoying though... > > HTH, > Dave >

2 1

Generating exe using Twisted py2exe
by Gregory Pereira June 28, 2005

June 28, 2005

Hi, we developed an application that uses Twisted to implement a Client, Server for a board game. We know have to generate an exe for the Game. So far it seems that Py2exe is the best way to do it. How difficult will it be including the Twisted Application? The main problems i may experience are where installed packages use the Windows registry - it's obviously impossible for py2exe to pick up on this. Regards Gregory Pereira Tel: +27 12 672 6203 Mobile: +27 83 359 5093 Fax +27 12 672 6022 E-Mail: gpereira(a)grintek.com Avitronics

3 2

memory leak with metaclass
by Tazzo June 28, 2005

June 28, 2005

We use metaclass for building on the fly new rendering pages and we found a memory leak. Managing with garbage collector we found that new classes are not released for unknown reason. Here a basic source that leak memory: #----------------------- print 'running server' import nevow from nevow import appserver from twisted.application import service from twisted.application import internet class Controller(nevow.rend.Page): def locateChild(self,context,segments): return type('Leaker',(nevow.rend.Page,),{'docFactory':nevow.loaders.stan(nevow.tags.h1['I leak memory'])})(),[] application=service.Application('leaker') webservice = internet.TCPServer(12345,appserver.NevowSite(Controller())) webservice.setServiceParent(application) ##AND USE THIS FOR SEE MEMORY LEAK ##import urllib ##for i in range(99999): ## urllib.urlopen('http://localhost:12345/') #----------------------- Is it a bug? Is there any way to avoid the memory leak? #tazzo #http://www.pytypus.org/ -- Email.it, the professional e-mail, gratis per te: http://www.email.it/f Sponsor: Occhialeria.it Scopri le migliori marche a prezzi imbattibili Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=879&d=20050625

3 2

Nevow 0.4.1 release tarball shitlist; nevow 0.4.1-1 deb in incoming
by Tommi Virtanen June 27, 2005

June 27, 2005

Hi. I just (finally) packaged nevow 0.4.1 for Debian, it's sitting in Incoming and will hit unstable a bit later. Meanwhile, here are the reasons the nevow 0.4.1 release tarball sucks: - .svn dirs - compared to 0.4.1 tag, missing: doc/man/nevow-xmlgettext.1 examples/i18n/update-l10n examples/pastebin/TODO examples/pastebin/static/robots.txt examples/wsgi/README examples/wsgi/test-cgi.py extras/xhtml-nevow.rnc nevow/canvas.as nevow/Canvas.fla test.py win32/nevow.nsi

1 0

Twisted network package for python implementation problem with GUI
by Gregory Pereira June 27, 2005

June 27, 2005

Hi We've got the client and server running with twisted, and it works well only for one problem. When exiting the game, it seems like the connection is not closing properly. This is the error that it gives. sys.exit() exceptions.SystemExit: Unhandled error in Deferred: Traceback from remote host -- Traceback unavailable We have basically two main loops namely, reactor.run() used to get the the connection and listening on port 8000, spinner.run() used to watch the events. We used the reactor.iterate() in the spinner.run and it worked well. All the networking works well but we can't close the connection on exit. We tried to use reactor.stop() but it doesn't seem to reach it. Greg

1 0

Uploading files
by Alberto Trujillo June 22, 2005

June 22, 2005

Could somebody explain me how can I upload a file to the server, or how could I sent it by mail. Thanks.

1 0

Returning an HTTP error from render method
by Jonathan Blocksom June 22, 2005

June 22, 2005

Hi, I'm trying to figure out how to return an HTTP error in the render method of a custom Resource. I can't find any docs on it, but from poking around in the code I came up with the code below that doesn't do it. Can somebody point me in the right direction for returning an error from the render method? Or should I be returning the error somewhere else? Thanks. --- from twisted.web import server from twisted.internet import reactor from twisted.web.resource import Resource class Sample404(Resource): "Returns a 404 error for everything except /hello" isLeaf = True def render_GET(self, request): if (request.path != '/hello'): request.setResponseCode(404) request.finish() return server.NOT_DONE_YET else: return "<html>You found me!</html>" if __name__=='__main__': site = server.Site(Sample404()) reactor.listenTCP(8080, site) reactor.run()

2 1

Nevow, formless: complex widgets
by Little June 22, 2005

June 22, 2005

How can i build a "complex control" which consists of a number of standart widgets? For example "DateWidget" which consits of a three standard annotate.Listwidgets DayList, MonthList and YearList? This is a simple example. More complex example is a String+List widget, where the List data is based on String data submitted. Explain: 1. String is empty. List is Emply. 2. User types a phrase into String and presses Submit button. 3. Same form, with ERROR in List (whis is not selected) but List is not emply, it's filled from DataBase using String value like filter in query. 4. A user selects a value from List and presses submit again The most copmlex example is a String+List with a live XMLHTTP-request when List filled from XMLHTTPrequest. I do it in past on PHP, but i can't find any way how i can do it in nevow (using formless, not manual form submitting)

1 0

domish.elementStream()
by Ross Jekel June 21, 2005

June 21, 2005

Hello, Neither or the two possible element stream classes when parsing correctly hook the root element to its children. This is pretty easy to see why from the code (below) but thought I would check to see if this is on purpose or something. What I wanted to do was just save off the root element passed to DocumentStartEvent from either class and be able to do XPathQueries on it. Since the root element returned doesn't have any children though, it fails. The code for both classes is similar to this: # Construct the new element e = Element(qname, self.defaultNsStack[-1], attrs) # Document already started if self.documentStarted == 1: if self.currElem != None: self.currElem.children.append(e) e.parent = self.currElem self.currElem = e # New document else: self.documentStarted = 1 self.DocumentStartEvent(e) Since self.currElem isn't set to e for the first element, e never gets it's children list filled in. Below is a sample program that shows what I'm talking about. It prints "0 []" when it should print "1 [<<twisted.xish.domish.Element object...>, ...]". from twisted.xish import domish from twisted.xish import xpath # Taken from twisted/xish/test/test_xpath.py xmlBlob = """ <foo xmlns='testns' attrib1='value1' attrib3="user@host/resource"> somecontent <bar> <foo> <gar>DEF</gar> </foo> </bar> somemorecontent <bar attrib2="value2"> <bar> <foo/> <gar>ABC</gar> </bar> </bar> </foo> """ p = domish.SuxElementStream() elem = list() p.DocumentStartEvent = lambda e: elem.append(e) p.ElementEvent = lambda e: None p.DocumentEndEvent = lambda: None p.parse(xmlBlob) print xpath.matches("/foo/bar", elem[0]), elem[0].children Thanks, Ross

1 0

Shutting down a xmlrpc server with an xmlrpc call
by Rasjid Wilcox June 20, 2005

June 20, 2005

Hi, I am new to twisted-web, so my apologies if this is obvious. I am happy to be pointed at the appropriate documentation. I am writing an xmlrpc server, and would like to there to be an xmlrpc call to shutdown the server gracefully. Ideally upon the call being made, it would return a 'success' response to the client, the reactor would stop taking any new requests, would finish any existing ones, and then exit cleanly. What I currently have is - def xmlrpc_quit(self): reactor.stop() but then the caller gets: >>> import xmlrpclib >>> s = xmlrpclib.Server('http://localhost:7080/') >>> s.quit() Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.3/xmlrpclib.py", line 1029, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.3/xmlrpclib.py", line 1316, in __request verbose=self.__verbose File "/usr/lib/python2.3/xmlrpclib.py", line 1070, in request headers xmlrpclib.ProtocolError: <ProtocolError for localhost:7080/: -1 > All suggestions / hints would be greatly appreciated. Cheers, Rasjid. -- Rasjid Wilcox Melbourne, Australia (UTC +10 hrs)

3 3