Mailman 3 June 2005 - Twisted

[Twisted-Python] Weekly Bug Summary
by James Y Knight June 22, 2005

June 22, 2005

Bug summary ______________________ Since 2005-06-15 01:17:07.440000: Bugs opened: 9 Bugs closed: 1 Total open bugs: 396 (+8) Urgent: 9 (+2) Bug: 167 (+4) Feature: 144 (+1) Wish: 76 (+1) New / Reopened Bugs ______________________ ==== Urgent ==== [CORE] LineReceiver Producer/Consumer support is broken (2 days ago) http://twistedmatrix.com/bugs/issue1087 opened by exarkun [CORE, DOCUMENTATION] gtk2reactor thread initialization confuses python-2.3 … [View More]

1 0

[Twisted-Python] How could I distinguish each TCP connection?
by Eric Hsu June 22, 2005

June 22, 2005

Hi all, I'm familiar with the thread-style network programming (create a single thread for each connected client). With twisted, how could I know which client I'm talking to? (or, how could I maintain the connections list?) I'm thinking of creating a Client object to represent each connected client Could I save the transport to the Client object so that I can retrive that transport back to talk to a specific client? Please point me to somewhere I could find the answer or enlighten me with … [View More]

3 5

[Twisted-Python] defers, reactor, idiomatic/proper usage -- new user questions.
by Richard Meraz June 21, 2005

June 21, 2005

Hello all, Just getting started with Twisted. Thanks to the community for tremendous work. I diligently went through the entire archives of the mailing list -- reading choice threads -- and I read through all the documentation on the twisted-matrix website. I've never done any event-driven programming, but there was enough on the site for me to start getting a handle on things. Below is a short script that crawls the links from a URI looking for RSS-type feeds. I'm hoping that some of … [View More]the more experienced developers would be willing to give some advice about whether I'm using twisted.internet.reactor and twisted.web.client.getPage correctly. I've put some comments labeled as #Question where I'm unsure wheter I understand exactly what I'm doing -- Im hoping someone can refute/chastise/critique the understanding that is implied by the code and questions. I have a thick skin so the more talented/vitriolic the response the better. # Find RSS Feeds. # Richard Meraz -- rfmeraz(a)gmail.com. from twisted.internet import reactor from twisted.web.client import getPage import feedlib # Includes modified version of M. Pilgrims feedfinder.py and modified version of # D. Mertz code for url extraction from p. 228 of __TPIP__. MAXTIME = 60 # Kill crawl after this time TIMEOUT = 20 # Kill page retrieval after this time inactive MAXDEPTH = 3 # Recurse this depth when crawling page. # Question: There seem to be many idioms to aggregate information from different defered call-back chains in twisted.. Since everything runs in a single thread I just stuck my stuff in a global class and everybody modifies the vars there as I pass it around to the call-backs that should see it. Seems okay for a small script like this? class StateVars: '''Keep Global state for starting/stopping feedfinding and a record of links we have checked and their status''' connections = 1 links_checked = {} # Structure: {url: (is RSS/ATOM/RDF, page-content)} # Question: start_feed_crawl is where I set up my defers. getPage returns a defer and I attach my call-back process_link. # addCallbacks adds a callback/errback in parallel so only one or the other is called? so I need to add # the final errback to catch errors from callback process_link ? def start_feed_crawl(uri,depth): '''Harvest feeds from a uri''' # Question: how to time-out this deferred chain if getPage is taking too long to finish its work. # what exactly does the argument timeout to getPage do, does it timeout the socket after a no-response # or does it put an upper-bound on how long getPage has to finish its work? getPage(uri, timeout=TIMEOUT).addCallbacks (callback=process_link, callbackArgs=(uri, depth, StateVars), errback = process_error, errbackArgs=(uri,StateVars) ).addErrback(process_error, uri, StateVars) def process_link(data,uri,depth,state): '''Recursive link processing callback. Determines whether a link is a RSS/ATOM/RDF feed. If not then extracts all xml-like links that could point to feeds and starts crawl on those.''' if feedlib.couldBeFeedData(data): #print 'Feed: %s' % uri state.links_checked[uri] = (True,data) else: state.links_checked[uri] = (False,None) if depth <= MAXDEPTH: alinks = feedlib.getALinks(data,uri) links = feedlib.getLinks(data,uri) rawurls = feedlib.extract_urls(data) links_to_check = [feedlib.makeFullURI(u) for u in set(alinks+links+rawurls) if feedlib.isXMLRelatedLink(u)] for l in links_to_check: # Don't need to see it again. if state.links_checked.has_key(l): continue else: state.connections += 1 # Question: since I'm starting up these defers in a callback they are # being created after I've called reactor.run() since we call start_feed_crawl # as we find new links that meet our criteria. Am I doing anything bad here? # All the examples I've seen (eg. p. 548-552 Python Cookbook, great eg by V. Volonghi # and P. Cogolo) have their data up-front and therefore set-up all the defers before calling # reactor.run(). Here I'm discovering my data as I go along and setting up deferrs while # the reactor is spinning. Here is my fundamental lack of understanding. While this script # seems to run okay, is it okay to do this? start_feed_crawl(l,depth+1) state.connections -= 1 # Question: Is this how I kill the reactor -- ie. using some sort of state condition. Is there a better way, # should I try better to understand deferred-list. For example. A top-level deferred-list that contains # other deferred-lists which get created to hold all the defers (created by start_feed_crawl) for the # links on a given page. Could this deferred-list be told to stop the reactor when the other lists have # fired their callback (after the component defers have finished) ? (Sorry for the convoluted question here # I'm new at this) if state.connections <= 0: reactor.stop() return def process_error(error,uri,state): '''Catch errors in link processing''' state.connections -= 1 if state.connections <= 0: reactor.stop() return '' if __name__ == '__main__': import sys if len(sys.argv) < 2: print 'feedfinder_new.py <uri>' sys.exit() uri = feedlib.makeFullURI(sys.argv[1]) start_feed_crawl(uri,1) # Question: I'm killing the process after a pre-determined amount of time. However reactor.stop() seems # to kill network connections. is there a way to stop the reactor but let the connections finish. # Hack to blow out any connections that are hung or uncalled after MAXTIME reactor.callLater(MAXTIME, reactor.stop) reactor.run() for l in StateVars.links_checked: if StateVars.links_checked[l][0]: print l Final question: occasionally I get errors that come from the http.py code in twisted. This get printed to the console, but don't necessarily stop my program. Should my errbacks be catching these? How do I keep errors from getting logged to the console (beside redirecting stderr). I can post an example if necessary of the errors I'm getting. Thanks for your help. Richard F. Meraz -- Never think there is anything impossible for the soul. It is the greatest heresy to think so. If there is sin, this is the only sin – to say that you are weak, or others are weak. Swami Vivekananda [View Less]

1 0

[Twisted-Python] Asynchronous File I/O
by Mustafa Sakalsiz June 21, 2005

June 21, 2005

Is there a way to read and writes files asynchronously using twisted. The files may be of regular files or devices (/dev/*). Saki

2 1

[Twisted-Python] Re: Using twisted for web crawling
by Graham Stratton June 21, 2005

June 21, 2005

Hello all! I'm trying to write a python utility to fetch large numbers of web pages. I originally wrote a multi-threaded app, but found that I had no control of it. I then re-wrote it using asyncore, which is fine apart from the fact that a few connections open sockets which never get closed, and eventually I run out of sockets (well, file descriptors). I see Twisted can avoid this problem by setting an interval after which an inactive connection is closed. I've been browsing the Twisted … [View More]

3 3

[Twisted-Python] newbie help (fwd)
by zunbeltz June 21, 2005

June 21, 2005

Hi, I have the following problem. I'm developing a GUI program (wxPython). This program has to comunicate (TCP) whit other program that controls a laboratory machine to do a measurement. I have a dialog box, wiht two buttoms "Start measurement" and "Stop". "Start" executes a function that do the measurement in the following way. 1) My program send a socket. 2) The other program recives it an send confirmation. 3) My program waits for the confirmation and send the next when confirmation … [View More]

1 0

RE: [Twisted-Python] TM t-shirts
by Henning.Ramm＠mediapro-gmbh.de June 20, 2005

June 20, 2005

> You can order shirts here: http://saph.twistedmatrix.com/blog/ Nice, but what colors (T-Shirt/print) are available? Perhaps you could make a selector for that also? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

2 1

RE: RE: [Twisted-Python] TM t-shirts
by Jen Dockter June 20, 2005

June 20, 2005

Message: 6 Date: Mon, 20 Jun 2005 16:15:57 +0200 From: <Henning.Ramm(a)mediapro-gmbh.de> Subject: RE: [Twisted-Python] TM t-shirts To: <saphirra(a)gmail.com>, <twisted-python(a)twistedmatrix.com> Message-ID: <F92274058556DE42B94B148EEFB9183949E725(a)exchange-03.suedkurier.de> Content-Type: text/plain; charset="iso-8859-1" >> You can order shirts here: http://saph.twistedmatrix.com/blog/ > >Nice, but what colors (T-Shirt/print) are available? Perhaps you … [View More]

1 0

[Twisted-Python] TM t-shirts
by Jen Dockter June 20, 2005

June 20, 2005

I am making Twisted Matrix t-shirts; I have presently just finished blanking out the screen and am now taking orders. The shirts will be using the small TMLabs logo (not the full pyramid snake, but just the top bit) on the front, approximately 7x5" with the URL ( www.twistedmatrix.com <http://www.twistedmatrix.com>) in web-friendly Verdana Bold on the back. These are high-quality, hand-made silk screenings; this is not crummy iron-on CafePress work. The cost per shirt is $14 for US … [View More]

1 0

[Twisted-Python] Fwd: [Twisted-commits] r11032 - Made LineReceiver about 3x as fast, and clarified some of the docstrings.
by Jp Calderone June 19, 2005

June 19, 2005

Begin forwarded message: > From: James Knight <foom(a)wolfwood.twistedmatrix.com> > Date: Tue, 06 Jul 2004 13:53:09 -0600 > To: twisted-commits(a)twistedmatrix.com > Subject: [Twisted-commits] r11032 - Made LineReceiver about 3x as fast, and clarified some of the docstrings. > Reply-to: twisted-python(a)twistedmatrix.com > > Author: foom >Date: Tue Jul 6 13:53:08 2004 >New Revision: 11032 > >Modified: > trunk/twisted/protocols/basic.py >Log: &… [View More]gt;Made LineReceiver about 3x as fast, and clarified some of the docstrings. > > > >Modified: trunk/twisted/protocols/basic.py >============================================================================== >--- trunk/twisted/protocols/basic.py (original) >+++ trunk/twisted/protocols/basic.py Tue Jul 6 13:53:08 2004 >@@ -204,24 +204,31 @@ > rawDataReceived, depending on mode.) > """ > self.__buffer = self.__buffer+data >+ lastoffset=0 > while self.line_mode: >- try: >- line, self.__buffer = self.__buffer.split(self.delimiter, 1) >- except ValueError: >+ offset=self.__buffer.find(self.delimiter, lastoffset) >+ if offset == -1: >+ self.__buffer=self.__buffer[lastoffset:] > if len(self.__buffer) > self.MAX_LENGTH: >- line, self.__buffer = self.__buffer, '' >+ line=self.__buffer >+ self.__buffer='' > return self.lineLengthExceeded(line) > break >- else: >- linelength = len(line) >- if linelength > self.MAX_LENGTH: >- line, self.__buffer = self.__buffer, '' >- return self.lineLengthExceeded(line) >- why = self.lineReceived(line) >- if why or self.transport and self.transport.disconnecting: >- return why >+ >+ line=self.__buffer[lastoffset:offset] >+ lastoffset=offset+len(self.delimiter) >+ >+ if len(line) > self.MAX_LENGTH: >+ line=self.__buffer[lastoffset:] >+ self.__buffer='' >+ return self.lineLengthExceeded(line) >+ why = self.lineReceived(line) >+ if why or self.transport and self.transport.disconnecting: >+ self.__buffer = self.__buffer[lastoffset:] >+ return why > else: >- data, self.__buffer = self.__buffer, '' >+ data=self.__buffer[lastoffset:] >+ self.__buffer='' > if data: > return self.rawDataReceived(data) > This changes dataReceived to call down into user code while the LineReceiver's instance attributes are in an inconsistent state. This can lead to duplicate data delivery if dataReceived is called into from lineReceived. Normally this sounds like a ridiculous thing to do, but notice the implementation of resumeProducing(): def resumeProducing(self): self.paused = False self.dataReceived('') self.transport.resumeProducing() Since resumeProducing is called immediately when a producer is registered, this means LineReceivers can no longer register themselves as producers for their transports from lineReceived, which is a much more reasonable thing to do. Demonstration of the bug in action attached. Jp [View Less]

2 3