Mailman 3 April 2004 - Twisted

[Twisted-Python] making C extension module types Copyable
by Damon Fasching April 6, 2004

April 6, 2004

Hi, I am attempting to set up a system around Perspective Broker. A large part of the data which I need to pass around is instances of a rather complex type which I have defined in a C extension module. This type is built of a deeply nested hierarchy of C structures. At some levels of the hierarchy, structure elements include malloced data buffers of arbitrary size. I was hoping for some sort of miracle (see attached code) and, no surprises, it didn't happen :( How do I get my class to … [View More]

1 0

[Twisted-Python] Twisted and Zope3 components systems
by Christopher Armstrong April 5, 2004

April 5, 2004

I don't think anybody mentioned this on the list yet. AIUI the concensus made at PyCon was that Twisted would switch to Zope3's component system (with backwards compatibility). Their system seems very hookable and it'll provide everything we need. Their component system will be moved out into a third-party package and both teams will work on it. Itamar has taken responsibility for it for our end of the deal, last I heard. There's an issue in the tracker where progress will hopefully be … [View More]

5 9

Re: [Twisted-Python] example code for imap client
by exarkun＠divmod.com April 3, 2004

April 3, 2004

On Sat, 3 Apr 2004 17:46:02 +0200, Burkhard.Kayser(a)t-online.de (Burkhard Kayser) wrote: >I want to develop an imap client which filters my mail and moves incomming > mails to certain folders. Is there any example code to start with ? > See either http://divmod.org/cvs/*checkout*/trunk/quotient/grabbers.py?content-type=te… or http://svn.twistedmatrix.com/cvs/*checkout*/trunk/doc/examples/imap4client.… or http://svn.twistedmatrix.com/cvs/*checkout*/trunk/twisted/test/test_imap.py… Jp

1 0

[Twisted-Python] example code for imap client
by Burkhard.Kayser＠t-online.de April 3, 2004

April 3, 2004

I want to develop an imap client which filters my mail and moves incomming mails to certain folders. Is there any example code to start with ? Burkhard

1 0

[Twisted-Python] nevow debugging
by Lee Harr April 2, 2004

April 2, 2004

Hi; I tried to contact the twisted-web list, but it seems to be having some issues right now... I hope it is ok to ask here. Does anyone have an example using nevow in conjunction with other web service types? I really like the simplicity of "mktap web --path foo" but my understanding is that twisted web development is going to be nevow based. Can these two things work together? Is there a way to serve nevow pages from a .rpy? I am looking for a way to debug that does not require … [View More]

2 1

Re: [Twisted-Python] help w/simple echo example w/deferred and threads
by exarkun＠divmod.com April 2, 2004

April 2, 2004

On Fri, 02 Apr 2004 16:11:16 -0500, "john nielsen" <jn(a)who.net> wrote: >I am trying to make a very simple echo server use threads to understand how twisted handles blocking code. I think I have the server ok but I am not sure how to have the client grab the data from a thread that finished some time later. If I comment out the sleep call, everything works fine. I looked at deferred but am a little confused. What is the simple code the client missing? > > Thanks for any help, … [View More]

1 0

[Twisted-Python] Scalability of an rss-aggregator
by Valentino Volonghi aka Dialtone April 2, 2004

April 2, 2004

Hi all, attached you will find my rss-aggregator made with twisted. It's really fast although when I tried with 745 feeds I got some problems. When the download reached 300 parsed feeds (more or less) it locked till I pressed Ctrl+C and then it processed the remaining 340 feeds in less than 30 seconds... I think that my design has at least an issue but I cannot find it so easily and I hope someone on this list can help me to improve it. The "script" is heavily commented. BTW When it … [View More]finishes (with all 740 feeds) it reports an awesome 330 seconds which is an impressive time, less than half a second for each feed, and It downloads more than 50Mb of feeds from the net (with 745 feeds to download). Thx for your help. -- Valentino Volonghi aka Dialtone Linux User #310274, Gentoo Proud User X Python Newsreader developer http://sourceforge.net/projects/xpn/ from twisted.internet import reactor, protocol, defer from twisted.web import client import feedparser, time, out rss_feeds = out.rss_feed # This is the default site list #rss_feeds = [('http://www.nongnu.org/straw/news.rss','straw'), # ('http://googlenews.74d.com/rss/google_it.rss','google'), # ('http://www.pythonware.com/daily/rss.xml','pythonware'), # ('http://www.theinquirer.net/inquirer.rss','inq'), # ('http://www.groklaw.net/backend/GrokLaw.rdf','grok'), # ('http://www.livejournal.com/users/moshez/data/rss','zadka'), # ('http://www.pythonware.com/news.rdf','pwn')] # michele(a)berthold.com INTER_QUERY_TIME = 300 class FeederProtocol(object): def __init__(self): self.parsed = 0 # This dict structure will be the following: # { 'URL': (TIMESTAMP, value) } self.cache = {} def gotError(self, data=None, extra_args=None): # An Error as occurred, print traceback infos and go on print data self.parsed += 1 print "="*20 print "Trying to go on..." def getFeeds(self, where=None): #print "getting feeds" # This is to get the feeds we want if not where: # We don't have a database, then we use the local # variabile rss_feeds return rss_feeds else: return None def memoize(self, feed, site=None, extra=None): # extra is the second element of each tuple of rss_feeds # site is the address of the feed, also the first element of each tuple # of rss_feeds print "Memoizing",site,"..." self.cache.setdefault(site, (time.time(),feed)) return feed def stopWorking(self, data=None): print "Closing connection number %d..."%(self.parsed,) print "-"*20 # This is here only for testing. When a protocol/interface will be # created to communicate with this rss-aggregator server, we won't need # to die after we parsed some feeds just one time. self.parsed += 1 if self.parsed >= len(rss_feeds): print "Closing all..." #for i in self.cache: # print i print time.time()-tp #reactor.stop() def getPageFromMemory(self, key=None): #print "getting from memory" # Getting the second element of the tuple which is the parsed structure # of the feed at address key, the first element of the tuple is the # timestamp d = defer.succeed(self.cache.get(key,key)[1]) return d def parseFeed(self, feed): # This is self explaining :) return feedparser.parse(feed) def startDownloading(self, site): #print "Looking if",site[0],"cached...", # Try to get the tuple (TIMESTAMP, FEED_STRUCT) from the dict if it has # already been downloaded. Otherwise assign None to already_got already_got = self.cache.get(site[0], None) # Ok guys, we got it cached, let's see what we will do if already_got: # Well, it's cached, but will it be recent enough? #print "It is\n Looking if timestamp for",site[0],"is recent enough...", elapsed_time = time.time() - already_got[0] # Woooohooo it is, elapsed_time is less than INTER_QUERY_TIME so I # can get the page from the memory, recent enough if elapsed_time < INTER_QUERY_TIME: #print "It is" return self.getPageFromMemory(site[0]) else: # Uhmmm... actually it's a bit old, I'm going to get it from the # Net then, then I'll parse it and then I'll try to memoize it # again #print "Getting",site[0],"from the Net because old" return self.downloadPage(site) else: # Well... We hadn't it cached in, so we need to get it from the Net # now, It's useless to check if it's recent enough, it's not there. #print "Getting",site[0],"from the Net" return self.downloadPage(site) def downloadPage(self, site): #print "Now downloading..." # Self-explanatory d = client.getPage(site[0]) # Uncomment the following if you want to make everything crash :), since # it will save the feed on a file, but with the memoize feature it will # crash everything cuz it will break the get-->parse-->memoize chain #d = client.downloadPage(site[0],site[1]) # Parse the feed and if there's some errors call self.gotError d.addCallbacks(self.parseFeed, self.gotError) # Now memoize it, if there's some error call self.getError d.addCallbacks(self.memoize, self.gotError, site) return d def workOnPage(self, parsed_feed=None, site=None, extra_args=None, extra_key=None): print "-"*20 #print "finished retrieving" print "Feed Version:",parsed_feed.get('version','Unknown') # # Uncomment the following if you want to print the feeds # chan = parsed_feed.get('channel', None) if chan: print chan.get('title', '') #print chan.get('link', '') #print chan.get('tagline', '') #print chan.get('description','') print "-"*20 #items = parsed_feed.get('items', None) #if items: # for item in items: # print '\tTitle: ', item.get('title','') # print '\tDate: ', item.get('date', '') # print '\tLink: ', item.get('link', '') # print '\tDescription: ', item.get('description', '') # print '\tSummary: ', item.get('summary','') # print "-"*20 #print "got",site #print "="*40 def start(self, data=None): # Here we gather all the urls for the feeds #self.factory.tries += 1 for feed in self.getFeeds(): # Now we start telling the reactor that it has # to get all the feeds one by one... d = self.startDownloading(feed) # The it will pass the result of # startDownloading to workOnPage (this is hidden in twisted) # together with the feed url (just to use some extra infos # in the workOnPage method) d.addCallbacks(self.workOnPage, self.gotError, feed) # And when the for loop is ended we put # stopWorking on the callback for the last # feed gathered d.addCallbacks(self.stopWorking, self.gotError) # This is to try the memoize feature #if self.factory.tries<3: # d.addCallback(self.start) class FeederFactory(protocol.ClientFactory): protocol = FeederProtocol() def __init__(self): # tries is used to make more connection to use the # memoizing feature #self.tries = 0 # Here we put in the FeederProtocol instance a reference to # FeederFactory under the name of self.factory (seen from protocol) self.protocol.factory = self self.protocol.start() f = FeederFactory() tp = time.time() reactor.run() [View Less]

3 17

[Twisted-Python] help w/simple echo example w/deferred and threads
by john nielsen April 2, 2004

April 2, 2004

I am trying to make a very simple echo server use threads to understand how twisted handles blocking code. I think I have the server ok but I am not sure how to have the client grab the data from a thread that finished some time later. If I comment out the sleep call, everything works fine. I looked at deferred but am a little confused. What is the simple code the client missing? Thanks for any help, john Here is the server ############### from twisted.spread import pb from twisted.internet … [View More]

1 0

Re: [Twisted-Python] ESMTPClient
by exarkun＠divmod.com April 2, 2004

April 2, 2004

On Thu, 11 Mar 2004 16:33:35 +0100, "Koen Van Herck" <koen_van_herck(a)yahoo.com> wrote: >Does anyone have an example of using twisted.protocols.smtp.ESMTPClient ? > I.e., I would like to write a client for sending mail to an SMTP server > which requires authentication. ESMTPClient (along with several other client implementations in Twisted) supports authentication by allowing you to specify a dictionary mapping authentication schemes to authentication logic objects. For … [View More]

1 0

Re: [Twisted-Python] Can I connect multiple class instances to 1 Twisted factory?
by exarkun＠divmod.com April 2, 2004

April 2, 2004

On Thu, 1 Apr 2004 07:52:33 -0500, "Kaleb Brasee" <qpalzm(a)alltel.net> wrote: > Is this even possible? I'm designing a simple chat and game > client/server as an intro to Python, and it would be nice to represent > the different interfaces (public chat room, private chat rooms, game > instances) as totally seperate classes and show them in totally seperate > windows. > > Given my basic understanding of Twisted, I see no way of doing this, and > my whole GUI is … [View More]

1 0