Mailman 3 April 2004 - Twisted

[Twisted-Python] making C extension module types Copyable
by Damon Fasching 06 Apr '04

06 Apr '04

Hi, I am attempting to set up a system around Perspective Broker. A large part of the data which I need to pass around is instances of a rather complex type which I have defined in a C extension module. This type is built of a deeply nested hierarchy of C structures. At some levels of the hierarchy, structure elements include malloced data buffers of arbitrary size. I was hoping for some sort of miracle (see attached code) and, no surprises, it didn't happen :( How do I get my class to inherit from Copyable. (I suspect the answer is I can't.) If I can't, how can I make it copyable (lowercase)? I was hoping that by wrapping MyType in a simple Python class, MyTypeWrapper, which does inherit from Copyable I would be able to pass it around. I still get an InsecureJelly exception for MyType. So it seems classes need to be registered with setCopierForClass even if they are used only as attributes of a class which is being copied. That seems reasonable. But, with a type defined in a C extension module, I don't have a class to setCopierFor... Also, does MyType also need to subclass Copyable in order to setCopierForClass for it? Don't think I can do that either... The second problem is probably more serious, if that's possible. I can't imagine a world generous enough where MyType instances would actually be serialized, deserialized and reconstructed properly on the receiving end, nested structures, malloced buffers and all. Can some suggest a path? I am writing both the client and server code and so was hoping to use Perspective Broker. Serializing the object by hand is going to be a pain, but not impossible. If that's what needs to be done, I am still not sure how to package it for transmission. Thanks for your help, Damon

1 0

[Twisted-Python] Twisted and Zope3 components systems
by Christopher Armstrong 05 Apr '04

05 Apr '04

I don't think anybody mentioned this on the list yet. AIUI the concensus made at PyCon was that Twisted would switch to Zope3's component system (with backwards compatibility). Their system seems very hookable and it'll provide everything we need. Their component system will be moved out into a third-party package and both teams will work on it. Itamar has taken responsibility for it for our end of the deal, last I heard. There's an issue in the tracker where progress will hopefully be reported: http://www.twistedmatrix.com/bugs/issue572 -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/

5 9

Re: [Twisted-Python] example code for imap client
by exarkun＠divmod.com 03 Apr '04

03 Apr '04

On Sat, 3 Apr 2004 17:46:02 +0200, Burkhard.Kayser(a)t-online.de (Burkhard Kayser) wrote: >I want to develop an imap client which filters my mail and moves incomming > mails to certain folders. Is there any example code to start with ? > See either http://divmod.org/cvs/*checkout*/trunk/quotient/grabbers.py?content-type=te… or http://svn.twistedmatrix.com/cvs/*checkout*/trunk/doc/examples/imap4client.… or http://svn.twistedmatrix.com/cvs/*checkout*/trunk/twisted/test/test_imap.py… Jp

1 0

[Twisted-Python] example code for imap client
by Burkhard.Kayser＠t-online.de 03 Apr '04

03 Apr '04

I want to develop an imap client which filters my mail and moves incomming mails to certain folders. Is there any example code to start with ? Burkhard

1 0

[Twisted-Python] nevow debugging
by Lee Harr 02 Apr '04

02 Apr '04

Hi; I tried to contact the twisted-web list, but it seems to be having some issues right now... I hope it is ok to ask here. Does anyone have an example using nevow in conjunction with other web service types? I really like the simplicity of "mktap web --path foo" but my understanding is that twisted web development is going to be nevow based. Can these two things work together? Is there a way to serve nevow pages from a .rpy? I am looking for a way to debug that does not require restarting the server every time I change the nevow .tac file. (like how I can use import mymod reload(mymod) in the .rpy while I am testing). I hope this makes sense. I would probably just go with something very simple, but I need to password protect some of the pages and that seems to indicate using nevow.guard Thank you for your time. _________________________________________________________________ The new MSN 8: advanced junk mail protection and 2 months FREE* http://join.msn.com/?page=features/junkmail

2 1

Re: [Twisted-Python] help w/simple echo example w/deferred and threads
by exarkun＠divmod.com 02 Apr '04

02 Apr '04

On Fri, 02 Apr 2004 16:11:16 -0500, "john nielsen" <jn(a)who.net> wrote: >I am trying to make a very simple echo server use threads to understand how twisted handles blocking code. I think I have the server ok but I am not sure how to have the client grab the data from a thread that finished some time later. If I comment out the sleep call, everything works fine. I looked at deferred but am a little confused. What is the simple code the client missing? > > Thanks for any help, > > john > > Here is the server > ############### > from twisted.spread import pb > from twisted.internet import reactor > from twisted.python import threadable > threadable.init() > import time,random > > class t: > def __init__(self): > self.result='' > def test(self,st='same'): > time.sleep(1) > print 'done sleeping' > self.result=st+':'+str(time.time()) > > class Echoer(pb.Root): > def __init__(self): > self.a=t() > def remote_echo(self, st): > reactor.callInThread(self.a.test,st) > print 'from thread',self.a.result > return self.a.result Instead of the above: class t: def test(self, st='same'): time.sleep(1) return st + ':' + str(time.time()) from twisted.internet import threads class Echoer(pb.Root): def __init__(self): self.a = t() def remote_echo(self, st): return threads.deferToThread(self.a.test, st) This is a good example if "time.sleep(1)" is only a hand-wave put in place of real, computational-expensive code, which I assume it is. If you simply want to delay the result for one section, threads are not necessary: from twisted.internet import defer def remote_echo(self, st): d = defer.Deferred() reactor.callLater(1, d.callback, st + ':' + str(time.time() + 1)) return d Be sure to read the Deferred howto in the documentation section of the website. Jp

1 0

[Twisted-Python] Scalability of an rss-aggregator
by Valentino Volonghi aka Dialtone 02 Apr '04

02 Apr '04

Hi all, attached you will find my rss-aggregator made with twisted. It's really fast although when I tried with 745 feeds I got some problems. When the download reached 300 parsed feeds (more or less) it locked till I pressed Ctrl+C and then it processed the remaining 340 feeds in less than 30 seconds... I think that my design has at least an issue but I cannot find it so easily and I hope someone on this list can help me to improve it. The "script" is heavily commented. BTW When it finishes (with all 740 feeds) it reports an awesome 330 seconds which is an impressive time, less than half a second for each feed, and It downloads more than 50Mb of feeds from the net (with 745 feeds to download). Thx for your help. -- Valentino Volonghi aka Dialtone Linux User #310274, Gentoo Proud User X Python Newsreader developer http://sourceforge.net/projects/xpn/ from twisted.internet import reactor, protocol, defer from twisted.web import client import feedparser, time, out rss_feeds = out.rss_feed # This is the default site list #rss_feeds = [('http://www.nongnu.org/straw/news.rss','straw'), # ('http://googlenews.74d.com/rss/google_it.rss','google'), # ('http://www.pythonware.com/daily/rss.xml','pythonware'), # ('http://www.theinquirer.net/inquirer.rss','inq'), # ('http://www.groklaw.net/backend/GrokLaw.rdf','grok'), # ('http://www.livejournal.com/users/moshez/data/rss','zadka'), # ('http://www.pythonware.com/news.rdf','pwn')] # michele(a)berthold.com INTER_QUERY_TIME = 300 class FeederProtocol(object): def __init__(self): self.parsed = 0 # This dict structure will be the following: # { 'URL': (TIMESTAMP, value) } self.cache = {} def gotError(self, data=None, extra_args=None): # An Error as occurred, print traceback infos and go on print data self.parsed += 1 print "="*20 print "Trying to go on..." def getFeeds(self, where=None): #print "getting feeds" # This is to get the feeds we want if not where: # We don't have a database, then we use the local # variabile rss_feeds return rss_feeds else: return None def memoize(self, feed, site=None, extra=None): # extra is the second element of each tuple of rss_feeds # site is the address of the feed, also the first element of each tuple # of rss_feeds print "Memoizing",site,"..." self.cache.setdefault(site, (time.time(),feed)) return feed def stopWorking(self, data=None): print "Closing connection number %d..."%(self.parsed,) print "-"*20 # This is here only for testing. When a protocol/interface will be # created to communicate with this rss-aggregator server, we won't need # to die after we parsed some feeds just one time. self.parsed += 1 if self.parsed >= len(rss_feeds): print "Closing all..." #for i in self.cache: # print i print time.time()-tp #reactor.stop() def getPageFromMemory(self, key=None): #print "getting from memory" # Getting the second element of the tuple which is the parsed structure # of the feed at address key, the first element of the tuple is the # timestamp d = defer.succeed(self.cache.get(key,key)[1]) return d def parseFeed(self, feed): # This is self explaining :) return feedparser.parse(feed) def startDownloading(self, site): #print "Looking if",site[0],"cached...", # Try to get the tuple (TIMESTAMP, FEED_STRUCT) from the dict if it has # already been downloaded. Otherwise assign None to already_got already_got = self.cache.get(site[0], None) # Ok guys, we got it cached, let's see what we will do if already_got: # Well, it's cached, but will it be recent enough? #print "It is\n Looking if timestamp for",site[0],"is recent enough...", elapsed_time = time.time() - already_got[0] # Woooohooo it is, elapsed_time is less than INTER_QUERY_TIME so I # can get the page from the memory, recent enough if elapsed_time < INTER_QUERY_TIME: #print "It is" return self.getPageFromMemory(site[0]) else: # Uhmmm... actually it's a bit old, I'm going to get it from the # Net then, then I'll parse it and then I'll try to memoize it # again #print "Getting",site[0],"from the Net because old" return self.downloadPage(site) else: # Well... We hadn't it cached in, so we need to get it from the Net # now, It's useless to check if it's recent enough, it's not there. #print "Getting",site[0],"from the Net" return self.downloadPage(site) def downloadPage(self, site): #print "Now downloading..." # Self-explanatory d = client.getPage(site[0]) # Uncomment the following if you want to make everything crash :), since # it will save the feed on a file, but with the memoize feature it will # crash everything cuz it will break the get-->parse-->memoize chain #d = client.downloadPage(site[0],site[1]) # Parse the feed and if there's some errors call self.gotError d.addCallbacks(self.parseFeed, self.gotError) # Now memoize it, if there's some error call self.getError d.addCallbacks(self.memoize, self.gotError, site) return d def workOnPage(self, parsed_feed=None, site=None, extra_args=None, extra_key=None): print "-"*20 #print "finished retrieving" print "Feed Version:",parsed_feed.get('version','Unknown') # # Uncomment the following if you want to print the feeds # chan = parsed_feed.get('channel', None) if chan: print chan.get('title', '') #print chan.get('link', '') #print chan.get('tagline', '') #print chan.get('description','') print "-"*20 #items = parsed_feed.get('items', None) #if items: # for item in items: # print '\tTitle: ', item.get('title','') # print '\tDate: ', item.get('date', '') # print '\tLink: ', item.get('link', '') # print '\tDescription: ', item.get('description', '') # print '\tSummary: ', item.get('summary','') # print "-"*20 #print "got",site #print "="*40 def start(self, data=None): # Here we gather all the urls for the feeds #self.factory.tries += 1 for feed in self.getFeeds(): # Now we start telling the reactor that it has # to get all the feeds one by one... d = self.startDownloading(feed) # The it will pass the result of # startDownloading to workOnPage (this is hidden in twisted) # together with the feed url (just to use some extra infos # in the workOnPage method) d.addCallbacks(self.workOnPage, self.gotError, feed) # And when the for loop is ended we put # stopWorking on the callback for the last # feed gathered d.addCallbacks(self.stopWorking, self.gotError) # This is to try the memoize feature #if self.factory.tries<3: # d.addCallback(self.start) class FeederFactory(protocol.ClientFactory): protocol = FeederProtocol() def __init__(self): # tries is used to make more connection to use the # memoizing feature #self.tries = 0 # Here we put in the FeederProtocol instance a reference to # FeederFactory under the name of self.factory (seen from protocol) self.protocol.factory = self self.protocol.start() f = FeederFactory() tp = time.time() reactor.run()

3 17

[Twisted-Python] help w/simple echo example w/deferred and threads
by john nielsen 02 Apr '04

02 Apr '04

I am trying to make a very simple echo server use threads to understand how twisted handles blocking code. I think I have the server ok but I am not sure how to have the client grab the data from a thread that finished some time later. If I comment out the sleep call, everything works fine. I looked at deferred but am a little confused. What is the simple code the client missing? Thanks for any help, john Here is the server ############### from twisted.spread import pb from twisted.internet import reactor from twisted.python import threadable threadable.init() import time,random class t: def __init__(self): self.result='' def test(self,st='same'): time.sleep(1) print 'done sleeping' self.result=st+':'+str(time.time()) class Echoer(pb.Root): def __init__(self): self.a=t() def remote_echo(self, st): reactor.callInThread(self.a.test,st) print 'from thread',self.a.result return self.a.result if __name__ == '__main__': reactor.listenTCP(8789, pb.PBServerFactory(Echoer())) reactor.run() #Here is the client from twisted.spread import pb from twisted.internet import reactor, defer from twisted.python import util factory = pb.PBClientFactory() reactor.connectTCP("localhost", 8789, factory) d = factory.getRootObject() d.addCallback(lambda object: object.callRemote("echo", "hello network")) d.addCallback(lambda echo: 'server echoed: '+echo) d.addErrback(lambda reason: 'error: '+str(reason.value)) d.addCallback(util.println) d = factory.getRootObject() d.addCallback(lambda object: object.callRemote("echo", "hello network")) d.addCallback(lambda echo: 'server echoed: '+echo) d.addErrback(lambda reason: 'error: '+str(reason.value)) d.addCallback(util.println) d.addCallback(lambda _: reactor.stop()) -- _______________________________________________ Get your free email from http://www.iname.com

1 0

Re: [Twisted-Python] ESMTPClient
by exarkun＠divmod.com 02 Apr '04

02 Apr '04

On Thu, 11 Mar 2004 16:33:35 +0100, "Koen Van Herck" <koen_van_herck(a)yahoo.com> wrote: >Does anyone have an example of using twisted.protocols.smtp.ESMTPClient ? > I.e., I would like to write a client for sending mail to an SMTP server > which requires authentication. ESMTPClient (along with several other client implementations in Twisted) supports authentication by allowing you to specify a dictionary mapping authentication schemes to authentication logic objects. For example, to use only Cram-MD5 authentication from twisted.protocols import smtp from twisted.protocols.imap4 import CramMD5ClientAuthenticator class ESMTPClientFactory(protocol.ClientFactory): def __init__(self, *a, **kw): self.a, self.kw = a, kw def buildProtocol(self, addr): p = smtp.ESMTPClient('password') p.factory = self p.requireAuthentication = True p.authenticators = {'CRAM-MD5': CramMD5ClientAuthenticator('username')} return p Having had to write out the above, it is even more apparent to me how much work is left to be done on the client side of many of our protocol implementations. The code is untested; please let me know if you run into any problems with it. Jp

1 0

Re: [Twisted-Python] Can I connect multiple class instances to 1 Twisted factory?
by exarkun＠divmod.com 02 Apr '04

02 Apr '04

On Thu, 1 Apr 2004 07:52:33 -0500, "Kaleb Brasee" <qpalzm(a)alltel.net> wrote: > Is this even possible? I'm designing a simple chat and game > client/server as an intro to Python, and it would be nice to represent > the different interfaces (public chat room, private chat rooms, game > instances) as totally seperate classes and show them in totally seperate > windows. > > Given my basic understanding of Twisted, I see no way of doing this, and > my whole GUI is contained in one class. The project is doable this way, > it would just be much better with multiple class and multiple window > seperation. Twisted doesn't enforce any design strategies with respect to any GUI code you use it with. If you simply require that your ServerFactory or ClientFactory be able to reference multiple GUI objects, this seems as simple as: f = YourFactory(chatRoomGUIObject, privateRoomGUIObject, gameGUIObject, ...) Not that I would go down that particular road myself. > > I am using Tkinter to design my GUIs, using the specialized Twisted > support code to setup the reactor. > > If anyone can give me any direction, I would appreciate it very much. > Think about what information needs to move between the network layer (where information is received from sockets), the application layer (where messages from a particular user are differentiated from messages from another particular user), and the UI layer (where messages are rendered in some fashion so as to be visible to the end user). Try and come up with the narrowest interface possible that lets that information move between those layers. In this instance, "narrow" would mean not passing every GUI class you have to your factory. Perhaps an event broadcasting system would be useful, or maybe a single "GUI" class that knows how to delegate messages down to the various other GUI classes you have defined. > Thanks Jp >

1 0