Mailman 3 March 2004 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 05 Aug '21

05 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Scalability of an rss-aggregator
by Valentino Volonghi aka Dialtone 02 Apr '04

02 Apr '04

Hi all, attached you will find my rss-aggregator made with twisted. It's really fast although when I tried with 745 feeds I got some problems. When the download reached 300 parsed feeds (more or less) it locked till I pressed Ctrl+C and then it processed the remaining 340 feeds in less than 30 seconds... I think that my design has at least an issue but I cannot find it so easily and I hope someone on this list can help me to improve it. The "script" is heavily commented. BTW When it finishes (with all 740 feeds) it reports an awesome 330 seconds which is an impressive time, less than half a second for each feed, and It downloads more than 50Mb of feeds from the net (with 745 feeds to download). Thx for your help. -- Valentino Volonghi aka Dialtone Linux User #310274, Gentoo Proud User X Python Newsreader developer http://sourceforge.net/projects/xpn/ from twisted.internet import reactor, protocol, defer from twisted.web import client import feedparser, time, out rss_feeds = out.rss_feed # This is the default site list #rss_feeds = [('http://www.nongnu.org/straw/news.rss','straw'), # ('http://googlenews.74d.com/rss/google_it.rss','google'), # ('http://www.pythonware.com/daily/rss.xml','pythonware'), # ('http://www.theinquirer.net/inquirer.rss','inq'), # ('http://www.groklaw.net/backend/GrokLaw.rdf','grok'), # ('http://www.livejournal.com/users/moshez/data/rss','zadka'), # ('http://www.pythonware.com/news.rdf','pwn')] # michele(a)berthold.com INTER_QUERY_TIME = 300 class FeederProtocol(object): def __init__(self): self.parsed = 0 # This dict structure will be the following: # { 'URL': (TIMESTAMP, value) } self.cache = {} def gotError(self, data=None, extra_args=None): # An Error as occurred, print traceback infos and go on print data self.parsed += 1 print "="*20 print "Trying to go on..." def getFeeds(self, where=None): #print "getting feeds" # This is to get the feeds we want if not where: # We don't have a database, then we use the local # variabile rss_feeds return rss_feeds else: return None def memoize(self, feed, site=None, extra=None): # extra is the second element of each tuple of rss_feeds # site is the address of the feed, also the first element of each tuple # of rss_feeds print "Memoizing",site,"..." self.cache.setdefault(site, (time.time(),feed)) return feed def stopWorking(self, data=None): print "Closing connection number %d..."%(self.parsed,) print "-"*20 # This is here only for testing. When a protocol/interface will be # created to communicate with this rss-aggregator server, we won't need # to die after we parsed some feeds just one time. self.parsed += 1 if self.parsed >= len(rss_feeds): print "Closing all..." #for i in self.cache: # print i print time.time()-tp #reactor.stop() def getPageFromMemory(self, key=None): #print "getting from memory" # Getting the second element of the tuple which is the parsed structure # of the feed at address key, the first element of the tuple is the # timestamp d = defer.succeed(self.cache.get(key,key)[1]) return d def parseFeed(self, feed): # This is self explaining :) return feedparser.parse(feed) def startDownloading(self, site): #print "Looking if",site[0],"cached...", # Try to get the tuple (TIMESTAMP, FEED_STRUCT) from the dict if it has # already been downloaded. Otherwise assign None to already_got already_got = self.cache.get(site[0], None) # Ok guys, we got it cached, let's see what we will do if already_got: # Well, it's cached, but will it be recent enough? #print "It is\n Looking if timestamp for",site[0],"is recent enough...", elapsed_time = time.time() - already_got[0] # Woooohooo it is, elapsed_time is less than INTER_QUERY_TIME so I # can get the page from the memory, recent enough if elapsed_time < INTER_QUERY_TIME: #print "It is" return self.getPageFromMemory(site[0]) else: # Uhmmm... actually it's a bit old, I'm going to get it from the # Net then, then I'll parse it and then I'll try to memoize it # again #print "Getting",site[0],"from the Net because old" return self.downloadPage(site) else: # Well... We hadn't it cached in, so we need to get it from the Net # now, It's useless to check if it's recent enough, it's not there. #print "Getting",site[0],"from the Net" return self.downloadPage(site) def downloadPage(self, site): #print "Now downloading..." # Self-explanatory d = client.getPage(site[0]) # Uncomment the following if you want to make everything crash :), since # it will save the feed on a file, but with the memoize feature it will # crash everything cuz it will break the get-->parse-->memoize chain #d = client.downloadPage(site[0],site[1]) # Parse the feed and if there's some errors call self.gotError d.addCallbacks(self.parseFeed, self.gotError) # Now memoize it, if there's some error call self.getError d.addCallbacks(self.memoize, self.gotError, site) return d def workOnPage(self, parsed_feed=None, site=None, extra_args=None, extra_key=None): print "-"*20 #print "finished retrieving" print "Feed Version:",parsed_feed.get('version','Unknown') # # Uncomment the following if you want to print the feeds # chan = parsed_feed.get('channel', None) if chan: print chan.get('title', '') #print chan.get('link', '') #print chan.get('tagline', '') #print chan.get('description','') print "-"*20 #items = parsed_feed.get('items', None) #if items: # for item in items: # print '\tTitle: ', item.get('title','') # print '\tDate: ', item.get('date', '') # print '\tLink: ', item.get('link', '') # print '\tDescription: ', item.get('description', '') # print '\tSummary: ', item.get('summary','') # print "-"*20 #print "got",site #print "="*40 def start(self, data=None): # Here we gather all the urls for the feeds #self.factory.tries += 1 for feed in self.getFeeds(): # Now we start telling the reactor that it has # to get all the feeds one by one... d = self.startDownloading(feed) # The it will pass the result of # startDownloading to workOnPage (this is hidden in twisted) # together with the feed url (just to use some extra infos # in the workOnPage method) d.addCallbacks(self.workOnPage, self.gotError, feed) # And when the for loop is ended we put # stopWorking on the callback for the last # feed gathered d.addCallbacks(self.stopWorking, self.gotError) # This is to try the memoize feature #if self.factory.tries<3: # d.addCallback(self.start) class FeederFactory(protocol.ClientFactory): protocol = FeederProtocol() def __init__(self): # tries is used to make more connection to use the # memoizing feature #self.tries = 0 # Here we put in the FeederProtocol instance a reference to # FeederFactory under the name of self.factory (seen from protocol) self.protocol.factory = self self.protocol.start() f = FeederFactory() tp = time.time() reactor.run()

3 17

[Twisted-Python] help ssh forwarding.
by Oisin Mulvihill 01 Apr '04

01 Apr '04

Hi, I have a series of linux based internet kiosks at a variety of sites. I need to be able to connect back into the kiosk from the main server. However direct connections aren't possible as kiosks are behind there own router/firewall. The solution to this is to forward port 22 on the kiosk to a port on the server. I was thinking of just spawning the command "ssh -R11122:my_server:22 user@my_server" from the kiosk, when ever I need to connect to a machine. However I'm hoping to do this with twisted, since the kiosk app uses it already, and spawning the ssh command won't play well with twisted's event system. I decided to try and hack together a simple version of what I want to do, based on the example sshsimpleclient.py. I don't seem to be able to figure out how to forward from the server to the kiosk. My program (attached) just forwards from 11122 on the kiosk to 22 on the server which isn't what I want. Can anyone suggest what I'm doing wrong? Can twisted actually do this? Thanks in advance, om -- Oisin Mulvihill Engines Of Creation Email: oisin(a)enginesofcreationsoftware.com Email: oisin(a)enginesofcreation.ie Mobile: +353 868191540

2 1

[Twisted-Python] help ssh forwarding.
by Oisin Mulvihill 01 Apr '04

01 Apr '04

Sorry If you get two of these, I sent the last one with the wrong email account. Hi, I have a series of linux based internet kiosks at a variety of sites. I need to be able to connect back into the kiosk from the main server. However direct connections aren't possible as kiosks are behind there own router/firewall. The solution to this is to forward port 22 on the kiosk to a port on the server. I was thinking of just spawning the command "ssh -R11122:my_server:22 user@my_server" from the kiosk, when ever I need to connect to a machine. However I'm hoping to do this with twisted, since the kiosk app uses it already, and spawning the ssh command won't play well with twisted's event system. I decided to try and hack together a simple version of what I want to do, based on the example sshsimpleclient.py. I don't seem to be able to figure out how to forward from the server to the kiosk. My program (attached) just forwards from 11122 on the kiosk to 22 on the server which isn't what I want. Can anyone suggest what I'm doing wrong? Can twisted actually do this? Thanks in advance, om -- Oisin Mulvihill Engines Of Creation Email: oisin(a)enginesofcreationsoftware.com Email: oisin(a)enginesofcreation.ie Mobile: +353 868191540

3 4

[Twisted-Python] adbapi results through xmlrpc
by Zach Thompson 31 Mar '04

31 Mar '04

Hello, I'm trying to connect 2 scripts (a server and client) xmlrpc. The server just needs to run a db query and return the result set to the client for display. Here's part of the server class: def xmlrpc_get_count(self): from twisted.enterprise import adbapi pool = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="zach") return pool.runQuery("select count(*) from asdf") And part of the client class: def OnButton(self, event): from twisted.web.xmlrpc import Proxy proxy = Proxy("http://localhost:8080") proxy.callRemote('get_count').addCallback(self.UpdateCount) def UpdateCount(self, count): self.label.SetLabel(count) Here's what the client prints when I press the button: Unhandled error in Deferred: Failure: xmlrpclib.Fault: <Fault 8002: "can't serialize output"> I can understand that xmlrpc needs to serialize the Deferred object to send it through http. I'm guessing there's a reference to the ConnectionPool object contained in the Deferred, and it makes sense that you wouldn't be able to serialize something like that. Any ideas? BTW, the client is running on a wxreactor, if that matters.. Thanks, Zach

3 3

Re: [Twisted-Python] adbapi results through xmlrpc
by dave＠krondo.com 31 Mar '04

31 Mar '04

Returning a deferred from the xmlrpc method will work fine -- twisted will use the result of the deferred and not the deferred itself when sending the return xml-rpc value. The problem is that the result of the query is not serializable, probably because it contains some pgsql objects somewhere. Instead of sending the result directly, you'll probably want to add a callback that picks the count value out of the row and column. Also, the type of the count may not be a regular integer but some pgsql-specific type. You'll have to do some debugging to see what's going on. Also, you probably don't want to create a connection pool for every call, but if you do, don't forget to close the pool or you may end up with lots of open connections. dave ---- Original Message ---- From: Zach Thompson Date: Wed 3/31/04 13:22 To: twisted-python(a)twistedmatrix.com Subject: [Twisted-Python] adbapi results through xmlrpc Hello, I'm trying to connect 2 scripts (a server and client) xmlrpc. The server just needs to run a db query and return the result set to the client for display. Here's part of the server class: def xmlrpc_get_count(self): from twisted.enterprise import adbapi pool = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="zach") return pool.runQuery("select count(*) from asdf") And part of the client class: def OnButton(self, event): from twisted.web.xmlrpc import Proxy proxy = Proxy("http://localhost:8080") proxy.callRemote('get_count').addCallback(self.UpdateCount) def UpdateCount(self, count): self.label.SetLabel(count) Here's what the client prints when I press the button: Unhandled error in Deferred: Failure: xmlrpclib.Fault: <Fault 8002: "can't serialize output"> I can understand that xmlrpc needs to serialize the Deferred object to send it through http. I'm guessing there's a reference to the ConnectionPool object contained in the Deferred, and it makes sense that you wouldn't be able to serialize something like that. Any ideas? BTW, the client is running on a wxreactor, if that matters.. Thanks, Zach

1 0

[Twisted-Python] Twisted Euphrates (Modified by Glyph Lefkowitz)
by Glyph Lefkowitz 29 Mar '04

29 Mar '04

Unfortunately, I've been out of the loop on Twisted development for some time. During the course of the PyCon sprints, I've been catching up with what's been going on and trying to do some planning for future directions in development. Yesterday Itamar and I had our quarterly screaming argument about obscure details, and it didn't come to blows, which I believe means that spring is right around the corner! The major issue we've been discussing is splitting up Twisted into multiple packages. Twisted has been encountering some growing pains, due to a mismatch between the way it's developed and the way it's distributed. We (T.M.Labs) are really a community of developers working on several interrelated projects with different goals, stability, and release cycles, but we are distributing one monolithic system, with code of vastly different qualities in the same tarball and sometimes even in the same package. The primary problem that this leads to is revision coupling. This is probably old news to anyone trying to deploy a large system which relies on any part of Twisted besides the base reactor (and maybe twisted.web). Scenarios like this happen all the time: - Twisted v.N.M.0 is released. - Project P uses twisted components X and Y. (Generally X and Y are marked 'Unstable' or 'Semi-Stable', but are very useful or heavily marketed or otherwise convince unsuspecting users to trust their stability.) - Project Q uses twisted components Y and Z. - Project P v. 1 is released. - Project Q v. 1 is released. - Project P gives useful feedback to twisted developer A who then fixes several bugs in component X which are critical to the functioning of the as-yet-unreleased P v.2 - Twisted developer B working on project Q commits substantial "improvements" to component Y, which change the functionality in a way that breaks project X. - Radix prepares an alpha/bugfix release, v.N.M.1 At this point, Project Q is perfectly happy, because the most recent alpha works for their code. However, Project P is very unhappy, because there is no release which includes their bugfix but does NOT include code which forces them to upgrade some functionality in an a way which is incompatible with the previous micro-version release of Twisted which their users presumably already have installed. The release notes have to be very specific, and because nobody reads them anyway, a lot of confusion results. While there are problems with splitting Twisted into multiple projects, I think that the time has come to do so. Originally the feedback I heard from most users was that it was great to have lots of different packages available so readily; the "batteries included" philosophy does have a lot going for it. Now, most of what I'm hearing is complaints about the above scenario, not the least of which from my own employer :-). We are more often in the role of project Q than project P (although we have been in both more than once) and I'd prefer not to have a few projects who employ active Twisted developers control the release cycle based on their requirements. Because splitting the project involves moving some code around, we should take the opportunity to clean up another problem: our code has no effective separation between interface and implementation. Within the project this is fine, because we can easily refactor any uses of deprecated APIs. While we aren't going to break the existing package-structure interfaces (or even deprecate them, for the time being), this means that in the future, imports for twisted projects will probably look like: from twisted_core import reactor from twisted_web import Resource The way this will be implemented, a top-level "twisted_core" module (or possibly, package, but I would like as little nesting as possible here) will import explicitly public names from a _twisted_core implementation package. For the forseeable future, the twisted.* package hierarchy will continue to work for imports; however, considering how informally specified this interface has been, I imagine improved documentation will emerge for the newer interface packages. Personally I wanted to call these modules "twisted_web_1", and I also wanted to talk about versioning here, but amidst strenuous objections from Anthony Baxter and Itamar ST I will refrain :). Suffice it to say that I believe that regardless of many potential strategies available for versioning, a separate module for interface vs. implementation will make it possible to determine what is public and usable from a deployed application using the Python interactive prompt or reading the code. Since many packages don't have stability declarations anyway, I think this will result in many fewer unintentional uses of private code. The first package that we are going to try this on is twisted.web, because there is a major rewrite pending integration of all the fixes and redesign that has happened in the Nevow package. If this goes well, other packages will use the same naming conventions. In any case, we will be migrating packages out of the Twisted core. for example: lore, conch, manhole, and mail. As part of this process we hope to improve the website to reflect the name "Twisted" as primarily a part of "Twisted Matrix Labs", the group of developers working on these projects. Similar to tigris.org, TML is "a mid-sized open source community focused on building better tools for networked application development". Package maintainers: each project should have its own sub-site, so consider a project description and any other documentation you might want on that site. As far as our development process, we will be keeping all the code in one subversion repository and still running BuildBot over the entire system in a way similar to how we're doing it now. There are lots of possible improvements to the process, all the way up to an independent package management system using distutils: please, don't recommend any of these. We don't want to make a huge number of changes at the same time, and while this moving / renaming is fairly mechanical, it will touch almost every file in the world; we need to make sure it all works before we begin adding features based on it.

8 12

[Twisted-Python] IOCP Reactor Problem
by Justin Johnson 29 Mar '04

29 Mar '04

When I try to use the new IOCP reactor (latest from svn) I get the following errors. The only thing being started up is a nevow web site. This worked when using win32reactor. Anyone have any ideas? Let me know if more info is needed. Thanks. -Justin 2004/03/29 15:44 Central Standard Time [-] Traceback (most recent call last): File "C:\Python23\Lib\site-packages\twisted\protocols\http.py", line 554, in requestReceived self.process() File "C:\svn\RatControl\nevow\appserver.py", line 130, in process return self.site.getResourceFor( File "C:\svn\RatControl\nevow\appserver.py", line 277, in getResourceFor return self.resourceAtPathOfRequest(request) File "C:\svn\RatControl\nevow\appserver.py", line 290, in resourceAtPathOfRequest return defer.maybeDeferred(res.locateChild, request, path).addCallback( --- <exception caught here> --- File "C:\Python23\Lib\site-packages\twisted\internet\defer.py", line 129, in maybeDeferred result = f(*args, **kw) File "C:\svn\RatControl\nevow\guard.py", line 188, in locateChild prepath = request.prePathURL() File "C:\Python23\Lib\site-packages\twisted\web\server.py", line 345, in prePathURL inet, addr, port = self.getHost() exceptions.TypeError: unpack non-sequence 2004/03/29 15:44 Central Standard Time [-] html is not a string: None 2004/03/29 15:44 Central Standard Time [-] None - - [29/Mar/2004:21:44:54 +0000] "GET /results HTTP/1.1" 500 5773 "http://localhost:8888/rmvob" "Mozilla/5.0 (Wi ndows; U; Windows NT 5.0; en-US; rv:1.6) Gecko/20040206 Firefox/0.8"

1 0

RE: [Twisted-Python] Windows IOCP reactor Question
by Schneider, Michael 29 Mar '04

29 Mar '04

If it is possible, that would be great!!!! Thanks Mike ---------------------------------------------------------------- Michael Schneider Senior Software Engineering Consultant UGS PLM Solutions - an EDS Company "The Greatest Performance Improvement Is the transitioning from a non-working state to the working state" > -----Original Message----- > From: twisted-python-admin(a)twistedmatrix.com > [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of Cory Dodt > Sent: Monday, March 29, 2004 11:26 AM > To: twisted-python(a)twistedmatrix.com > Subject: Re: [Twisted-Python] Windows IOCP reactor Question > > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > | It would be very nice if no c compiler would be required to > | install twisted IOCP reactor from CVS. > | > > Maybe it's time to have win32 buildbot start publishing win32 > builds somewhere > so people on Windows can download nightlies. > > C > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.3-nr1 (Windows 2000) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFAaE4b3A5SrXAiHQcRAoaRAJ44QyCeJsN6H4wKUWN2JFg2jtKBvQCfVpMe > nWqo+4V78VMn9Nw+YzebnrE= > =Rwjb > -----END PGP SIGNATURE----- > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >

1 0

[Twisted-Python] Windows IOCP reactor Question
by Schneider, Michael 29 Mar '04

29 Mar '04

Thanks for the work on IOCP, I really appriecaite the efforts and will make use of it. I am an old UNIX hacker, and I am delivering a Windows project with twisted. I am about as far away from a windows programmer that you can get. I did have to get memory info from the Widows OS. I was going to use C, but then I found ctypes and the struct python C struct wrapper. It was nice to have a python only solution. As I was looking through the code in iocpcoreobject.c. I noticed that most (maybe all) of the functionality could be accessed by the ctypes module and the Struct package. Being a newbie to windows programming, I am not comfortable making the definitive statement that it should use this, and elimintate the C code. Instead I would like to ask the question. Does anyone see a problem replacing the C code in iocpcoreobject.c with a python module that leverages ctype. I attached a simple python script that gets the memory from a windows box using ctypes. It would be very nice if no c compiler would be required to install twisted IOCP reactor from CVS. Thanks, Mike ------------------------------------------------------- #!/usr/bin/env python # Simple WMI and ctypes example # Michael Schnedier import os import time import wmi from ctypes import * from ctypes.wintypes import * import win32api class MEMORYSTATUS(Structure): _fields_ = [ ('dwLength', DWORD), ('dwMemoryLoad', DWORD), ('dwTotalPhys', DWORD), ('dwAvailPhys', DWORD), ('dwTotalPageFile', DWORD), ('dwAvailPageFile', DWORD), ('dwTotalVirtual', DWORD), ('dwAvailVirtual', DWORD), ] class ComputeNode: def __init__(self): self.memInfo = None self.lastUpdate = time.time() self.update() def genMemoryXmlStringRep(self): self.validate() xmlString = \ """<?xml version="1.0"?> <MemoryInfo platform="windows" TotalPhysicalMemory="%d" AvailablePhysicalMemory="%d" TotalPageFileSize="%d" AvailPageFileSize="%d" GetTotalVirtualMemory="%d" GetAvailableVirtualMemory="%d" /> """%( self.getTotalPhysicalMemory(), self.getAvailPhysicalMemory(), self.getTotalPageFileSize(), self.getAvailPageFile(), self.getTotalVirtualMemory(), self.getAvailVirtualMemory()) return xmlString def getTotalPhysicalMemory(self): self.validate() return self.memInfo.dwTotalPhys/1024/1024 def getAvailPhysicalMemory(self): self.validate() return self.memInfo.dwAvailPhys/1024/1024 def getTotalPageFileSize(self): self.validate() return self.memInfo.dwTotalPageFile/1024/1024 def getAvailPageFile(self): self.validate() return self.memInfo.dwAvailPageFile/1024/1024 def getTotalVirtualMemory(self): self.validate() return self.memInfo.dwTotalVirtual/1024/1024 def getAvailVirtualMemory(self): self.validate() return self.memInfo.dwAvailVirtual/1024/1024 def validate(self, deltaTrigger=60 ): deltaTime = time.time() - self.lastUpdate if (self.lastUpdate < 0) or (deltaTime > deltaTrigger): self.update() def update(self): self.lastUpdate = time.time() newStatus = MEMORYSTATUS() windll.kernel32.GlobalMemoryStatus(byref(newStatus)) self.memInfo = newStatus machineInfo = ComputeNode() print machineInfo.genMemoryXmlStringRep() Since the C code seems to be a wrapper to COM services, and registration of COM service I am curious ---------------------------------------------------------------- Michael Schneider Senior Software Engineering Consultant UGS PLM Solutions - an EDS Company "The Greatest Performance Improvement Is the transitioning from a non-working state to the working state" >

2 1