Mailman 3 July 2010 - Twisted

[Twisted-Python] msn file transfer
by code prophecy 11 Jul '10

11 Jul '10

Hi, Can anyone provide an example of msn file transfer ? Twisted msn implementation does have the filesender/recver classes but i cant figue out how to use them.Any help is welcome. Also, can anyone suggest a way to implement avatar/display image functionality to the existing module? i'm ready to do it but need some pointers or hint as to how to go abt it. Thanks

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 11 Jul '10

11 Jul '10

1 0

[Twisted-Python] Lore to Sphinx Conversion Progress Report 6
by Kevin Horn 11 Jul '10

11 Jul '10

It's been a while, I know you were all waiting with baited breath... First a few fixes: - Thanks to Tim Allen and Steve Steiner, several theme issues were fixed. This fixes a few minor display bugs and should make pages validate properly. - inline markup will now include child contents. This causes rst not to display properly in cases of "nested inline markup", but should make it easier to manually fix these places later - :download: links are now generated for non-rst files ending in .py (most of the examples), as welll as .tac and .sql files - toc entries which end in a "/" character now have "index appended to them in the ReST output, which fixes a few issues where the built-Sphinx docs would end up with broken navigation links. A couple of issues were also fixed at the PyCon sprints: - <cite> tags are now handled...sort of. According to Glyph, these aren't really very important, so we just return the contents of the tag without any special markup. If we feel the need to put citation markup back into the docs we can do that post-transition. - Glyph also found an easy way to improve the CSS of the theme somewhat. The issue where the sidebar drops below the main content should now be less of a problem, though it's not gone entirely. - Glyph and I also discussed a few different ways to handle links to the API docs and agreed on a way forward. Basically, well change the <code class='API'> tags into something like: :api:`path.to.documented object <label>`, and then create a docutils extension to convert that into a link. The beginnings of this are in the hg repos, but it looks like it won't be quite as simple as we thought. First, the way pydoctor generates links is deceptively simple. If you look up a module, like "twisted.internet.defer", the link looks like this: http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.html If you look up a class, like "twisted.internet.defer.Deferred" for example, the link looks like this: http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Defer… Pretty simple right? But if you look up a function or method like "twisted.internet.defer.Deferred.callback", the link has an anchor, like so: http://twistedmatrix.com/documents/9.0.0/api/twisted.internet.defer.Deferre… So we'll need to do some checking on either the api docs or on twisted itself to determine what sort of object is being linked to. Shouldn't be super-difficult, just haven't had time yet. Second, I haven't yet figured out how to access the Sphinx config object from a docutils role function. I'm sure this is just a matter of finding an example or getting an answer from the mailing list, but I just haven't gotten to it yet. I think the API links are the only major issue still remaining until we start doing the conversion, so once this issue is handled, I'll probably file a ticket and create a branch in SVN for the conversion. If anyone has any docs tickets open, now is the time to get them fixed! And if anyone has some great ideas about how to organize the new docs in the repository, I'd love to hear them. As always, converted docs are here: http://twistedsphinx.funsize.net/ The lore2Sphinx Hg repos is here: http://bitbucket.org/khorn/lore2sphinx/ Comments, ideas, patches and pull requests gratefully accepted (or at least discussed). Kevin Horn

7 21

[Twisted-Python] Uploading multiple files using ftpclient in Twisted
by Jaepyoung Kim 11 Jul '10

11 Jul '10

Dear twisted-python expert, I am build engineer and I need to upload binary to ftp server. Binary files are about 30 and each file size is about 30M. The current script is uploading using ftplib and it takes time about 1 hour. I want to change this script to use twisted asynchronous function. I thought if I use asynchronous function in twisted like following, then file uploading will be executed in parallel. But this was executed sequentially. Uploading second file starts afer completing first file upload. Could you check what was wrong in my source code? Or Am I wrong in understanding asynchronous function? for file in fileList: ftpClient.cwd("/tmp").addCallbacks(uploadFiles, fail, callbackArgs=(file,ftpClient)) Thanks, Jaepyoung Full source code: def filesucess(ab): print ab def uploadFiles(result, file,ftpClient): d1,d2=ftpClient.storeFile('/tmp/'+file) d1.addCallback(cbStore, file).addErrback(fileTransferFail) d1.addCallback(filesucess) return d2 def fileTransferFail(failure): failure.printTraceback() reactor.stop() def showBuffer(result, bufferProtocol): print 'Got data:' print bufferProtocol.buffer.getvalue() class Options(usage.Options): optParameters = [['host', 'h', 'localhost'], ['port', 'p', 21], ['username', 'u', 'user'], ['password', None, 'password'], ['passive', None, 0], ['debug', 'd', 1], ] def cbStore(consumer, filename): fs = FileSender() print filename+" cbstror" d = fs.beginFileTransfer(open(filename, 'r'), consumer) d.addCallback(lambda _: consumer.finish()).addErrback(fileTransferFail) return d def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) print config.opts['password'] creator.connectTCP(config.opts['host'], config.opts['port'],timeout=10).addCallback(connectionMade).addErrback(connectionFailed) reactor.run() def connectionFailed(f): print "Connection Failed:", f reactor.stop() def connectionMade(ftpClient): # Get the current working directory ftpClient.pwd().addCallbacks(success, fail) fileList = os.listdir('./temp') for file in fileList: ftpClient.cwd("/tmp").addCallbacks(uploadFiles, fail, callbackArgs=(file,ftpClient)) print "connectionmade"

2 2

[Twisted-Python] Congrats to ITA
by Jason J. W. Williams 09 Jul '10

09 Jul '10

Just wanted to drop a note of congrats to ITA on their acquisition by Google. It'll be great to see Twisted in play at the Googleplex for travel. -J

2 1

[Twisted-Python] twisted 10.1 ppa
by m.beyer5＠gmx.de 09 Jul '10

09 Jul '10

Hi, I'm anxious to try out the new 10.1 release. Please push it to the ppa to make it easily accessable. Thanks :) Marcus -- GMX DSL: Internet-, Telefon- und Handy-Flat ab 19,99 EUR/mtl. Bis zu 150 EUR Startguthaben inklusive! http://portal.gmx.net/de/go/dsl -- GMX DSL: Internet-, Telefon- und Handy-Flat ab 19,99 EUR/mtl. Bis zu 150 EUR Startguthaben inklusive! http://portal.gmx.net/de/go/dsl

2 1

[Twisted-Python] conceptually, why a deferred can't be used more than once?
by Jean Daniel 09 Jul '10

09 Jul '10

Dear list, Concerning the concept of the deferred, why is it more useful to go with a deferred which gets consumed and can only be fired once? In my small script I realize I need to take special care that the deferred has not been used, and that I must recreate explicitly a deferred for each network request. In a parallel world someone might have come up with a deferred concept which happily fires the callback as many times as there data coming back from the server. Is it a dumb idea? Could the deferred design be part of the solution to the network problem of two requests passing each other as each ends is not yet aware that the other has just sent a request? Buggy networks nodes would expect a response but get a request instead and go crazy... Thanks for your help,

2 1

Re: [Twisted-Python] conceptually, why a deferred can't be used more than once?
by Terry Jones 08 Jul '10

08 Jul '10

Hi Jean > Concerning the concept of the deferred, why is it more useful to go with > a deferred which gets consumed and can only be fired once? I can't speak for the Twisted devs, but my #1 reason would be that it's just simpler that way (and Deferreds are already complex enough). > In my small script I realize I need to take special care that the > deferred has not been used, and that I must recreate explicitly a > deferred for each network request. In a parallel world someone might have > come up with a deferred concept which happily fires the callback as many > times as there data coming back from the server. Is it a dumb idea? No, I don't think so. I've done some thinking and playing along these lines. At one point I thought about a Deferred class that could be called multiple times, but decided the code changes were too ugly. Recently I did some experiments in which Deferreds could be re-used. The first observation was that the time taken to create Deferreds is really negligible. That is, just re-using an already allocated object doesn't save you much. More interestingly, the adding of call/errback functions to a Deferred does take significant time. I don't have the details of what I did handy, but from memory I managed to get about a 2x speedup in a simple test when Deferreds were having about 5 call/errbacks added to them. I wrote a class that allowed you to reset a Deferred, leaving the call/errback chain on it (snipping off other things that might have been added to the chain), and there was also a DeferredPool class that could hand you back an already prepared Deferred (or make you a new one if the pool was empty). I.e., if you're in a situation where you're repeatedly using Deferreds and putting exactly the same call/errbacks on them, you might get a win by resetting and reusing already fired Deferred. On the other hand, you'd also be a social pariah :-) If anyone wants more details, I'm happy to dig up my experimental code. It was fun and it seemed to work just fine. > Could the deferred design be part of the solution to the network problem > of two requests passing each other as each ends is not yet aware that the > other has just sent a request? Buggy networks nodes would expect a > response but get a request instead and go crazy... OK, I'll leave that one for someone else... T

1 0

[Twisted-Python] Creating a COM object in a thread
by Don Dwiggins 08 Jul '10

08 Jul '10

I need to create a COM object in a Windows application, and call it. Since the call will take some time to execute, I wrap it in a deferToThread. I've found that, when I create the object inline, it works. However, when I defer it, it hangs up in the win32com.client.Dispatch call. I've tried several things, including digging into the guts of Dispatch -- the hangup occurs during the creation of the object. I've discussed this with Mark Hammond, who suggests one lead: "The Windows message loop is used by the COM marshalling process. IIRC, the first thread to initialize COM in a process is the thread in which single-threaded objects will always end up being called in. If a different thread creates the object, COM uses Windows messages to marshall all calls back to that main thread. IOW, your second thread makes a call - even to create the object - which results in that thread sending a windows message to the main thread to act on the request. What this probably means in practice is that twisted needs to use a reactor which calls MsgWaitForMultipleObjects() and runs a message loop when the function detects a new message is in the queue. I'm not sure if there is an existing reactor which does this." I'm posting this on the off chance that someone else has wandered into this corner of Windows arcana, and has come up with something useful. I'm about to give up on doing this "inside" the twisted app, and running it in a separate app that I'll call from the thread. TIA, -- Don Dwiggins Advanced Publishing Technology

4 8

[Twisted-Python] 10.1 release follow-up
by Jonathan Lange 06 Jul '10

06 Jul '10

Hello everybody, As you've seen, the Twisted 10.1.0 release is out there. I've started this thread for any discussions about the way the release was done and for the next release. Glyph Lefkowitz has kindly offered to be the RM for the 10.2 release, which is scheduled for September 15, 2010. Note that the Ubuntu feature freeze is August 12th, so we can expect 10.1 to be the release that goes into Ubuntu 10.10. There were four open tickets on the 10.1 milestone. I have moved them off the milestone and closed them. When they are fixed, I'll be happy to consider re-opening the milestone and rolling out a 10.1.1. As part of this release, there have been many improvements to the release process document, see http://twistedmatrix.com/trac/wiki/ReleaseProcess. I would greatly appreciate it if the document could be reviewed for clarity and correctness. The biggest blockers to a swift release, from my perspective, were the following: * myself having work travel and then vacation around the scheduled time of the release * delaying to wait for feedback on the release candidate * waiting for the review for the release ticket * waiting for someone else to solve the buildbot issues for Windows I've filed a ticket for moving the release document into version control, http://twistedmatrix.com/trac/ticket/4543. I'm happy to do so, but I'd like at least a rubber-stamp from someone before I do the work. jml

6 13