Mailman 3 March 2007 - Twisted

[Twisted-Python] Couple of conch bug-fixes
by John Popplewell 12 Mar '07

12 Mar '07

Hi, I've been using the Twisted SSH server and client code and found a couple of bugs in the SFTP code: In 'twisted/conch/ssh/filetransfer.py': FILEXFER_ATTR_ACMODTIME 0x00000009 should be replaced by this: FILEXFER_ATTR_ACMODTIME 0x00000008 According to this document anyway: http://tools.ietf.org/html/draft-ietf-secsh-filexfer-02#section-5 This makes the 'preserve timestamp' option in WinSCP3 work. In 'twisted/conch/scripts/cftp.py': d = self.client.openFile(remote, filetransfer.FXF_WRITE|filetransfer.FXF_CREAT, {}) should be replaced, in *both* places, by something like this: flags = filetransfer.FXF_WRITE|filetransfer.FXF_CREAT|filetransfer.FXF_TRUNC d = self.client.openFile(remote, flags, {}) This makes file tranfer work when an *existing* file is being over-written by a *smaller* file. It took me a while to find this one :-) Hope this is a suitable place for these minor fixes, regards, John.

3 3

[Twisted-Python] Strange twistd behaviour
by Jarek Zgoda 12 Mar '07

12 Mar '07

I have an application ran as a twistd plugin. It spawns a "worker thread" that gets objects from Queue.Queue and runs in a "while 1: " loop. The problem I got is that when I run the application without daemonizing it (using -n switch for twistd) everything works fine, but as soon as it is run as daemon, the thread does not perform the "looping". I am bit stuck, as I don't know even where to look for possible causes... Can anybody put some light here? -- Jarek Zgoda "We read Knuth so you don't have to."

4 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 11 Mar '07

11 Mar '07

Bug summary ______________________ Summary for 2007-03-04 through 2007-03-11 Bugs opened: 14 Bugs closed: 13 Total open bugs: 878 (+1) |== Type Changes |== Priority Changes |== Component Changes |Defect: +2 |Highest: -5 |Core: +3 |Enhancement: +0 |High: +1 |Lore: +1 |Normal: +4 |Pydoctor: +0 |Low: +2 |Release Management: +0 |Trial: +0 |Web: +1 |Web2: -3 |Words: +0 New / Reopened Bugs ______________________ ===== Highest ===== [#2509] removing event triggers while in an event trigger callback does bad things (opened by jack) defect core http://twistedmatrix.com/trac/ticket/2509 ===== High ===== [#2507] missing 'plugins' directory breaks trial (opened by mlum(a)redhat.com) defect trial http://twistedmatrix.com/trac/ticket/2507 ===== Normal ===== [#2504] Parameterize reactor in ircsupport to simplify testing (opened by exarkun) enhancement words http://twistedmatrix.com/trac/ticket/2504 [#2505] twisted.xmlrpc.XMLRPC.render need more error handling (opened by yasusii) defect web http://twistedmatrix.com/trac/ticket/2505 [#2508] TypeError: <lambda>() got multiple values for keyword argument 'asc' (opened by tsal) (CLOSED, wontfix) defect release management http://twistedmatrix.com/trac/ticket/2508 [#2510] add a howto explaining how to communicate between two protocol instances (opened by exarkun) enhancement core http://twistedmatrix.com/trac/ticket/2510 [#2511] AttributeError when generating docs with pydoctor (opened by kumar303) (CLOSED, fixed) defect pydoctor http://twistedmatrix.com/trac/ticket/2511 [#2513] Infinite recursion in udp write() function (opened by p1mrx) defect core http://twistedmatrix.com/trac/ticket/2513 [#2514] Documentation bug in the list of possible <span/> classes (opened by edwards) defect lore http://twistedmatrix.com/trac/ticket/2514 [#2515] "python setup.py build" fails for Twisted (opened by edwards) defect core http://twistedmatrix.com/trac/ticket/2515 [#2516] wxreactor and wx.FileDialog stops twisted reactor loop (opened by line72) defect core http://twistedmatrix.com/trac/ticket/2516 [#2515] "python setup.py build" fails for Twisted (opened by edwards) defect core http://twistedmatrix.com/trac/ticket/2515 ===== Low ===== [#2506] Memcached client (opened by therve) enhancement core http://twistedmatrix.com/trac/ticket/2506 [#2512] simple typo in twisted/internet/error.py (opened by thomasvs) enhancement core http://twistedmatrix.com/trac/ticket/2512 Closed Bugs ______________________ ===== Highest ===== [#2104] HTTP digest authentication forced for each page refresh (opened by Cyrus Daboo, closed by dreid, fixed) defect web2 http://twistedmatrix.com/trac/ticket/2104 [#2460] HTTPAuthResource doesn't provide any way to support anonymous access. (opened by dreid, closed by dreid, fixed) enhancement web2 http://twistedmatrix.com/trac/ticket/2460 [#2172] kqueue connection failure results in bad fd message (opened by ghazel, closed by alus, fixed) defect core http://twistedmatrix.com/trac/ticket/2172 [#2495] Trial never passes a traceback to pyunit-API TestResult objects (opened by jml, closed by jml, fixed) defect trial http://twistedmatrix.com/trac/ticket/2495 [#2461] words.protocols.jabber.sasl uses base64.b64encode, which breaks compatibility with Python 2.3.x (opened by murphy_slaw, closed by ralphm, fixed) defect words http://twistedmatrix.com/trac/ticket/2461 [#460 ] [PATCH] example cred checker for pypgsql/adbapi. (opened by matiu, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/460 ===== Normal ===== [#2280] DigestCredential needs a new interface that supports verifying credentials when given a prehashed user:realm:password (opened by dreid, closed by dreid, fixed) enhancement web2 http://twistedmatrix.com/trac/ticket/2280 [#2508] TypeError: <lambda>() got multiple values for keyword argument 'asc' (opened by tsal, closed by exarkun, wontfix) defect release management http://twistedmatrix.com/trac/ticket/2508 [#2511] AttributeError when generating docs with pydoctor (opened by kumar303, closed by mwh, fixed) defect pydoctor http://twistedmatrix.com/trac/ticket/2511 [#2247] Wrong signature for addCallbacks in the Deferred Reference (opened by zseil, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2247 [#1603] Typo in client howto (opened by exarkun, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/1603 [#2515] "python setup.py build" fails for Twisted (opened by edwards, closed by exarkun, ) defect core http://twistedmatrix.com/trac/ticket/2515 [#1932] Make twisted installable with easy_install / setuptools (opened by philn, closed by radix, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/1932

1 0

[Twisted-Python] Need some pointers for writing asynchronous code for Twisted app
by Brian Costlow 11 Mar '07

11 Mar '07

I'm trying to rewrite an existing 'back end' server application. The old app worked something like this. Client boxes, using multiple methods (ftp, copy over nfs mount, even rsh sessions) create a file on disk on the server, which was either XML or a simple proprietary data format. The server was done in python. It loops over a direcory, looking for new files, and processes them into a MySQL database. I patched the client (not python, but proprietary vendor apps, which are glued together via TCL,) to just write the data to a TCP socket. Using Twisted, I now have a test TCP server running, which uses LineReceiver, with each line recieved added to a list, and a connection close callback that writes the data to a file. This last part isn't part of the final app, and it does block, i know. It's just for me to see the data we're gettiing. So we have a working system now that transmits the files via a unified method, to a server that can handle simultaneous connections. Cool. Now I have to do real work with the data. I have some architectural questions on how to proceed. Take the case of the XML data. In the old version, it reads the XML into an ElementTree, uses business logic to iterate through all or part of the tree, building a key, value dict, that dict is passed to another object whose methods construct sql inserts from the dict data and makes db calls. (That's simplified, the current db layer is a huge rube goldberg). So the easy way out, it seems to me, would be to make the LineRecever callback build the ElementTree as I get it. Then wrap minimally modfied versions of the code that processes the ElementTree to the dict, and the dict to the database, in a callInThread or deferToThread call. Which is a lot of use of the thread pool, which seems to violate the idea of a low-overhead asynchronous event loop. So is there a better way? For example, if I have a callback chain, when the first one fires, do they all fire in sequence as the prior callback returns, or does the chain yield to other events. If it does, I could potentially break the code into smaller chunks, say so each one processed enough tree data to generate 1 dict entry, and add the chunks as a callback chain on the connectionLost? Note: None of this code is tested, I'm just trying to get the basic logic worked out. Something like this? def connectionLost(self): d = defer.Deferrred() d.addCallback(chunkOne) d.addCallback(chunkTwo) d.addCallback(chunkThree) d.addCallback(chunkN...) d.addCallback(finish) d.callback(self.myElementTree) If I have a bunch of connections that close as simultaneously as the implementation allows, does that sequence all fire first for one closing connection, then the next, and so on? Or do they intermix? Or do I need to set up a chain of deferreds with explicit scheduling? Something like: def connectionLost(self): self.myDict = {} finish() def finish(self) d = defer.Deferred def realFinish(d): do stuff to clean up d.addCallback(ChunkThree) d.addCallback(realFinish) reactor.callLater(0, d.callback, None) def chunkThree() d = defer.Deferred def realChunkThree(self.MyElementTree, self.myDict): do stuff to process one dict key d.addCallback(ChunkTwo) d.addCallback(realChunkThree) reactor.callLater(0, d.callback, None) return d etc, The above doesn't really seem much different than the first, it's just that we schedule the calls explicitly, and pass data around in multiple deferreds. The last thing I though about doing was something like this: def connectionLost(self): myDict = {} d.defer.Deferred() d.addCallback(finish) myIterObj = self.myElementTree.getIterator() def processChunk(): try: foo = myIterObj.next() do stuff with foo to process element to dict entry except StopIteration: d.callback(None) except: error handling stuff else reactor.callLater(0, processChunk) return d Except I found some really similar code in an old thread, where Bob Ippolito says, 'just use flow instead' http://twistedmatrix.com/pipermail/twisted-python/2003-July/005013.html But the current flow doc says: Don't use flow, write asynchronous code.

5 7

[Twisted-Python] seafelt uses twisted
by Justin Warren 09 Mar '07

09 Mar '07

Hey folks, This is sort of related to the SNMP replies I've made recently. It says in the wiki that you're interested in projects that use twisted, so I thought I'd let you know that seafelt (http://www.seafelt.com) now uses twisted at the core of the backend programs, for SNMP, ICMP and TCP discovery and polling, as well as asynchronous database access to PostgreSQL. seafelt is a network performance monitoring and reporting tool that I've been working on with Jim Crumpler for the past several years in our free time. We started it because we thought MRTG was too limited and the enterprise tools too unwieldy and expensive. We hope to do something commercial with it at some point, but it's free for personal and non-commercial use. The SNMP library is released under the MIT license, and I'd be happy to have the library components integrated into twisted, particularly bits like the ICMP code, since the twisted-pair stuff appears to have stalled? Incidentally, I wrote libsnmp quite a few years back because PySNMP had some performance issues at large scale, so I don't know if that's improved now that PySNMP is integrated with twisted. I recently ported all our old threaded code across to twisted and it's now even faster and more efficient, not to mention easier to debug. I think I now finally grok Deferreds, though I still struggle with Errback chains and Unhandled Deferred errors from time to time. So yeah, check it out if you're interested in this kind of thing. Big thumbs up to the twisted guys for providing us with such a great framework. You guys rock, but you knew that already. ;) -- Justin Warren <daedalus(a)eigenmagic.com>

4 4

[Twisted-Python] how to get the result of a callLater-scheduled func?
by Stefan Rank 08 Mar '07

08 Mar '07

Hi, I am scheduling a function with `callLater`. The function happens to return a Deferred (but it might return anything), and I was searching for an easy way to add a callback in order to get the result. I know a workaround by creating a Deferred first, adding the call as callback and another callback to get the result, but this seems overly complicated (see below). Am I missing something? So I set up some function to be scheduled (happens to be an inlineCallbacks-generator for testing, but I think this should not be relevant here):: >>> from twisted.internet import defer, reactor >>> @defer.inlineCallbacks ... def test(name): ... print name, 'start' ... yield # yielding a non-Deferred does nothing ... print name, 'end' ... yield defer.returnValue('uiuiui') I know this works:: >>> deftest = defer.Deferred().addCallback(test) >>> jamesdelcall = reactor.callLater(0, deftest.callback, 'James') >>> def stopverbose(whatever): ... print 'stopping, result:', whatever ... reactor.stop() >>> deftest.addCallback(stopverbose) >>> reactor.run() James start James end stopping, result: uiuiui What I expected to be able to do:: >>> jamesdelcall = reactor.callLater(0, test, 'James') >>> def stopverbose(whatever): ... print 'stopping, result:', whatever ... reactor.stop() >>> jamesdelcal.gethisdeferred.addCallback(stopverbose) # <- MAGIC >>> reactor.run() James start James end stopping, result: uiuiui I looked at the code of `ReactorBase.callLater` and `ReactorBase.runUntilCurrent`: Would it be possible to add a `Deferred` to `DelayedCall` (possibly created lazily, or maybe even make DelayedCall a subclass of Deferred), so that such code is possible, or would this somehow destroy the mainloop logic? cheers, stefan

2 2

[Twisted-Python] mktap and twistd
by John Pote 08 Mar '07

08 Mar '07

Hello all, I'm just putting together a twisted app that will have an http server for a control UI and a TCP server running my own protocol. I've read the 'Configuring and using the Twisted.Web Server' but am a bit confused as to the relation between rolling my own as described at the beginning of the howto and the use of mktap and twistd. Do I take it correctly that mktap makes a simple application that is not particularly adaptable? Assuming I roll my own as shown at the beginning of the howto how does twistd fit in? I seem to remember reading somewhere that twistd was the preferred means of starting a twisted app. I've tried making a simple web server with mktap and twistd which works fine except on my winXP box does not run in the background. This is a winXP limitation I presume. Any help appreciated, John Pote

2 2

[Twisted-Python] is IOCP reactor on win2k mature enough fo r production use?
by yangjun_yulin＠126.com 07 Mar '07

07 Mar '07

i want to replace some old programs which run on win2k, the program is simple but have to deal with a lot of TCP connections, 1K on average and up to 5K+ on max. the program is mainly used to collect data,every connection send data in sevaral seconds(from 3 sec to 30 sec),every time the data is less than 100 Bytes. thanks for any suggestion!

1 0

[Twisted-Python] Any examples of SNMP enabling a Twisted Server?
by Jarrod Roberson 06 Mar '07

06 Mar '07

I was just wondering if there were any twisted server ( NOT CLIENT code ) that are SNMP enabled. I found lots of SNMP client code via google. But nothing on SNMP interface for a custom service?

3 2

[Twisted-Python] Twisted and setuptools
by Brian Granger 06 Mar '07

06 Mar '07

There was some talk at PyCon 2007 of making twisted egg friendly during the sprints. Was any progress made on that front? It was great to meet everyone at PyCon! Cheers, Brian

2 1