Mailman 3 March 2007 - Twisted

[Twisted-Python] Couple of conch bug-fixes
by John Popplewell March 12, 2007

March 12, 2007

Hi, I've been using the Twisted SSH server and client code and found a couple of bugs in the SFTP code: In 'twisted/conch/ssh/filetransfer.py': FILEXFER_ATTR_ACMODTIME 0x00000009 should be replaced by this: FILEXFER_ATTR_ACMODTIME 0x00000008 According to this document anyway: http://tools.ietf.org/html/draft-ietf-secsh-filexfer-02#section-5 This makes the 'preserve timestamp' option in WinSCP3 work. In 'twisted/conch/scripts/cftp.py': d = self.client.openFile(remote, … [View More]

3 3

[Twisted-Python] Strange twistd behaviour
by Jarek Zgoda March 12, 2007

March 12, 2007

I have an application ran as a twistd plugin. It spawns a "worker thread" that gets objects from Queue.Queue and runs in a "while 1: " loop. The problem I got is that when I run the application without daemonizing it (using -n switch for twistd) everything works fine, but as soon as it is run as daemon, the thread does not perform the "looping". I am bit stuck, as I don't know even where to look for possible causes... Can anybody put some light here? -- Jarek Zgoda "We read Knuth so … [View More]

4 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 11, 2007

March 11, 2007

1 0

[Twisted-Python] Need some pointers for writing asynchronous code for Twisted app
by Brian Costlow March 11, 2007

March 11, 2007

I'm trying to rewrite an existing 'back end' server application. The old app worked something like this. Client boxes, using multiple methods (ftp, copy over nfs mount, even rsh sessions) create a file on disk on the server, which was either XML or a simple proprietary data format. The server was done in python. It loops over a direcory, looking for new files, and processes them into a MySQL database. I patched the client (not python, but proprietary vendor apps, which are glued together via … [View More]TCL,) to just write the data to a TCP socket. Using Twisted, I now have a test TCP server running, which uses LineReceiver, with each line recieved added to a list, and a connection close callback that writes the data to a file. This last part isn't part of the final app, and it does block, i know. It's just for me to see the data we're gettiing. So we have a working system now that transmits the files via a unified method, to a server that can handle simultaneous connections. Cool. Now I have to do real work with the data. I have some architectural questions on how to proceed. Take the case of the XML data. In the old version, it reads the XML into an ElementTree, uses business logic to iterate through all or part of the tree, building a key, value dict, that dict is passed to another object whose methods construct sql inserts from the dict data and makes db calls. (That's simplified, the current db layer is a huge rube goldberg). So the easy way out, it seems to me, would be to make the LineRecever callback build the ElementTree as I get it. Then wrap minimally modfied versions of the code that processes the ElementTree to the dict, and the dict to the database, in a callInThread or deferToThread call. Which is a lot of use of the thread pool, which seems to violate the idea of a low-overhead asynchronous event loop. So is there a better way? For example, if I have a callback chain, when the first one fires, do they all fire in sequence as the prior callback returns, or does the chain yield to other events. If it does, I could potentially break the code into smaller chunks, say so each one processed enough tree data to generate 1 dict entry, and add the chunks as a callback chain on the connectionLost? Note: None of this code is tested, I'm just trying to get the basic logic worked out. Something like this? def connectionLost(self): d = defer.Deferrred() d.addCallback(chunkOne) d.addCallback(chunkTwo) d.addCallback(chunkThree) d.addCallback(chunkN...) d.addCallback(finish) d.callback(self.myElementTree) If I have a bunch of connections that close as simultaneously as the implementation allows, does that sequence all fire first for one closing connection, then the next, and so on? Or do they intermix? Or do I need to set up a chain of deferreds with explicit scheduling? Something like: def connectionLost(self): self.myDict = {} finish() def finish(self) d = defer.Deferred def realFinish(d): do stuff to clean up d.addCallback(ChunkThree) d.addCallback(realFinish) reactor.callLater(0, d.callback, None) def chunkThree() d = defer.Deferred def realChunkThree(self.MyElementTree, self.myDict): do stuff to process one dict key d.addCallback(ChunkTwo) d.addCallback(realChunkThree) reactor.callLater(0, d.callback, None) return d etc, The above doesn't really seem much different than the first, it's just that we schedule the calls explicitly, and pass data around in multiple deferreds. The last thing I though about doing was something like this: def connectionLost(self): myDict = {} d.defer.Deferred() d.addCallback(finish) myIterObj = self.myElementTree.getIterator() def processChunk(): try: foo = myIterObj.next() do stuff with foo to process element to dict entry except StopIteration: d.callback(None) except: error handling stuff else reactor.callLater(0, processChunk) return d Except I found some really similar code in an old thread, where Bob Ippolito says, 'just use flow instead' http://twistedmatrix.com/pipermail/twisted-python/2003-July/005013.html But the current flow doc says: Don't use flow, write asynchronous code. [View Less]

5 7

[Twisted-Python] seafelt uses twisted
by Justin Warren March 9, 2007

March 9, 2007

Hey folks, This is sort of related to the SNMP replies I've made recently. It says in the wiki that you're interested in projects that use twisted, so I thought I'd let you know that seafelt (http://www.seafelt.com) now uses twisted at the core of the backend programs, for SNMP, ICMP and TCP discovery and polling, as well as asynchronous database access to PostgreSQL. seafelt is a network performance monitoring and reporting tool that I've been working on with Jim Crumpler for the past … [View More]

4 4

[Twisted-Python] how to get the result of a callLater-scheduled func?
by Stefan Rank March 8, 2007

March 8, 2007

Hi, I am scheduling a function with `callLater`. The function happens to return a Deferred (but it might return anything), and I was searching for an easy way to add a callback in order to get the result. I know a workaround by creating a Deferred first, adding the call as callback and another callback to get the result, but this seems overly complicated (see below). Am I missing something? So I set up some function to be scheduled (happens to be an inlineCallbacks-generator for testing, … [View More]

2 2

[Twisted-Python] mktap and twistd
by John Pote March 8, 2007

March 8, 2007

Hello all, I'm just putting together a twisted app that will have an http server for a control UI and a TCP server running my own protocol. I've read the 'Configuring and using the Twisted.Web Server' but am a bit confused as to the relation between rolling my own as described at the beginning of the howto and the use of mktap and twistd. Do I take it correctly that mktap makes a simple application that is not particularly adaptable? Assuming I roll my own as shown at the beginning of … [View More]

2 2

[Twisted-Python] is IOCP reactor on win2k mature enough fo r production use?
by yangjun_yulin＠126.com March 7, 2007

March 7, 2007

i want to replace some old programs which run on win2k, the program is simple but have to deal with a lot of TCP connections, 1K on average and up to 5K+ on max. the program is mainly used to collect data,every connection send data in sevaral seconds(from 3 sec to 30 sec),every time the data is less than 100 Bytes. thanks for any suggestion!

1 0

[Twisted-Python] Any examples of SNMP enabling a Twisted Server?
by Jarrod Roberson March 6, 2007

March 6, 2007

I was just wondering if there were any twisted server ( NOT CLIENT code ) that are SNMP enabled. I found lots of SNMP client code via google. But nothing on SNMP interface for a custom service?

3 2

[Twisted-Python] Twisted and setuptools
by Brian Granger March 6, 2007

March 6, 2007

There was some talk at PyCon 2007 of making twisted egg friendly during the sprints. Was any progress made on that front? It was great to meet everyone at PyCon! Cheers, Brian

2 1