...

...

-> Do I need a thread for every copy/move operation (large files via network)?

Nope. That's the beauty of twisted code. I've written this sort of handling multiple packet trains many times. Write a protocol.

Hm, sounds like using FTP and not shutil.move via system's SMB? Or is there already something better in twisted?

If I'd write my own protocol to wrap shutil.move, the operation would be blocking because I don't work on file block level and the protocol doesn't make sense. I'm not feeling like writing file transfer code on file block level. Or do you think it would be worth the hassle?

It is so darned easy to write your own protocols that there's really not much point in reusing like you are attempting to do. Here's a simple example: from twisted.internet.protocol import Protocol, ClientFactory from sys import stdout from string import unpack class FileXferProtocol(Protocol): """Handles receiving files. Fire off via the factory.""" def dataReceived(self, data): if not hasattr(self, "filesize"): # First block of the file; get the final size self.filesize = unpack("!L", data[:4]) data = data[4:] # Receiving blocks of the file self.openfile.write(data) self.recvdlen += len(data) if self.recvdlen == self.filesize: self.openfile.close() self.transport.loseConnection() return class FileXferFactory(ClientFactory): """Creates protocols to receive files""" protocol = FileXferProtocol def __init__(self, filename): self.filename = filename self.openfile = file(filename) if __name__ == '__main__': # First, create the factory protocol and initialize it f = FileXferFactory("tempfile") # Second, connect to a server to get a file (127.0.0.1:9999 in this case) reactor.connectTCP("127.0.0.1", 9999, f) reactor.run() # Call the reactor You could get hung up on worrying about blocks and reassembly and whatnot (and I have done so in the past), but why bother? About the only thing necessary here is to install some sort of error checking. In the factory code it might be appropriate to do some error checking (make sure the entire file was received) and then to call the next piece of code (the file handler). On the server side, simply read some bytes and shovel it onto the port. It would look something like this: class fileXferSend(Protocol): """Sends files""" def connectionMade(self): # The factory already set this up with a file to send self.openfile = open(self.filename) # Somewhat braindead code because the standard object won't give length self.openfile.seek(0,2) self.length = self.openfile.tell() # Start over self.openfile.seek(0, 0) block = self.openfile.read(500) # Prime the system self.transport.write(pack("!L", self.length) + block) reactor.callLater(0, self.sendBlocks) def sendBlocks(self): # Keep sending out blocks of the file until it is done block = self.openfile.read(1000) if len(block): # Keep repeating as long as there is data self.transport.write(block) reactor.callLater(0, self.sendBlocks) else: self.openfile.close() return This works very easily because TCP already handles packet assembly and ordering and checksums on all the blocks. So all you have to do is shovel the data across the network. At least theoretically, you don't even have to keep track of file length. Also, you could alternatively simply prepend a byte at the beginning of the file. The byte could be say "C" for a continuation or "E" for "eof of file". The code would work almost the same. Or you could be very careful about monitoring the error codes that come out of connectionLost to be sure whether it was a "normal" session close from the host or a dropped session (error), and don't bother implementing ANY sort of "find the end of the file" stuff.

So not using DOM but per-line XML handling. Not that convenient, because I use only small bits of all the XML data that are spread all over the files, but I guess it would become better twisted code.

Perhaps. I'm not really familiar with minidom all that well. The above code contains a perfect example of what I mean. If I had set the parameter in the file.read() command to -1 or omitted it, then the .read() function would have dumped the entire file into a string. Although this may have serious memory implications, the basic problem is that it will also block twisted while it is reading the entire file into memory. Instead, the above code reads a chunk of file, sends a packet, and then returns to the reactor in a short loop (reactor.callLater(0, ...)) which allows the reactor to intersperse calls with other events.

That's perhaps a general problem with twisted: There are great solutions for everything, but you need to know them in detail to know which fits your problem. Or you must know how you should reshape your problem to fit in some twisted solution...

Did you ever notice the same problem with basically every other framework out there? I suggested that you read up on flows for a specific reason. Flows allow you to do what you are suggesting in a very twisted way...you can sequence your XML procedure so that you break it up into short bits of execution that by themselves are effectively non-blocking. That is the "twisted way". Flows let you manage this situation when you have a full blown state machine, not just a linear sequence of steps. BUT, the documentation for Flows and Deferred's is really good at explaining how to break your code up into small non-blocking pieces. So I wasn't really pushing you to USE the Flows module, but to use the concepts that are in it (just read the introductory parts to get the idea).

I know twisted "does it all"(TM), but was is "it"? ;-)

That is the trouble with frameworks. WxWidgets is one of the best GUI frameworks available that works well with Python (via wxPython). But interestingly enough, wxWidgets includes it's own sockets library! However, before you ask, it is not easy to get wxPython (and wxWidgets) to play well with twisted. It is also reactor based, but unlike Python, the reactor in wxWidgets is very unfriendly to all other reactor-based systems.

I'm just trying to write a simple directory watcher (I need this at every corner of my app), Patrick Lauber wrote an answer to my initial question on that, but that wasn't really what I needed.

It works so far that it calls a deferred callback if it gets a notify on a new/changed file, but only once; next time I get an "AlreadyCalledError" - looks like I don't yet understand deferreds.

Common error. A deferred is a promise to call back at some time in the future, but only once (no more, no less)! Quite often, reactor.callLater(0, xxx) is what you want to do. On the face of it, reactor.callLater() appears to be a timer mechanism. But what happens if you call it with the twisted idiom reactor.callLater(0, function, parameters)? This is a very common twisted idiom. What it does is schedule another function to run immediately when the reactor is allowed to schedule (assuming that there aren't several more functions that are already ready to run... otherwise it waits in line). And (subject to buffer limits on pending calls), you can call a function via the reactor as many times as you want. Otherwise, your code can call back again and receive another deferred, and eventually another callback. Also, you may be looking instead for "deferredList". For instance, let's say that you are processing a list of files in parallel. In threading-based code, you'd fire off a thread for each file and then wait for each one to return (or perhaps never wait). In deferred's, you'd do something similar: dlist = list() for i in filelist: dlist.append(handleFile(i) # handleFile returns a deferred return defer.deferredList(dlist) This routine will return a single Deferred, but the callback results will be a list of the results (and their errback/callback status) from ALL of the handleFile() calls.

At the moment it's inherited from pb.Root, because I'll need it to run remotely sometimes, but perhaps it would be better to use a service or something else -- it should run "all the time" if not stopped and call a callback for every file. I attached the file, perhaps someone can point out my biggest mistakes?

pb is useful if you intend on using the Perspective Broker in the future for twisted's own version of RPC's. If not, it is probably wise to stay away. PB makes it very easy to refactor your code into PB form later on if you so desire. My only problem with it is that you REALLY need to control both ends of the pipe and you have to live within the limitations of TCP/IP (a big limit in the certain P2P code which is better off with very light weight RPC's).

...

first but once you get used to it, deferred's seem just well...obvious.

I hope to get into that higher state of mind soon. ;-)

Everywhere that you anticipate your code blocking on a procedure call, the code itself needs to return a deferred early on (before it blocks). Then later on, it uses the deferred to pass a result. Frequently, you will have bits of code that read something like this: d = defer.Deferred() reactor.callLater(0.00001, nextStep, d) return d def nextStep(d): ...does something... d.callback(real return) Then the caller commonly does something like: state.x = <state is a utility class to pass around function state> d=callDeferredCode(xxx) d.addcallback(responseHandler, state) return def responseHandler(response, state): ... This totally decouples the two routines. Essentially, the calling function and the called function coordinate an orderly shutdown of the calling function's code. Then the callee goes about it's business of running some lengthy function (perhaps waiting on network transmissions) before finally returning with a value in hand. The caller then picks up via the second function and the saved state. This pattern is a bit ugly but at least it is reasonably readable and it gets around so many ugly details. Once you've written a couple of these, you'll start to think about when and where and how to place the deferred and reactor.callLater calls appropriately. At first, it's just a bit of a challenge wrapping your head around the concept of continuations. Oh...and the state thing...this gets mentioned once in a while and once you use it, it is highly intuitive but new users frequently miss the concept. First, create a "utility class": class Utility: pass What good is an empty class? Plenty! Within a class, you can always use the self object for this. But outside of that, use the utility class as a temporary storage bin with named slots. By this I mean, state = Utility() state.filename = "the file I don't want to forget about" state.status = "The number I'll need later" Also one other thing...once you create a deferred (defer.Deferred), you can chain off of it as much as you want in both the caller and callee. For instance, the callee may not bother creating the deferred but may instead make calls to a deeper function and simply addCallback() before returning the SAME deferred variable to the caller. Then when the deferred actually fires, it can pre-proces the returned results before returning them to the top-level caller. Clear as much, right? Well, this situation happens for instance if you have a function to clean up/process the raw results from a network I/O call before returning the answers to a higher level. For instance, if you are writing your own RPC handler to use UDP packets (which I've done), the lowest level is responsible for handling network I/O. The next level up is responsible for detecting and handling retransmissions. The next level up is responsible for splitting/concatenating data that is too big to fit inside a single packet. And the next level up is responsible for doing a version of pickle/unpickle. So that the higher level routines communicate essentially with "rpcSend(method, param1, param2, param3...) calls while the lower level routines completely obscure the details (and are in turn obscured from the lower level details of the protocol).

I enjoyed being able to switch the logging output, e.g. from file to database or email per config file without the need to go into the code. I don't feel like re-inventing the wheel, but as Glyph pointed out, the config syntax of standard logging is just ugly and messy; the config syntax of log4[j|perl|net] ist much more logical. Perhaps someone should write a log4twisted module...

More likely, just a log4python, with sufficient room that log4twisted doesn't really require too much. For instance, log4python can use the .next() call to iterate over log entries (when reading from it). twisted code will just use this interface (instead of .dumpEntireLog) to sanely read the log in chunks.