RE: [Twisted-Python] App design / Twisted Logging?
Paul, thank you very much for taking the time to answer my confusion.
-> Do I need a thread for every copy/move operation (large files via network)?
Nope. That's the beauty of twisted code. I've written this sort of handling multiple packet trains many times. Write a protocol.
Hm, sounds like using FTP and not shutil.move via system's SMB? Or is there already something better in twisted? If I'd write my own protocol to wrap shutil.move, the operation would be blocking because I don't work on file block level and the protocol doesn't make sense. I'm not feeling like writing file transfer code on file block level. Or do you think it would be worth the hassle?
Another part reads some XML files (using minidom) and writes ... I ran into long processing problems before. There's a simple solution. Pick something convenient such as simply chunking everything up into processing one line at a time.
So not using DOM but per-line XML handling. Not that convenient, because I use only small bits of all the XML data that are spread all over the files, but I guess it would become better twisted code.
your constructs using deferreds. And the Flow module allows you to go even further than the hard coded version I'm suggesting here. Try reading through the Flow module documentation. Just don't get hung up about using it. For my purposes, it was usually better just to have the basic concept in mind than to actually use Flow.
That's perhaps a general problem with twisted: There are great solutions for everything, but you need to know them in detail to know which fits your problem. Or you must know how you should reshape your problem to fit in some twisted solution... I know twisted "does it all"(TM), but was is "it"? ;-)
Yep. Frequently, this is where I'd run a deferred() just after sending the request to the known-to-be-slow process. Read that section carefully too. Then your code gets called back immediately when the process completes. You will need to understand deferred's to fully utilize twisted's threading interface, and then you'll realize that you probably don't need threads.
I'm just trying to write a simple directory watcher (I need this at every corner of my app), Patrick Lauber wrote an answer to my initial question on that, but that wasn't really what I needed. It works so far that it calls a deferred callback if it gets a notify on a new/changed file, but only once; next time I get an "AlreadyCalledError" - looks like I don't yet understand deferreds. At the moment it's inherited from pb.Root, because I'll need it to run remotely sometimes, but perhaps it would be better to use a service or something else -- it should run "all the time" if not stopped and call a callback for every file. I attached the file, perhaps someone can point out my biggest mistakes?
first but once you get used to it, deferred's seem just well...obvious.
I hope to get into that higher state of mind soon. ;-)
As to logging itself...it might be easier to just write your own. Usually I haven't been overly thrilled with the mechanical-looking output of off-the-shelf loggers and logging is little more than just printing to a file anyways, sometimes with some extra (chop it up and reclaim space) features added.
I enjoyed being able to switch the logging output, e.g. from file to database or email per config file without the need to go into the code. I don't feel like re-inventing the wheel, but as Glyph pointed out, the config syntax of standard logging is just ugly and messy; the config syntax of log4[j|perl|net] ist much more logical. Perhaps someone should write a log4twisted module... Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.
participants (1)
-
Henning.Ramm@mediapro-gmbh.de