Mailman 3 March 2005 - Twisted

[Twisted-Python] Developers, I need your help with 2.0
by Christopher Armstrong 11 Mar '05

11 Mar '05

I plan on releasing 2.0 *for real* early next week. First, please take a look at the bug tracker. There are a few urgent issues left which I'm interested in, as well as a bunch of trivial-looking [PATCH]es that would be cool to get in for 2.0. I plan on doing some of that this weekend. I'm also planning on getting the totally reworked project-oriented website (thanks much to Mary Gardiner) polished off and deployed some time this weekend. I need help on doing the release notes for 2.0. Traditionally this has been Itamar's job, but James also recently volunteered. It'd be really awesome if someone could do this for me so I don't have to think about it. :-) Project Maintainers! Twisted: Glyph, Itamar, Jp Conch: Paul Flow: Clark Lore: Andrew, Moshe Mail, Names, News: Jp Pair: Tommi Runner: Andrew Web: Donovan Words: Jp, Glyph, ...? Xish: Uh, what's that guy's name? Dave or something? I don't think he's been very involved. Please update your README files and tell me what version number you want your project to have. Currently I'm assuming everything is at 0.1.0 except for conch and web which are both at 0.5.0. Also take a look at your setup.py file and any other topfiles for your project to make sure they're what you want (In case you've been under a rock, those are located in Twisted/twisted/<yourproj>/topfiles/ in the repository). Packagers! Get cracking. If I think of anything else, feel safe knowing I will bug you about it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash

4 4

Re: [Twisted-Python] deferreds and hung connections
by Lenny G Arbage 11 Mar '05

11 Mar '05

--- Jp Calderone <exarkun(a)divmod.com> wrote: > > As was recently mentioned on the list, timeout > support needs > to be part of the result-generating API. Luckily, > in the case > of getPage, it is, as demonstrated by this manhole > session: > > >>> from twisted.web import client > >>> client.getPage('http://google.com', > timeout=0.001) > <Deferred #0> > Deferred #0 failed: 'Getting http://google.com > took longer than 0.001 > seconds.' > >>> Thanks for the pointer, Jp. I just stumbled on some of the threads on the mailing list stretching back to as far as 2002 about whether being able to cancel Deferreds was a good idea or not. So the above solution works. Where do I catch, or otherwise react to, the error printed above? When the Deferred times out, is there a way to inject another callback in there ("timeoutcallback"?) so that I can do some processing on timeout? It looks like I could override HTTPClientFactory's timeout() method if I had a handle to it, but getPage() doesn't give me access. Am I barking up the wrong tree? Lenny __________________________________ Celebrate Yahoo!'s 10th Birthday! Yahoo! Netrospective: 100 Moments of the Web http://birthday.yahoo.com/netrospective/

5 6

Re: [Twisted-Python] Developers, I need your help with 2.0
by Christopher Armstrong 10 Mar '05

10 Mar '05

On Wed, 9 Mar 2005 07:57:27 -0700, Dave Smith <dizzyd(a)gmail.com> wrote: > I am David "dizzy" Smith, and I endorse this message. :) > On Wed, 9 Mar 2005 15:17:43 +0100, Ralph Meijer <twisted(a)ralphm.ik.nu> wrote: > > In short, and in agreement with Dave, I'd like to offer maintainership of these > > modules. I have some plans about future developments, but that can be discussed > > later, I guess. Well, ok. I suggest people commit ralphm's patches if they have unit tests and seem reasonable. Eventually we can work out SVN access (man I wish we had separate repositories). -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash

1 0

[Twisted-Python] Twisted Logging?
by Paul Campbell 10 Mar '05

10 Mar '05

> > The whole thing becomes a workflow manager that collects information about processed newspaper pages. I'm planning to connect several modules (running on different machines) with PB. The server will handle a database connection and perhaps cache some data that the clients might request. Later the server will provide some status information via http (praise twisted). > > One part is an Acrobat Distiller wrapper (via win32com.client): > > I created a library with makepy and implemented an event handler class that catches Distiller's events, e.g. "OnJobStart". > That handler class is inherited from a base class by makepy. Each event calls one predefined method. > The handler class is connected with the distiller class like > dist = acrobatdistiller.PdfDistiller() > win32com.client.WithEvents(dist, DistEventHandler) > There can be only one Distiller instance. I guess I should keep that as long as possible, for several files to distill. > -> How should I handle these events with twisted? Do I need multiple inheritance to inherit my handler also from some twisted class? This problem has come up before. Basically, you need to queue up your series of commands and then send them one at a time. Obviously, blocking is a bad thing. In this case, your PDF handler can be isolated into a separate process and/or thread. Twisted supports this. Read the threading stuff very carefully. The threading support has specific hooks designed to run a process in just this situation. So architecturally, you'd have your PDF application running in a thread. Every time it finishes a job, it sends a signal over to the twisted code signalling that the job is complete. The twisted code then schedules another one on an as-needed basis. On the twisted side, you can write a server. The server either queues a job or sends it immediately. Either way, it can return immediately and it won't be a problem blocking-wise since the time to execute the code is minimal (you hid all the blocking stuff in the side-thread where your PDF driver runs). The "return" function essentially is either an outright "pull a command out of the queue and send it over" function or you could set it up as a reactor.calllater(0.0001, functionToQcheduleQueuedPdfJob) > Before a PostScript file gets distilled [dist.FileToPDF(source, target, joboptions)], I must patch it (write some pdfmarks into it). I guess I should use a thread for those blocking operations (distilling can take several minutes). > -> One thread for both, just work blocking? This can be in your threaded code as mentioned earlier since it's all one long process. > -> Do I need a thread for every copy/move operation (large files via network)? Nope. That's the beauty of twisted code. I've written this sort of handling multiple packet trains many times. Write a protocol. There are two ways to do this. First, you can write one protocol per file transfer. The factory sets up the protocol to do the file transfer. At the end of the file, the protocol then saves the file to disk and tears down the connection. Then on return to the factory, the factory can call the code to write the PDF file. The protocol code simply shovels the data from the network to a file. As each packet arrives, it executes for a short period of time before returning (which releases control back to the reactor). No blocking is necessary at all. The protocol itself will be little more than a modification of the Echo example protocol except you may want to add some headers or something to send a file size and/or CRC to make sure that the file comes through intact. > Another part reads some XML files (using minidom) and writes the mangled content into a database. > I guess I can't split XML parsing into smaller non-blocking portions -> one thread for every file? I ran into long processing problems before. There's a simple solution. Pick something convenient such as simply chunking everything up into processing one line at a time. Store your state in a "utility class". A utility class is something like this: class Utility: pass Then you can store anything you want. First, initialize it: storage = Utility() Then attach whatever data you need for your state: storage.data = something storage.header = something else. Then, whenever you want to chunk your code into discrete steps, simply pass it along in a chain of functions, each one ending with: reactor.calllater(0.0001, nextstep, storage) The functions will look like: def nextstep(storage): ... There are some other ways of handling this. You can get really creative with your constructs using deferreds. And the Flow module allows you to go even further than the hard coded version I'm suggesting here. Try reading through the Flow module documentation. Just don't get hung up about using it. For my purposes, it was usually better just to have the basic concept in mind than to actually use Flow. > For the database stuff I'm using runInteraction, seems to be the best solution. Yep. Frequently, this is where I'd run a deferred() just after sending the request to the known-to-be-slow process. Read that section carefully too. Then your code gets called back immediately when the process completes. You will need to understand deferred's to fully utilize twisted's threading interface, and then you'll realize that you probably don't need threads. > A general issue is logging. I'm used to log4perl/log4net (same as log4j from apache), but dropped log4py in favour of the similar Python 2.3 standard logging. I already asked in my last mail to this list how I can integrate that with twisted. > -> Is it possible to get correct linenumbers if using deferreds? Or can I trust logging to be non-blocking? Deferred's work fine. It's just a programming construct that is a work-around for a lack of a real continuation. On the other hand, deferred's are much easier to use than continuations. The code seems VERY unnatural to write at first but once you get used to it, deferred's seem just well...obvious. As to logging itself...it might be easier to just write your own. Usually I haven't been overly thrilled with the mechanical-looking output of off-the-shelf loggers and logging is little more than just printing to a file anyways, sometimes with some extra (chop it up and reclaim space) features added.

1 0

RE: [Twisted-Python] Twisted Logging?
by Henning.Ramm＠mediapro-gmbh.de 10 Mar '05

10 Mar '05

>For more discussion of stdlib logging specifically: >http://twistedmatrix.com/bugs/issue307 Thank you. I won't question that your points are valid. I'm very much used to "standard logging" with its possibilities to write all the stuff to a file, send a mail on error and filter everything after source, level etc. And because it's kind of standard I can simply use the same logging format from Perl, VB.NET and Python. (Draw your wreath of garlic, but here's a lot of different code to keep together.) I'm convinced twisted's possibilities are better, but its logging isn't that extensive (yet). And I find the logging API rather chaotic. Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

1 0

[Twisted-Python] setTimeout in Deferred
by Yun Mao 10 Mar '05

10 Mar '05

This is the doc of setTimeout: """Set a timeout function to be triggered if I am not called. @param seconds: How long to wait (from now) before firing the timeoutFunc. @param timeoutFunc: will receive the Deferred and *args, **kw as its arguments. The default timeoutFunc will call the errback with a L{TimeoutError}. DON'T USE THIS! It's a bad idea! Use a function called by reactor.callLater instead to accomplish the same thing! YOU HAVE BEEN WARNED! """ Any reason why not? It seems to me that setTimeout just calls reactor.callLater() Yun

3 3

Re: [Twisted-Python] deferreds and hung connections
by Lenny G Arbage 10 Mar '05

10 Mar '05

--- Christopher Armstrong <radeex(a)gmail.com> wrote: > It's just a regular errback. Handle it with > d.addErrback(myHandler). Thanks Christopher. I made the newbie mistake of not always returning from my chained call/errbacks, so I wasn't seeing this. But after fixing that, it works just as described. Very slick. I've spent some time (in a previous life) doing libasync coding, and twisted compares very favorably (and doesn't have to go through so many hoops since it relies on python language goodness that C doesn't have by default). Nice stuff. Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

[Twisted-Python] deferreds and hung connections
by Lenny G Arbage 09 Mar '05

09 Mar '05

Suppose I am using twisted.web.client.getPage() to retrieve webpages. I am setting a callback and an errback for the result, and this all works fantastically. But what about the case where the webserver simply hangs with the connection open, never returning with either a result or an error? I assume that my errback will get called if the socket times out and closes (right?). But what if I don't want to wait that long -- what if I want to do my own timeout after N seconds/minutes? I've been looking for examples of how to do this, and monkeying with reactor.callLater() with partial success. If I save the id from callLater(), I can cancel the timeout in callback() or errback(). However, if the timeout does get fired, I get: [Failure instance: Traceback: twisted.internet.error.AlreadyCalled, Tried to cancel an already-called event. /usr/lib/python2.3/site-packages/twisted/internet/defer.py:338:_runCallbacks /tmp/deferred2.py:21:errback /usr/lib/python2.3/site-packages/twisted/internet/base.py:82:cancel ] Am I going about this the wrong way? Perhaps twisted has some built-ins that I'm not aware of to do essentially the same thing? I have a modified example from the Defer Howto that exhibits this behavior. I can post if the above explanation isn't enough. thanks, Lenny __________________________________ Celebrate Yahoo!'s 10th Birthday! Yahoo! Netrospective: 100 Moments of the Web http://birthday.yahoo.com/netrospective/

2 1

[Twisted-Python] Twisted Logging?
by Henning.Ramm＠mediapro-gmbh.de 09 Mar '05

09 Mar '05

Is there already an integration of Python 2.3 standard logging module into twisted? Or can I just use it without any asynchronous wrapper? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

3 2

[Twisted-Python] application design / threads
by Henning.Ramm＠mediapro-gmbh.de 09 Mar '05

09 Mar '05

Hi again! Being a bloody newbie I'm quite unsure how to construct my application; twisted has so much possibilities, perhaps you can provide some hints. The whole thing becomes a workflow manager that collects information about processed newspaper pages. I'm planning to connect several modules (running on different machines) with PB. The server will handle a database connection and perhaps cache some data that the clients might request. Later the server will provide some status information via http (praise twisted). One part is an Acrobat Distiller wrapper (via win32com.client): I created a library with makepy and implemented an event handler class that catches Distiller's events, e.g. "OnJobStart". That handler class is inherited from a base class by makepy. Each event calls one predefined method. The handler class is connected with the distiller class like dist = acrobatdistiller.PdfDistiller() win32com.client.WithEvents(dist, DistEventHandler) There can be only one Distiller instance. I guess I should keep that as long as possible, for several files to distill. -> How should I handle these events with twisted? Do I need multiple inheritance to inherit my handler also from some twisted class? Before a PostScript file gets distilled [dist.FileToPDF(source, target, joboptions)], I must patch it (write some pdfmarks into it). I guess I should use a thread for those blocking operations (distilling can take several minutes). -> One thread for both, just work blocking? -> Do I need a thread for every copy/move operation (large files via network)? Another part reads some XML files (using minidom) and writes the mangled content into a database. I guess I can't split XML parsing into smaller non-blocking portions -> one thread for every file? For the database stuff I'm using runInteraction, seems to be the best solution. A general issue is logging. I'm used to log4perl/log4net (same as log4j from apache), but dropped log4py in favour of the similar Python 2.3 standard logging. I already asked in my last mail to this list how I can integrate that with twisted. -> Is it possible to get correct linenumbers if using deferreds? Or can I trust logging to be non-blocking? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

1 0