Mailman 3 March 2005 - Twisted

[Twisted-Python] Developers, I need your help with 2.0
by Christopher Armstrong March 10, 2005

March 10, 2005

I plan on releasing 2.0 *for real* early next week. First, please take a look at the bug tracker. There are a few urgent issues left which I'm interested in, as well as a bunch of trivial-looking [PATCH]es that would be cool to get in for 2.0. I plan on doing some of that this weekend. I'm also planning on getting the totally reworked project-oriented website (thanks much to Mary Gardiner) polished off and deployed some time this weekend. I need help on doing the release notes for 2.0. … [View More]

4 4

Re: [Twisted-Python] deferreds and hung connections
by Lenny G Arbage March 10, 2005

March 10, 2005

--- Jp Calderone <exarkun(a)divmod.com> wrote: > > As was recently mentioned on the list, timeout > support needs > to be part of the result-generating API. Luckily, > in the case > of getPage, it is, as demonstrated by this manhole > session: > > >>> from twisted.web import client > >>> client.getPage('http://google.com', > timeout=0.001) > <Deferred #0> > Deferred #0 failed: 'Getting http://google.com … [View More]

5 6

Re: [Twisted-Python] Developers, I need your help with 2.0
by Christopher Armstrong March 10, 2005

March 10, 2005

On Wed, 9 Mar 2005 07:57:27 -0700, Dave Smith <dizzyd(a)gmail.com> wrote: > I am David "dizzy" Smith, and I endorse this message. :) > On Wed, 9 Mar 2005 15:17:43 +0100, Ralph Meijer <twisted(a)ralphm.ik.nu> wrote: > > In short, and in agreement with Dave, I'd like to offer maintainership of these > > modules. I have some plans about future developments, but that can be discussed > > later, I guess. Well, ok. I suggest people commit ralphm's patches if they … [View More]

1 0

[Twisted-Python] Twisted Logging?
by Paul Campbell March 10, 2005

March 10, 2005

> > The whole thing becomes a workflow manager that collects information about processed newspaper pages. I'm planning to connect several modules (running on different machines) with PB. The server will handle a database connection and perhaps cache some data that the clients might request. Later the server will provide some status information via http (praise twisted). > > One part is an Acrobat Distiller wrapper (via win32com.client): > > I created a library with makepy … [View More]and implemented an event handler class that catches Distiller's events, e.g. "OnJobStart". > That handler class is inherited from a base class by makepy. Each event calls one predefined method. > The handler class is connected with the distiller class like > dist = acrobatdistiller.PdfDistiller() > win32com.client.WithEvents(dist, DistEventHandler) > There can be only one Distiller instance. I guess I should keep that as long as possible, for several files to distill. > -> How should I handle these events with twisted? Do I need multiple inheritance to inherit my handler also from some twisted class? This problem has come up before. Basically, you need to queue up your series of commands and then send them one at a time. Obviously, blocking is a bad thing. In this case, your PDF handler can be isolated into a separate process and/or thread. Twisted supports this. Read the threading stuff very carefully. The threading support has specific hooks designed to run a process in just this situation. So architecturally, you'd have your PDF application running in a thread. Every time it finishes a job, it sends a signal over to the twisted code signalling that the job is complete. The twisted code then schedules another one on an as-needed basis. On the twisted side, you can write a server. The server either queues a job or sends it immediately. Either way, it can return immediately and it won't be a problem blocking-wise since the time to execute the code is minimal (you hid all the blocking stuff in the side-thread where your PDF driver runs). The "return" function essentially is either an outright "pull a command out of the queue and send it over" function or you could set it up as a reactor.calllater(0.0001, functionToQcheduleQueuedPdfJob) > Before a PostScript file gets distilled [dist.FileToPDF(source, target, joboptions)], I must patch it (write some pdfmarks into it). I guess I should use a thread for those blocking operations (distilling can take several minutes). > -> One thread for both, just work blocking? This can be in your threaded code as mentioned earlier since it's all one long process. > -> Do I need a thread for every copy/move operation (large files via network)? Nope. That's the beauty of twisted code. I've written this sort of handling multiple packet trains many times. Write a protocol. There are two ways to do this. First, you can write one protocol per file transfer. The factory sets up the protocol to do the file transfer. At the end of the file, the protocol then saves the file to disk and tears down the connection. Then on return to the factory, the factory can call the code to write the PDF file. The protocol code simply shovels the data from the network to a file. As each packet arrives, it executes for a short period of time before returning (which releases control back to the reactor). No blocking is necessary at all. The protocol itself will be little more than a modification of the Echo example protocol except you may want to add some headers or something to send a file size and/or CRC to make sure that the file comes through intact. > Another part reads some XML files (using minidom) and writes the mangled content into a database. > I guess I can't split XML parsing into smaller non-blocking portions -> one thread for every file? I ran into long processing problems before. There's a simple solution. Pick something convenient such as simply chunking everything up into processing one line at a time. Store your state in a "utility class". A utility class is something like this: class Utility: pass Then you can store anything you want. First, initialize it: storage = Utility() Then attach whatever data you need for your state: storage.data = something storage.header = something else. Then, whenever you want to chunk your code into discrete steps, simply pass it along in a chain of functions, each one ending with: reactor.calllater(0.0001, nextstep, storage) The functions will look like: def nextstep(storage): ... There are some other ways of handling this. You can get really creative with your constructs using deferreds. And the Flow module allows you to go even further than the hard coded version I'm suggesting here. Try reading through the Flow module documentation. Just don't get hung up about using it. For my purposes, it was usually better just to have the basic concept in mind than to actually use Flow. > For the database stuff I'm using runInteraction, seems to be the best solution. Yep. Frequently, this is where I'd run a deferred() just after sending the request to the known-to-be-slow process. Read that section carefully too. Then your code gets called back immediately when the process completes. You will need to understand deferred's to fully utilize twisted's threading interface, and then you'll realize that you probably don't need threads. > A general issue is logging. I'm used to log4perl/log4net (same as log4j from apache), but dropped log4py in favour of the similar Python 2.3 standard logging. I already asked in my last mail to this list how I can integrate that with twisted. > -> Is it possible to get correct linenumbers if using deferreds? Or can I trust logging to be non-blocking? Deferred's work fine. It's just a programming construct that is a work-around for a lack of a real continuation. On the other hand, deferred's are much easier to use than continuations. The code seems VERY unnatural to write at first but once you get used to it, deferred's seem just well...obvious. As to logging itself...it might be easier to just write your own. Usually I haven't been overly thrilled with the mechanical-looking output of off-the-shelf loggers and logging is little more than just printing to a file anyways, sometimes with some extra (chop it up and reclaim space) features added. [View Less]

1 0

RE: [Twisted-Python] Twisted Logging?
by Henning.Ramm＠mediapro-gmbh.de March 10, 2005

March 10, 2005

>For more discussion of stdlib logging specifically: >http://twistedmatrix.com/bugs/issue307 Thank you. I won't question that your points are valid. I'm very much used to "standard logging" with its possibilities to write all the stuff to a file, send a mail on error and filter everything after source, level etc. And because it's kind of standard I can simply use the same logging format from Perl, VB.NET and Python. (Draw your wreath of garlic, but here's a lot of different code to keep … [View More]

1 0

[Twisted-Python] setTimeout in Deferred
by Yun Mao March 10, 2005

March 10, 2005

This is the doc of setTimeout: """Set a timeout function to be triggered if I am not called. @param seconds: How long to wait (from now) before firing the timeoutFunc. @param timeoutFunc: will receive the Deferred and *args, **kw as its arguments. The default timeoutFunc will call the errback with a L{TimeoutError}. DON'T USE THIS! It's a bad idea! Use a function called by reactor.callLater instead to accomplish the … [View More]

3 3

Re: [Twisted-Python] deferreds and hung connections
by Lenny G Arbage March 9, 2005

March 9, 2005

--- Christopher Armstrong <radeex(a)gmail.com> wrote: > It's just a regular errback. Handle it with > d.addErrback(myHandler). Thanks Christopher. I made the newbie mistake of not always returning from my chained call/errbacks, so I wasn't seeing this. But after fixing that, it works just as described. Very slick. I've spent some time (in a previous life) doing libasync coding, and twisted compares very favorably (and doesn't have to go through so many hoops since it … [View More]

1 0

[Twisted-Python] deferreds and hung connections
by Lenny G Arbage March 9, 2005

March 9, 2005

Suppose I am using twisted.web.client.getPage() to retrieve webpages. I am setting a callback and an errback for the result, and this all works fantastically. But what about the case where the webserver simply hangs with the connection open, never returning with either a result or an error? I assume that my errback will get called if the socket times out and closes (right?). But what if I don't want to wait that long -- what if I want to do my own timeout after N seconds/minutes? I've been … [View More]

2 1

[Twisted-Python] Twisted Logging?
by Henning.Ramm＠mediapro-gmbh.de March 9, 2005

March 9, 2005

Is there already an integration of Python 2.3 standard logging module into twisted? Or can I just use it without any asynchronous wrapper? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

3 2

[Twisted-Python] application design / threads
by Henning.Ramm＠mediapro-gmbh.de March 9, 2005

March 9, 2005

Hi again! Being a bloody newbie I'm quite unsure how to construct my application; twisted has so much possibilities, perhaps you can provide some hints. The whole thing becomes a workflow manager that collects information about processed newspaper pages. I'm planning to connect several modules (running on different machines) with PB. The server will handle a database connection and perhaps cache some data that the clients might request. Later the server will provide some status information … [View More]

1 0