Itamar Shtull-Trauring wrote:

[...] What you mean by "traditional" is actually a pull parser. Parsing APIs can be pull or push (i.e. asynchronous). Well-designed parsers are always push, because push parsers can be trivially converted to blocking pull parsers, but not vice-versa. Some examples of push/asynch parsers: twisted's Protocol class, or the SAX API. Sorry, I think my example was somewhat misleading and it also becomes clear to me that I haven't used the word "asynchronous" correctly. I didn't consider that one can also register callbacks with a parser, for example, and call this type of programming asynchronous. (The principle "Don't call us, we call you" would apply here, too, of course.)

No, what I really meant by "traditional" was to write parsers and generators which traverse the document as a whole in one large step, without giving a chance to the twisted reactor to process any other events. Let's assume I got a dom tree from pythons XML parser. First, I'd traverse that tree and build up another tree consisting of element objects. Each element object is an instance of a class corresponding to a tag, for example for tag <chapter> I'd create a class "chapter". This is necessary because there's not always a one-to-one correspondence between tags and my document elements and to associate some additional attributes with such elements later, for example automatically generated chapter numbers. I'd then use the generator to traverse that element tree, calling "render_element" methods on my way. For element chapter with attribute title I'd call "render_chapter( node )", which then generates "<h1>chapter_title</h1>". Let's assume I had some element with child elements. Without knowing about twisted at all, I'd have created a foreach loop to process each child like this: foreach child_node in root_elem.children: if child_node.type = chapter: processChapter( child_node ) My idea now is that depending on the number of child elements, looping could take some time. So instead I'd use twisted's reactor, specifically its callLater method like this (it's only pseudo code!): class Generator: def generate_html( self ): self.d = defer.Deferred() self.startProcessing() return self.d def startProcessing( self ): self.current_element = root_elem self.processNextElement() def processNextElement( self ): if more elements to process: if current_element.type = chapter reactor.callLater( 0, processChapter, current_element ) ..... else: d.callback( "finished" ) In this way any twisted user could get a Deferred from the generate_html method and get called when the Generator has generated all HTML. The problem with this is that I couldn't ever use such code without also installing twisted, of course. It's more or less clear to me how to divide the traversal of such a dom tree into discrete steps, but it's not so clear how to call the processNextElement with reactor.callLater from the outside. Although, after I've read the other answers, it seems to me I'm not far from a solution. I think I could also create two classes: the Generator class, which would provide a processNextElement method and doesn't need to depend on the twisted framework, and a TwistedGenerator class, which would do exactly the same like the code above and repeatedly call processNextElement with reactor.callLater. But the internal housekeeping which element to process next could be more difficult than with the solution above, couldn't it? (Because instead of seperate methods like "processChapter", "processList", etc. I'd only have one method to call from outside, "processNextElement" (and something like "moreElementsToProcess"). The TwistedGenerator wrapper shouldn't know about the internal state of the Generator, I think.) Many greetings, Jürgen

Toby Dickenson wrote:

[...]

Set twisted aside for a moment. I propose one good pythonic interface to your html creation code may be a generate_html_iter() method, which returns an iterator over the documents html fragments. You can implement this using the processNextElement approach you suggested, although python generator functions may be more convenient.

A thin wrapper around this iterator could use reactor.callLater to schedule it, then fire a callback when complete. An equally thin wrapper could use PostMessage to calculate the document in the background of a win32 gui. Or it could feed a pull producer. Alternatively it could be run in another thread with deferToThread(lambda:''.join(g.generate_html_iter()))

I hope this helps,

Some seconds before I read your message I've posted my reply to Christopher Armstrong. This message contains some code of mine that already uses some sort of wrapper class. Though it doesn't use an iterator and maybe isn't as pythonic, it wraps a synchronous interface. Not sure if this is similar to the thin wrapper you had in mind? Thank you very much, Jürgen

Christopher Armstrong wrote:

You don't need to make the module depend on Twisted, but you also don't need to force users to use a thread. Just make sure the library knows how to parse and process incrementally; then your asynchronous users can pass in chunks of data as they receive them and your other users can pass in everything at once. Basically, it's a matter of inverting your library's loop that would otherwise go "read; process;" to "when process_more_data is called, process that data".

As I already explained in my replies to Jean-Paul Calderone and Itamar Shtull-Trauring, I'm not sure if I have fully understood how this works in particular. The example of a parser probably wasn't well chosen, because in all replies to my original posting people seem to assume that I need to do something in response to an external event (e.g. I/O events). Please, better think of a long running CPU bound algorithm perhaps. From twisted, I'd like to start that algorithm in the background and get notified when it has finished. It shouldn't ever block the application for too long. In other applications, I'd like to call the algorithm in a synchronous way. It is most likely that my difficulty in understanding results from the fact that in such a case there is no external event that could trigger the processing of the individual steps the algorithm has. The best I've come up with so far uses a wrapper around a synchronous interface that provides methods for calling the next step of the algorithm and for testing if the algorithm has finished. What I'd like to know is if this solution basically is the style one has to use when writing algorithms divided into chunks and if the approach to integrate it into twisted is well chosen. Also, I don't fully understand the implications this programming style has on long running loops. I have the impression that I would need to divide loops into several methods: loopInit, loopCondition, loopNextStep and loopFinished. Is this correct? Following is my code for a counter class which also uses the wrapper class I've described above. Using the wrapper class, it is possibly to run several counters concurrently. The Counter class is already decoupled from twisted. In case I want to use the synchronous version, I'd use Counter, in case I want to use the asynchronous version with deferreds, I'd use TwistedCounter. ---- from twisted.internet import reactor, defer class Counter: def __init__( self, id, limit ): self.id = id self.limit = limit self.count = 0 def limitNotReached( self ): return self.count < self.limit def increment( self ): if self.limitNotReached(): print "Counter %d: %d" % ( self.id, self.count ) self.count = self.count + 1 def run( self ): while self.limitNotReached(): self.increment() class TwistedCounter: def __init__( self, counter ): self.counter = counter def run( self ): self.d = defer.Deferred() self.increment() return self.d def increment( self ): if self.counter.limitNotReached(): self.counter.increment() reactor.callLater( 0, self.increment ) else: self.d.callback( "finished" ) def counters_finished( result ): print " traditional counters finished." reactor.stop() print "-- Run traditional counter:\n" traditional_c = Counter( 1, 130 ) traditional_c.run() print "\n-- Run twisted counters concurrently:\n" c1 = TwistedCounter( Counter( 2, 100 ) ) c2 = TwistedCounter( Counter( 3, 150 ) ) d1 = c1.run() d2 = c2.run() d = defer.DeferredList( [ d1, d2 ] ) d.addCallback( counters_finished ) reactor.run() ---- What I can't fully imagine is if I can use this technique in all possible cases. In case I need to nest calls or algorithms, I'll run into the problem of not being able to call any of the asynchronous wrappers from the synchronous parts of my library, because those wrappers return deferreds which are part of twisted. So I'd end up with two versions of my library again. Jürgen

Jean-Paul Calderone schrieb:

On Thu, 18 Oct 2007 14:41:38 +0200, Jürgen Strass wrote:

...

To give some example, I'd like to develop my own simplified document format in XML and a corresponding parser. [...]

Have you seen Lore?

Not yet. I'll have a look at it, though. I guess to some degree I'm reinventing the wheel, but I rather see this a an exercise.

...

The question is this: since parsing and generating large documents could block the reactor in a twisted app, should I use any of twisted's asynchronous programming features in this module (for better integration with twisted) or should I rather develop it in a traditional way and run it in a thread? Incremental parsing is often useful and simpler than the alternative. If you are accepting a document over the network, why buffer it yourself and then parse it when you could just be giving each piece directly to the parser? Done this way, it often is the case that even large documents can be parsed without blocking for an unreasonable amount of time. Okay, I agree. To get away somewhat from parsers, the question arises for other programming tasks as well, of course.

So I already asked myself how one would translate the example of a factorial function in twisted's core documentation to use the reactor's scheduling mechanism instead of running it in a thread. I think an example of how to divide it into chunks and how to use the reactor would be great. What I tried at first was programming a simple counter this way. It would look much similar to the code I presented in reply to Itamar Shtull-Trauring's answer. What I'm not sure about is if this is the correct way to go for. Many thanks for all the other points you've answered, it made a lot of things much clearer to me. Jürgen

Johann Borck schrieb:

[...] I think the main misunderstanding is "[..]to use the reactor's scheduling mechanism instead of running it in a thread." Twisteds reactor is not a superior multi-purpose scheduler (as JP mentioned), but a domain-specific event handler for networking. While your use-case might (that's my guess) profit from choosing 'chunking' over pythons threading, it still wouldn't from choosing it over the scheduling of your OS.

hm, did I get you right there?

Oh yes, you're right. The whole time I thought of the reactor as a multi-purpose scheduler and didn't get JP's answer right. The misunderstanding in part results from the core documentation. In chapter 1.3, it is said: "This document will give you a high level overview of concurrent programming (interleaving several tasks) and of Twisted's concurrency model: non-blocking code or asynchronous code." Then, two examples follow: (1) CPU bound tasks and (2) tasks that wait for data. Because I thought the introduction applied to both type of examples, I also assumed that Twisted's concurrency model would apply for both types of tasks. Of course, I already wondered about not being able to find any examples for (1), while the whole rest of the docs deals with (2). ;-) Moreover, I once read Douglas Schmidt's book "Pattern Oriented Software Architecture (2)", which describes several patterns - including the reactor pattern - for middleware-oriented applications. The book led to some confusion on mine about when to use which pattern and what concurrency mechanism would be best for a particular situation. The - maybe wrong - conclusion I've drawn from that book is that context switching overhead (be it threads or processes) isn't only bad for I/O bound tasks, but also for most other concurrent tasks. As you said, function calls in python are expensive, nevertheless I thought they were less expensive than the overhead caused by context switching between threads and processes, at least on a single processor system. Or have I made a mistake here? Moreover, couldn't the creation of whole new processes be even more expensive? I mean, with "long" running algorithms I really meant tasks that could take some minutes. Processes would be very fine here. But what about - for example - CPU bound tasks that only take some hundred milliseconds, but nevertheless would block the reactor? Would you use processes in this case, too? Maybe prespawned processes? Or should I use rather threads in such a case? Many thanks for your enlighting reply, Jürgen