Jürgen Strass wrote:
My idea now is that depending on the number of child elements, looping could take some time. So instead I'd use twisted's reactor, specifically its callLater method like this (it's only pseudo code!):
class Generator:
def generate_html( self ): self.d = defer.Deferred() self.startProcessing() return self.d
def startProcessing( self ): self.current_element = root_elem self.processNextElement()
def processNextElement( self ): if more elements to process: if current_element.type = chapter reactor.callLater( 0, processChapter, current_element ) ..... else: d.callback( "finished" )
This is not so good. You have taken away your users option to control which thread this processing is performed in, because it has to run in the reactor's thread to avoid breaking Deferred's threading rules. Also your users are not getting any benefit from the incremental nature of this code. For example they cant get access to the first chunk of html until after callback("finished"). At least, not without more complexity. They cant decide to stop processing early because they were only interested in the html <head>.
It's more or less clear to me how to divide the traversal of such a dom tree into discrete steps
It seems like you are confusing this goal - the ability to perform work in incremental steps - with the use of twisted's reactor to schedule those steps. Set twisted aside for a moment. I propose one good pythonic interface to your html creation code may be a generate_html_iter() method, which returns an iterator over the documents html fragments. You can implement this using the processNextElement approach you suggested, although python generator functions may be more convenient. A thin wrapper around this iterator could use reactor.callLater to schedule it, then fire a callback when complete. An equally thin wrapper could use PostMessage to calculate the document in the background of a win32 gui. Or it could feed a pull producer. Alternatively it could be run in another thread with deferToThread(lambda:''.join(g.generate_html_iter())) I hope this helps,