On 10/18/07, Jürgen Strass
To give some example, I'd like to develop my own simplified document format in XML and a corresponding parser. ... This module could be useful outside any twisted application, of course. ... The question is this: since parsing and generating large documents could block the reactor in a twisted app, should I use any of twisted's asynchronous programming features in this module (for better integration with twisted) or should I rather develop it in a traditional way and run it in a thread?
You don't need to make the module depend on Twisted, but you also don't need to force users to use a thread. Just make sure the library knows how to parse and process incrementally; then your asynchronous users can pass in chunks of data as they receive them and your other users can pass in everything at once. Basically, it's a matter of inverting your library's loop that would otherwise go "read; process;" to "when process_more_data is called, process that data".
The last question is what criteria I could use to divide long lasting operations into chunks. In almost all books about asynchronous programming I only read that if they're too big, they could block the event loop. Of course, but how big is too big? And what's the measure for it? Milliseconds, number of operations, number of code lines - or what? Doesn't it depend entirely on the application at hand and how reactive it has to be?
"Bigness" here refers to time spent. And yes, it depends on the application. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/