Re: [Twisted-Python] deciding to use twisted or not

On Wed, Aug 26, 2009 at 6:41 PM, Glyph Lefkowitz wrote:

On Wed, Aug 26, 2009 at 4:56 PM, Martin-Louis Bright wrote:

...

I would like to write a small daemon that monitors (tails) a server log, parses the entries and sends HTTP requests based on some of those entries. I would like it if the reading of the log file and the sending of http requests were asynchronous. Should I use twisted for this? Or is twisted overkill...

Twisted would be perfectly appropriate for this! The HTTP requests will certainly be non-blocking, assuming you use twisted.web.client (or the new, better thing which I hope will be released before we all get old).

In fact, if anything, Twisted is under-kill; you'll need to go a bit further. The one minor issue is that Twisted has no explicit way to do asynchronous file I/O (because most operating systems provide a bewildering array of not-really-working systems for non-blocking file I/O, so it would be challenging for Twisted to do in a way that was useful). There are a number of ways to emulate it though; for log files (which are fairly low volume, and almost by definition will immediately be in your filesystem cache as the parts you want to read become available) you can just do a LoopingCall() which does .read() on a file that it leaves open to retrieve the next chunk of data, and that will probably be good enough. In practice you won't actually block doing this read() because your kernel is just going to pull the bytes directly out of filesystem cache memory and hand them back to you. It is of course possible to have to wait for the disk or even the network, depending on your underlying filesystem.

I just filed http://twistedmatrix.com/trac/ticket/3983 for a more thorough solution (mostly in the hopes that somebody will close it as a duplicate and point to some pre-existing issue I couldn't find through search) so you can monitor future discussion there if you like.

It would certainly be nice if Twisted supported async file io, but in this case wouldn't a ProcessProtocol around 'tail -f' be a good solution as well? -Cary

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-- 01100011 01100001 01110010 01111001