CPU usage while reading a named pipe

[Ned Deily]

In article 
<da2362e0-ec68-467b-b50b-6067057d76e2 at y36g2000yqh.googlegroups.com>,
 Miguel P <prosper.spurius at gmail.com> wrote:
> I've been working on parsing (tailing) a named pipe which is the
> syslog output of the traffic for a rather busy haproxy instance. It's
> a fair bit of traffic (upto 3k hits/s per server), but I am finding
> that simply tailing the file  in python, without any processing, is
> taking up 15% of a CPU core. In contrast HAProxy takes 25% and syslogd
> takes 5% with the same load. `cat < /named.pipe` takes 0-2%
> 
> Am I just doing things horribly wrong or is this normal?
> 
> Here is my code:
> 
> from collections import deque
> import io, sys
> 
> WATCHED_PIPE = '/var/log/haproxy.pipe'
> 
> if __name__ == '__main__':
>     try:
>         log_pool = deque([],10000)
>         fd = io.open(WATCHED_PIPE)
>         for line in fd:
>             log_pool.append(line)
>     except KeyboardInterrupt:
>         sys.exit()
> 
> Deque appends are O(1) so that's not it. And I am using 2.6's io
> module because it's supposed to handle named pipes better. I have
> commented the deque appending line and it still takes about the same
> CPU.

Be aware that the io module in Python 2.6 is written in Python and was 
viewed as a prototype.  In the current svn trunk, what will be Python 
2.7 has a much faster C implementation of the io module backported from 
Python 3.1.

-- 
 Ned Deily,
 nad at acm.org