[Tutor] *nix tail -f multiple log files with Thread

[bob gailer]

David wrote:
> Hi everyone,
>
> I copied a program from C to track multiple log files. I would like to 
> be able to print a label when a log file is updated.

Did the C program accomplish that? If so, it might help to see that 
code. If not why mention it?

Your example output falls short of your stated goal as I understand it! 
My guess is that you want what you showed:
[snip]
<access_log>
127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
/theme/custom_corners/favicon.ico HTTP/1.1" 200 894
127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
/lib/speller/spellChecker.js HTTP/1.1" 200 15980
127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET / HTTP/1.1" 200 13718
127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
/theme/custom_corners/styles.php HTTP/1.1" 200 30709
followed by, for example:
<error_log>
[Fri Feb 13 10:08:01 2009] [error] [client 127.0.0.1] File does not 
exist: /var/www/localhost/htdocs/moodle/favicon.ico
When tail discovers more records in the error log.

Is this accurate?

If so I think solutions might lie in either redirecting tail output to 
files that would be post-processed later or monitoring the logs using 
python rather than the tail command.

How shall we proceed?

> Here is the program;
> <code>
> #!/usr/bin/python
> from threading import Thread
> import subprocess
> from Queue import Queue
>
> num_threads = 3
> queue = Queue()
> logfiles = ["/var/log/messages",
>         "/var/log/apache2/access_log",
>         "/var/log/apache2/error_log"]
>
> def logtailer(i, q,):
>     while True:
>         lfile = q.get()
>         sudo = "sudo"
>         tail = "tail"
>         arg = "-f"
>         ret = subprocess.call([sudo, tail, arg, lfile])
>         q.task_done()
>
> for i in range(num_threads):
>     worker = Thread(target=logtailer, args=(i, queue))
>     worker.setDaemon(True)
>     worker.start()
>
> for lfile in logfiles:
>     queue.put(lfile)
>
> queue.join()
> </code>
> And here is a sample of the output;
>
> [Fri Feb 13 08:58:48 2009] [error] [client 127.0.0.1] File does not 
> exist: /var/www/localhost/htdocs/moodle/favicon.ico
> [Fri Feb 13 08:58:51 2009] [error] [client 127.0.0.1] File does not 
> exist: /var/www/localhost/htdocs/moodle/favicon.ico
> Feb 13 09:02:26 opteron sudo:    david : TTY=pts/4 ; PWD=/home/david ; 
> USER=root ; COMMAND=/bin/tail -f /var/log/apache2/error_log
> Feb 13 09:02:26 opteron sudo: pam_unix(sudo:session): session opened 
> for user root by david(uid=0)
> Feb 13 09:02:26 opteron sudo: pam_unix(sudo:session): session closed 
> for user root
> Feb 13 09:10:01 opteron cron[10633]: (root) CMD (test -x 
> /usr/sbin/run-crons && /usr/sbin/run-crons )
> Feb 13 09:18:33 opteron su[10678]: Successful su for root by david
> Feb 13 09:18:33 opteron su[10678]: + pts/6 david:root
> Feb 13 09:18:33 opteron su[10678]: pam_unix(su:session): session 
> opened for user root by david(uid=1000)
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /theme/custom_corners/favicon.ico HTTP/1.1" 200 894
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /lib/speller/spellChecker.js HTTP/1.1" 200 15980
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET / HTTP/1.1" 200 13718
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /theme/custom_corners/styles.php HTTP/1.1" 200 30709
>
> I would like to be able to add a label like;
>
> <error_log>
> [Fri Feb 13 08:58:48 2009] [error] [client 127.0.0.1] File does not 
> exist: /var/www/localhost/htdocs/moodle/favicon.ico
> [Fri Feb 13 08:58:51 2009] [error] [client 127.0.0.1] File does not 
> exist: /var/www/localhost/htdocs/moodle/favicon.ico
> <syslog>
> Feb 13 09:02:26 opteron sudo:    david : TTY=pts/4 ; PWD=/home/david ; 
> USER=root ; COMMAND=/bin/tail -f /var/log/apache2/error_log
> Feb 13 09:02:26 opteron sudo: pam_unix(sudo:session): session opened 
> for user root by david(uid=0)
> Feb 13 09:02:26 opteron sudo: pam_unix(sudo:session): session closed 
> for user root
> Feb 13 09:10:01 opteron cron[10633]: (root) CMD (test -x 
> /usr/sbin/run-crons && /usr/sbin/run-crons )
> Feb 13 09:18:33 opteron su[10678]: Successful su for root by david
> Feb 13 09:18:33 opteron su[10678]: + pts/6 david:root
> Feb 13 09:18:33 opteron su[10678]: pam_unix(su:session): session 
> opened for user root by david(uid=1000)
> <access_log>
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /theme/custom_corners/favicon.ico HTTP/1.1" 200 894
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /lib/speller/spellChecker.js HTTP/1.1" 200 15980
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET / HTTP/1.1" 200 13718
> 127.0.0.1 - - [13/Feb/2009:09:18:47 -0500] "GET 
> /theme/custom_corners/styles.php HTTP/1.1" 200 30709
>
> I can create the label like this;
> def label()
>     print "<%s\n>" % lfile
>
> But I have no idea how to get it to work when a thread is updated.
> thanks
> -david
>
>
>


-- 
Bob Gailer
Chapel Hill NC
919-636-4239