[Tutor] Logfile multiplexing
alan.gauld at btinternet.com
Wed Nov 11 16:47:39 CET 2009
> Why would that be admitting defeat?
> Well, it mean admitting defeat on solving the problem in python. Yes
You could still use Python to do the sorting. But it breaks the problem
into two separate and simpler processes. One simply sorts a file given
a particular data layout and sort algorithm optimised for sorting where
a) the volumes dictate doing it out of memory, and
b) the data is almost sorted already.
and the other merges pre-sorted files into one, filtering out unwanted
lines as you go.
cron can be used to launch the sorting jobs - one per file if that
is a constant, or via a shell script that checks the files and launches
the python sort per file found as a background process. (You could
replace the shell with Python too but shell is designed for this kind
cron can also be used to then launch the merge job which can check
that the sorting has completed, with no errors and merge the results.
> programming exercise, learning how to sort a number of files into one
> is something I'd like to crack.
The problem hasn't changed, nor has the solution technology, its how
you architect the solution that is different. Just because its Python
doesn't mean it all has to be in a single program.
> Maybe the real lesson here is knowing which battles to fight, and a
> good developer uses the right tools for the job.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Tutor