shutil.tail(file, lines)
This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here: http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-py... ...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read). What do you think? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
Edit: after a better look it seems data gets stored in memory increasingly. We can see whether that is fixable someway though. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/11/8 Giampaolo Rodolà <g.rodola@gmail.com>:
There is also a more general solution: the reverse file iterator, started here but never finished: http://bugs.python.org/issue1677872 - John On Tue, Nov 8, 2011 at 1:27 PM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
2011/11/8 John O'Connor <jxo6948@rit.edu>:
Oh, nice! That would certainly be better. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
On Tue, 8 Nov 2011 19:20:45 +0100 Giampaolo Rodolà <g.rodola@gmail.com> wrote:
Well, is it supposed to be a text file or a binary file? With a binary file the above approach is ok (you can use an adaptative average line length if you want to be smarter). With a text file and a generic encoding (possible weird or nasty) you have no other solution than reading the file from the start. Regards Antoine.
Edit: after a better look it seems data gets stored in memory increasingly. We can see whether that is fixable someway though. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/11/8 Giampaolo Rodolà <g.rodola@gmail.com>:
There is also a more general solution: the reverse file iterator, started here but never finished: http://bugs.python.org/issue1677872 - John On Tue, Nov 8, 2011 at 1:27 PM, Giampaolo Rodolà <g.rodola@gmail.com> wrote:
2011/11/8 John O'Connor <jxo6948@rit.edu>:
Oh, nice! That would certainly be better. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
On Tue, 8 Nov 2011 19:20:45 +0100 Giampaolo Rodolà <g.rodola@gmail.com> wrote:
Well, is it supposed to be a text file or a binary file? With a binary file the above approach is ok (you can use an adaptative average line length if you want to be smarter). With a text file and a generic encoding (possible weird or nasty) you have no other solution than reading the file from the start. Regards Antoine.
participants (4)
-
Antoine Pitrou -
Giampaolo Rodolà -
John O'Connor -
Matt Joiner