shutil.tail(file, lines)
This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here: http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-py... ...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read). What do you think? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
Edit: after a better look it seems data gets stored in memory
increasingly. We can see whether that is fixable someway though.
--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
2011/11/8 Giampaolo Rodolà
This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here: http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-py... ...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read).
What do you think?
--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
There is also a more general solution: the reverse file iterator, started
here but never
finished: http://bugs.python.org/issue1677872
- John
On Tue, Nov 8, 2011 at 1:27 PM, Giampaolo Rodolà
Edit: after a better look it seems data gets stored in memory increasingly. We can see whether that is fixable someway though.
--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
2011/11/8 Giampaolo Rodolà
: This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here:
http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-py...
...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read).
What do you think?
--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
2011/11/8 John O'Connor
There is also a more general solution: the reverse file iterator, started here but never finished: http://bugs.python.org/issue1677872 - John
Oh, nice! That would certainly be better. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/
+1 This is an excellent idea. I reimplement this very often.
On Wed, Nov 9, 2011 at 5:43 AM, Giampaolo Rodolà
2011/11/8 John O'Connor
: There is also a more general solution: the reverse file iterator, started here but never finished: http://bugs.python.org/issue1677872 - John
Oh, nice! That would certainly be better.
--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
Addendum: I implement the form that continues tailing, waiting for new
data. I think the naive form "last X lines" is a little trivial to put
in stdlib.
On Wed, Nov 9, 2011 at 8:24 AM, Matt Joiner
+1 This is an excellent idea. I reimplement this very often.
On Wed, Nov 9, 2011 at 5:43 AM, Giampaolo Rodolà
wrote: 2011/11/8 John O'Connor
: There is also a more general solution: the reverse file iterator, started here but never finished: http://bugs.python.org/issue1677872 - John
Oh, nice! That would certainly be better.
--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Tue, 8 Nov 2011 19:20:45 +0100
Giampaolo Rodolà
This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here: http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-py... ...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read).
Well, is it supposed to be a text file or a binary file? With a binary file the above approach is ok (you can use an adaptative average line length if you want to be smarter). With a text file and a generic encoding (possible weird or nasty) you have no other solution than reading the file from the start. Regards Antoine.
participants (4)
-
Antoine Pitrou
-
Giampaolo Rodolà
-
John O'Connor
-
Matt Joiner