> > > > Challenge 3: do it faster and with less code.
> > it should work even if the file is too large
> > to fit in memory (as long as each individual line fits in memory).
def getline(filename, lineno):
if lineno < 1:
return ''
f = open(filename)
g = f.readline
return (reduce(lambda x,y: g(), range(lineno), ''), f.close())[0]
This incorporates a couple of ideas inspired by Tim's post.
Raymond Hettinger
Tim, Jeremy, and I are having so much fun...
:)
-Barry
-------------------- snip snip --------------------
Python 2.3a0 (#14, Apr 23 2002, 18:26:18)
[GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 3 .__class__
<type 'int'>
[9903 refs]
>>> 3 .__class__ = int
[9903 refs]
>>> 3 .__class__ = float
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: __class__ assignment: 'float' object layout differs from 'int'
[9939 refs]
>>> 3 .__class__ = bool
[9939 refs]
>>> 3
True
[9939 refs]
>>> 3L .__class__ = int
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'long' object has only read-only attributes (assign to .__class__)
[9939 refs]
>>> 23000 .__class__ = bool
Debug memory block at address p=0x814f544:
485823496 bytes originally requested
the 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb):
at p-4: 0x7a *** OUCH
at p-3: 0x61 *** OUCH
at p-2: 0xc8 *** OUCH
at p-1: 0x3c *** OUCH
the 4 pad bytes at tail=0x250a094c are Segmentation fault
Here's a cute use for enumerate(), now that it's checked in.
I was reading the section on files in the Python Cookbook manuscript.
It recommends using linecache() if you need to get a particular line
from a file. I agree that this is a good approach if you may need
multiple lines from the same file, but if you know that all you need
is one line from one file, linecache does too much (it reads the whole
file, and keeps all of it in memory).
A function that reads forward until the right line is of course easily
written for any version of Python; but enumerate() seems a
particularly good fit here since it avoids the need to have a separate
counter. This also shows how useful it is that files are iterators!
def getline(filename, lineno):
if lineno < 1:
return ''
lineno -= 1
f = open(filename)
for i, line in enumerate(f):
if i == lineno:
break
else:
line = ''
f.close()
return line
Challenge 1: do it faster.
Challenge 2: do it with less code.
Challenge 3: do it faster and with less code.
Rules: the semantics must be the same: always close the file, return
'' for lineno out of range; lineno < 1 should not open the file.
--Guido van Rossum (home page: http://www.python.org/~guido/)
Word to the wise: learn how _PyString_Resize works. I just fixed dozens of
misues in the code base, and beefed up the docs. Here's the checkin
comment:
"""
Repair widespread misuse of _PyString_Resize. Since it's clear people
don't understand how this function works, also beefed up the docs. The
most common usage error is of this form (often spread out across gotos):
if (_PyString_Resize(&s, n) < 0) {
Py_DECREF(s);
s = NULL;
goto outtahere;
}
The error is that if _PyString_Resize runs out of memory, it automatically
decrefs the input string object s (which also deallocates it, since its
refcount must be 1 upon entry), and sets s to NULL. So if the "if"
branch ever triggers, it's an error to call Py_DECREF(s): s is already
NULL! A correct way to write the above is the simpler:
if (_PyString_Resize(&s, n) < 0)
goto outtahere;
"""
Even better, *most* of the time you can just do
_PyString_Resize(&s, n);
return s;
at the end of a function. This works correctly in all respects regardless
of whether _PyString_Resize() succeeds or fails.
Note that there's also no need to store a trailing \0 byte yourself before
calling _PyString_Resize: it does that automatically (for all the PyString_
alloc and realloc functions, the size you pass in is one less than the
number of bytes actually allocated for string space, and a trailing \0 byte
is always appended by magic in successful cases).
xreadlines' default CHUNKSIZE is 8K. minidom defaults to double that.
Have those values been keeping up with machine RAM inflation and should
they? I presume others have thought this through but my probably-naive
impression is that getting data in such little bytes is a waste of time
on machines with tens or hundreds of megabytes of RAM.
Paul Prescod
Theres one word that was left out from the discussion.
numerate()
As far as I can tell, the dictionary meaning is pretty much the same as
for enumerate, but theres a much stronger association with numbering
things (especially for non-latin speakers).
> Challenge 3: do it faster and with less code.
def getline(filename, lineno):
if lineno < 1:
return ''
f = open(filename)
i, line = zip(xrange(lineno), f)[-1]
f.close()
if i+1 == lineno:
return line
return ''
To keep to the spirit of the challenge, I'm ignoring that
the function is i/o bound which would lead to using
an 'rb' read and doing .finds or .counts on '\n'.
The approach is to vectorize, trading away memory
allocation time and xrange time to save the overhead
of the pure Python loop and test cycle.
The test is saved by taking advantage of zip's feature
which stops when the first iterator is exhausted.
Raymond Hettinger
(I'm not subscribed so this won't thread right, but oh well)
This doesn't preserve the same semantics as the original, in that it
always reads the entire file, but it is awfully short:
def getline(filename, lineno):
return (lineno < 1 and ('',) or
([ln for (i,ln) in enumerate(open(filename)) if
i==lineno-1] or ('',)))[0]
--
BTR
YOU HEAR ME, BEN WOLFSON?!? YOUR SUPERCILIOUS CONTEMPT POWERS CANNOT
WITHSTAND THE POWER OF MY BOORISH *DIMWIT* *RAY*!!!!
-- revjack
Should the use of deprecated functions in the stdlib be removed?
string.atoi -> int
string.atof -> float
sys.exc_trackback -> sys.exc_info()[2]
Should the methods which use gopher also be deprecated?
ConfigParser.py:311: string.atoi is deprecated
ConfigParser.py:314: string.atof is deprecated
Cookie.py:331: string.atoi is deprecated
inspect.py:658: sys.exc_traceback is deprecated
urllib2.py:103: Module gopherlib is deprecated
urllib.py:381: Module gopherlib is deprecated
(Line #s aren't exactly correct, but problems exist in CVS.)
Neal
Barry,
email package in Python CVS significantly differs from the mimelib
version. Shoould I wait while you update it or should I recreate my patches
against Python CVS version?
Oleg.
--
Oleg Broytmann http://www.zope.org/Members/phd/ phd(a)phd.pp.ru
Programmers don't die, they just GOSUB without RETURN.
