Python-Dev April 2002

python-dev@python.org

121 participants
266 discussions

Fw: [Python-Dev] Use for enumerate()
by Raymond Hettinger 27 Apr '02

27 Apr '02

> > > > Challenge 3: do it faster and with less code. > > it should work even if the file is too large > > to fit in memory (as long as each individual line fits in memory). def getline(filename, lineno): if lineno < 1: return '' f = open(filename) g = f.readline return (reduce(lambda x,y: g(), range(lineno), ''), f.close())[0] This incorporates a couple of ideas inspired by Tim's post. Raymond Hettinger

2 1

Fun with numbers
by barry＠zope.com 27 Apr '02

27 Apr '02

Tim, Jeremy, and I are having so much fun... :) -Barry -------------------- snip snip -------------------- Python 2.3a0 (#14, Apr 23 2002, 18:26:18) [GCC 2.96 20000731 (Mandrake Linux 8.2 2.96-0.76mdk)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> 3 .__class__ <type 'int'> [9903 refs] >>> 3 .__class__ = int [9903 refs] >>> 3 .__class__ = float Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: __class__ assignment: 'float' object layout differs from 'int' [9939 refs] >>> 3 .__class__ = bool [9939 refs] >>> 3 True [9939 refs] >>> 3L .__class__ = int Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: 'long' object has only read-only attributes (assign to .__class__) [9939 refs] >>> 23000 .__class__ = bool Debug memory block at address p=0x814f544: 485823496 bytes originally requested the 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb): at p-4: 0x7a *** OUCH at p-3: 0x61 *** OUCH at p-2: 0xc8 *** OUCH at p-1: 0x3c *** OUCH the 4 pad bytes at tail=0x250a094c are Segmentation fault

4 3

Use for enumerate()
by Guido van Rossum 27 Apr '02

27 Apr '02

Here's a cute use for enumerate(), now that it's checked in. I was reading the section on files in the Python Cookbook manuscript. It recommends using linecache() if you need to get a particular line from a file. I agree that this is a good approach if you may need multiple lines from the same file, but if you know that all you need is one line from one file, linecache does too much (it reads the whole file, and keeps all of it in memory). A function that reads forward until the right line is of course easily written for any version of Python; but enumerate() seems a particularly good fit here since it avoids the need to have a separate counter. This also shows how useful it is that files are iterators! def getline(filename, lineno): if lineno < 1: return '' lineno -= 1 f = open(filename) for i, line in enumerate(f): if i == lineno: break else: line = '' f.close() return line Challenge 1: do it faster. Challenge 2: do it with less code. Challenge 3: do it faster and with less code. Rules: the semantics must be the same: always close the file, return '' for lineno out of range; lineno < 1 should not open the file. --Guido van Rossum (home page: http://www.python.org/~guido/)

7 12

_PyString_Resize
by Tim Peters 27 Apr '02

27 Apr '02

Word to the wise: learn how _PyString_Resize works. I just fixed dozens of misues in the code base, and beefed up the docs. Here's the checkin comment: """ Repair widespread misuse of _PyString_Resize. Since it's clear people don't understand how this function works, also beefed up the docs. The most common usage error is of this form (often spread out across gotos): if (_PyString_Resize(&s, n) < 0) { Py_DECREF(s); s = NULL; goto outtahere; } The error is that if _PyString_Resize runs out of memory, it automatically decrefs the input string object s (which also deallocates it, since its refcount must be 1 upon entry), and sets s to NULL. So if the "if" branch ever triggers, it's an error to call Py_DECREF(s): s is already NULL! A correct way to write the above is the simpler: if (_PyString_Resize(&s, n) < 0) goto outtahere; """ Even better, *most* of the time you can just do _PyString_Resize(&s, n); return s; at the end of a function. This works correctly in all respects regardless of whether _PyString_Resize() succeeds or fails. Note that there's also no need to store a trailing \0 byte yourself before calling _PyString_Resize: it does that automatically (for all the PyString_ alloc and realloc functions, the size you pass in is one less than the number of bytes actually allocated for string space, and a trailing \0 byte is always appended by magic in successful cases).

2 2

CHUNKSIZE
by Paul Prescod 27 Apr '02

27 Apr '02

xreadlines' default CHUNKSIZE is 8K. minidom defaults to double that. Have those values been keeping up with machine RAM inflation and should they? I presume others have thought this through but my probably-naive impression is that getting data in such little bytes is a waste of time on machines with tens or hundreds of megabytes of RAM. Paul Prescod

4 5

re: PEP 279 revisited
by Damien Morton 27 Apr '02

27 Apr '02

Theres one word that was left out from the discussion. numerate() As far as I can tell, the dictionary meaning is pretty much the same as for enumerate, but theres a much stronger association with numbering things (especially for non-latin speakers).

9 12

Re: [Python-Dev] Use for enumerate()
by Raymond Hettinger 27 Apr '02

27 Apr '02

> Challenge 3: do it faster and with less code. def getline(filename, lineno): if lineno < 1: return '' f = open(filename) i, line = zip(xrange(lineno), f)[-1] f.close() if i+1 == lineno: return line return '' To keep to the spirit of the challenge, I'm ignoring that the function is i/o bound which would lead to using an 'rb' read and doing .finds or .counts on '\n'. The approach is to vectorize, trading away memory allocation time and xrange time to save the overhead of the pure Python loop and test cycle. The test is saved by taking advantage of zip's feature which stops when the first iterator is exhausted. Raymond Hettinger

3 3

Use for enumerate
by Ben Wolfson 26 Apr '02

26 Apr '02

(I'm not subscribed so this won't thread right, but oh well) This doesn't preserve the same semantics as the original, in that it always reads the entire file, but it is awfully short: def getline(filename, lineno): return (lineno < 1 and ('',) or ([ln for (i,ln) in enumerate(open(filename)) if i==lineno-1] or ('',)))[0] -- BTR YOU HEAR ME, BEN WOLFSON?!? YOUR SUPERCILIOUS CONTEMPT POWERS CANNOT WITHSTAND THE POWER OF MY BOORISH *DIMWIT* *RAY*!!!! -- revjack

1 1

More deprecations
by Neal Norwitz 26 Apr '02

26 Apr '02

Should the use of deprecated functions in the stdlib be removed? string.atoi -> int string.atof -> float sys.exc_trackback -> sys.exc_info()[2] Should the methods which use gopher also be deprecated? ConfigParser.py:311: string.atoi is deprecated ConfigParser.py:314: string.atof is deprecated Cookie.py:331: string.atoi is deprecated inspect.py:658: sys.exc_traceback is deprecated urllib2.py:103: Module gopherlib is deprecated urllib.py:381: Module gopherlib is deprecated (Line #s aren't exactly correct, but problems exist in CVS.) Neal

11 23

email, rfc 2231
by Oleg Broytmann 26 Apr '02

26 Apr '02

Barry, email package in Python CVS significantly differs from the mimelib version. Shoould I wait while you update it or should I recreate my patches against Python CVS version? Oleg. -- Oleg Broytmann http://www.zope.org/Members/phd/ phd(a)phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

2 3

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev April 2002