Mailman 3 April 2001 - Python-Dev

Iterators, generators and 2.2 (was RE: do...until wisdom needed...)
by Tim Peters April 21, 2001

April 21, 2001

[Aahz] > Generators (called "iterators") are going to be 2.2. They'll be > more powerful that Icon's generators; it's not clear whether they'll > be a full-fledged substitute for coroutines. [Neelakantan Krishnaswami] > {\mr_burns Excellent.} > > Is this the iterator idea that Ping posted about a couple of months > back? What is the PEP number for this? I'm curious how the existing > iteration protocol will interact with the new iterators. This is getting confused. … [View More]Iterators != generators (sorry, Aahz! it's more involved than that). Aahz gave you the PEP number for iterators, and last night Guido checked an initial implementation into the 2.2 CVS tree. In Python terms, "for" setup looks for an __iter__ method first, and if it doesn't find it but does find __getitem__, builds a lightweight iterator around the __getitem__ method instead. So the "for" loop works only with iterators now, but there's an adapter providing iterators by magic for old sequence objects that don't know about iterators: C:\Code\python\dist\src\PCbuild>python Python 2.2a0 (#16, Apr 20 2001, 23:16:12) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. >>> def f(s): ... for i in s: ... print i ... >>> from dis import dis >>> dis(f) 0 SET_LINENO 1 3 SET_LINENO 2 6 SETUP_LOOP 25 (to 34) 9 LOAD_FAST 0 (s) 12 GET_ITER >> 13 SET_LINENO 2 16 FOR_ITER 14 19 STORE_FAST 1 (i) 22 SET_LINENO 3 25 LOAD_FAST 1 (i) 28 PRINT_ITEM 29 PRINT_NEWLINE 30 JUMP_ABSOLUTE 13 33 POP_BLOCK >> 34 LOAD_CONST 0 (None) 37 RETURN_VALUE >>> The backward compatibility layer described above is hiding in the new GET_ITER opcode. Of course builtin lists (and so on) define the iterator slot directly now, so GET_ITER simply returns their iterator directly. Loops are less complicated (internally) now, and run significantly faster. User-defined types and classes no longer *need* to (ab)use __getitem__ to implement iteration (which is of particular interest to Greg Wilson right now, who is implementing a Set class and doesn't *want* to define __getitem__ because it's semantically senseless). None of that should be controversial in the least. More controversial is that iteration over dict keys has been tentatively added (and note that this is another thing made *possible* by breaking the old connection between __getitem__ and iteration): >>> dict = {"one": 1, "two": 2} >>> for k in dict: ... print k ... one two >>> This is significantly faster, and unboundedly more memory-efficient, than doing "for k in dict.keys()". The dict.__contains__ slot was also filled in, so that "k in dict" is synonymous with "dict.has_key(k)", but again runs significantly faster: >>> "one" in dict 1 >>> "three" in dict 0 >>> File objects have also grown iterators, so that, e.g., for line in sys.stdin: print line now works. Iterators can be explicitly materialized too, via the new iter() builltin function, and invoked apart from the "for" protocol: >>> i1 = iter(dict) >>> i1 <dictionary-iterator object at 0079A250> >>> dir(i1) ['next'] >>> print i1.next.__doc__ it.next() -- get the next value, or raise StopIteration >>> i2 = iter(dict) >>> i1.next() 'one' >>> i2.next() 'one' >>> i1.next() 'two' >>> i2.next() 'two' >>> i1.next() Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> Note that this allows a simple memory-efficient implementation of parallel sequence iteration too. For example, this program: class zipiter: def __init__(self, seq1, *moreseqs): seqs = [seq1] seqs.extend(list(moreseqs)) self.seqs = seqs def __iter__(self): self.iters = [iter(seq) for seq in self.seqs] return self def next(self): return [i.next() for i in self.iters] for i, j, k in zipiter([1, 2, 3], "abc", (5., 6., 7., 8.)): print i, j, k prints 1 a 5.0 2 b 6.0 3 c 7.0 Now all that is just iteration in a thoroughly conventional sense. There is no support here for generators or coroutines or microthreads, except in the sense that breaking the iteration==__getitem__ connection makes it easier to think about *how* generators may be implemented, and having an explicit iterator object "should" make it possible to go beyond Icon's notion of generators (which can only be driven implicitly by control context). Neil Schemenauer is currently thinking hard about that "in his spare time", but there's no guarantee anything will come of it in 2.2. Iterators are a sure thing, though (not least because they're already implemented!). not-only-implemented-but-feel-exactly-right-ly y'rs - tim [View Less]

1 0

Modifying the PyUnicode_FromUnicode result
by Martin v. Loewis April 21, 2001

April 21, 2001

Currently, a number of routines assume that the result of PyUnicode_FromUnicode can be modified, i.e. they mutate the resulting unicode object. Look at unicodeobject.c:fixup for an example, and assume that fixfct is fixtitle (*). This is different from PyString_FromStringAndSize, whose result can be only modified if the str argument is NULL. These routines broke after I applied my caching patch, since now PyUnicode_FromUnicode may return an existing string. Is that difference intentional? My … [View More]

2 5

Re: Cygwin Python Distribution (was ANNOUNCE: Python 2.1 final release)
by Jason Tishler April 21, 2001

April 21, 2001

On Fri, Apr 20, 2001 at 07:37:01PM -0500, Clark C. Evans wrote: > On Tue, 17 Apr 2001, Jason Tishler wrote: > > I have contributed Python to the standard Cygwin distribution. Cygwin > > Python is located in the contrib/python directory on the Cygwin mirrors. > > Cygwin's setup.exe will automatically install Cygwin Python the next > > time one installs or updates from a mirror. > > This is interesting. From what I understand, if you link > against cygwin.… [View More]

2 2

re: string slicing and method consistency
by Greg Wilson April 20, 2001

April 20, 2001

> > Greg Wilson: > > if "abbc"[-1] is "c", and if > > "abbc".replace("b", "x", 1) is "axbc", then shouldn't > > "abbc".replace("b", "x", -1) be "abxc" (i.e. negative > > numbers replace the *last* occurrences of the value)? > > Same argument for "split", etc. > Guido van Rossum: > Dubious hypergeneralization. Greg Wilson: Do you have an editor macro set up yet to generate that phrase? :-) > Guido van Rossum: > The thing is that this parameter, … [View More]

2 1

Re: Class Methods
by Rainer Deyke April 20, 2001

April 20, 2001

"Thomas Heller" <thomas.heller(a)ion-tof.com> wrote in message news:mailman.987779255.16686.python-list@python.org... > There are some signs :-) that Python's object > model is going to be revised even _before_ > Python 3000. > > Is someone willing to join me fighting > for class methods (I mean 'real' class-methods > in the Smalltalk style here, _not_ static > methods like in Java or C++). I posted this in another thread, but I think it bears repeating here. I … [View More]

4 3

pydoc script
by Paul F. Dubois April 20, 2001

April 20, 2001

Ka-Ping, Your pydoc script can fail because the pydoc executed is not the pydoc (and therefore the python) in the users' current path if they start it explicitly with a full path. I suggest a similar trick to this one: #!/bin/csh -f unsetenv PYTHONHOME set bindir = `dirname $0` set path=(${bindir} $path);rehash #in case of respawns, get our python exec ${bindir}/python -O -c 'import browser;browser.tk_cdat.main()'

1 0

Shall I start adding iterators to Python 2.2?
by Guido van Rossum April 20, 2001

April 20, 2001

I've got a fairly complete implementation of iterators along the lines of Ping's PEP (slightly updated). This is available for inspection through CVS: just check out the CVS branch of python/dist/src named "iter-branch" from SourceForge: cvs checkout -r iter-branch -d <directory> python/src/dist (This branch was forked off the trunk around the time of version 2.1b1, so it's not up to date with Python 2.1, but it's good enough to show off iterators.) My question is: should I just … [View More]merge this code onto the trunk (making it part of 2.2), or should we review the design more before committing to this implementation? Brief overview of what I've got implemented: - There is a new built-in operation, spelled as iter(obj) in Python and as PyObject_GetIter(obj) in C; it calls the tp_iter slot in obj's type. This returns an iterator, which can be any object that implements the iterator protocol. The iterator protocol defines one method, next(), which returns the next value or raises the new StopIteration exception. - For backwards compatibility, if obj's type does not have a valid tp_iter slot, iter(obj) and PyObject_GetIter(obj) create a sequence iterator that iterates over a sequence. - "for x in S: B" is implemented roughly as __iter = iter(S) while 1: try: x = __iter.next() except StopIteration: break B (except that the semantics of break when there's an else clause are not different from what this Python code would do). - The test "key in dict" is implemented as "dict.has_key(key)". (This was done by implementing the sq_contains slot. - iter(dict) returns an iterator that iterates over the keys of dict without creating a list of keys first. This means that "for key in dict" has the same effect as "for key in dict.keys()" as long as the loop body doesn't modify the dictionary (assignment to existing keys is okay). - There's an operation to create an iterator from a function and a sentinel value. This is spelled as iter(function, sentinel). For example, for line in iter(sys.stdin.readline, ""): ... is an efficient loop over the lines of stdin. - But even cooler is this, which is totally equivalent: for line in sys.stdin: ... - Not yet implemented, but part of the plan, is to use iterators for all other implicit loops, like map/reduce/filter, min/max, and the "in" test for sequences that don't define sq_contains. --Guido van Rossum (home page: http://www.python.org/~guido/) [View Less]

6 8

configuration "feature"
by Greg Wilson April 20, 2001

April 20, 2001

I just checked out a fresh copy of Python from Sourceforge on a Solaris 5.8 machine, and typed: ./configure -with-cxx rather than ./configure -with-cxx=g++ It generates a makefile with "CXX=yes", so "make" produces: bash-2.03$ make yes -c -g -O2 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H \ -o Modules/ccpython.o ./Modules/ccpython.cc make: yes: Command not found :-) Greg

2 1

Re: [Python-Dev] configuration "feature"
by Michael Hudson April 20, 2001

April 20, 2001

Greg Wilson <Greg.Wilson(a)baltimore.com> writes: > I just checked out a fresh copy of Python from Sourceforge > on a Solaris 5.8 machine, and typed: > > ./configure -with-cxx > > rather than > > ./configure -with-cxx=g++ > > It generates a makefile with "CXX=yes", so "make" produces: > > bash-2.03$ make > yes -c -g -O2 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H > \ > -o Modules/ccpython.o ./Modules/… [View More]

1 0

off-topic, sorry ;)
by Thomas Wouters April 20, 2001

April 20, 2001

My batch of PythonLabs T-Shirts arrived yesterday (thanx, by the way, Melissa!) but there was something twilight-zonish about the box that I just had to share ;) My colleague Scott took delivery of the box, and knew without looking at the sender or description that it was something Python related. It had this sticker on it: http://www.xs4all.nl/~thomas/SPAM.jpg -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

1 0