Mailman 3 April 2001 - Python-Dev

Iterators, generators and 2.2 (was RE: do...until wisdom needed...)
by Tim Peters 21 Apr '01

21 Apr '01

[Aahz] > Generators (called "iterators") are going to be 2.2. They'll be > more powerful that Icon's generators; it's not clear whether they'll > be a full-fledged substitute for coroutines. [Neelakantan Krishnaswami] > {\mr_burns Excellent.} > > Is this the iterator idea that Ping posted about a couple of months > back? What is the PEP number for this? I'm curious how the existing > iteration protocol will interact with the new iterators. This is getting confused. Iterators != generators (sorry, Aahz! it's more involved than that). Aahz gave you the PEP number for iterators, and last night Guido checked an initial implementation into the 2.2 CVS tree. In Python terms, "for" setup looks for an __iter__ method first, and if it doesn't find it but does find __getitem__, builds a lightweight iterator around the __getitem__ method instead. So the "for" loop works only with iterators now, but there's an adapter providing iterators by magic for old sequence objects that don't know about iterators: C:\Code\python\dist\src\PCbuild>python Python 2.2a0 (#16, Apr 20 2001, 23:16:12) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. >>> def f(s): ... for i in s: ... print i ... >>> from dis import dis >>> dis(f) 0 SET_LINENO 1 3 SET_LINENO 2 6 SETUP_LOOP 25 (to 34) 9 LOAD_FAST 0 (s) 12 GET_ITER >> 13 SET_LINENO 2 16 FOR_ITER 14 19 STORE_FAST 1 (i) 22 SET_LINENO 3 25 LOAD_FAST 1 (i) 28 PRINT_ITEM 29 PRINT_NEWLINE 30 JUMP_ABSOLUTE 13 33 POP_BLOCK >> 34 LOAD_CONST 0 (None) 37 RETURN_VALUE >>> The backward compatibility layer described above is hiding in the new GET_ITER opcode. Of course builtin lists (and so on) define the iterator slot directly now, so GET_ITER simply returns their iterator directly. Loops are less complicated (internally) now, and run significantly faster. User-defined types and classes no longer *need* to (ab)use __getitem__ to implement iteration (which is of particular interest to Greg Wilson right now, who is implementing a Set class and doesn't *want* to define __getitem__ because it's semantically senseless). None of that should be controversial in the least. More controversial is that iteration over dict keys has been tentatively added (and note that this is another thing made *possible* by breaking the old connection between __getitem__ and iteration): >>> dict = {"one": 1, "two": 2} >>> for k in dict: ... print k ... one two >>> This is significantly faster, and unboundedly more memory-efficient, than doing "for k in dict.keys()". The dict.__contains__ slot was also filled in, so that "k in dict" is synonymous with "dict.has_key(k)", but again runs significantly faster: >>> "one" in dict 1 >>> "three" in dict 0 >>> File objects have also grown iterators, so that, e.g., for line in sys.stdin: print line now works. Iterators can be explicitly materialized too, via the new iter() builltin function, and invoked apart from the "for" protocol: >>> i1 = iter(dict) >>> i1 <dictionary-iterator object at 0079A250> >>> dir(i1) ['next'] >>> print i1.next.__doc__ it.next() -- get the next value, or raise StopIteration >>> i2 = iter(dict) >>> i1.next() 'one' >>> i2.next() 'one' >>> i1.next() 'two' >>> i2.next() 'two' >>> i1.next() Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> Note that this allows a simple memory-efficient implementation of parallel sequence iteration too. For example, this program: class zipiter: def __init__(self, seq1, *moreseqs): seqs = [seq1] seqs.extend(list(moreseqs)) self.seqs = seqs def __iter__(self): self.iters = [iter(seq) for seq in self.seqs] return self def next(self): return [i.next() for i in self.iters] for i, j, k in zipiter([1, 2, 3], "abc", (5., 6., 7., 8.)): print i, j, k prints 1 a 5.0 2 b 6.0 3 c 7.0 Now all that is just iteration in a thoroughly conventional sense. There is no support here for generators or coroutines or microthreads, except in the sense that breaking the iteration==__getitem__ connection makes it easier to think about *how* generators may be implemented, and having an explicit iterator object "should" make it possible to go beyond Icon's notion of generators (which can only be driven implicitly by control context). Neil Schemenauer is currently thinking hard about that "in his spare time", but there's no guarantee anything will come of it in 2.2. Iterators are a sure thing, though (not least because they're already implemented!). not-only-implemented-but-feel-exactly-right-ly y'rs - tim

1 0

Modifying the PyUnicode_FromUnicode result
by Martin v. Loewis 21 Apr '01

21 Apr '01

Currently, a number of routines assume that the result of PyUnicode_FromUnicode can be modified, i.e. they mutate the resulting unicode object. Look at unicodeobject.c:fixup for an example, and assume that fixfct is fixtitle (*). This is different from PyString_FromStringAndSize, whose result can be only modified if the str argument is NULL. These routines broke after I applied my caching patch, since now PyUnicode_FromUnicode may return an existing string. Is that difference intentional? My feeling is that it is an error to modify a unicode object, unless it is known not to be initialized. Regards, Martin P.S. This was actually the first failure case when running test_unicodedata under my patch.

2 5

Re: Cygwin Python Distribution (was ANNOUNCE: Python 2.1 final release)
by Jason Tishler 21 Apr '01

21 Apr '01

On Fri, Apr 20, 2001 at 07:37:01PM -0500, Clark C. Evans wrote: > On Tue, 17 Apr 2001, Jason Tishler wrote: > > I have contributed Python to the standard Cygwin distribution. Cygwin > > Python is located in the contrib/python directory on the Cygwin mirrors. > > Cygwin's setup.exe will automatically install Cygwin Python the next > > time one installs or updates from a mirror. > > This is interesting. From what I understand, if you link > against cygwin.dll, the software must be released under > the GPL. Thus, is the licensing debate over? Do you > have the right to re-license python under the GPL? Or am > I missing something fundmental here? > > $ objdump -p python2.1.exe | grep DLL > vma: Hint Time Forward DLL First > DLL Name: KERNEL32.dll > DLL Name: cygwin1.dll > DLL Name: libpython2.1.dll > > Sorry to bring this up... I'm just curious. Clark brings up a valid point. Did I screw up from a licensing point of view? I found the following on the Python web site: However, we expect that Python 2.0 and later versions, released by BeOpen PythonLabs, will be released under a GPL-compatible license. IANAL, any guidance regarding this matter would be greatly appreciated. Thanks, Jason -- Jason Tishler Director, Software Engineering Phone: +1 (732) 264-8770 x235 Dot Hill Systems Corp. Fax: +1 (732) 264-8798 82 Bethany Road, Suite 7 Email: Jason.Tishler(a)dothill.com Hazlet, NJ 07730 USA WWW: http://www.dothill.com

2 2

re: string slicing and method consistency
by Greg Wilson 20 Apr '01

20 Apr '01

> > Greg Wilson: > > if "abbc"[-1] is "c", and if > > "abbc".replace("b", "x", 1) is "axbc", then shouldn't > > "abbc".replace("b", "x", -1) be "abxc" (i.e. negative > > numbers replace the *last* occurrences of the value)? > > Same argument for "split", etc. > Guido van Rossum: > Dubious hypergeneralization. Greg Wilson: Do you have an editor macro set up yet to generate that phrase? :-) > Guido van Rossum: > The thing is that this parameter, > called maxsplit, is not really an index -- it's a count. Greg Wilson: Understood; I'm asking whether changing its name and interpretation (in a way that doesn't break any existing code) would be worthwhile: >>> path = "/some/long/path/to/file.html" >>> main, parent, file = path.split("/", -2) >>> main "/some/long/path" >>> parent "to" >>> file "file.html" > > Greg Wilson: > > Turns out that "abbc".replace("b", "x", -1) is "axxc" > > (i.e. negative arguments are ignored). I would have > > expected this to raise a ValueError, if anything. Is > > there a reason for this behavior? Greg Wilson again: Question still stands --- if these are counts, then shouldn't negative values raise exceptions? Thanks, Greg

2 1

Re: Class Methods
by Rainer Deyke 20 Apr '01

20 Apr '01

"Thomas Heller" <thomas.heller(a)ion-tof.com> wrote in message news:mailman.987779255.16686.python-list@python.org... > There are some signs :-) that Python's object > model is going to be revised even _before_ > Python 3000. > > Is someone willing to join me fighting > for class methods (I mean 'real' class-methods > in the Smalltalk style here, _not_ static > methods like in Java or C++). I posted this in another thread, but I think it bears repeating here. I consider classes/instances a special case of namespaces: one which allows (multiple) instantiation and inheritance. In actual pratice not all of the classes I design are designed for multiple instantiation, or instantiation at all for that matter. What I would like to see is a separation of concerns ("Does this namespace have __special__ attributes for operator overloading?" inpdependent from "Does this namespace require instantiation?") followed by a culling of irrelevant features ("Don't want operator overloading? Don't name your function '__add__' then."). The resulting programming language might look something like this: namespace A: # Create a named unique object pass namespace B(A): # Similar to 'from ... import', but done through linking # instead of copying class C(A): # 'class' = namespace that requires/supports instantiation # class inherits from namespace => functions in namespace # are treated as "static" functions pass namespace D(C()): # namespace inherits from instance of class pass The module itself would be a 'namespace' object. Overall, I think this has the potential of creating a much simpler and more regular language. Separate keywords for 'class' and 'namespace' might even turn out to be unnecessary. In this context, class methods would either be automatically included, or turn out to be truly redundant, or both. -- Rainer Deyke (root(a)rainerdeyke.com) Shareware computer games - http://rainerdeyke.com "In ihren Reihen zu stehen heisst unter Feinden zu kaempfen" - Abigor

4 3

pydoc script
by Paul F. Dubois 20 Apr '01

20 Apr '01

Ka-Ping, Your pydoc script can fail because the pydoc executed is not the pydoc (and therefore the python) in the users' current path if they start it explicitly with a full path. I suggest a similar trick to this one: #!/bin/csh -f unsetenv PYTHONHOME set bindir = `dirname $0` set path=(${bindir} $path);rehash #in case of respawns, get our python exec ${bindir}/python -O -c 'import browser;browser.tk_cdat.main()'

1 0

Shall I start adding iterators to Python 2.2?
by Guido van Rossum 20 Apr '01

20 Apr '01

I've got a fairly complete implementation of iterators along the lines of Ping's PEP (slightly updated). This is available for inspection through CVS: just check out the CVS branch of python/dist/src named "iter-branch" from SourceForge: cvs checkout -r iter-branch -d <directory> python/src/dist (This branch was forked off the trunk around the time of version 2.1b1, so it's not up to date with Python 2.1, but it's good enough to show off iterators.) My question is: should I just merge this code onto the trunk (making it part of 2.2), or should we review the design more before committing to this implementation? Brief overview of what I've got implemented: - There is a new built-in operation, spelled as iter(obj) in Python and as PyObject_GetIter(obj) in C; it calls the tp_iter slot in obj's type. This returns an iterator, which can be any object that implements the iterator protocol. The iterator protocol defines one method, next(), which returns the next value or raises the new StopIteration exception. - For backwards compatibility, if obj's type does not have a valid tp_iter slot, iter(obj) and PyObject_GetIter(obj) create a sequence iterator that iterates over a sequence. - "for x in S: B" is implemented roughly as __iter = iter(S) while 1: try: x = __iter.next() except StopIteration: break B (except that the semantics of break when there's an else clause are not different from what this Python code would do). - The test "key in dict" is implemented as "dict.has_key(key)". (This was done by implementing the sq_contains slot. - iter(dict) returns an iterator that iterates over the keys of dict without creating a list of keys first. This means that "for key in dict" has the same effect as "for key in dict.keys()" as long as the loop body doesn't modify the dictionary (assignment to existing keys is okay). - There's an operation to create an iterator from a function and a sentinel value. This is spelled as iter(function, sentinel). For example, for line in iter(sys.stdin.readline, ""): ... is an efficient loop over the lines of stdin. - But even cooler is this, which is totally equivalent: for line in sys.stdin: ... - Not yet implemented, but part of the plan, is to use iterators for all other implicit loops, like map/reduce/filter, min/max, and the "in" test for sequences that don't define sq_contains. --Guido van Rossum (home page: http://www.python.org/~guido/)

6 8

configuration "feature"
by Greg Wilson 20 Apr '01

20 Apr '01

I just checked out a fresh copy of Python from Sourceforge on a Solaris 5.8 machine, and typed: ./configure -with-cxx rather than ./configure -with-cxx=g++ It generates a makefile with "CXX=yes", so "make" produces: bash-2.03$ make yes -c -g -O2 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H \ -o Modules/ccpython.o ./Modules/ccpython.cc make: yes: Command not found :-) Greg

2 1

Re: [Python-Dev] configuration "feature"
by Michael Hudson 20 Apr '01

20 Apr '01

Greg Wilson <Greg.Wilson(a)baltimore.com> writes: > I just checked out a fresh copy of Python from Sourceforge > on a Solaris 5.8 machine, and typed: > > ./configure -with-cxx > > rather than > > ./configure -with-cxx=g++ > > It generates a makefile with "CXX=yes", so "make" produces: > > bash-2.03$ make > yes -c -g -O2 -Wall -Wstrict-prototypes -I. -I./Include -DHAVE_CONFIG_H > \ > -o Modules/ccpython.o ./Modules/ccpython.cc > make: yes: Command not found > > :-) Teehee. Try it on a linux box though; there is a /usr/bin/yes, and it doesn't do anything too helpful... Cheers, M. -- [Perl] combines all the worst aspects of C and Lisp: a billion different sublanguages in one monolithic executable. It combines the power of C with the readability of PostScript. -- Jamie Zawinski

1 0

off-topic, sorry ;)
by Thomas Wouters 20 Apr '01

20 Apr '01

My batch of PythonLabs T-Shirts arrived yesterday (thanx, by the way, Melissa!) but there was something twilight-zonish about the box that I just had to share ;) My colleague Scott took delivery of the box, and knew without looking at the sender or description that it was something Python related. It had this sticker on it: http://www.xs4all.nl/~thomas/SPAM.jpg -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

1 0