Mailman 3 March 2005 - Python-Dev

RE: [Python-Dev] Faster Set.discard() method?
by Tony Meyer 18 Mar '05

18 Mar '05

> To avoid the exception in the discard method, it could be > implemented as: > > def discard(self, element): > """Remove an element from a set if it is a member. > > If the element is not a member, do nothing. > """ > try: > self._data.pop(element, None) > except TypeError: > transform = getattr(element, > "__as_temporarily_immutable__", None) > if transform is None: > raise # re-raise the TypeError exception we caught > del self._data[transform()] [...] > But the dict.pop method is about 12 times faster. Is this worth doing? The 2.4 builtin set's discard function looks like it does roughly the same as the 2.3 sets.Set. Have you tried comparing a C version of your version with the 2.4 set to see if there are speedups there, too? IMO keeping the sets.Set version as clean and readable as possible is nice, since the reason this exists is for other implementations (Jython, PyPy, ...) and documentation, right? OTOH, speeding up the CPython implementation is nice and it's read by many fewer people. =Tony.Meyer

2 1

Faster Set.discard() method?
by Andrew McNamara 18 Mar '05

18 Mar '05

To avoid the exception in the discard method, it could be implemented as: def discard(self, element): """Remove an element from a set if it is a member. If the element is not a member, do nothing. """ try: self._data.pop(element, None) except TypeError: transform = getattr(element, "__as_temporarily_immutable__", None) if transform is None: raise # re-raise the TypeError exception we caught del self._data[transform()] Currently, it's implemented as the much clearer: try: self.remove(element) except KeyError: pass But the dict.pop method is about 12 times faster. Is this worth doing? -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 0

code blocks using 'for' loops and generators
by Brian Sabbey 18 Mar '05

18 Mar '05

I would like to get some feedback on a proposal to introduce smalltalk/ruby-like "code blocks" to python. Code blocks are, among other things, a clean way to solve the "acquire/release" problem [1][2]. This proposal also touches on some of the problems PEP 288 [3] deals with. The best discussion I have been able to find on this topic is in the thread of ref. [4]. Since generators, when used with 'for' loops, are so similar to code blocks [5], one can imagine two ways to implement code blocks in python: (1, parallel) keep them as similar as possible to 'for' loops so that a normal user doesn't distinguish the two, or (2, orthogonal) make them so much different than 'for' loops that the normal user doesn't notice the similarities. Anything between these extremes will probably be confusing. Here I will describe an attempt at (1). (I) Give generators a __call__ method as an alternative to 'next'. Method __call__ should take a single parameter: a function. Using __call__ will cause the generator to start executing normally, but when a 'yield' is reached, the generator will invoke the function passed to __call__ instead of activating the currently implemented 'yield' mechanism. Since __call__ will be an alternative to 'next', it will raise an exception if 'next' has already been called and vice-versa. Also, any calls to 'next' will raise an exception if there is a 'yield' in the try of a try/finally. Such yields will no longer trigger a SyntaxError because they will not a problem when using __call__. (II) Have a method of generators, __blockcall__, which will be equal to __call__, but will only exist if (1) the generator contains a "try/finally try" yield, or (2) the user explicitly defines it, for example, with a function decorator (@completion_required would be a descriptive name). Have 'for' loops use __blockcall__ if it is available, and __iter__ otherwise. Pass to __blockcall__ the block of code in the 'for' loop. Scope rules for the passed block of code should mimic current 'for' loop behavior. Behavior of 'break' and 'return' should be mimicked, perhaps with special exceptions catchable only by the 'for' loop. Mimicking 'yield' will be a problem unless/until multi-level yields are allowed. (performance and implementation difficulties for all of this? I don't know). The thunk shouldn't be savable for later use because the 'for' loop will no longer be around to deal with 'break' and 'return'. This means that __blockcall__ will not be implementable as a function that takes a function as an argument. (III) Allow 'continue' to pass values to 'yield' (something similar previously rejected here [6]). As far as I know, all other control statements that transfer execution to a different frame (yield, return, raise) pass values, and I don't see why this case should be any different. I do not see such a mechanism as gimmicky; being able to cleanly pass values when changing scope is an inherent part of nearly every programming language. As an example of the syntax I am suggesting, here is something I was desiring recently, a generator to open, unpickle, repickle and close a file: def pickled_file(name): f = open(name, 'r') l yield pickle.load(f) f.close() f = open(name, 'w') pickle.dump(l, f) f.close() The unpickled object is sent to the caller at the yield statement, and the modified object is received back at the same statement. Note the suggested 'yield' syntax and the conspicuous absence of '='. This syntax is backwardly compatible with current yield syntax. Also, this syntax does not require yield to appear as a function; it is still clear that this is a unique control-flow statement. This function would be used like this: for l in pickled_file('greetings.pickle'): l.append('hello') l.append('howdy') continue l The above code would have the same effect as: def named(l): l.append('hello') l.append('howdy') return l pickled_file('greetings.pickle')(named) (IV) Allow 'yield' to return no value; in this case a new keyword, 'with', will be required instead of an awkward 'for': "with f():" instead of "for in f():" (V) For the same reasons as in (III), allow generators to return values. These values can be sent with the StopIteration exception if 'next' is being used for iteration. An obvious syntax for receiving these values is shown by this example: with dt = stopwatch(): f() g() print 'it took', dt, 'seconds' Although "with stopwatch() result dt:" might not be so bad. [1] PEP 310 Reliable Acquisition/Release Pairs http://www.python.org/peps/pep-0310.html [2] PEP 325 Resource-Release Support for Generators http://www.python.org/peps/pep-0325.html [3] PEP 288 Generators Attributes and Exceptions http://www.python.org/peps/pep-0288.html [4] http://mail.python.org/pipermail/python-dev/2003-February/032800.html [5] http://mail.python.org/pipermail/python-dev/2003-February/032826.html [6] http://mail.python.org/pipermail/python-dev/2002-March/021923.html -Brian

7 31

itertools.walk()
by Raymond Hettinger 18 Mar '05

18 Mar '05

Some folks on comp.lang.python have been pushing for itertools to include a flatten() operation. Unless you guys have some thoughts on the subject, I'm inclined to accept the request. Rather than calling it flatten(), it would be called "walk" and provide a generalized capability to descend through nested iterables (similar to what os.walk does for directories). The one wrinkle is having a stoplist argument to specify types that should be considered atomic eventhough they might be iterable (strings for example). Raymond Hettinger

3 4

Rationale for sum()'s design?
by Guido van Rossum 17 Mar '05

17 Mar '05

There are a few design choices we could have made for sum(); in particular, for non-empty sequences we could not have used the identity element (the optional second argument). As it is, we get unjustified but understandable complaints that sum() "only supports numbers". An alternative design could have returned: - the identity (defaulting to 0) if the sequence is empty - the first and only element if the sequence only has one element - (...(((A + B) + C) + D) + ...) if the sequence has more than one element This has surprises too (in particular of returning 0 when invoked without overriding the identity argument for a seqence of addable non-numbers) but works without using the second argument for a larger set of inputs I believe it is often used in such a way that the input is known to be non-empty). I'd be happy to be pointed to a past discussion where this was considered and rejected with a good reason; then I can post that to the blog (where the deficiency in sum() is being berated a bit excessively). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

13 35

Problems with definition of _POSIX_C_SOURCE
by Jack Jansen 17 Mar '05

17 Mar '05

On a platform I won't mention here I'm running into problems compiling Python, because of it defining _POSIX_C_SOURCE. It turns out that on this platform the definition causes all sorts of declarations in sys/types.h to be skipped (presumably because they're not official Posix identifiers), which in turn causes platform-specific headers to fail. The comment in pyconfig.h suggests that defining _POSIX_C_SOURCE may enable certain features, but the actual system headers appear to work the other way around: it seems that defining this will disable features that are not strict Posix. Does anyone know what the real meaning of this define is? Because if it is the former then Python is right, but if it is the latter Python really has no business defining it: in general Python isn't 100% posix-compliant because it'll use all sorts of platform-dependent (and, thus, potentially non-posix-compliant) code... This problem is currently stopping Python 2.4.1 to compile on this platform, so if anyone can provide any insight that would be very helpful... -- Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

3 2

Rationale for sum()'s design?
by Michael Walter 17 Mar '05

17 Mar '05

On Thu, 17 Mar 2005 14:34:23 +1300, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Not to mention that if the seq is empty, there's no > way of knowing what T to instantiate... You just use the same T as inferred for seq : sequence[T] <wink>. Michael

1 0

RE: code blocks using 'for' loops and generators
by Jim Jewett 17 Mar '05

17 Mar '05

It may be time to PEP (or re-PEP), if only to clarify what people are actually asking for. Brian Sabbey's example from message http://mail.python.org/pipermail/python-dev/2005-March/052202.html *seems* reasonably clear, but I don't see how it relates in any way to "for" loops or generators, except as one (but not the only) use case. def add(thunk1, thunk2, other): print thunk1(1,2) + thunk2(3,4) + other with x,y from add(100): value x*y also a,b: # yikes?? value a*b # this is thunk2 To make my own understanding explicit: (1) Calls for "Ruby blocks" or "thunks" are basically calls for placeholders in a function. These placeholders will be filled with code from someplace else, but will execute in the function's own local namespace. (2) A function as a parameter isn't good enough, because the passed-in function can't see bindings in the surrounding larger function. (It still sees the lexical scope it which it was defined.) (3) A function as a parameter isn't good enough, because the passed-in function can't modify bindings in the surrounding larger function. (4) A thunk could be used today be creating a string (rather than a pre-compiled function) and substituting in the thunk's string (rather than the code to which it would compile) before execution, though this would be ugly. (5) A thunk *might* be do-able with exec or eval if a function's locals were still a dictionary. (6) This has nothing whatsoever to do with for loops or generators, except as a common use case. In particular, thunks may be used to lock/unlock or unwind-protect some resource. Generators are not the only place this is needed, but they have made the "what do you mean, __del__ might never get called?" problem even worse, and Ruby often solves these with a Ruby Block. (7) A __leave__ or __exit__ special method really turns into another name for __del__. It might still be useful if it came with semantics of "I explicitly don't care what order cycles are broken; call me as soon as my object is garbage and who cares if it gets resurrected." There have been recipes (from Tim?) for emulating this by using a proxy to the resource, so that no __del__ cycles can form. -jJ

4 3

RE: [Python-Dev] rationale for the no-new-features approach
by Tony Meyer 17 Mar '05

17 Mar '05

[Bob Ippolito] >>>> try: >>>> set >>>> except NameError: >>>> from sets import Set as set >>>> >>>> You don't need the rest. [Skip Montanaro] >>> Sure, but then pychecker bitches about a statement that appears to >>> have no effect. ;-) [Bob Ippolito] >> Well then fix PyChecker to look for this pattern :) +1. [Gregory P. Smith] > or make it even uglier to hide from pychecker by writing that as: > > exec(""" > try: > set > except NameError: > from sets import Set as set > """) I presume that was somewhat tongue-in-cheek, but if it wasn't, please reconsider. Modulefinder isn't able to realise that set (or sets.Set) is needed with the latter (a problem of this very nature was just fixed with bsddb), which causes trouble for people later on. =Tony.Meyer

2 1

rationale for the no-new-features approach
by Anthony Baxter 17 Mar '05

17 Mar '05

So it's only fair that I write down my rationale for why I'm being anal about the no-new-features approach. Comments are more than welcome - ideally, after discussion, I'll put some more words in the bugfix PEP. Goal 1: When we cut a bugfix release, people will upgrade to it. - This helps the users (they have bugs fixed) - This helps us (python-dev) because people won't be logging bugs against already-fixed-bugs. - This makes us (Python) look good, because people using Python have the most bug-free experience possible. Goal 2: Packagers of Python will package up releases of Python that are as close to the "official" release as possible. - This, to me, is a huge win. If we don't have to worry about whether someone is running 2.4.1, or DistroFoo's version of 2.4.1 with a couple of backported fixes from 2.4.2, or some other horrible frankenstein's monster of a release, it makes our lives much easier. (This is also why I've been on a bit of a stomping exercise to attempt to get distros that broke Python into pieces to stop doing this (notably, Debian/Ubuntu's distutils- in-python-devel pain)) - And yes, we're always going to have the problem of some distros stuck on old Python releases (Redhat -> Python 1.5.2, Debian stable -> Python 2.1, &c) but we can hopefully encourage them to at least roll out the latest bugfix version of whatever version they're stuck on. Goal 3: Good PR. - I've read a disturbing amount of words about other languages (*cough*Java*cough*) that have no sanity about minor and major releases. This seems to piss people off a great deal. One of Python's selling points is that we're very cautious and a suitable choice for grownups to use in business. - This is good for us (Python community) because it makes it less likely we'll be stuck in a job where we're forced to code Perl, or C++, or Java, or some other horrible fate. It also boosts the size of the community, meaning that there will be more volunteers to work on Python (hurrah!) Goal 4: Try and prevent something like try: True, False except NameError: True, False = 1, 0 from ever ever happening again. - This, I hope, needs no further explanation <wink> Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

9 14