Python-ideas August 2009

python-ideas@python.org

56 participants
20 discussions

Heap data type
by Facundo Batista 07 Dec '09

07 Dec '09

Using the module heapq, I found that it's not easy to use. Or, at least, that it could be straightforward to use. My main issue is that for correct usage: - user should provide an external list, that shouldn't use for anything else to don't break the invariant - to alter the heap queue, a bunch of functions must be used, passing always the external list I think that changing "external list" for "internal attribute", and "bunch of functions " for "methods", it will leave the module easier to use, safer, and more object oriented. So, I basically coded it. What do you think about including this class in the heap module? """ class Heap(object): '''Maintains a heapified list, always conserving the invariant. Heaps are arrays for which heap[k] <= heap[2*k+1] and heap[k] <= heap[2*k+2] for all k, counting elements from zero. ''' def __init__(self, iterable=None): '''Initializes the heap from any iterable. >>> Heap([1, 2]) Heap([1, 2]) >>> Heap([]) Heap([]) >>> Heap() Heap([]) >>> Heap((1, 2, 3)) Heap([1, 2, 3]) >>> Heap(x**2 for x in range(3)) Heap([0, 1, 4]) ''' if iterable is None: self._queue = [] else: self._queue = list(iterable) heapq.heapify(self._queue) def __repr__(self): return "Heap(%s)" % self._queue def push(self, item): '''Push the item to the heap queue. >>> h = Heap() >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.push(2) >>> h.push(1) >>> h.push(0) >>> h Heap([0, 2, 1, 5, 3, 4]) ''' heapq.heappush(self._queue, item) def pop(self): '''Pop one item from the heap queue. >>> h = Heap([0, 2, 1, 5, 3, 4]) >>> h.pop() 0 >>> h.pop() 1 >>> h.pop() 2 >>> h.pop() 3 >>> h.pop() 4 >>> h.pop() 5 >>> h Heap([]) >>> h.pop() Traceback (most recent call last): ... IndexError: index out of range ''' return heapq.heappop(self._queue) def pushpop(self, item): '''Push the item and pop one from the heap queue. Note that this method is more efficient than calling both push() and pop() separately. >>> h = Heap() >>> h.pushpop(7) 7 >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.pushpop(7) 3 >>> h.pushpop(7) 4 >>> h Heap([5, 7, 7]) ''' return heapq.heappushpop(self._queue, item) def replace(self, item): '''Pop one item and push the received one into the heap queue Note that this method is more efficient than calling both pop() and push() separately. >>> h = Heap() >>> h.replace(3) Traceback (most recent call last): ... IndexError: index out of range >>> h.push(7) >>> h.replace(2) 7 >>> h Heap([2]) >>> h.push(3) >>> h Heap([2, 3]) >>> h.replace(1) 2 >>> h.replace(9) 1 >>> h Heap([3, 9]) ''' return heapq.heapreplace(self._queue, item) """ Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

11 14

Python Bytecode Verifier
by Kornél Pál 20 Sep '09

20 Sep '09

Hi, I've created a Python Bytecode Verifier in Python. I'm not a Python guru so I borrowed coding patterns from C/C++. I also created this with C portability in mind. The only reason I used Python was to experiment with Python and was easier to morph code during development. If this program were ported to C it would only need 8 bytes per opcode (some additional storage to track blocks) and a single pass. I haven't found backward jumps to previously unused code in any compiled Python code but it can easily be supported. In that case some more partial passes are required. I also was able to successfully verify all Python files in the Python-2.5.2 source package. The verification algorythm should be quite complete but I may have missed some limitations of the interpreter that could be checked by the verifier as well. The ability to create this verfier proved that although Python bytecode is designed for a dynamically typed interpreter, is still easily verifiable. I am willing port this code C but only in the case if there is any chance to be included in Python. I believe that Python in general would benefit having the ability to safely load .pyc files and create code objects on the fly. Both Java and .NET have the ability to safely load compiled byte code. .NET Framework, just like Python also has the ability to create and execute new code at run-time. You may feel that enabling closed source applications and/or creating a multi-language runtime would hurt Python but both of these have contributed to the success of Java (both the language and the runtime). Kornél

2 2

Add a builtin method to 'int' for base/radix conversion
by Yuvgoog Greenle 17 Sep '09

17 Sep '09

I believe int(s, base) needs an inverse function to allow string representation with different bases. An example use case is 'hashing' a counter like video ID's on youtube, you could use a regular int internally and publish a shorter base-62 id for links. This subject was discussed 2.5 years ago: http://mail.python.org/pipermail/python-dev/2006-January/059789.html I opened a feature request ticket: http://bugs.python.org/issue6783 Some of the questions that remain: 1. Whether this should be a method for int or a regular function in a standard library module like math. 2. What should the method/function be called? (base_convert, radix, etc) What do you guys think? RunThePun

10 20

expose {set,get}res{uid,gid} in posixmodule.c
by travis+ml-python-ideas＠subspacefield.org 27 Aug '09

27 Aug '09

So there has been a little discussion on the python mailing list regarding a request I had to expose {set,get}res{uid,gid}. The general consensus there is that even though it isn't a POSIX standard, it is shared among many Unix OSes and belongs in the same module as setuid and friends, namely os, which maps to posixmodule.c Previous discusion and background as to why this is desirable: http://bugs.python.org/issue6508 If this is the Right Way to do it, I'll brush up on autotools and see about submitting a patch... -- Obama Nation | My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email john(a)subspacefield.org to get blacklisted.

2 1

bitwise operations on bytes
by Eric Eisner 26 Aug '09

26 Aug '09

Hello, As previously mentioned on python-ideas [1] (circa 2006), it would make sense to be able to perform bitwise operations on bytes/bytearray. Stealing the example from the original suggestion: Suppose I have a string (say, read in from a binary file) that has a 4-byte field in it. Bit 11 indicates whether the accompanying data is in glorped form (something I'm just making up for this example). For example, if the field has 0x000480c4, the data is not glorped. If the data is glorped, bit 11 would be on, i.e., 0x001480c4. Let's say I want to turn on the glorp bit; what I have to do now: GLORPED = 0x10 newchar = flags[1] | GLORPED flags = flags[0] + newchar + flags[2:] What I'd like to be able to do is something like: GLORPED = b"\x00\x10\x00\x00" flags |= GLORPED # test if the glorped bit is on any(flags & GLORPED) I have run into this a few times, at least when reading/writing binary formats etc. This approach is more intuitive than the typical/archaic way of converting to an integer, performing a bitwise operation on the integer, converting back to bytes. Arguably, bitwise operations on a high-level integer type don't make sense, as base 2 is an implementation detail. At the very least, bytes and bytearray should be usable with the ~ ^ | & operators Example behavior: >>> ~b'\x55\xff' b'\xaa\x00' >>> b'\xf5\x60' & b'\xa9\x3c' b'\xa1\x20' Unresolved problems: If the two arguments are of different length, either it could either raise a ValueError or mimic the behavior of ints. Xoring an int to a byte seems less than well defined in general, due to endianness ambiguity of the int and size ambiguity. I would think this should not be allowed. Also conceivable is using the shift operators >> and << on bytes, but I personally would use that less often, and the result of such an operation is ambiguous due to endianness. -Eric Eisner [1] http://mail.python.org/pipermail/python-ideas/2006-December/000001.html

15 46

exception based conditional expression, similar to if-else conditional expression
by Jeff McAninch 23 Aug '09

23 Aug '09

I would like to propose an expression, similar to the if-else expression, that responds to exceptions. I had originally posted this (probably mistakenly) on py-dev. This current posting is a cleaned up version of the idea, based on responses I got on from the earlier posting. _*Abstract: *_Proposal for a conditional expression, similar to the if-else expression, that responds to exceptions. _*Motivation: *_An expression syntax that responds to exceptions, and which reproduces the readability and conciseness of the if-else conditional expression, would simplify some exception-handling cases, especially within list comprehensions. _*Very Simple Example - type coercion: *_Current approach: try: x = float(string) except: x = float('nan') Proposed solution using exception-based conditional expression: x = float(string) except ValueError: float('nan') _*Simple Example - type coercion in a list comprehension: *_Current approach: def safe_float(string): try: x = float(string) except ValueError: x = float('nan') return x ... xs = (safe(float(string)) for string in strings) Proposed solution using exception-based conditional expression: xs = ((float(string) except ValueError: float('nan')) for string in strings) _*Discussion: *_In my own python coding, I find I make common use of the if-else conditional expression, especially within list comprehensions. (In one of my packages, which has ~5800 lines of code, I found if-else expressions in ~1% of the lines.) Here is a slightly more involved example than the examples presented above. In data processing, I often string together a sequence of iterable list comprehensions, corresponding to a sequence of operations on a given dataset "ys" to produce a processed dataset "x": xs = (operation_A(x) for x in ys) xs = (operation_B(x) for x in xs if filter_B(x)) xs = (operation_C(x) if (some_condition(x)) else operation_D(x) for x in xs if filter_C(x)) # final, explicit list of values xs = [ x for x in xs ] This is often a flexible way for me to define processing and filtering sequences which also seems to have good performance on very large datasets. One advantage is that I can quickly mix-and-match from existing processes like this to make a new process. An exception-based conditional would go nicely into many of these process sequences, keeping them both robust and flexible. xs = (operation_N(x) except exceptionN: operation_Nprime(x) for x in xs) I also often have object classes which have some common method or attribute. For instance, some of my objects have scope-dependent values: x = y.evaluate(scope)) where scope is typically locals(), globals(), or some other dictionary-like container. But, to keep my code modular, I want to handle, in the same lines of code, objects which do not have some particular method, which leads me to lines of code like: x = y.evaluate(locals()) if ('evaluate' in y.__dict__) else y This seems not very "Pythonic", similar to using type-testing instead of try-except. (My impression was that there was a long-standing trend in the evolution of Python to remove tests like this, and I thought that was the original motivation for the try-except syntax.) I would much rather write: x = y.evaluate(locals()) except AttributeError: y or, in the list comprehension example: xs = (y.evaluate(locals()) except AttributeError: y for y in ys) Clearly this can be handled in several ways with the language as it is. One way is to define a new function, as in the second simple example above: def safe_evaluate(y,scope): try: x = y.evaluate(scope) except AttributeError: x = y return x ... xs = (safe_evaluate(y,locals()) for y in ys) but this quickly (in my packages at least) leads to an annoying proliferation of "safe_" functions. Again, this seems not to be in the "Pythonic" spirit, and is also less concise, less readable. (I also suspect, but have not verified, that this is in general less efficient than in-line expressions -- wasn't that part of the original motivation for list comprehensions?). In the thread of my previous post to py-dev, there were comments, questions, and suggestions concerning the details of the syntax. Having reflected on this for a couple weeks, I am now most strongly supportive of what is essentially just an inline compression of the current try-except syntax. So the following examples would be allowed: x = expression0 except: default_expression x = expression0 except exception1: expression1 except exception2: expression2 except: default_expression Or, more generally: x = expression0\ except exception1: expression1\ except exception2: expression2\ ... except exceptionI: expressionI\ ... except: default_expression In this last example, the behaviour would be as follows: - evaluate expression0. If no exception is encountered, return the result. - if an exception is encountered, search for the matching exception in the except clauses. - if a matching exception ("exceptionI") is found, evaluate the corresponding expression ("expressionI"), and return the result. - if no matching exception is found, and a default except: clause (i.e., one without and exception) is given, evaluate default_expression, and return the result. - if no matching exception is found, and no default except clause if given, pass the exception on to the caller. - if a new exception is encountered while evaluating an an except expression ("expressionI"), pass the exception on to the caller. I hope I have made a convincing case here. This seems to me to be a natural ("Pythonic") addition to the language. Jeff McAninch -- ========================== Jeffrey E. McAninch, PhD Physicist, X-2-IFD Los Alamos National Laboratory Phone: 505-667-0374 Email: mcaninch(a)lanl.gov ==========================

11 17

Interrogate alternate namespace keyword and concept
by Prozacgod 21 Aug '09

21 Aug '09

A while back i had learned about the 'with' statement and at first I naively thought that it worked similarly to a with statement I was familiar with, from delphi/pascal - obviously it doesn't and I was instantly hit with the idea of how useful it would be to have a keyword that could use a namespace from an object, hitting the familiar __getattr__ functions and related. the keyword I proposed would be 'interrogate' (which is rather long, could use another one like 'using' or something) but.. thats not really important to me, the idea is, so class test(): def __init__(self): self.x = 42 foo = test() bar = test() x = 33 bar.x = 22 interrogate foo, bar: print x ----- output is 42 since x is found inside of foo before bar, and locals would be interrogated last, but a more usefull example would be .. class test(): def __getattr__(self, name): if len(name) == 1 and (ord(name) in range(ord('a'), ord('z'))) : return ord(name) test_ns = test() interrogate test_ns: print a ---- output is 97 using the above example lexical closures may be awkward to implement in the language itself, this is a bit of a contrived example.. but.. eh.. # if I understand python enough, then I believe that when we enter this block, and create variables, they go out of scope, # and there are no exceptions to this rule, ever so .. in order to implement mine obviously scope would be the same # mechanism so I have oh_bother declared in the local namespace first oh_bother = None interrogate test_ns: def geronimo(): return z oh_bother = geronimo print oh_bother() --- output is 122 -- -Prozacgod "Prozac may heal the mind, but friends can mend the soul"

6 19

[Fwd: Re: exception based conditional expression, similar to if-else conditional expression]
by Jeff McAninch 21 Aug '09

21 Aug '09

Calvin Spealman wrote: > -1 on colons in the expression like that. I like the idea of being > able to handle an exception in generator expressions and the like, but > I've never seen a syntax I liked. I think I've favored the idea of > something like `float(x) except float('nan') if ValueError` thinking > it reads more naturally as an expression, puts the real logic > ("convert x to a float or get a NaN float") together, which I think > makes sense. > > Yes, I agree about the colons. They have no purpose. I was just blindly following the try-except. (Duh! on my part) So, in the simple example: x = float(string) except ValueError float('nan') But possibly the exception tuples now have to be explicitly tuples? x = float(string) except (ValueError,) float('nan') So the general case would be something like exception_expression :== nominal_value {except exception_tuple exception_value}* {except default_value} Jeff McAninch -- ========================== Jeffrey E. McAninch, PhD Physicist, X-2-IFD Los Alamos National Laboratory Phone: 505-667-0374 Email: mcaninch(a)lanl.gov ==========================

4 5

Re: [Python-ideas] exception based conditional expression, similar to if-else conditional expression
by Jeff McAninch 21 Aug '09

21 Aug '09

> > -1, if this goes through it will be extended to all other compound > statements over time and we'll end up with 2 or 3 ways to do every > thing. > > A much better idea would be to find a way to make all compound > statements into expressions, future-proofing the decision and avoiding > the redundancy between "compound statement statement" and "compound > statement expression". > Python's list of compound statements is pretty thin (just 7), and 4 of these already have expression forms. for ==> list comprehension, iterator expression while ==> list comprehension, iterator if ==> if-else conditional expression def ==> already have lambda expressions try ==> my proposal class ==> Okay, so this one doesn't have a compound statement, but where would you use it? with ==> I haven't had an occasion to use with, so can't think of an example where it could be useful as a compound expression. Seems to me try-except is kind of sticking out here, and I've tried to show what I think are natural and common places where a try-except expression would be appropriate. -- ========================== Jeffrey E. McAninch, PhD Physicist, X-2-IFD Los Alamos National Laboratory Phone: 505-667-0374 Email: mcaninch(a)lanl.gov ==========================

5 4

pep 312 - implicit lambdas idea
by ilya 20 Aug '09

20 Aug '09

I was thinking about a good syntax for implicit lambdas for a while and today I had this idea: make ``_:`` a shortcut for ``lambda _=None:`` For example: map( _: _ + 5, some_list) register_callback( _: True) def apply_transform(..., transform = _:_, ... ): but still addition = lamba x, y: x + y The rationale is that you only want to get rid of lambda keyword to create a *very* simple function, the one that will be called either without parameters or with only one parameter. For everything more complicated, you really should go and write the explicit function signature using lambda. Even though ``:`` could theoretically denote implicit lambda, it's too easy to miss it. The combination ``_:`` is much easier to notice. It also makes explicit that there is at most one parameter and it's name is ``_``. Since it's very short, it can easily be used in a long function call or as a default parameter, as above Your thoughts?

9 17

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas August 2009