Mailman 3 September 2009 - Python-ideas

Heap data type
by Facundo Batista 07 Dec '09

07 Dec '09

Using the module heapq, I found that it's not easy to use. Or, at least, that it could be straightforward to use. My main issue is that for correct usage: - user should provide an external list, that shouldn't use for anything else to don't break the invariant - to alter the heap queue, a bunch of functions must be used, passing always the external list I think that changing "external list" for "internal attribute", and "bunch of functions " for "methods", it will leave the module easier to use, safer, and more object oriented. So, I basically coded it. What do you think about including this class in the heap module? """ class Heap(object): '''Maintains a heapified list, always conserving the invariant. Heaps are arrays for which heap[k] <= heap[2*k+1] and heap[k] <= heap[2*k+2] for all k, counting elements from zero. ''' def __init__(self, iterable=None): '''Initializes the heap from any iterable. >>> Heap([1, 2]) Heap([1, 2]) >>> Heap([]) Heap([]) >>> Heap() Heap([]) >>> Heap((1, 2, 3)) Heap([1, 2, 3]) >>> Heap(x**2 for x in range(3)) Heap([0, 1, 4]) ''' if iterable is None: self._queue = [] else: self._queue = list(iterable) heapq.heapify(self._queue) def __repr__(self): return "Heap(%s)" % self._queue def push(self, item): '''Push the item to the heap queue. >>> h = Heap() >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.push(2) >>> h.push(1) >>> h.push(0) >>> h Heap([0, 2, 1, 5, 3, 4]) ''' heapq.heappush(self._queue, item) def pop(self): '''Pop one item from the heap queue. >>> h = Heap([0, 2, 1, 5, 3, 4]) >>> h.pop() 0 >>> h.pop() 1 >>> h.pop() 2 >>> h.pop() 3 >>> h.pop() 4 >>> h.pop() 5 >>> h Heap([]) >>> h.pop() Traceback (most recent call last): ... IndexError: index out of range ''' return heapq.heappop(self._queue) def pushpop(self, item): '''Push the item and pop one from the heap queue. Note that this method is more efficient than calling both push() and pop() separately. >>> h = Heap() >>> h.pushpop(7) 7 >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.pushpop(7) 3 >>> h.pushpop(7) 4 >>> h Heap([5, 7, 7]) ''' return heapq.heappushpop(self._queue, item) def replace(self, item): '''Pop one item and push the received one into the heap queue Note that this method is more efficient than calling both pop() and push() separately. >>> h = Heap() >>> h.replace(3) Traceback (most recent call last): ... IndexError: index out of range >>> h.push(7) >>> h.replace(2) 7 >>> h Heap([2]) >>> h.push(3) >>> h Heap([2, 3]) >>> h.replace(1) 2 >>> h.replace(9) 1 >>> h Heap([3, 9]) ''' return heapq.heapreplace(self._queue, item) """ Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

11 14

bump version of openssl (was "adding digital signature and encryption 'hashes' to hashlib?")
by CTO 30 Sep '09

30 Sep '09

Right now the version of openssl shipping with python is 9.8, where version 1.0 was released some months back and provides access to EVP_PKEY_CTX, which is useful in interacting with the public key interface now provided by evp. Would it be possible to bump it up to 1.0, or is that prohibitively difficult? Geremy Condra

1 1

filter
by Josef Eschgfaeller 27 Sep '09

27 Sep '09

It would be nice to have a syntax like [f(x) for x in u and cond(x)] instead of list(map(f,filter(cond,u))). For avoiding interference with a common and it could be [f(x) for x in u andcond cond(x)] or perhaps [f(x) for x in u with cond(x)] if this is not interfering. Josef Eschgfaeller

3 2

Why is there a callable predicate, but no iterable?
by Andrey Fedorov 27 Sep '09

27 Sep '09

So there was a discussion back in April [0] about the lack of an "iterable" predicate, which Pascal pointing out that the intention may be to use "isinstance(obj, Iterable)" instead. That seems inconsistent with the existence of collections.Callable (so, isinstance(obj, Callable) instead of callable(obj)). Which direction is this more likely to be resolved? Should I write iterable(obj) or expect callable(obj) to be deprecated? - Andrey 0. http://mail.python.org/pipermail/python-ideas/2009-April/004382.html

5 7

[Wild Idea] Static Ducks
by Dj Gilcrease 26 Sep '09

26 Sep '09

I do not expect this idea to gain much traction but it has been an idea rolling around in my head for a year or two, so I decided to write it all down and see what other thought. Abstract: Python does not currently support static typing of variables, which is something I like, but there are times it would be nice to have static typing for variables, or method/function return values. Static Ducks should be able to run along side or imported to code that does not use Static Ducks. Actual runtime enforcement of Static Ducks should be off by default and require activation either via the command line or at the top of a file (similar to the __future__ requirement) where it would affect only that file (and any Static Ducks imported to it) Design Goals: The new syntax should * Be simple to read * Make it obvious what is happening; at the very least it should be obvious that new users can safely ignore it when writing their own code * Allow future compilers to optimize for Static Typing. * Work within a function or method definition to ensure a passed variable is the proper type * Allow Functions/Methods return values to be typed and checked * Raise an TypeError exception when the variable is set to something improper, or the function/method tries to return an improper value Issues: If this PEP is implemented it may conflict with PEP 362 (Function Signature Object) Use Cases: Python to C compilers could be enhanced by allowing static ducks. IDE's code complete could be enhanced by having static ducks available IDE's issues prediction could be enhanced to detect type issues for code that uses static ducks (Like any static typed language) Syntax Idea: """$type variable = value $returnType def function($type variable): #do stuff return value $returnType def function($type variable): #do stuff return value class test: $type variable = value $returnType def method(self, $type variable): #do stuff return value""" The reason I selected $type is because I couldn't think of anything else that $ was used for and I wanted a syntax similar to Decorators and similar to other statically typed languages. All types allow None or their proper type when setting, this so people can have typed variables that can be "undefined" for checking Example: $int count = 0 $bool def check($string testValue): return testValue.lower() == "true" class test: $string _name = None $string def get_name(self): return self._name def set_name(self, $string name): self._name = name Usage Details: When a variable has been statically type, any time it is set later its type needs to be verified. I see this working in a similar way to how properties work. Also using this style would allow people to create their own static ducks. """psudo-code class StaticDuckBase(object): _value = None class IntTypeClass(StaticDuckBase): def _get_value(self): return self._value def _set_value(self, val): if isinstance(val, (int, type(None))): self._value = val else: try: val = int(val) self._value = val except: raise TypeError("Expected int, got " + str(type(val)) + " instead!") value = property(_get_value, _set_value, None) class MyTypeClass(StaticDuckBase): def _get_value(self): return self._value def _set_value(self, val): if isinstance(val, MyType)): self._value = val else: raise TypeError("Expected MyType, got " + str(type(val)) + " instead!") value = property(_get_value, _set_value, None) staticducks.add('mytype', MyTypeClass) """ Though there will need to be some way to check if a variable is statically typed so that when you do $int count = 0 then later count = 5 it uses the static typing. Also if no static duck is specified it should revert to nonstatic but emit a warning that is only show when run with the -W option Type List: Base Types: $int $float $bool $string $tuple $namedtuple $list $set $frozenset $dict $byte $bytearray $memoryview $nonstatic # This means treat it as if it were not statically typed Composite type $bytes -> $byte | $bytearray $number -> $int | $float | Decimal $seq -> $tuple | $list | $set | $frozenset | $namedtuple $sets -> $set | $frozenset Special Cases #lets you require a specific type for keys and values $dict($type key[, $type value]) """eg $dict($int, $string) tmp = {} # creates a dict that requires ints as keys and strings as values $dict($nonstatic, $bool) tmp = {} # creates a dict that requires values to be bool, but keys can be anything """ $tuple($type value) # lets you specify the type for the tuples values $list($type value) # lets you specify the type for the lists values $set($type value) # lets you specify the type for the sets values $frozenset($type value) # lets you specify the type for the sets values $sets($type value) # lets you specify the type for the sets values $seq($type value) # lets you specify the type for the sequence values $string(encoding="UTF-8", normalized=False) $normalized # This would be equiv of doing $string(normalized=True) $normalized(encoding="UTF-8")

22 75

adding digital signature and encryption "hashes" to hashlib?
by Bill Janssen 26 Sep '09

26 Sep '09

A couple of weeks ago, there was a discussion on python-dev about adding the ability to load modules from encrypted zip files. I'm not sure the discussion went anywhere, and I was on vacation when it took place. However, it reminded me of an idea from a couple of years ago: extend the hashlib module to produce two additional kinds of hashes: a digital signature for some sequence of bytes, and an encrypted/decrypted version of a sequence of bytes. Basically, the would bring more of the OpenSSL EVP API out to Python (hashlib already uses OpenSSL EVP for various hash formations). http://www.openssl.org/docs/crypto/evp.html With this, it would be fairly trivial to implement strong encryption of zip files (or anything else), and this could then be used to do the import feature. I'd envision adding new constructors to hashlib: sig = hashlib.signature([data] [, keyfile=...] [, signature_algorithm=...]) This would have the regular update() method, and digest() and hexdigest(), but would also support the method sig.verify(existing_sig) which would return a boolean saying the "existing sig" is a verified signature for that data. Similarly, encryption/decryption would be enc = hashlib.encryptor([plaintext] [, keyfile=...] [, ciphers=...]) enc.digest() would give the ciphertext. And dec = hashlib.decryptor([ciphertext] [, keyfile=...] [, ciphers=...]) and dec.digest would yield the plaintext. The encryptor and decryptor constructors could take either "key" or "keyfile" parameters. Using "key" would support symmetric encrytion, while using "keyfile" would produce EVP envelope encryption/decryption. Bill

8 28

wild idea : target decorators
by Boris Borcic 25 Sep '09

25 Sep '09

Simply put, augmented assignment of the form x op= expr invites pythonistas to a particular intuition of the DRY principle which feels denied whenever the need is encountered to write code of the form x = f(x,y) or even, more simply x = f(x) on another hand, Python already admits an implied x = f(x) with decorators, since by definition of decorators @f def x(...) : suite means the same as def x(...) : suite x=f(x) the embryonic proposal is to extend decorator syntax to arbitrary assignment targets, to permit use of it to formulate x=f(x) DRY-ly, e.g. @f x together with a further syntax extension allowing to write the above on a single line, by postfixing the assignment target with the decorator, e.g. x @f or perhaps x @= f to be more in line with other augmented assignment operators, by analogy, say, with how x *= 2 can replace x = 2*x while removing the need to repeat x ok, this is clearly an completely embryonic idea shown (and motivated) only by (and through) how it would apply to the simplest imaginable use case. The displaying of it is intended to stimulate community discussion on whether and how it might extend to a more useful wider and consistent extension of the python language, and not to invite approval ratings over its current, well, egg. IOW, does anybody feel like me there is a challenge to see better whether/how these two disjoint python corners, e.g. decorators on the one hand, and augmented assignment on the other hand, may surprisingly reveal they can fit a unifying extension of both ? Cheers, BB

4 7

Python Bytecode Verifier
by Kornél Pál 20 Sep '09

20 Sep '09

Hi, I've created a Python Bytecode Verifier in Python. I'm not a Python guru so I borrowed coding patterns from C/C++. I also created this with C portability in mind. The only reason I used Python was to experiment with Python and was easier to morph code during development. If this program were ported to C it would only need 8 bytes per opcode (some additional storage to track blocks) and a single pass. I haven't found backward jumps to previously unused code in any compiled Python code but it can easily be supported. In that case some more partial passes are required. I also was able to successfully verify all Python files in the Python-2.5.2 source package. The verification algorythm should be quite complete but I may have missed some limitations of the interpreter that could be checked by the verifier as well. The ability to create this verfier proved that although Python bytecode is designed for a dynamically typed interpreter, is still easily verifiable. I am willing port this code C but only in the case if there is any chance to be included in Python. I believe that Python in general would benefit having the ability to safely load .pyc files and create code objects on the fly. Both Java and .NET have the ability to safely load compiled byte code. .NET Framework, just like Python also has the ability to create and execute new code at run-time. You may feel that enabling closed source applications and/or creating a multi-language runtime would hurt Python but both of these have contributed to the success of Java (both the language and the runtime). Kornél

2 2

Wild idea: Exception.format
by Georg Brandl 19 Sep '09

19 Sep '09

To make Exceptions where some object(s) are involved more useful, it is often necessary to put the objects on the exception *in addition to* formatting them into a string representation for the message. This little classmethod would make that easier:: class BaseException: @classmethod def format(cls, fmt, *args): return cls(fmt.format(*args), *args) Example usage:: ext = 'foo' raise LookupError.format('Extension {0} not registered', ext) 'foo' could then be accessed as ``exc.args[1]``. A similar, but also very useful implementation would be :: def format(cls, fmt, **kwds): exc = cls(fmt.format(**kwds)) exc.__dict__.update(kwds) return exc with example usage being:: raise LookupError.format('Extension {ext} not registered', ext='foo') and 'foo' being accessible as ``exc.ext``. I realize this is probably too obscure for Python core, but I wanted to show it to you anyway, maybe it'll be found useful. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

4 4

IDEA: do not alter default SIGINT handling
by Bernie Innocenti 17 Sep '09

17 Sep '09

On startup, the Python interpreter changes the default behavior of SIGINT, which results in many Python programs to ignore the keyboard interrupt exactly in the situations when users are most likely to use it (i.e.: when the program becomes unresponsive). Minimal testcase: $ echo "void foo() { for(;;) {} }" >foo.c $ gcc -shared -o foo.so foo.c $ python -c 'import ctypes;ctypes.CDLL("./foo.so").foo()' ^C^C^C ^C ^C DAMN! ^C This scenario mimics a Python program calling some blocking library function. It can also happen with IO-bound functions if they loop on read() and don't abort on short reads. One might be tempted to say "this behavior of the Python intepreter is by design" and suggest users to use CTRL-\ instead of CTRL-C. However, this non-standard behavior is very annoying for users who expect ^C to work on UNIX systems. In fact, no other compiled or interpreted language I know of behaves this way, and Python should not be the only exception. While I see the usefulness of KeyboardInterrupt from the programmer point of view, only a minority of programs actually need to trap SIGINT and do something with it. Other language runtimes require the programmer to manually trap SIGINT when needed. The Python interpreter could maintain backwards compatibility by enabling automatic SIGINT trapping when entering a "try" block that would intercept KeyboardInterrupt. For 2 years, I've been using this workaround in my /usr/lib64/python2.6/sitecustomize.py: ----cut----- import signal signal.signal(signal.SIGINT, signal.SIG_DFL) ----cut----- CTRL-C has been working perfectly ever since. So far, I have not yet found a single Python program where restoring the default behavior of SIGINT causes real issues, but there may certainly be a few. Granted, this is just a kludge, not a perfect fix, but from a user perspective, it already improves upon the current behavior (i.e. more pros than cons). At least, this is my personal experience. If you're skeptical, please try the above workaround yourself for a few months and let me know what breaks for you. If we could break the syntax of "print" statements, I'm sure we can also find a satisfactory compromise for CTRL-C handling that won't affect more than 0.1% of existing Python programs. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/

16 39