Mailman 3 January 2011 - Python-ideas

A sorted version of **kwargs
by Don Spaulding 20 Jan '11

20 Jan '11

Hi there python-ideas! Does it bother anyone else that it's so cumbersome to instantiate an OrderedDict with ordered data? >>> from collections import OrderedDict >>> OrderedDict(b=1,a=2) OrderedDict([('a', 2), ('b', 1)]) # Order lost. Boooo. >>> OrderedDict([('b',1),('a',2)]) OrderedDict([('b', 1), ('a', 2)]) Obviously, OrderedDict's __init__ method (like all other functions) never gets a chance to see the kwargs dict in the order it was specified. It's usually faked by accepting the sequence of (key, val) tuples, as above. I personally think it would be nice to be able to ask the interpreter to keep track of the order of the arguments to my function, something like: def sweet_function_name(*args, **kwargs, ***an_odict_of_kwargs): pass I'm not married to the syntax. What do you think about the idea?

13 17

Adding salt and Modular Crypt Format to crypt library.
by Sean Reifschneider 17 Jan '11

17 Jan '11

Over the years I've written the same code over and over to create a random salt string of 2 characters. Worse, the Modular Crypt Format is difficult to find documentation on, so creating stronger hashed passwords is difficult to get right. By this, I mean things like: crypt.crypt('password', 'xJ') crypt.crypt('password', '$1$/gL8bA.z') crypt.crypt('password', '$6$/uPNNoSGrlc0Kf0go') To that end, I'm proposing the addition of a "mksalt()" method which will generate a salt, and several METHOD_* values to select which hashing method to use. I also figure there will need to be a "methods()" call that figures out what methods are available in the library crypt() and return a list of the available ones. If we have a way to generate a salt, then I figure we could drop the salt argument of crypt.crypt(), and if not specified to generate one. So to hash a password you could do: "crypt.crypt('password')". I figure that the best way to accomplish this is to implement this all in Python and move the existing C crypt module to _crypt. I've created an issue: http://bugs.python.org/issue10924 with this description and a patch to accomplish the above. Thoughts and review? Thanks, Sean -- I have a large collection of sea shells, which I keep scattered on beaches around the world. Maybe you've seen it... -- Steven Wright Sean Reifschneider, Member of Technical Staff <jafo(a)tummy.com> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability

2 2

Module aliases and/or "real names"
by Nick Coghlan 12 Jan '11

12 Jan '11

Disclaimer: this is a currently half-baked idea that needs some discussion here if it is going to turn into something a bit more coherent :) On and off, I've been pondering the problem of the way implementation details (like the real file structures of the multiprocessing and unittest packages, or whether or not an interpreter use the pure Python or the C accelerated version of various modules) leak out into the world via the __module__ attribute on various components. This mostly comes up when discussing pickle compatibility between 2.x and 3.x, but in can show up in various guises whenever you start relying on dynamic introspection. As, I see it, there are 3 basic ways of dealing with the problem: 1. Allow objects to lie about their source module This is likely a terrible idea, since a function's global namespace reference would disagree with its module reference. I suspect much weirdness would result. 2. A pickle-specific module alias registry, since that is where the problem comes up most often A possible approach, but not necessarily a good one (since it isn't really a pickle-specific problem). 3. An inspect-based module alias registry That is, an additional query API (get_canonical_module_name?) in the inspect module that translates from the implementation detail module name to the "preferred" module name. The implementation could be as simple as a "__canonical__" attribute in the module namespace. I actually quite like option 3, with various things (such as pydoc) updated to show *both* names when they're different. That way people will know where to find official documentation for objects from pseudo-packages and acceleration modules (i.e. under the canonical name), without hiding where the actual implementation came from. Pickle *generation* could then be updated to only send canonical module names during normal operation, reducing the exposure of implementation details like pseudo-packages and acceleration modules. Whether or not runpy should set __canonical__ on the main module would be an open question (probably not, *unless* runpy was also updated to add the main module to sys.modules under its real name as well __main__). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 34

Add irange with large integer step support to itertools
by Martin Manns 11 Jan '11

11 Jan '11

Hi I would like to propose an addition of an "irange" function to itertools. This addition could reduce testing effort when developing applications, in which large integers show up. Both, xrange (Python 2.x) and range (Python 3.x) have limited support for large integer step values, for example: Python 3.1.3 (r313:86834, Nov 28 2010, 10:01:07) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> range(10**10000, 10**10000+10**1000, 10**900)[5] Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: Python int too large to convert to C ssize_t The code below is untested and for clarification only. It has been taken and modified from [issue7721] http://bugs.python.org/issue7721 With irange, no OverflowError is thrown: >>> from itertools import islice >>> from irange import irange >>> def nth(iterable, n, default=None): ... "Returns the nth item or a default value" ... return next(islice(iterable, n, None), default) ... >>> nth(irange(10**10000, 10**10000+10**1000, 10**900), 5) 100000000000000 ... ## Code snippet (untested) from itertools import count, takewhile def irange(start, stop=None, step=1): """Range for long integers Usage: irange([start], stop, [step]) Parameters ---------- start: Integer stop: Integer step: Integer, defaults to 1 """ if start is None: raise TypeError("range() integer argument expected, got NoneType") if stop is None: stop = start start = 0 if step is None: step = 1 if step > 0: if stop < start: return (_ for _ in []) cond = lambda x: x < stop elif step < 0: if stop > start: return (_ for _ in []) cond = lambda x: x > stop else: raise ValueError("irange() step argument must not be zero") return takewhile(cond, (start + i * step for i in count())) ## End code snippet Does such an addition make sense in your eyes? Regards Martin

5 10

python -c "..." should output result like the interpreter
by Michael Foord 05 Jan '11

05 Jan '11

Hello all, I often use python as a calculator or for simple operations using -c. It would be enormously useful if python -c "..." would output on stdout the result of the last evaluated expression in the same way that the interactive interpreter does. The following outputs nothing: python -c "12 / 4.1" So you always have to prefix expressions with print to see the result. Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

7 19

An improved `ContextManager`
by cool-RR 03 Jan '11

03 Jan '11

Hello folks. Ever since Michael Foord talked about `ContextDecorator` in python-ideas I've been kicking around an idea for my own take on it. It's a `ContextManager` class which provides the same thing that Foord's `ContextDecorator` does, but also provides a few more goodies, chief of which being the `manage_context` method. I've been working on this for a few days and I think it's ready for review. It's well-tested and extensively documented. I started using it wherever I have context managers in GarlicSim. I'll be happy to get your opinions on my approach and any critiques you may have. If there are no problems with this approach, I'll probably release it with GarlicSim 0.6.1 and blog about it. Here is my `context_manager` module<https://github.com/cool-RR/GarlicSim/blob/first_context_manager_review/garl…>. Here are its tests<https://github.com/cool-RR/GarlicSim/tree/first_context_manager_review/garl…> . Following is the module's docstring which explains the module in more detail. Ram. Defines the `ContextManager` and `ContextManagerType` classes. These classes allow for greater freedom both when (a) defining context managers and when (b) using them. Inherit all your context managers from `ContextManager` (or decorate your generator functions with `ContextManagerType`) to enjoy all the benefits described below. Defining context managers ------------------------- There are 3 different ways in which context managers can be defined, and each has their own advantages and disadvantages over the others. 1. The classic way to define a context manager is to define a class with `__enter__` and `__exit__` methods. This is allowed, and if you do this you should still inherit from `ContextManager`. Example: class MyContextManager(ContextManager): def __enter__(self): pass # preparation def __exit__(self, type_=None, value=None, traceback=None): pass # cleanup 2. As a decorated generator, like so: @ContextManagerType def MyContextManager(): try: yield finally: pass # clean-up This usage is nothing new; It's also available when using the standard library's `contextlib.contextmanager` decorator. One thing that is allowed here that `contextlib` doesn't allow is to yield the context manager itself by doing `yield SelfHook`. 3. The third and novel way is by defining a class with a `manage_context` method which returns a decorator. Example: class MyContextManager(ContextManager): def manage_context(self): do_some_preparation() try: with some_lock: yield self finally: do_some_cleanup() This approach is sometimes cleaner than defining `__enter__` and `__exit__`; Especially when using another context manager inside `manage_context`. In our example we did `with some_lock` in our `manage_context`, which is shorter and more idiomatic than calling `some_lock.__enter__` in an `__enter__` method and `some_lock.__exit__` in an `__exit__` method. These were the different ways of *defining* a context manager. Now let's see the different ways of *using* a context manager: Using context managers ---------------------- There are 2 different ways in which context managers can be used: 1. The plain old honest-to-Guido `with` keyword: with MyContextManager() as my_context_manager: do_stuff() 2. As a decorator to a function @MyContextManager() def do_stuff(): pass # doing stuff When the `do_stuff` function will be called, the context manager will be used. This functionality is also available in the standard library of Python 3.2+ by using `contextlib.ContextDecorator`, but here it is combined with all the other goodies given by `ContextManager`. That's it. Inherit all your context managers from `ContextManager` (or decorate your generator functions with `ContextManagerType`) to enjoy all these benefits. -- Sincerely, Ram Rachum

1 0

read-only attributes
by K. Richard Pixley 03 Jan '11

03 Jan '11

It seems to me that one of the more common reasons for using @property is to create a read-only attribute. I wonder if it would make sense to simply create a read-only decorator. Compare: class Foo(object): _size = 4 @property def size(self): return _size against: class Foo(object): @read-only size = 4 This gets more interesting if decorators nest: class Foo(object): __metaclass__ = abc.ABCMeta @abstract @classattribute @read-only size = None --rich

3 2

Optimizing builtins
by Guido van Rossum 03 Jan '11

03 Jan '11

[Changed subject *and* list] > 2010/12/31 Maciej Fijalkowski <fijall(a)gmail.com> >> How do you know that range is a builtin you're thinking >> about and not some other object? On Fri, Dec 31, 2010 at 7:02 AM, Cesare Di Mauro <cesare.di.mauro(a)gmail.com> wrote: > By a special opcode which could do this work. ]:-) That can't be the answer, because then the question would become "how does the compiler know it can use the special opcode". This particular issue (generating special opcodes for certain builtins) has actually been discussed many times before. Alas, given Python's extremely dynamic promises it is very hard to do it in a way that is *guaranteed* not to change the semantics. For example, I could have replaced builtins['range'] with something else; or I could have inserted a variable named 'range' into the module's __dict__. (Note that I am not talking about just creating a global variable named 'range' in the module; those the compiler could recognize. I am talking about interceptions that a compiler cannot see, assuming it compiles each module independently, i.e. without whole-program optimizations.) Now, *in practice* such manipulations are rare (with the possible exception of people replacing open() with something providing hooks for e.g. a virtual filesystem) and there is probably some benefit to be had. (I expect that the biggest benefit might well be from replacing len() with an opcode.) I have in the past proposed to change the official semantics of the language subtly to allow such optimizations (i.e. recognizing builtins and replacing them with dedicated opcodes). There should also be a simple way to disable them, e.g. by setting "len = len" at the top of a module, one would be signalling that len() is not to be replaced by an opcode. But it remains messy and nobody has really gotten very far with implementing this. It is certainly not "low-hanging fruit" to do it properly. I should also refer people interested in this subject to at least three PEPs that were written about this topic: PEP 266, PEP 267 and PEP 280. All three have been deferred, since nobody was bold enough to implement at least one of them well enough to be able to tell if it was even worth the trouble. I haven't read either of those in a long time, and they may well be outdated by current JIT technology. I just want to warn folks that it's not such a simple matter to replace "for i in range(....):" with a special opcode. (FWIW, optimizing "x[i] = i" would be much simpler -- I don't really care about the argument that a debugger might interfere. But again, apart from the simplest cases, it requires a sophisticated parser to determine that it is really safe to do so.) -- --Guido van Rossum (python.org/~guido)

9 20

Re: [Python-ideas] Optimizing builtins
by Guido van Rossum 01 Jan '11

01 Jan '11

On Sat, Jan 1, 2011 at 5:11 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Sat, Jan 1, 2011 at 11:32 AM, Cesare Di Mauro wrote: >> Yes, I know it, but the special opcode which I was talking about has a very >> different usage. >> The primary goal was to speed-up fors, generating specialized code when the >> proper range builtin is found at runtime, and it's convenient to have such >> optimized code. >> As you stated, the compiler don't know if range is a builtin until runtime >> (at the precise moment of for execution), so it'll generate two different >> code paths. The function's bytecode will look like that: >> 0 SETUP_LOOP 62 >> 2 JUMP_IF_BUILTIN_OR_LOAD_GLOBAL 'range', 40 >> # Usual, slow, code starts here >> 40 LOAD_CONSTS (4, 3, 2, 1, 0) # Loads the tuple on the stack >> 44 LOAD_FAST_MORE_TIMES x, 5 # Loads x 5 times on the stack >> 46 LOAD_CONSTS (4, 3, 2, 1, 0) # Loads the tuple on the stack >> 48 STACK_ZIP 3, 5 # "zips" 3 sequences of 5 elements each on the stack >> 52 STORE_SUBSCR >> 54 STORE_SUBSCR >> 56 STORE_SUBSCR >> 58 STORE_SUBSCR >> 60 STORE_SUBSCR >> 62 POP_BLOCK >> 64 RETURN_CONST 'None' >> It's just an example; cde can be different based on the compiler >> optimizations and opcodes available in the virtual machine. >> The most important thing is that the semantic will be preserved (I never >> intended to drop it! ;) > > The thing is, having a JIT, this all is completely trivial (as well as > bunch of other stuff like avoiding allocating ints at all). Right. That's a much saner solution than trying to generate bulky bytecode as Cesare proposed. The advantage of a JIT is also that it allows doing these optimizations only in those places where it matters. In general I am not much in favor of trying to optimize Python's bytecode. I prefer the bytecode to be dead simple. This probably makes it an easy target for CS majors interested in code generation, and it probably is a great exercise trying to do something like that, but let's please not confuse that with actual speed improvements to Python -- those come from careful observation (& instrumentation) of real programs, not from looking at toy bytecode samples. (Most of the bytecode improvements that actually made a difference were done in the first 5 years of Python's existence.) > Generating two different code paths has a tendency to lead to code > explosion (even exponential if you're not careful enough), which has > it's own set of problems (including cache locality, because code > executed is no longer a small continuous chunk of memory). What we > (PyPy) do, is to compile only the common path (using JIT) and then > have unlikely path fall back to the interpreter. This generally solves > all of nasty problems you can possibly encounter. Great observation! -- --Guido van Rossum (python.org/~guido)

3 4

ML Style Pattern Matching for Python
by Eike Welk 01 Jan '11

01 Jan '11

After learning a bit of Ocaml I started to like its pattern matching features. Since then I want to have a "match" statement in Python. I wonder if anybody else would like this too. ML style pattern matching is syntactic sugar, that combines "if" statements with tuple unpacking, access to object attributes, and assignments. It is a compact, yet very readable syntax for algorithms, that would otherwise require nested "if" statements. It is especially useful for writing interpreters, and processing complex trees. Instead of a specification in BNF, here is a function written with the proposed pattern matching syntax. It demonstrates the features that I find most important. The comments and the print statements explain what is done. Proposed Syntax --------------- def foo(x): match x with | 1 -> # Equality print("x is equal to 1") | a:int -> # Type check print("x has type int: %s" % a) | (a, b) -> # Tuple unpacking print("x is a tuple with length 2: (%s, %s)" % (a, b)) | {| a, b |} -> # Attribute existence and access print("x is an object with attributes 'a' and 'b'.") print("a=%s, b=%s" % (a, b)) # Additional condition | (a, b, c) with a > b -> print("x is a tuple with length 3: (%s, %s, %s)" % (a, b, c)) print("The first element is greater than the second element.") # Complex case | {| c:int, d=1 |}:Foo -> print("x has type Foo") print("x is an object with attributes 'c' and 'd'.") print("'c' has type 'int', 'd' is equal to 1.") print("c=%s, d=%s" % (c, d)) # Default case | _ -> print("x can be anything") Equivalent Current Python ------------------------- The first four cases could be handled more simply, but handling all cases in the same way leads IMHO to more simple code overall. def foo(x): while True: # Equality if x == 1: print("x is equal to 1") break # Type check if isinstance(x, int): a = x print("x is an integer: %s" % a) break # Tuple unpacking if isinstance(x, tuple) and len(x) == 2: a, b = x print("x is a tuple with length 2: (%s, %s)" % (a, b)) break # Attribute existence testing and access if hasattr(x, "a") and hasattr(x, "b"): a, b = x.a, x.b print("x is an object with attributes 'a' and 'b'.") print("a=%s, b=%s" % (a, b)) break # Additional condition if isinstance(x, tuple) and len(x) == 3: a, b, c = x if a > b : print("x is a tuple with length 3: (%s, %s, %s)" % (a, b, c)) print("The first element is greater than the second " "element.") break # Complex case if isinstance(x, Foo) and hasattr(x, "c") and hasattr(x, "d"): c, d = x.c, x.d if isinstance(c, int) and d == 1: print("x has type Foo") print("x is an object with attributes 'c' and 'd'.") print("'c' has type 'int', 'd' is equal to 1.") print("c=%s, d=%s" % (c, d)) break # Default case print("x can be anything") break Additional Code to Run Function "foo" ------------------------------------- class Bar(object): def __init__(self, a, b): self.a = a self.b = b class Foo(object): def __init__(self, c, d): self.c = c self.d = d foo(1) # Equality foo(2) # Type check foo((1, 2)) # Tuple unpacking foo(Bar(1, 2)) # Attribute existence testing and access foo((2, 1, 3)) # Additional condition foo(Foo(2, 1)) # Complex case foo("hello") # Default case I left out dict and set, because I'm not sure how they should be handled. I think list should be handled like tuples. Probably there should be a universal matching syntax for all sequences, similarly to the already existing syntax: a, b, *c = s I don't really like the "->" digraph at the end of each match case. A colon would be much more consistent, but I use colons already for type checking (a:int). I generally think that Python should acquire more features from functional languages. In analogy to "RPython" it should ultimately lead to "MLPython", a subset of the Python language that can be type checked and reasoned about by external tools, similarly to what is possible with Ocaml. Eike.

14 30