Mailman 3 January 2007 - Python-ideas

fixing mutable default argument values
by Chris Rebert May 2, 2007

May 2, 2007

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] … [View More]

9 25

tab completion in pdb
by stephen emslie Feb. 8, 2007

Feb. 8, 2007

I make frequent use of the python's built-in debugger, which I think is brilliant in its simplicity. However an important feature seems to be missing: bash-like tab completion similar to that provided by the rlcompleter module. By default, Pdb and other instances of Cmd complete names for commands only. However in the context of pdb, I think it is more useful to complete identifiers and keywords in its current scope than to complete names of commands (most of which have single letter … [View More]

2 4

proto-PEP: Fixing Non-constant Default Arguments
by Chris Rebert Feb. 3, 2007

Feb. 3, 2007

The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default … [View More]arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=<baz>): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/ [View Less]

10 42

Import and '..', '../..' in serach path.
by Ron Adam Feb. 1, 2007

Feb. 1, 2007

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package. Module in package: import sys sys.path = ['..'] + sys.path import package.module # Imports module in "this!" package. Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of … [View More]the path. Module in sub-package: import sys sys.path = ['../..'] + sys.path import package.subpackage.module # finds "self" (subpackage) reliably. By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path. Adding this tip to the documentation some where would be nice. (providing there is no major surprising side effects.) Of course I may have missed some obvious way to do this. If so, it wasn't in an obvious place to be found. I looked. ;-) ---------------------------------- It might be useful to have a built-in function to do this. A function could also check for __init__ files and raise errors if they are missing. set_package_name(dotted.name) # Replaces import sys & path modification Where dotted.name is the full package + sub-package name the current module is located in. The function would search upwards to get the root package directory and add that to the *front* of sys.path. Module in package: set_package_name('package') # Add parent directory to front of sys.path import packagename.module # Finds module in "this!" package reliably. Module in subpackage: set_package_name('package.subpackage') import package.subpackage.module # Finds "self" (subpackage) reliably. ---------------------------------- It may also be able to modify the import behavior to allow relative imports to work when the module is run as script. set_package_name('package') from . import module1 # Imports modules from "this" package. from . import module2 Currently an exception is raised you try to run a module with relative references as a script. ValueError: Attempted relative import in non-package I think it is very handy to be able to run tests as scripts and keep them in a sub-package. Especially while I'm writing them. Another benefit of using relative imports with an absolute specified package name, is if you rename a package or relocate a submodule, you only have one line to change. Cheers, Ron [View Less]

2 7

Mutable default arguments - another approach
by Piotr Duda Jan. 31, 2007

Jan. 31, 2007

I think that this problem can be solved by the following change of default argument behavior: default arguments are evaluated in definition time (like in current implementation), but right after being evaluated, result object is checked if it's mutable (for example by checking of presence __copy__ special method or being instance of built in (sub)class list/dict/set), if object is mutable, argument is marked by COPY_DEF_ARG flag. There are two reasons for this check being done there: 1. … [View More]

6 7

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by Eduardo "EdCrypt" O. Padoan Jan. 30, 2007

Jan. 30, 2007

On 1/30/07, Eduardo EdCrypt O. Padoan <eopadoan(a)altavix.com> wrote: > > If I have time and figure out the right regexes I'll try and come up with > > some more numbers on the entire stdlib, and the ammount of uses of =None. > > > > Some uses of the spam=[] and ham=None in Python projects, including > Python itself: > > http://www.google.com/codesearch?q=def.*%5C(.*%3D%5C%5B%5C%5D.*%5C)%3A%20la… > http://www.google.com/codesearch?hl=en&lr=&q=… [View More]

1 0

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

there really is no need to have something like that built into the language. most default arguments are immutable anyhow, and i'd guess most of the mutable defaults can be addressed with the suggested @copydefaults decorator. as for uncopyable or stateful defaults (dev = sys.stdout), which require reevaluation, you can just use this modified version of my first suggested decorator: >>> def reeval(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... … [View More]

1 0

simplification pep-like thing
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

this text needs more thought and rephrasing, but i think that it covers all the details. -------------------------------------------------------------------------- Abstract ========= The simplification API provides a mechanism to get the state of an object, as well as reconstructing an object with a given state. This new API would affect pickle, copy_reg, and copy modules, as well as the writers of serializers. The idea is to separate the state-gathering from the encoding: * State-gathering … [View More]is getting the state of the object * Encoding is the converting the state (which is an object) into a sequence of bytes (textual or binary) This is somewhat similar to the ISerializable interface of .NET, which defines only GetObjectData(); the actual encoding is done by a Formatter class, which has different implementations for SOAP-formatting, binary-formatting, and other formats. Motivation ========== There are many generic and niche serializers out there, including pickle, banana/jelly, Cerealizer, brine, and lots of serializers targeting XML output. Currently, all serializers have their own methods of retrieving the contents, or state of the object. This API attempts to solve this issue by providing standard means for getting object state, and creating objects with their state restored. Another issue is making the serialization process "proxy-friendly". Many frameworks use object proxies to indirectly refer to another object (for instance, RPC proxies, FFI, etc.). In this case, it's desirable to simplify the referenced object rather than the proxy, and this API addresses this issue too. Simplification ============== Simplification is the process of converting a 'complex' object into its "atomic" components. You may think of these atomic components as the *contents* of the object. This proposal does not state what "atomic" means -- this is open to the decision of the class. The only restriction imposed is, collections of any kind must be simplified as tuples. Moreover, the simplification process may be recursive: object X may simplify itself in terms of object Y, which in turn may go further simplification. Simplification Protocol ======================= This proposal introduces two new special methods: def __simplify__(self): return type, state @classmethod def __rebuild__(cls, state): return new_instance_of_cls_with_given_state __simplify__ takes no arguments (bar 'self'), and returns a tuple of '(type, state)', representing the contents of 'self': * 'type' is expected to be a class or a builtin type, although it can be any object that has a '__rebuild__' method. This 'type' will be used later to reconstruct the object (using 'type.__rebuild__(state)'). * 'state' is any other that represents the inner state of 'self', in a simplified form. This can be an atomic value, or yet another complex object, that may go further simplification. __rebuild__ is a expected to be classmethod that takes the state returned by __simplify__, and returns a new instance of 'cls' with the given state. If a type does not wish to be simplified, it may throw a TypeError in its __simplify__ method; however, this is not recommended. Types that want to be treated as atomic elements, such as file, should just return themselves, and let the serializer handle them. Default Simplification ===================== All the built in types would grow a __simplify__ and __rebuild__ methods, which would follow these guidelines: Primitive types (int, str, float, ...) are considered atomic. Composite types (I think 'complex' is the only type), are broken down into their components. For the complex type, that would be a tuple of (real, imaginary). Container types (tuples, lists, sets, dicts) represent themselves as tuples of items. For example, dicts would be simplified according to this pseudocode: def PyDict_Simplifiy(PyObject * self): return PyDictType, tuple(self.items()) Built in types would be considered atomic. User-defined classes can be simplified into their metaclass, __bases__, and __dict__. The type 'object' would simplify instances by returning their __dict__ and any __slots__ the instance may have. This is the default behavior; classes that desire a different behavior would override __simplify__ and __rebuild__. Example of default behavior: >>> class Foo(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... >>> f = Foo() >>> cls, state = f.__simplify__() >>> cls <class '__main__.Foo'> >>> state {"a" : 5, "b" : "spam"} >>> shallow_copy = cls.__rebuild__(state) >>> state.__simplify__() (<type 'dict'>, (("a", 5), ("b", "spam"))) Example of customized behavior >>> class Bar(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... def __simplify__(self): ... return Bar, 17.5 ... @clasmethod ... def __rebuild__(cls, state) ... self = cls.__new__(cls) ... if state == 17.5: ... self.a = 5 ... self.b = "spam" ... return self ... >>> b = Bar() >>> b.__simplify__() (<class '__main__.Bar'>, 17.5) Code objects ============= I wish that modules, classes and functions would also be simplifiable, however, there are some issues with that: * How to serialize code objects? These can be simplified as tuple of their co_* attributes, but these attributes are very implementation- specific. * How to serialize cell variables, or other globals? It would be nice if .pyc files where generated like so: import foo pickle.dump(foo) It would also allow sending of code between machines, just like any other object. Copying ======== Shallow copy, as well as deep copy, can be implemented using the semantics of this new API. The copy module should be rewritten accordingly: def copy(obj): cls, state = obj.__simplify__() return cls.__rebuild__(state) deepcopy() can be implemented similarly. Deprecation =========== With the proposed API, copy_reg, __reduce__, __reduce_ex__, and possibly other modules become deprecated. Apart from that, the pickle and copy modules need to be updated accordingly. C API ====== The proposal introduces two new C-API functions: PyObject * PyObject_Simplify(PyObject * self); PyObject * PyObject_Rebuild(PyObject * type, PyObject * state); Although this is only a suggestion. I'd like to hear ideas from someone with more experience in the core. I do not see a need for a convenience routine such as simplify(obj) <--> obj.__simplify__(), since this API is not designed for everyday usage. This is the case with __reduce__ today. Object Proxying =============== Because __simplify__ returns '(type, state)', it may choose to "lie" about it's actual type. This means that when the state is reconstructed, another type is used. Object proxies will use this mechanism to serialize the referred object, rather than the proxy. class Proxy(object): [...] def __simplify__(self): # this returns a tuple with the *real* type and state return self.value.__simplify__() Serialization ============ Serialization is the process of converting fully-simplified objects into byte sequences (strings). Fully simplified objects are created by a recursive simplifier, that simplifies the entire object graph into atomic components. Then, the serializer would convert the atomic components into strings. Note that this proposal does not define how atomic objects are to be converted to strings, or how a 'recursive simplifier' should work. These issues are to be resolved by the implementation of the serializer. For instance, file objects are atomic; one serializer may be able to handle them, by storing them as (filename, file-mode, file-position), while another may not be, so it would raise an exception. Recursive Simplifier =================== This code demonstrates the general idea of how recursive simplifiers may be implemented: def recursive_simplifier(obj): cls, state = obj.__simplify__() # simplify all the elements inside tuples if type(state) is tuple: nested_state = [] for item in state: nested_state.append(recursive_simplifier(item)) return cls, nested_state # see if the object is atomic; if not, dig deeper if (cls, state) == state.__simplify__(): # 'state' is an atomic object, no need to go further return cls, state else: # this object is not atomic, so dig deeper return cls, recusrive_simplifier(state) -tomer [View Less]

3 3

my take on mutable default arguments
by tomer filiba Jan. 28, 2007

Jan. 28, 2007

i thought this code be solved nicely with a decorator... it needs some more work, but it would make a good cookbook recipe: .>>> from copy import deepcopy. .>>> .>>> def defaults(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k,v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = deepcopy(v) ... return func(*args, **kwargs) ... return wrapper ... … [View More]

3 4

new pickle semantics/API
by tomer filiba Jan. 26, 2007

Jan. 26, 2007

---------- Forwarded message ---------- From: tomer filiba <tomerfiliba(a)gmail.com> Date: Jan 24, 2007 10:45 PM Subject: new pickle semantics/API To: Python-3000(a)python.org i'm having great trouble in RPyC with pickling object proxies. several users have asked for this feature, but no matter how hard i try to "bend the truth", pickle always complains. it uses type(obj) for the dispatching, which "uncovers" the object is actually a proxy, rather than a real object. recap: RPyC uses … [View More]local proxies that refer to objects of a remote interpreter (another process/machine). if you'd noticed, every RPC framework has its own serializer. for example banna/jelly in twisted and bunch of other XML serializers, and what not. for RPyC i wrote yet another serializer, but for different purposes, so it's not relevant for the issue at hand. what i want is a standard serialization *API*. the idea is that any framework could make use of this API, and that it would be generic enough to eliminate copy_reg and other misfortunes. this also means the built in types should be familiarized with this API. - - - - - - - - for example, currently the builtin types don't support __reduce__, and require pickle to use it's own internal registry. moreover, __reduce__ is very pickle-specific (i.e., it takes the protocol number). what i'm after is an API for "simplifying" complex objects into simpler parts. here's the API i'm suggesting: def __getstate__(self): # return a tuple of (type(self), obj), where obj is a simplified # version of self @classmethod def __setstate__(cls, state): # return an instance of cls, with the given state well, you may already know these two, although their semantics are different. but wait, there's more! the idea is of having the following simple building blocks: * integers (int/long) * strings (str) * arrays (tuples) all picklable objects should be able to express themselves as a collection of these building blocks. of course this will be recursive, i.e., object X could simplify itself as object Y, where object Y might go further simplification, until we are left with building blocks only. for example: * int - return self * float - string in the format "[+-]X.YYYe[+-]EEE" * complex - two floats * tuple - tuple of its simplified elements * list - tuple of its simplified elements * dict - a tuple of (key, value) tuples * set - a tuple of its items * file - raises TypeError("can't be simplified") all in all, i choose to call that *simplification* rather than *serialization*, as serialization is more about converting the simplified objects into a sequence of bytes. my suggestion leaves that out for the implementers of specific serializers. so this is how a typical serializer (e.g., pickle) would be implemented: * define its version of a "recursive simplifier" * optionally use a "memo" to remember objects that were already visited (so they would be serialized by reference rather than by value) * define its flavor of converting ints, strings, and arrays to bytes (binary, textual, etc. etc.) - - - - - - - - the default implementation of __getstate__, in object.__getstate__, will simply return self.__dict__ and any self.__slots__ this removes the need for __reduce__, __reduce_ex__, and copy_reg, and simplifies pickle greatly. it does require, however, adding support for simplification for all builtin types... but this doesn't call for much code: def PyList_GetState(self): state = tuple(PyObject_GetState(item) for item in self) return PyListType, state also note that it makes the copy module much simpler: def copy(obj): state = obj.__getstate__() return type(obj).__setstate__(state) - - - - - - - - executive summary: simplifying object serialization and copying by revising __getstate__ and __setstate__, so that they return a "simplified" version of the object. this new mechanism should become an official API to getting or setting the "contents" of objects (either builtin or user-defined). having this unified mechanism, pickling proxy objects would work as expected. if there's interest, i'll write a pep-like document to explain all the semantics. -tomer [View Less]

4 14