Mailman 3 January 2007 - Python-ideas

fixing mutable default argument values
by Chris Rebert May 2, 2007

May 2, 2007

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)

9 25

tab completion in pdb
by stephen emslie Feb. 8, 2007

Feb. 8, 2007

I make frequent use of the python's built-in debugger, which I think is brilliant in its simplicity. However an important feature seems to be missing: bash-like tab completion similar to that provided by the rlcompleter module. By default, Pdb and other instances of Cmd complete names for commands only. However in the context of pdb, I think it is more useful to complete identifiers and keywords in its current scope than to complete names of commands (most of which have single letter abbreviations). I believe this makes pdb a far more usable introspection tool. Implementation: I've attached a patch to pdb.py (on Python 2.4.4c1). The only real difference to rlcompleter's default complete method is that because pdb changes scope as you step through a program, rlcompleter's namespace is updated to reflect the current local and global namespace. This is my first attempt at a python patch. Any suggestions or improvements are welcome. Stephen Emslie

2 4

proto-PEP: Fixing Non-constant Default Arguments
by Chris Rebert Feb. 3, 2007

Feb. 3, 2007

The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=<baz>): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/

10 42

Import and '..', '../..' in serach path.
by Ron Adam Feb. 1, 2007

Feb. 1, 2007

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package. Module in package: import sys sys.path = ['..'] + sys.path import package.module # Imports module in "this!" package. Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of the path. Module in sub-package: import sys sys.path = ['../..'] + sys.path import package.subpackage.module # finds "self" (subpackage) reliably. By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path. Adding this tip to the documentation some where would be nice. (providing there is no major surprising side effects.) Of course I may have missed some obvious way to do this. If so, it wasn't in an obvious place to be found. I looked. ;-) ---------------------------------- It might be useful to have a built-in function to do this. A function could also check for __init__ files and raise errors if they are missing. set_package_name(dotted.name) # Replaces import sys & path modification Where dotted.name is the full package + sub-package name the current module is located in. The function would search upwards to get the root package directory and add that to the *front* of sys.path. Module in package: set_package_name('package') # Add parent directory to front of sys.path import packagename.module # Finds module in "this!" package reliably. Module in subpackage: set_package_name('package.subpackage') import package.subpackage.module # Finds "self" (subpackage) reliably. ---------------------------------- It may also be able to modify the import behavior to allow relative imports to work when the module is run as script. set_package_name('package') from . import module1 # Imports modules from "this" package. from . import module2 Currently an exception is raised you try to run a module with relative references as a script. ValueError: Attempted relative import in non-package I think it is very handy to be able to run tests as scripts and keep them in a sub-package. Especially while I'm writing them. Another benefit of using relative imports with an absolute specified package name, is if you rename a package or relocate a submodule, you only have one line to change. Cheers, Ron

2 7

Mutable default arguments - another approach
by Piotr Duda Jan. 31, 2007

Jan. 31, 2007

I think that this problem can be solved by the following change of default argument behavior: default arguments are evaluated in definition time (like in current implementation), but right after being evaluated, result object is checked if it's mutable (for example by checking of presence __copy__ special method or being instance of built in (sub)class list/dict/set), if object is mutable, argument is marked by COPY_DEF_ARG flag. There are two reasons for this check being done there: 1. performance 2. it can be controlled by "from __future__ import ..." statement in per-file manner Then if default argument is needed in function call, first COPY_DEF_ARG flag is checked if not set, default argument behaves exactly like in current implementation, if the flag is set, it's shallow copy is used instead. Adding following classes/functions to stdlib allow reproduce old behavior as well as add some new. class IterateDefaultArg(object): def __init__(self, iterator): self.__iterator = iterator def __copy__(self): return self.__iterator.next() class DefaultArgWrapper(object): def __init__(self, generatorfunc): self.__generatorfunc = generatorfunc def __call__(self, *args, **kwargs): return DefaultArgObject(self.__generatorfunc(*args, **kwargs)) @DefaultArgWrapper def nocopyarg(obj): while 1: yield obj With this current definition like: def foo(cache = {}): ... need to be replaced by: def foo(cache = nocopyarg({})): ... If one want to use deep copy instead copy, it might be done like this: @DefaultArgWrapper def deepcopyarg(obj): from copy import deepcopy while 1: yield deepcopy(obj) def foo(x = deepcopyarg(<some expression>)): ... P.S. sorry for my bad English -- 闇に隠れた黒い力弱い心を操る

6 7

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by Eduardo "EdCrypt" O. Padoan Jan. 30, 2007

Jan. 30, 2007

On 1/30/07, Eduardo EdCrypt O. Padoan <eopadoan(a)altavix.com> wrote: > > If I have time and figure out the right regexes I'll try and come up with > > some more numbers on the entire stdlib, and the ammount of uses of =None. > > > > Some uses of the spam=[] and ham=None in Python projects, including > Python itself: > > http://www.google.com/codesearch?q=def.*%5C(.*%3D%5C%5B%5C%5D.*%5C)%3A%20la… > http://www.google.com/codesearch?hl=en&lr=&q=def.*%5C%28.*%3DNone.*%5C%29%3… > > In this second search, I need a way to search that, in the body of the > function, we have something like "if foo is not None: foo = []" (and > foo = {} too) > > -- > EduardoOPadoan (eopadoan->altavix::com) > Bookmarks: http://del.icio.us/edcrypt > Blog: http://edcrypt.blogspot.com > Jabber: edcrypt at jabber dot org > ICQ: 161480283 > GTalk: eduardo dot padoan at gmail dot com > MSN: eopadoan at altavix dot com > -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com

1 0

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

there really is no need to have something like that built into the language. most default arguments are immutable anyhow, and i'd guess most of the mutable defaults can be addressed with the suggested @copydefaults decorator. as for uncopyable or stateful defaults (dev = sys.stdout), which require reevaluation, you can just use this modified version of my first suggested decorator: >>> def reeval(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k, v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = v() # <--- this is the big change ... return func(*args, **kwargs) ... return wrapper ... return deco ... the defaults are now provided as *functions*, which are evaluated at the time of calling the function. >>> @reeval(device = lambda: sys.stdout) ... def say(text, device): ... device.write("%s: %s\n" % (device.name, text)) ... device.flush() this means you can do things like -- >>> say("hello1") <stdout>: hello1 >>> >>> say("hello2", device = sys.stderr) <stderr>: hello2 >>> >>> sys.stdout = sys.stderr >>> say("hello3") <stderr>: hello3 >>> >>> sys.stdout = sys.__stdout__ >>> say("hello4") <stdout>: hello4 decorators are powerful enough for this not-so-common case. it would be nice to have a collection of useful decorators as part of stdlib (such as these ones, but including memoize and many others)... but that's a different issue. maybe we should really start a list of useful decorators to be included as an stdlib module. -tomer

1 0

simplification pep-like thing
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

this text needs more thought and rephrasing, but i think that it covers all the details. -------------------------------------------------------------------------- Abstract ========= The simplification API provides a mechanism to get the state of an object, as well as reconstructing an object with a given state. This new API would affect pickle, copy_reg, and copy modules, as well as the writers of serializers. The idea is to separate the state-gathering from the encoding: * State-gathering is getting the state of the object * Encoding is the converting the state (which is an object) into a sequence of bytes (textual or binary) This is somewhat similar to the ISerializable interface of .NET, which defines only GetObjectData(); the actual encoding is done by a Formatter class, which has different implementations for SOAP-formatting, binary-formatting, and other formats. Motivation ========== There are many generic and niche serializers out there, including pickle, banana/jelly, Cerealizer, brine, and lots of serializers targeting XML output. Currently, all serializers have their own methods of retrieving the contents, or state of the object. This API attempts to solve this issue by providing standard means for getting object state, and creating objects with their state restored. Another issue is making the serialization process "proxy-friendly". Many frameworks use object proxies to indirectly refer to another object (for instance, RPC proxies, FFI, etc.). In this case, it's desirable to simplify the referenced object rather than the proxy, and this API addresses this issue too. Simplification ============== Simplification is the process of converting a 'complex' object into its "atomic" components. You may think of these atomic components as the *contents* of the object. This proposal does not state what "atomic" means -- this is open to the decision of the class. The only restriction imposed is, collections of any kind must be simplified as tuples. Moreover, the simplification process may be recursive: object X may simplify itself in terms of object Y, which in turn may go further simplification. Simplification Protocol ======================= This proposal introduces two new special methods: def __simplify__(self): return type, state @classmethod def __rebuild__(cls, state): return new_instance_of_cls_with_given_state __simplify__ takes no arguments (bar 'self'), and returns a tuple of '(type, state)', representing the contents of 'self': * 'type' is expected to be a class or a builtin type, although it can be any object that has a '__rebuild__' method. This 'type' will be used later to reconstruct the object (using 'type.__rebuild__(state)'). * 'state' is any other that represents the inner state of 'self', in a simplified form. This can be an atomic value, or yet another complex object, that may go further simplification. __rebuild__ is a expected to be classmethod that takes the state returned by __simplify__, and returns a new instance of 'cls' with the given state. If a type does not wish to be simplified, it may throw a TypeError in its __simplify__ method; however, this is not recommended. Types that want to be treated as atomic elements, such as file, should just return themselves, and let the serializer handle them. Default Simplification ===================== All the built in types would grow a __simplify__ and __rebuild__ methods, which would follow these guidelines: Primitive types (int, str, float, ...) are considered atomic. Composite types (I think 'complex' is the only type), are broken down into their components. For the complex type, that would be a tuple of (real, imaginary). Container types (tuples, lists, sets, dicts) represent themselves as tuples of items. For example, dicts would be simplified according to this pseudocode: def PyDict_Simplifiy(PyObject * self): return PyDictType, tuple(self.items()) Built in types would be considered atomic. User-defined classes can be simplified into their metaclass, __bases__, and __dict__. The type 'object' would simplify instances by returning their __dict__ and any __slots__ the instance may have. This is the default behavior; classes that desire a different behavior would override __simplify__ and __rebuild__. Example of default behavior: >>> class Foo(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... >>> f = Foo() >>> cls, state = f.__simplify__() >>> cls <class '__main__.Foo'> >>> state {"a" : 5, "b" : "spam"} >>> shallow_copy = cls.__rebuild__(state) >>> state.__simplify__() (<type 'dict'>, (("a", 5), ("b", "spam"))) Example of customized behavior >>> class Bar(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... def __simplify__(self): ... return Bar, 17.5 ... @clasmethod ... def __rebuild__(cls, state) ... self = cls.__new__(cls) ... if state == 17.5: ... self.a = 5 ... self.b = "spam" ... return self ... >>> b = Bar() >>> b.__simplify__() (<class '__main__.Bar'>, 17.5) Code objects ============= I wish that modules, classes and functions would also be simplifiable, however, there are some issues with that: * How to serialize code objects? These can be simplified as tuple of their co_* attributes, but these attributes are very implementation- specific. * How to serialize cell variables, or other globals? It would be nice if .pyc files where generated like so: import foo pickle.dump(foo) It would also allow sending of code between machines, just like any other object. Copying ======== Shallow copy, as well as deep copy, can be implemented using the semantics of this new API. The copy module should be rewritten accordingly: def copy(obj): cls, state = obj.__simplify__() return cls.__rebuild__(state) deepcopy() can be implemented similarly. Deprecation =========== With the proposed API, copy_reg, __reduce__, __reduce_ex__, and possibly other modules become deprecated. Apart from that, the pickle and copy modules need to be updated accordingly. C API ====== The proposal introduces two new C-API functions: PyObject * PyObject_Simplify(PyObject * self); PyObject * PyObject_Rebuild(PyObject * type, PyObject * state); Although this is only a suggestion. I'd like to hear ideas from someone with more experience in the core. I do not see a need for a convenience routine such as simplify(obj) <--> obj.__simplify__(), since this API is not designed for everyday usage. This is the case with __reduce__ today. Object Proxying =============== Because __simplify__ returns '(type, state)', it may choose to "lie" about it's actual type. This means that when the state is reconstructed, another type is used. Object proxies will use this mechanism to serialize the referred object, rather than the proxy. class Proxy(object): [...] def __simplify__(self): # this returns a tuple with the *real* type and state return self.value.__simplify__() Serialization ============ Serialization is the process of converting fully-simplified objects into byte sequences (strings). Fully simplified objects are created by a recursive simplifier, that simplifies the entire object graph into atomic components. Then, the serializer would convert the atomic components into strings. Note that this proposal does not define how atomic objects are to be converted to strings, or how a 'recursive simplifier' should work. These issues are to be resolved by the implementation of the serializer. For instance, file objects are atomic; one serializer may be able to handle them, by storing them as (filename, file-mode, file-position), while another may not be, so it would raise an exception. Recursive Simplifier =================== This code demonstrates the general idea of how recursive simplifiers may be implemented: def recursive_simplifier(obj): cls, state = obj.__simplify__() # simplify all the elements inside tuples if type(state) is tuple: nested_state = [] for item in state: nested_state.append(recursive_simplifier(item)) return cls, nested_state # see if the object is atomic; if not, dig deeper if (cls, state) == state.__simplify__(): # 'state' is an atomic object, no need to go further return cls, state else: # this object is not atomic, so dig deeper return cls, recusrive_simplifier(state) -tomer

3 3

my take on mutable default arguments
by tomer filiba Jan. 28, 2007

Jan. 28, 2007

i thought this code be solved nicely with a decorator... it needs some more work, but it would make a good cookbook recipe: .>>> from copy import deepcopy. .>>> .>>> def defaults(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k,v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = deepcopy(v) ... return func(*args, **kwargs) ... return wrapper ... return deco ... .>>> @defaults(x = []) ... def foo(a, x): ... x.append(a) ... print x ... .>>> foo(5) [5] .>>> foo(5) [5] .>>> foo(5) [5] maybe it should be done by copying func_defaults... then it could be written as @copydefaults def f(a, b = 5, c = []): ... -tomer

3 4

new pickle semantics/API
by tomer filiba Jan. 26, 2007

Jan. 26, 2007

---------- Forwarded message ---------- From: tomer filiba <tomerfiliba(a)gmail.com> Date: Jan 24, 2007 10:45 PM Subject: new pickle semantics/API To: Python-3000(a)python.org i'm having great trouble in RPyC with pickling object proxies. several users have asked for this feature, but no matter how hard i try to "bend the truth", pickle always complains. it uses type(obj) for the dispatching, which "uncovers" the object is actually a proxy, rather than a real object. recap: RPyC uses local proxies that refer to objects of a remote interpreter (another process/machine). if you'd noticed, every RPC framework has its own serializer. for example banna/jelly in twisted and bunch of other XML serializers, and what not. for RPyC i wrote yet another serializer, but for different purposes, so it's not relevant for the issue at hand. what i want is a standard serialization *API*. the idea is that any framework could make use of this API, and that it would be generic enough to eliminate copy_reg and other misfortunes. this also means the built in types should be familiarized with this API. - - - - - - - - for example, currently the builtin types don't support __reduce__, and require pickle to use it's own internal registry. moreover, __reduce__ is very pickle-specific (i.e., it takes the protocol number). what i'm after is an API for "simplifying" complex objects into simpler parts. here's the API i'm suggesting: def __getstate__(self): # return a tuple of (type(self), obj), where obj is a simplified # version of self @classmethod def __setstate__(cls, state): # return an instance of cls, with the given state well, you may already know these two, although their semantics are different. but wait, there's more! the idea is of having the following simple building blocks: * integers (int/long) * strings (str) * arrays (tuples) all picklable objects should be able to express themselves as a collection of these building blocks. of course this will be recursive, i.e., object X could simplify itself as object Y, where object Y might go further simplification, until we are left with building blocks only. for example: * int - return self * float - string in the format "[+-]X.YYYe[+-]EEE" * complex - two floats * tuple - tuple of its simplified elements * list - tuple of its simplified elements * dict - a tuple of (key, value) tuples * set - a tuple of its items * file - raises TypeError("can't be simplified") all in all, i choose to call that *simplification* rather than *serialization*, as serialization is more about converting the simplified objects into a sequence of bytes. my suggestion leaves that out for the implementers of specific serializers. so this is how a typical serializer (e.g., pickle) would be implemented: * define its version of a "recursive simplifier" * optionally use a "memo" to remember objects that were already visited (so they would be serialized by reference rather than by value) * define its flavor of converting ints, strings, and arrays to bytes (binary, textual, etc. etc.) - - - - - - - - the default implementation of __getstate__, in object.__getstate__, will simply return self.__dict__ and any self.__slots__ this removes the need for __reduce__, __reduce_ex__, and copy_reg, and simplifies pickle greatly. it does require, however, adding support for simplification for all builtin types... but this doesn't call for much code: def PyList_GetState(self): state = tuple(PyObject_GetState(item) for item in self) return PyListType, state also note that it makes the copy module much simpler: def copy(obj): state = obj.__getstate__() return type(obj).__setstate__(state) - - - - - - - - executive summary: simplifying object serialization and copying by revising __getstate__ and __setstate__, so that they return a "simplified" version of the object. this new mechanism should become an official API to getting or setting the "contents" of objects (either builtin or user-defined). having this unified mechanism, pickling proxy objects would work as expected. if there's interest, i'll write a pep-like document to explain all the semantics. -tomer

4 14

oneof() and multi split and replace for stirngs
by Calvin Spealman Jan. 23, 2007

Jan. 23, 2007

I don't know if this would make sense to try to push to 2.6 or 3.0. I was talking with some people about how the ability to split or replace on multiple substrings would be added to python, without adding new methods or having ugly tuple passing requirents like s.split(('foo', 'bar'), 4). This idea came to mind, so I wanted to toss it out there for scrutination. It would be a builtin, but can be implemented in python like this, basically: class oneof(list): def __init__(self, *args): list.__init__(self) self.extend(args) def __eq__(self, o): return o in self assert 'bar' == oneof('bar', 'baz') In addition to the new type, .replace, .split, and other appropriate functions would be updated to take this as the substring argument to locate and would match any one of the substrings it contains. I've asked a few people and gotten good responses on the general idea so far, but what do you all think? 1) Would the multi-substring operations be welcomed? 2) Could this be a good way to add those to the API without breaking things? 3) What version would it target? 4) What all functions and methods should support this or generally might gain value from some similar solution? -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/

3 2

Re: [Python-ideas] oneof() and multi split and replace for stirngs
by Calvin Spealman Jan. 23, 2007

Jan. 23, 2007

On 1/23/07, Josiah Carlson <jcarlson(a)uci.edu> wrote: > Whether it is a tuple being passed, or a "magic" container, I don't > think it matters; though I would lean towards a tuple because it is 5 > less characters to type out, and one fewer data types to worry about. I had talked to others and the concensus was against the tuples for ugliness. s.split((a,b), c) wasn't a popular choice, but s.split(oneof(a, b), c) reads better. > This has been discussed before in python-dev, I believe the general > consensus was that it would be convenient at times, but I also believe > the general consensus was "use re"; It seems like we have a history of useful string operations being moved away from "use re" to "dont use re", such that the slightly recent startswith and endswith methods, and even split and replace themselves. I would like to see less reasons for people to worry with regular expressions until they actually need them. If we can provide a better way to get the job done, that seems like a great idea. ALSO The oneof type isn't just a single use thing. Usecode may often make use of it, and other types could benefit such as doing a lookup with d[oneof(1,2,3)] (where order would matter for priority). I think this semantic collection type would be very useful in a number of contexts where we would currently just loop or duplicate code. -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/

2 1

Fwd: multi-dispatch again
by tomer filiba Jan. 14, 2007

Jan. 14, 2007

Guido asked me to move it here. anyway, i might as well state my case better. in the formal definitions of object oriented programming, objects are said to encapsulate state and behavior. behavior is largely dependent on the type of the object, but the state is important nonetheless. for example, file objects are stateful: they can be opened for reading-only, writing-only, both, or be closed altogether. still, they are all instances of the file type. since generic functions/multi-dispatch come to help the programmer by reducing boilerplate early type-checking code and providing a more capable dispatch mechanism -- we can't overlook the state of the object. this early type-checking goes against the spirit of pure duck typing (which i'm fond of), but is crucial when the code has side effects. in this case, you can't just start executing the code and "hope it works", as the resources (i.e., files) involved are modified. in this kind of code, you want to check everything is alright *instead* of suddenly having an AttributeError/TypeError somewhere. here's an example: @dispatch def copy(src: file, dst: file): while True: buf = src.read(1000) if not buf: break dst.write(buf) suppose now that dst is mistakenly opened for reading. this means src would have already been modified, while dst.write is bound to fail. if src is a socket, for instance, this would be destructive. so if we already go as far as having multiple dispatch, which imposes constraints on the arguments a function accepts, we might as well base it on the type and state of the object, rather than only on it's type. we could say, for example: @dispatch def copy(src: file_for_reading, dst: file_for_writing): file_for_reading is not a type -- it's a checker. it may be defined as def file_for_reading(obj: file): return file.mode == "r" and not file.closed types, by default, would check using isinstance(), but costume checkers could check for stateful requirements too. and a note about performance: this check is required whether it's done explicitly or by the dispatch mechanism, and since most functions don't require so many overloads, i don't think it's an issue. besides, we can have a different decorator for dispatching by type or by checkers, i.e., @dispatch vs @stateful_dispatch, or something. the simple @dispatch would use a dictionary, while the stateful version would use a loop. -tomer ---------- Forwarded message ---------- From: tomer filiba <tomerfiliba(a)gmail.com> Date: Jan 14, 2007 2:28 PM Subject: multi-dispatch again To: Python-3000(a)python.org i just thought of a so-to-speak counter-example for ABCs... it's not really a counter-example, but i believe it shows a deficiency in the concept. theoretically speaking, objects are a made of type and state. ABCs, isinstance() and interfaces at general only check the type part. for example: @dispatch def log_to_file(text: str, device: file): file.write(text) this will constrain the *type* of the device, but not its *state*. practically speaking, i can pass a closed file, or a file open for reading-only, and it would pass silently. basing multi-dispatch on types is of course a leap forward, but if we already plan to take this leap, why not make it general enough to support more complex use-cases? this way we could rewrite the snippet above as @dispatch def log_to_file(text: str, device: open_file): file.write(text) where open_file isn't a type, but rather a "checker" that may also examine the state. by default, type objects would check for inheritance (via a special method), but checkers could extend this behavior. for efficiency purposes, we can have two decorators: @type_dispatch - dispatches based on type only @full_dispatch - dispatches based on type and state bottom line -- we can't just look at the type of the object for dispatching, overlooking its state. the state is meaningful, and we'd want the function not to be called at all if the state of the object is wrong. -tomer

1 0

Re: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ... #3: fix super()
by Eduardo "EdCrypt" O. Padoan Jan. 6, 2007

Jan. 6, 2007

> I was bitten by the urge to play with this today, and modified my > previous "self" hack to handle "super" also, so that the following > code works: > > class D (C): > @method > def sum(n): > return super.sum(n * 2) - self.base > > Posted as "evil2.py" here: > > http://www.lag.net/robey/code/surf/ > > Because hacking "super" requires having the class object handy, this > one needs a metaclass to do its magic, which is a shame. I guess if > it was implemented inside the cpython compiler, it would be less of a > problem. BTW, a "super-only" version of this decortor (that I think could be called "implement") has some more chances in Python. But think belongs more to Python-Ideas list, ok? -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com

1 1

Module local namespaces
by Matt Draisey Jan. 4, 2007

Jan. 4, 2007

A while ago, I developed a small PyGtk programme that could dynamically reload all the working callbacks and logic while the GUI was still running. I could get away with this because of the flexible way Python loads modules at runtime, but it ended up being a waste of time as implementing it took more time that actually using it. For sanity's sake it quickly becomes clear you almost never want to rely on being able to refer to a half initialized module. And wouldn't it be nice if Python enforced this. My suggestion is that module importing occur in a temporary local namespace that exists only until the end of the module code is executed, then a small function could copy everything from the temporary namespace into the module object. The usual closure semantics would guarantee that top-level functions could still call each other, but they would effectively become immutable after the namespace wraps up. The 'global' keyword could be used at the top level in a module to force it to be defined in the module immediately, and to ensure internal references to the object go through the module object. This would be a big change in module import semantics, but should have remarkably few consequences, as it really is an enforcement mechanism for good style. The copying from the temporary namespace into the module object would be a good place to insert a hook function to filter what objects are actually published to the module. You could by default not copy any object indentified by a leading underscore.

4 7

PEP 3107 Function Annotations: interoperability (again)
by Tony Lownds Jan. 3, 2007

Jan. 3, 2007

In discussing the Function Annotations PEP on python-list, interoperability between schemes came up again: http://mail.python.org/pipermail/python-list/2006-December/420645.html John Roth wrote: > Third, it's half of a proposal. Type checking isn't the only use > for metadata about functions/methods, classes, properties > and other objects, and the notion that there are only going to > be a small number of non-intersecting libraries out there is > an abdication of responsibility to think this thing through. This issue came up before in http://mail.python.org/pipermail/python-3000/2006-August/002796.html and a rather long thread followed. Here is the paragraph in the PEP that needs updating, at the least: There is no worry that these libraries will assign semantics at random, or that a variety of libraries will appear, each with varying semantics and interpretations of what, say, a tuple of strings means. The difficulty inherent in writing annotation interpreting libraries will keep their number low and their authorship in the hands of people who, frankly, know what they're doing. The notion that libraries don't intersect should be stripped from the PEP. The question in my mind is whether this PEP needs to take responsibility for interoperability. I contend that people who design an annotation-consuming library that ends up intersecting several others will likely be capable of finding a solution even if not ideal without a central mechanism, and will be far better situated to define a solution for a central mechanism. Any thoughts? Thanks -Tony

4 6

Re: [Python-ideas] new operators via backquoting
by George Sakkis Jan. 3, 2007

Jan. 3, 2007

Chris Rebert wrote: > In Haskell, foo `baz` bar means (baz foo bar), which translates to > baz(foo, bar) in Python. This allows Haskell programmers to use > functions as infix operators. > If I recall correctly, in Py3k, enclosing something in backticks will no > longer cause it to be repr()-ed, leaving the backtick without a meaning > in Python. Perhaps you're not aware of the "best hack of 2005", which works in current Python and goes a step further by allowing you to even "override" the operator's delimiter: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122 George

2 1

new operators via backquoting
by Chris Rebert Jan. 3, 2007

Jan. 3, 2007

In Haskell, foo `baz` bar means (baz foo bar), which translates to baz(foo, bar) in Python. This allows Haskell programmers to use functions as infix operators. If I recall correctly, in Py3k, enclosing something in backticks will no longer cause it to be repr()-ed, leaving the backtick without a meaning in Python. Thus, I propose one of the following as the new use for the backtick (`): [Note: In both, the characters between the backticks must be a valid Python identifier.] (A) `baz` is treated as an operator, named "baz", just as / is "div". foo `baz` bar thus causes python to try to call foo.__baz__(bar), and failing that, bar.__rbaz__(foo), and if both those fail, raise TypeError. This is, if I understand correctly, how the builtin operators work. (B) `baz` is a special way to call a callable. foo `baz` bar is translated to baz(foo, bar) with the standard lookup rules for resolving "baz" Example use cases, stolen from Haskell: The Craft of Functional Programming: 2 `max` 5 => 5 7 `cons` tail => ConsCell(val=7, next=tail) matrix1 `crossproduct` matrix2 => cross-product of the matrices [1, 2, 3] `zip` ['a', 'b', 'c'] => [[1, 'a'], [2, 'c'], [3, 'c']] I believe that this would improve the readability of code, such as Numeric, without going off the deep end and offering programmable syntax. - Chris Rebert

5 4

Attribute Docstrings and Annotations
by Tony Lownds Jan. 3, 2007

Jan. 3, 2007

I'd like to propose annotations and docstrings on attributes for Python 3000 with the following Grammar and properties: expr_stmt: test (':' test ['=' (yield_expr|testlist)] | augassign (yield_expr|testlist) | [',' testlist] ('=' (yield_expr|testlist))* ) * annotations can appear for attributes that are not defined yet * code to generate and populate __annotations__ and __attrdoc__ would appear for all modules and class bodies, not for functions. * attribute annotations allow a name as target only * attribute annotations without assignment are illegal for functions * attribute annotations with assignment should probably be illegal for functions * docstring annotations only apply to the first target, and only if it is a name * docstring annotations do not apply to augmented assignments * docstring and annotations on functions do not get populated in __annotations__ * the class docstring is not reused as a function docstring The basic rationale for annotations on attributes is completeness with PEP3107. I'm proposing attribute docstrings as well because I think it's preferable to have a spot for documentation that isn't tied to annotations, like functions' __doc__. Also attribute docstrings as specified look similar to function docstrings; both have statements consisting of only a string that is taken as documentation. Here is an interactive session and the code that might be generated. What do you think? Thanks -Tony >>> class X: ... "class docstring" ... foo: 1 = 1 ... bar: 2 ... "attribute docstring" ... attr = None ... "another attribute docstring" ... fields = __slots__ = ['fields', 'attr'] ... "docstring ignored" ... x, y = 1, 2 ... >>> X.__attrdoc__ {'fields': 'another attribute docstring', 'attr': 'attribute docstring'} >>> X.__annotations__ {'foo': 1, 'bar': 2} >>> X.foo 1 >>> X.bar Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: type object 'X' has no attribute 'bar' >>> def f(): ... x: 1 ... File "<stdin>", line 2 SyntaxError: annotation without assignment not allowed in function 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 6 BUILD_MAP 0 9 STORE_NAME 2 (__annotations__) 12 BUILD_MAP 0 15 STORE_NAME 3 (__attrdoc__) 3 18 LOAD_CONST 0 ('class docstring') 21 STORE_NAME 4 (__doc__) 5 24 LOAD_CONST 1 ('attribute docstring') 27 LOAD_NAME 3 (__attrdoc__) 30 LOAD_CONST 2 ('attr') 33 STORE_SUBSCR 34 LOAD_NAME 5 (None) 37 STORE_NAME 6 (attr) 6 40 LOAD_CONST 3 (1) 48 STORE_NAME 7 (foo) 44 LOAD_CONST 3 (1) 51 LOAD_NAME 2 (__annotations__) 54 LOAD_CONST 4 ('foo') 57 STORE_SUBSCR 7 58 LOAD_CONST 5 (2) 61 LOAD_NAME 2 (__annotations__) 64 LOAD_CONST 6 ('bar') 67 STORE_SUBSCR 9 68 LOAD_CONST 7 ('another attribute docstring') 71 LOAD_NAME 3 (__attrdoc__) 74 LOAD_CONST 8 ('fields') 77 STORE_SUBSCR 78 LOAD_CONST 8 ('fields') 81 LOAD_CONST 2 ('attr') 84 BUILD_LIST 2 87 DUP_TOP 88 STORE_NAME 8 (fields) 91 STORE_NAME 9 (__slots__) 94 LOAD_LOCALS 95 RETURN_VALUE

6 7

Re: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable ->
by Josiah Carlson Jan. 2, 2007

Jan. 2, 2007

To Tony and Kay, my short answer is: use __returns__ . Moving to python-ideas as per Guido's request: "Guido van Rossum" <guido(a)python.org> wrote: > This is sufficiently controversial that I believe it ought to go to > python-ideas first. If it comes to a PEP it should be a separate one > from PEP 3107. > On 1/1/07, Talin <talin(a)acm.org> wrote: > > Tony Lownds wrote: > > >> From: Tony Lownds <tony(a)pagedna.com> > > > What do people here think? > > 1) Normally, we don't name operators based on their shape - we don't > > call '/' the __slash__ operator, for example, nor do we call '|' the > > "__vbar__" operator. Certainly, but those two operators, and basically every other operator used in Python have long-standing semantics in basically every language that Python is even remotely related to. > > 2) I think that all operators should have a "suggested semantic". When > > someone overloads the '+' operator, its a good bet that the meaning of > > the overload has something to do with addition or accumulation in a > > general sense. This won't *always* be true, but it will be true often > > enough. I don't buy your "suggested semantic" argument. And even if I did, operator overloading allows people to choose the semantics of operations for themselves; suggesting a semantic for an operation would be a set of documentation that would never be read, and if it was read, ignored. > > But an arbitrary operator with no guidelines as to what it means is > > anyone's guess; It means that when we see a '->' operator embedded in > > the code, we have no idea what is being said. Ahh, but in this case there is precisely one place where the '->' operator is planned on being found for Py3k: def <name>(<arglist with or without annotations>) -> <annotation>: <body> In that sense, we don't need a fcn_obj.__becomes__(<annotation>) method, and that wasn't what the discussion was about, it was "what should the attribute be called for this already agreed upon *function annotation*?". Now, because this *particular* annotation was created to allow for the annotation of "returns", I agree with Kay's last suggestion, the attribute should be called __returns__, as that is *exactly* what the annotation was meant to convey. > > From an HCI perspective, punctuation symbols improve code readability > > only if their meanings are familiar to the reader; An operator whose > > meaning is constantly changing is a hindrance to readability rather than > > a help. Claiming "we want a particular operation to always refer to the same method/attribute" is only applicable if a particular operation has a chance of meaning more than one thing. Currently it means *exactly* one thing in the context of Python, 'this function returns X', so in my opinion, your argument isn't applicable. If you can manage to convince more people in python-ideas of arbitrary operations (as per your previous message(s) on the subject), and/or you can convince Guido to say "I would like more operations in Python", then your argument is applicable. However, I don't believe that you will be able to convince Guido that the large set of operations that you have previously posted about would be a good idea, and I certainly don't believe it would happen with sufficient time to make it into the first Py3k release. - Josiah

6 10