Mailman 3 February 2007 - Python-ideas

New syntax for 'dynamic' attribute access
by Ben North 09 Feb '07

09 Feb '07

Hi, I'd like to describe an addition I made to Python syntax which allows easier access to attributes where the attribute name is only known at run-time. For example: setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) As noted in the PEP-style description below, I mostly did this for fun, but I thought it might be worth bringing to the attention of python-ideas. A quick search through prior postings and Google for this idea didn't come up with anything. Ben. - - - - 8< - - - - PEP: XXX Title: Syntax For Dynamic Attribute Access Version: $Revision$ Last-Modified: $Date$ Author: Ben North <ben(a)redfrontdoor.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 29-Jan-2007 Post-History: Abstract Dynamic attribute access is currently possible using the "getattr" and "setattr" builtins. The present PEP suggests a new syntax to make such access easier, allowing the coder for example to write x.('foo_%d' % n) += 1 z = y.('foo_%d' % n).('bar_%s' % s) instead of attr_name = 'foo_%d' % n setattr(x, attr_name, getattr(x, attr_name) + 1) z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s) Note (I wrote this patch mostly to advance my own understanding of and experiment with the python language, but I've written it up in the style of a PEP in case it might be a useful idea.) Rationale Dictionary access and indexing both have a friendly invocation syntax: instead of x.__getitem__(12) the coder can write x[12]. This also allows the use of subscripted elements in an augmented assignment, as in "x[12] += 1". The present proposal brings this ease-of-use to dynamic attribute access too. Attribute access is currently possible in two ways: * When the attribute name is known at code-writing time, the ".NAME" trailer can be used, as in x.foo = 42 y.bar += 100 * When the attribute name is computed dynamically at run-time, the "getattr" and "setattr" builtins must be used: x = getattr(y, 'foo_%d' % n) setattr(z, 'bar_%s' % s, 99) The "getattr" builtin also allows the coder to specify a default value to be returned in the event that the object does not have an attribute of the given name: x = getattr(y, 'foo_%d' % n, 0) This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in: x = y.('foo_%d' % n, None) This 2-argument form of dynamic attribute access is not permitted as the target of an (augmented or normal) assignment. Finally, the new syntax can be used with the "del" statement, as in del x.(attr_name) Impact On Existing Code The proposed new syntax is not currently valid, so no existing well-formed programs have their meaning altered by this proposal. Across all "*.py" files in the 2.5 distribution, there are around 600 uses of "getattr", "setattr" or "delattr". They break down as follows (figures have some room for error because they were arrived at by partially-manual inspection): c.300 uses of plain "getattr(x, attr_name)", which could be replaced with the new syntax; c.150 uses of the 3-argument form, i.e., with the default value; these could be replaced with the 2-argument form of the new syntax (the cases break down into c.125 cases where the attribute name is a literal string, and c.25 where it's only known at run-time); c.5 uses of the 2-argument form with a literal string attribute name, which I think could be replaced with the standard "x.attribute" syntax; c.120 uses of setattr, of which 15 use getattr to find the new value; all could be replaced with the new syntax, the 15 where getattr is also involved would show a particular increase in clarity; c.5 uses which would have to stay as "getattr" because they are calls of a variable named "getattr" whose default value is the builtin "getattr"; c.5 uses of the 2-argument form, inside a try/except block which catches AttributeError and uses a default value instead; these could use 2-argument form of the new syntax; c.10 uses of "delattr", which could use the new syntax. As examples, the line setattr(self, attr, change_root(self.root, getattr(self, attr))) from Lib/distutils/command/install.py could be rewritten self.(attr) = change_root(self.root, self.(attr)) and the line setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) Alternative Syntax For The New Feature Other syntaxes could be used, for example braces are currently invalid in a "trailer", so could be used here, giving x{'foo_%d' % n} += 1 My personal preference is for the x.('foo_%d' % n) += 1 syntax though: the presence of the dot shows there is attribute access going on; the parentheses have an analogous meaning to the mathematical "work this out first" meaning. This is also the syntax used in the language Matlab [1] for dynamic "field" access (where "field" is the Matlab term analogous to Python's "attribute"). Error Cases Only strings are permitted as attribute names, so for instance the following error is produced: >>> x.(99) = 8 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: attribute name must be string, not 'int' This is handled by the existing PyObject_GetAttr function. Draft Implementation A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests. References [1] Using Dynamic Field Names :: Data Types (MATLAB Programming) http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.… Copyright This document has been placed in the public domain. [PAGE-BREAK GOES HERE BUT REMOVED FOR EMAIL] Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: - - - - 8< - - - - diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Grammar/Grammar Python-2.5/Grammar/Grammar --- ORIG__Python-2.5/Grammar/Grammar 2006-05-25 12:25:51.000000000 +0100 +++ Python-2.5/Grammar/Grammar 2007-02-01 18:07:04.133160000 +0000 @@ -119,7 +119,7 @@ listmaker: test ( list_for | (',' test)* [','] ) testlist_gexp: test ( gen_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test -trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME +trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] sliceop: ':' [test] diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Include/opcode.h Python-2.5/Include/opcode.h --- ORIG__Python-2.5/Include/opcode.h 2006-02-27 22:32:47.000000000 +0000 +++ Python-2.5/Include/opcode.h 2007-02-02 13:36:58.542848000 +0000 @@ -39,6 +39,10 @@ #define SLICE 30 /* Also uses 31-33 */ +/* LOAD_DYNAMIC_ATTR is below because it takes an argument. */ +#define STORE_DYNAMIC_ATTR 36 +#define DELETE_DYNAMIC_ATTR 37 + #define STORE_SLICE 40 /* Also uses 41-43 */ @@ -89,6 +93,9 @@ #define UNPACK_SEQUENCE 92 /* Number of sequence items */ #define FOR_ITER 93 +#define LOAD_DYNAMIC_ATTR 94 /* Whether default given; 0 = no, 1 = yes */ +/* STORE_DYNAMIC_ATTR, DELETE_DYNAMIC_ATTR are above; they take no argument. */ + #define STORE_ATTR 95 /* Index in name list */ #define DELETE_ATTR 96 /* "" */ #define STORE_GLOBAL 97 /* "" */ diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_dynattr.py Python-2.5/Lib/test/test_dynattr.py --- ORIG__Python-2.5/Lib/test/test_dynattr.py 1970-01-01 01:00:00.000000000 +0100 +++ Python-2.5/Lib/test/test_dynattr.py 2007-02-02 17:37:37.982390000 +0000 @@ -0,0 +1,91 @@ +import unittest +import warnings +import sys +from test import test_support + +class AttrHolder: + pass + +class TestDynAttr(unittest.TestCase): + + def test_simple_get(self): + a = AttrHolder() + a.foo = 100 + a.attr_42 = 1 + self.assertEqual(a.('foo'), 100) + self.assertEqual(a.('fo' + 'o'), 100) + self.assertEqual(a.('f' + 'o' + [('o' if True else 'obar')][0]), 100) + self.assertEqual(a.({'FOO': 'fo'}['FOO'] + 'o'), 100) + self.assertEqual(a.('fo%s' % 'o'), 100) + self.assertEqual(a.('attr_42'), 1) + self.assertEqual(a.('attr_%d' % 42), 1) + self.assertEqual(a.('foo' if True else 'attr_42'), 100) + + def test_nested_get(self): + a = AttrHolder() + a.b = AttrHolder() + a.b.c = 1 + attr_name_b = 'b' + attr_name_c = 'c' + self.assertEqual(a.(attr_name_b).(attr_name_c), 1) + + def test_defaulting_get(self): + a = AttrHolder() + a.foo = 100 + self.assertEqual(a.('foo', 99), 100) + self.assertEqual(a.('bar', 99), 99) + self.assertEqual(a.('baz', 99), 99) + self.assertEqual(a.('foo' if True else 'attr_42', 99), 100) + self.assertEqual(a.('foo' if False else 'attr_42', 99), 99) + + @staticmethod + def attempt_non_string_use(attr_name): + a = AttrHolder() + return a.(attr_name) + + def test_only_strings_allowed(self): + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 99) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, None) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 1.0) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, AttrHolder) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, sys) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, ()) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, []) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, {}) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, (1, 2)) + + def test_augassign(self): + a = AttrHolder() + a.foo = 100 + a.('foo') += 10 + self.assertEqual(a.foo, 110) + self.assertEqual(a.('fo' + 'o'), 110) + a.('f' + 'o' + 'o') *= 10 + self.assertEqual(a.foo, 1100) + self.assertEqual(a.('fo' + 'o'), 1100) + a.('foobar'[:3]) /= 5 + self.assertEqual(a.foo, 220) + self.assertEqual(a.('fo' + 'o'), 220) + a.(['foo', 'bar', 'baz'][0]) -= 40 + self.assertEqual(a.foo, 180) + self.assertEqual(a.('fo' + 'o'), 180) + + def test_setattr(self): + a = AttrHolder() + a.('foo') = 99 + self.assertEqual(a.foo, 99) + a.('bar' + '_baz') = 100 + self.assertEqual(a.bar_baz, 100) + + def test_delattr(self): + a = AttrHolder() + a.foo = 99 + del a.('foo') + self.assertEqual(hasattr(a, 'foo'), False) + + +def test_main(): + test_support.run_unittest(TestDynAttr) + +if __name__ == "__main__": + test_main() diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_syntax.py Python-2.5/Lib/test/test_syntax.py --- ORIG__Python-2.5/Lib/test/test_syntax.py 2006-05-19 07:43:50.000000000 +0100 +++ Python-2.5/Lib/test/test_syntax.py 2007-02-02 17:21:04.991119000 +0000 @@ -235,6 +235,18 @@ >>> f() += 1 Traceback (most recent call last): SyntaxError: illegal expression for augmented assignment (<doctest test.test_syntax[33]>, line 1) + +Outlawed uses of dynamic attributes: + +>>> x.('foo', 0) += 1 +Traceback (most recent call last): +SyntaxError: augmented assignment to 2-argument dynamic-attribute expression not possible (<doctest test.test_syntax[34]>, line 1) +>>> x.('foo', 0) = 1 +Traceback (most recent call last): +SyntaxError: can't assign to 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[35]>, line 1) +>>> del x.('foo', 0) +Traceback (most recent call last): +SyntaxError: can't delete 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[36]>, line 1) """ import re diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Parser/Python.asdl Python-2.5/Parser/Python.asdl --- ORIG__Python-2.5/Parser/Python.asdl 2006-04-04 05:00:23.000000000 +0100 +++ Python-2.5/Parser/Python.asdl 2007-02-02 12:39:36.665151000 +0000 @@ -72,6 +72,7 @@ -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) + | DynamicAttribute(expr value, expr attr, expr? dflt, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ast.c Python-2.5/Python/ast.c --- ORIG__Python-2.5/Python/ast.c 2006-09-05 04:56:01.000000000 +0100 +++ Python-2.5/Python/ast.c 2007-02-02 13:54:51.388446000 +0000 @@ -353,6 +353,13 @@ } e->v.Attribute.ctx = ctx; break; + case DynamicAttribute_kind: + if ((ctx == Store || ctx == Del) + && e->v.DynamicAttribute.dflt) + expr_name = "2-argument form of dynamic-attribute expression"; + else + e->v.DynamicAttribute.ctx = ctx; + break; case Subscript_kind: e->v.Subscript.ctx = ctx; break; @@ -1427,7 +1434,7 @@ static expr_ty ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) { - /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME + /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] */ @@ -1440,8 +1447,30 @@ return ast_for_call(c, CHILD(n, 1), left_expr); } else if (TYPE(CHILD(n, 0)) == DOT ) { + if (TYPE(CHILD(n, 1)) == NAME) return Attribute(left_expr, NEW_IDENTIFIER(CHILD(n, 1)), Load, LINENO(n), n->n_col_offset, c->c_arena); + else { + expr_ty e_val, e_dflt; + REQ(CHILD(n, 1), LPAR); + if (!(e_val = ast_for_expr(c, CHILD(n, 2)))) + return NULL; + + if (TYPE(CHILD(n, 3)) == RPAR) { + assert(NCH(n) == 3); + e_dflt = NULL; + } else { + assert(NCH(n) == 5); + REQ(CHILD(n, 3), COMMA); + REQ(CHILD(n, 5), RPAR); + if (!(e_dflt = ast_for_expr(c, CHILD(n, 4)))) + return NULL; + } + + return DynamicAttribute(left_expr, e_val, e_dflt, Load, + LINENO(n), n->n_col_offset, + c->c_arena); + } } else { REQ(CHILD(n, 0), LSQB); @@ -1964,6 +1993,15 @@ case Attribute_kind: case Subscript_kind: break; + case DynamicAttribute_kind: + if (expr1->v.DynamicAttribute.dflt) { + ast_error(ch, "augmented assignment to " + "2-argument dynamic-attribute " + "expression not possible"); + return NULL; + } + break; + default: ast_error(ch, "illegal expression for augmented " "assignment"); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ceval.c Python-2.5/Python/ceval.c --- ORIG__Python-2.5/Python/ceval.c 2006-08-13 19:10:10.000000000 +0100 +++ Python-2.5/Python/ceval.c 2007-02-08 12:56:39.637479000 +0000 @@ -1787,6 +1787,17 @@ if (err == 0) continue; break; + case STORE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + u = POP(); + err = PyObject_SetAttr(v, w, u); /* v.w = u */ + Py_DECREF(w); + Py_DECREF(v); + Py_DECREF(u); + if (err == 0) continue; + break; + case DELETE_ATTR: w = GETITEM(names, oparg); v = POP(); @@ -1795,6 +1806,14 @@ Py_DECREF(v); break; + case DELETE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + err = PyObject_SetAttr(v, w, (PyObject *)NULL); + Py_DECREF(w); + Py_DECREF(v); + break; + case STORE_GLOBAL: w = GETITEM(names, oparg); v = POP(); @@ -1994,6 +2013,28 @@ if (x != NULL) continue; break; + case LOAD_DYNAMIC_ATTR: + if (oparg) + u = POP(); + else + u = NULL; + w = POP(); + v = TOP(); + x = PyObject_GetAttr(v, w); + if (x == NULL && u != NULL + && PyErr_ExceptionMatches(PyExc_AttributeError)) + { + PyErr_Clear(); + Py_INCREF(u); + x = u; + } + Py_DECREF(v); + Py_DECREF(w); + Py_XDECREF(u); /* This one may be NULL (if no default) */ + SET_TOP(x); + if (x != NULL) continue; + break; + case COMPARE_OP: w = POP(); v = TOP(); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/compile.c Python-2.5/Python/compile.c --- ORIG__Python-2.5/Python/compile.c 2006-08-12 02:45:47.000000000 +0100 +++ Python-2.5/Python/compile.c 2007-02-08 12:55:51.958809000 +0000 @@ -1357,6 +1357,13 @@ case SLICE+3: return -1; + case LOAD_DYNAMIC_ATTR: + return -1 - oparg; + case STORE_DYNAMIC_ATTR: + return -3; + case DELETE_DYNAMIC_ATTR: + return -2; + case STORE_SLICE+0: return -2; case STORE_SLICE+1: @@ -3641,6 +3648,41 @@ return 0; } break; + case DynamicAttribute_kind: + { + int has_default_p = (e->v.DynamicAttribute.dflt != NULL); + + if (e->v.DynamicAttribute.ctx != AugStore) { + VISIT(c, expr, e->v.DynamicAttribute.value); + VISIT(c, expr, e->v.DynamicAttribute.attr); + if (has_default_p) + VISIT(c, expr, e->v.DynamicAttribute.dflt); + } + switch (e->v.DynamicAttribute.ctx) { + case AugLoad: + assert(!has_default_p); + ADDOP_I(c, DUP_TOPX, 2); + /* Fall through to Load */ + case Load: + ADDOP_I(c, LOAD_DYNAMIC_ATTR, has_default_p); + break; + case AugStore: + ADDOP(c, ROT_THREE); + /* Fall through to Store */ + case Store: + assert(!has_default_p); + ADDOP(c, STORE_DYNAMIC_ATTR); + break; + case Del: + ADDOP(c, DELETE_DYNAMIC_ATTR); + break; + default: + PyErr_SetString(PyExc_SystemError, + "invalid context in dynamic-attribute expression"); + return 0; + } + break; + } case Subscript_kind: switch (e->v.Subscript.ctx) { case AugLoad: @@ -3700,6 +3742,18 @@ auge->v.Attribute.ctx = AugStore; VISIT(c, expr, auge); break; + case DynamicAttribute_kind: + assert(e->v.DynamicAttribute.dflt == NULL); + auge = DynamicAttribute(e->v.DynamicAttribute.value, e->v.DynamicAttribute.attr, NULL, + AugLoad, e->lineno, e->col_offset, c->c_arena); + if (auge == NULL) + return 0; + VISIT(c, expr, auge); + VISIT(c, expr, s->v.AugAssign.value); + ADDOP(c, inplace_binop(c, s->v.AugAssign.op)); + auge->v.DynamicAttribute.ctx = AugStore; + VISIT(c, expr, auge); + break; case Subscript_kind: auge = Subscript(e->v.Subscript.value, e->v.Subscript.slice, AugLoad, e->lineno, e->col_offset, c->c_arena); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/symtable.c Python-2.5/Python/symtable.c --- ORIG__Python-2.5/Python/symtable.c 2006-08-12 02:43:40.000000000 +0100 +++ Python-2.5/Python/symtable.c 2007-02-08 12:53:36.279608000 +0000 @@ -1192,6 +1192,12 @@ case Attribute_kind: VISIT(st, expr, e->v.Attribute.value); break; + case DynamicAttribute_kind: + VISIT(st, expr, e->v.DynamicAttribute.value); + VISIT(st, expr, e->v.DynamicAttribute.attr); + if (e->v.DynamicAttribute.dflt) + VISIT(st, expr, e->v.DynamicAttribute.dflt); + break; case Subscript_kind: VISIT(st, expr, e->v.Subscript.value); VISIT(st, slice, e->v.Subscript.slice);

4 3

tab completion in pdb
by stephen emslie 08 Feb '07

08 Feb '07

I make frequent use of the python's built-in debugger, which I think is brilliant in its simplicity. However an important feature seems to be missing: bash-like tab completion similar to that provided by the rlcompleter module. By default, Pdb and other instances of Cmd complete names for commands only. However in the context of pdb, I think it is more useful to complete identifiers and keywords in its current scope than to complete names of commands (most of which have single letter abbreviations). I believe this makes pdb a far more usable introspection tool. Implementation: I've attached a patch to pdb.py (on Python 2.4.4c1). The only real difference to rlcompleter's default complete method is that because pdb changes scope as you step through a program, rlcompleter's namespace is updated to reflect the current local and global namespace. This is my first attempt at a python patch. Any suggestions or improvements are welcome. Stephen Emslie

2 4

Re: [Python-ideas] fixing mutable default argument values
by Jan Kanis 07 Feb '07

07 Feb '07

Other non-python stuff is taking my time, but I'll reply to this anyway. Better late than never (in this case). On Thu, 01 Feb 2007 16:19:00 +0100, Lucio Torre <lucio.torre(a)gmail.com> wrote: > On 1/31/07, Jan Kanis <jan.kanis(a)phil.uu.nl> wrote: >> > >> > Isnt the other argument the fact that always re evaluating will bring >> > similar gotchas? >> >> IMO, it won't. Caching will have to be implemented in a different way, >> but >> that wouldn't really be a gotcha (except for python 2.x programmers who >> didn't read the "what's new in python 3.0") >> > > Maybe is better to annoy newbies that didnt read the python FAQ than > seasoned python 2.x programmers! :) That statement I can agree with, but it isn't really applicable here. The applicable statement reads: "Maybe it's better to annoy newbies that didn't read the python FAQ than python 2.x programmers who don't know anything about 3.0 and assume it's just another new version of python." And I don't agree with that one. :) >> If the >> function depends on other state I'd imagine that most of the time you'd >> want a change in state to be reflected in the functions default value. >> >> example: >> def foo(x = exec_sql("SELECT someval FROM defaults_table WHERE >> some_expr")): >> # do something >> > > i was thinking more in the lines of the case when you dont want the > default to change because something has changed. Then don't make the default depend on something you're going to change. Well, what this is really about is about closure vars changing when the outer scope changes them instead of the value of the closure vars being captured by the new lexical scope. This has been proposed before (to change this behaviour on closure variables in general) and rejected. (though I am in favour of it) Python is a late binding language in just about everything, and that is considered to be a good thing. The point you're making is exactly the early-vs-late binding discussion. > I still think this would be optimizing towards the case where you > should put the code in the body. What if i want a static default > parameter, never to change, but i want to compute it at compile time? > Should i need to use extra syntax for that? Isnt this case more common > than the others? Something like: > > def foo( pixels = (640*480) ): pass > > Should overriding '*' change that default value? (can you do that in > python3k? i think you cant in 2.x) I'm not really sure if you can either. Anyway, the only cases I can think of when this makes sense is when there's a bug in '*' and you want to patch it, or if you're overloading '*' to work with other values, without changing the meaning for ints. In both of these cases it would be a good resp. neutral thing to late-bind '*' > On the other hand, i think that learning about compile time and run > time is a great thing and should be encouraged. Yup. But let's not fill the language with gotchas for the newbies to find and learn from :) >> > And having code that is invoked every time a function is called.. >> > shouldnt the place for this be the function body? >> >> The only reason to answer 'yes' is because it is what you're currently >> used to. > > Im not only used to it, i like it :) You'll get to like the new way :) >> If I see a function with a default argument of [], I expect it to >> use an empty list when I call it without the argument. >> > > That is true. I hate that. But i still like it better than the other > cases. This can be learnt easily and gives you the power to do > everything the other solutions have. The other solutions require more > syntax and more learning to achieve the same. Id love to find a > solution for this, but this options dont seem like one. Please note that I'm not in favour of any of the possible new syntaxes, I am only in favour of changing the semantics without new syntax (cf the pre-pep). Since many newbie python programmers run into this, it seems that it is easier to learn that python evaluates the default expressions at calltime, because that is what the rest of python makes the newbies expect when they first try it. So for newbies, there would actually be less learning involved. You will have to remember this change when you start using py 3.0, but this won't be the only thing to remember. And there are still several other ways to make some python code evaluate the expression at deftime. >> Having code that is eval'd when the function is defined, shouldn't the >> place for this be outside the function definition? > > i dont know, "def fname (...):" is executed when the code block is > executed (creating the function object), and the body is executed when > the function is called. So having a statement that has both compile > time and run time effects is kind of messy. As the body is part of the def statement, the def is already doing exactly that (being both compile/def time and run/call time). Consider the ...'s to be part of the function body in the new semantics. Instead of part of the code in a def being executed at deftime and part at calltime, everything is just run at calltime. There are some pretty large warnings in the docs about this distinction currently, they can be taken out. > I hope im still making sense and not playing out of my league, but > thats just my 2c. > > Lucio. - Jan

2 1

proto-PEP: Fixing Non-constant Default Arguments
by Chris Rebert 03 Feb '07

03 Feb '07

The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=<baz>): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/

10 42

Re: [Python-ideas] File system API
by Tal Einat 02 Feb '07

02 Feb '07

On 2/2/07, Eduardo EdCrypt O. Padoan <eopadoan(a)altavix.com> wrote: > I think that there aready exists a proposal for an Abstract FS Layer > for Python somewere. > I haven't really seen a proposal for such a thing. itools.vfs is the only actualy file-system API, or file-system abstraction layer, I know of. It is going more or less in the directions I am currently thinking of. But the fact is that it has not gained popularity makes me think that maybe the direction needs to be better thought out. And it's not really a proposal as much as a partial implementation, created mostly for the needs of the other itools modules. Most other proposals had to do with handling file-systems paths, and while some also mixed in some functionality for reading/writing files, browsing directory trees, etc., they do not provide a layer of abstraction for file-systems and file-system-like entities. This is why I think there is room for such an initiative. - Tal Einat reduce(lambda m,x:[m[i]+s[-1] for i,s in enumerate(sorted(m))], [[chr(154-ord(c)) for c in '.&-&,l.Z95193+179-']]*18)[3]

3 2

Re: [Python-ideas] File system API
by Eduardo "EdCrypt" O. Padoan 02 Feb '07

02 Feb '07

> > I think that there aready exists a proposal for an Abstract FS Layer > > for Python somewere. > > I haven't been able to find any mention of it. Maybe you could point me in > the right direction? > > - Tal > http://wiki.python.org/moin/CodingProjectIdeas/FileSystemVirtualization Another related reference is the recently announced UniPath module: http://sluggo.scrapping.cc/python/unipath/Unipath-current/README.html -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com

1 0

Re: [Python-ideas] File system API
by Eduardo "EdCrypt" O. Padoan 02 Feb '07

02 Feb '07

I think that there aready exists a proposal for an Abstract FS Layer for Python somewere. -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com

1 0

File system API
by Tal Einat 02 Feb '07

02 Feb '07

I'm proposing a general file-system API (for Python of course). Basically, that's it. I've been wondering how many thousands of times this has been discussed, but an hour's worth of searching the net actually gave very little on this subject. Most other proposals I've seen started far, far further into developing the idea, and I think it's worth giving it a fresh start from the beginning. Also, most proposals centered around either creating efficient fake file-systems, simplifying working with paths, or unifying functionality currently spread between several Python modules. I believe these can be achieved, and more - but let's first think what other goals can be achieved, which goals conflict with each other, and which are the most important. Just for a taste: The OS's file system would be an instance of a class implementing this API. Other examples of classes which would implement this API: FTP implementations, file system proxies in RPC frameworks, and "fake" file systems for testing purposes. (This last example - testing - is where most of the discussion I've seen on this subject was centered.) Surely more uses can be found - I think we should think through the uses cases before thinking of design. Personally, my need is for a remote file-system proxy - my application runs multiple remote servers, and I need to work with a virtual file system on each server. Currently the code for this is quite horrid (low-level hackery); with an API for file-systems I would just need to wrap the local file-system with a proxy class and be done. Obviously I have some initial ideas concerning design, but I haven't hacked at file-systems much yet, and I'd like to hear what more experienced people have to say. So let's leave out design and implementation ideas for later, and focus on: * What would be the uses for this (e.g. file-system proxies) * What would be the benefits and drawbacks * Is there any reason why this would not be possible * Is this a good idea at all (or are we better off without it) * How important do we think this is (for the betterment of Python) - Tal Einat reduce(lambda m,x:[m[i]+s[-1] for i,s in enumerate(sorted(m))], [[chr(154-ord(c)) for c in '.&-&,l.Z95193+179-']]*18)[3]

1 0

Import and '..', '../..' in serach path.
by Ron Adam 01 Feb '07

01 Feb '07

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package. Module in package: import sys sys.path = ['..'] + sys.path import package.module # Imports module in "this!" package. Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of the path. Module in sub-package: import sys sys.path = ['../..'] + sys.path import package.subpackage.module # finds "self" (subpackage) reliably. By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path. Adding this tip to the documentation some where would be nice. (providing there is no major surprising side effects.) Of course I may have missed some obvious way to do this. If so, it wasn't in an obvious place to be found. I looked. ;-) ---------------------------------- It might be useful to have a built-in function to do this. A function could also check for __init__ files and raise errors if they are missing. set_package_name(dotted.name) # Replaces import sys & path modification Where dotted.name is the full package + sub-package name the current module is located in. The function would search upwards to get the root package directory and add that to the *front* of sys.path. Module in package: set_package_name('package') # Add parent directory to front of sys.path import packagename.module # Finds module in "this!" package reliably. Module in subpackage: set_package_name('package.subpackage') import package.subpackage.module # Finds "self" (subpackage) reliably. ---------------------------------- It may also be able to modify the import behavior to allow relative imports to work when the module is run as script. set_package_name('package') from . import module1 # Imports modules from "this" package. from . import module2 Currently an exception is raised you try to run a module with relative references as a script. ValueError: Attempted relative import in non-package I think it is very handy to be able to run tests as scripts and keep them in a sub-package. Especially while I'm writing them. Another benefit of using relative imports with an absolute specified package name, is if you rename a package or relocate a submodule, you only have one line to change. Cheers, Ron

2 7