New syntax for 'dynamic' attribute access
f() += 1 Traceback (most recent call last): SyntaxError: illegal expression for augmented assignment (<doctest test.test_syntax[33]>, line 1)
Hi, I'd like to describe an addition I made to Python syntax which allows easier access to attributes where the attribute name is only known at run-time. For example: setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) As noted in the PEP-style description below, I mostly did this for fun, but I thought it might be worth bringing to the attention of python-ideas. A quick search through prior postings and Google for this idea didn't come up with anything. Ben. - - - - 8< - - - - PEP: XXX Title: Syntax For Dynamic Attribute Access Version: $Revision$ Last-Modified: $Date$ Author: Ben North <ben@redfrontdoor.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 29-Jan-2007 Post-History: Abstract Dynamic attribute access is currently possible using the "getattr" and "setattr" builtins. The present PEP suggests a new syntax to make such access easier, allowing the coder for example to write x.('foo_%d' % n) += 1 z = y.('foo_%d' % n).('bar_%s' % s) instead of attr_name = 'foo_%d' % n setattr(x, attr_name, getattr(x, attr_name) + 1) z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s) Note (I wrote this patch mostly to advance my own understanding of and experiment with the python language, but I've written it up in the style of a PEP in case it might be a useful idea.) Rationale Dictionary access and indexing both have a friendly invocation syntax: instead of x.__getitem__(12) the coder can write x[12]. This also allows the use of subscripted elements in an augmented assignment, as in "x[12] += 1". The present proposal brings this ease-of-use to dynamic attribute access too. Attribute access is currently possible in two ways: * When the attribute name is known at code-writing time, the ".NAME" trailer can be used, as in x.foo = 42 y.bar += 100 * When the attribute name is computed dynamically at run-time, the "getattr" and "setattr" builtins must be used: x = getattr(y, 'foo_%d' % n) setattr(z, 'bar_%s' % s, 99) The "getattr" builtin also allows the coder to specify a default value to be returned in the event that the object does not have an attribute of the given name: x = getattr(y, 'foo_%d' % n, 0) This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in: x = y.('foo_%d' % n, None) This 2-argument form of dynamic attribute access is not permitted as the target of an (augmented or normal) assignment. Finally, the new syntax can be used with the "del" statement, as in del x.(attr_name) Impact On Existing Code The proposed new syntax is not currently valid, so no existing well-formed programs have their meaning altered by this proposal. Across all "*.py" files in the 2.5 distribution, there are around 600 uses of "getattr", "setattr" or "delattr". They break down as follows (figures have some room for error because they were arrived at by partially-manual inspection): c.300 uses of plain "getattr(x, attr_name)", which could be replaced with the new syntax; c.150 uses of the 3-argument form, i.e., with the default value; these could be replaced with the 2-argument form of the new syntax (the cases break down into c.125 cases where the attribute name is a literal string, and c.25 where it's only known at run-time); c.5 uses of the 2-argument form with a literal string attribute name, which I think could be replaced with the standard "x.attribute" syntax; c.120 uses of setattr, of which 15 use getattr to find the new value; all could be replaced with the new syntax, the 15 where getattr is also involved would show a particular increase in clarity; c.5 uses which would have to stay as "getattr" because they are calls of a variable named "getattr" whose default value is the builtin "getattr"; c.5 uses of the 2-argument form, inside a try/except block which catches AttributeError and uses a default value instead; these could use 2-argument form of the new syntax; c.10 uses of "delattr", which could use the new syntax. As examples, the line setattr(self, attr, change_root(self.root, getattr(self, attr))) from Lib/distutils/command/install.py could be rewritten self.(attr) = change_root(self.root, self.(attr)) and the line setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) Alternative Syntax For The New Feature Other syntaxes could be used, for example braces are currently invalid in a "trailer", so could be used here, giving x{'foo_%d' % n} += 1 My personal preference is for the x.('foo_%d' % n) += 1 syntax though: the presence of the dot shows there is attribute access going on; the parentheses have an analogous meaning to the mathematical "work this out first" meaning. This is also the syntax used in the language Matlab [1] for dynamic "field" access (where "field" is the Matlab term analogous to Python's "attribute"). Error Cases Only strings are permitted as attribute names, so for instance the following error is produced: >>> x.(99) = 8 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: attribute name must be string, not 'int' This is handled by the existing PyObject_GetAttr function. Draft Implementation A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests. References [1] Using Dynamic Field Names :: Data Types (MATLAB Programming) http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.h... Copyright This document has been placed in the public domain. [PAGE-BREAK GOES HERE BUT REMOVED FOR EMAIL] Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: - - - - 8< - - - - diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Grammar/Grammar Python-2.5/Grammar/Grammar --- ORIG__Python-2.5/Grammar/Grammar 2006-05-25 12:25:51.000000000 +0100 +++ Python-2.5/Grammar/Grammar 2007-02-01 18:07:04.133160000 +0000 @@ -119,7 +119,7 @@ listmaker: test ( list_for | (',' test)* [','] ) testlist_gexp: test ( gen_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test -trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME +trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] sliceop: ':' [test] diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Include/opcode.h Python-2.5/Include/opcode.h --- ORIG__Python-2.5/Include/opcode.h 2006-02-27 22:32:47.000000000 +0000 +++ Python-2.5/Include/opcode.h 2007-02-02 13:36:58.542848000 +0000 @@ -39,6 +39,10 @@ #define SLICE 30 /* Also uses 31-33 */ +/* LOAD_DYNAMIC_ATTR is below because it takes an argument. */ +#define STORE_DYNAMIC_ATTR 36 +#define DELETE_DYNAMIC_ATTR 37 + #define STORE_SLICE 40 /* Also uses 41-43 */ @@ -89,6 +93,9 @@ #define UNPACK_SEQUENCE 92 /* Number of sequence items */ #define FOR_ITER 93 +#define LOAD_DYNAMIC_ATTR 94 /* Whether default given; 0 = no, 1 = yes */ +/* STORE_DYNAMIC_ATTR, DELETE_DYNAMIC_ATTR are above; they take no argument. */ + #define STORE_ATTR 95 /* Index in name list */ #define DELETE_ATTR 96 /* "" */ #define STORE_GLOBAL 97 /* "" */ diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_dynattr.py Python-2.5/Lib/test/test_dynattr.py --- ORIG__Python-2.5/Lib/test/test_dynattr.py 1970-01-01 01:00:00.000000000 +0100 +++ Python-2.5/Lib/test/test_dynattr.py 2007-02-02 17:37:37.982390000 +0000 @@ -0,0 +1,91 @@ +import unittest +import warnings +import sys +from test import test_support + +class AttrHolder: + pass + +class TestDynAttr(unittest.TestCase): + + def test_simple_get(self): + a = AttrHolder() + a.foo = 100 + a.attr_42 = 1 + self.assertEqual(a.('foo'), 100) + self.assertEqual(a.('fo' + 'o'), 100) + self.assertEqual(a.('f' + 'o' + [('o' if True else 'obar')][0]), 100) + self.assertEqual(a.({'FOO': 'fo'}['FOO'] + 'o'), 100) + self.assertEqual(a.('fo%s' % 'o'), 100) + self.assertEqual(a.('attr_42'), 1) + self.assertEqual(a.('attr_%d' % 42), 1) + self.assertEqual(a.('foo' if True else 'attr_42'), 100) + + def test_nested_get(self): + a = AttrHolder() + a.b = AttrHolder() + a.b.c = 1 + attr_name_b = 'b' + attr_name_c = 'c' + self.assertEqual(a.(attr_name_b).(attr_name_c), 1) + + def test_defaulting_get(self): + a = AttrHolder() + a.foo = 100 + self.assertEqual(a.('foo', 99), 100) + self.assertEqual(a.('bar', 99), 99) + self.assertEqual(a.('baz', 99), 99) + self.assertEqual(a.('foo' if True else 'attr_42', 99), 100) + self.assertEqual(a.('foo' if False else 'attr_42', 99), 99) + + @staticmethod + def attempt_non_string_use(attr_name): + a = AttrHolder() + return a.(attr_name) + + def test_only_strings_allowed(self): + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 99) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, None) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 1.0) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, AttrHolder) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, sys) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, ()) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, []) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, {}) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, (1, 2)) + + def test_augassign(self): + a = AttrHolder() + a.foo = 100 + a.('foo') += 10 + self.assertEqual(a.foo, 110) + self.assertEqual(a.('fo' + 'o'), 110) + a.('f' + 'o' + 'o') *= 10 + self.assertEqual(a.foo, 1100) + self.assertEqual(a.('fo' + 'o'), 1100) + a.('foobar'[:3]) /= 5 + self.assertEqual(a.foo, 220) + self.assertEqual(a.('fo' + 'o'), 220) + a.(['foo', 'bar', 'baz'][0]) -= 40 + self.assertEqual(a.foo, 180) + self.assertEqual(a.('fo' + 'o'), 180) + + def test_setattr(self): + a = AttrHolder() + a.('foo') = 99 + self.assertEqual(a.foo, 99) + a.('bar' + '_baz') = 100 + self.assertEqual(a.bar_baz, 100) + + def test_delattr(self): + a = AttrHolder() + a.foo = 99 + del a.('foo') + self.assertEqual(hasattr(a, 'foo'), False) + + +def test_main(): + test_support.run_unittest(TestDynAttr) + +if __name__ == "__main__": + test_main() diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_syntax.py Python-2.5/Lib/test/test_syntax.py --- ORIG__Python-2.5/Lib/test/test_syntax.py 2006-05-19 07:43:50.000000000 +0100 +++ Python-2.5/Lib/test/test_syntax.py 2007-02-02 17:21:04.991119000 +0000 @@ -235,6 +235,18 @@ + +Outlawed uses of dynamic attributes: + +>>> x.('foo', 0) += 1 +Traceback (most recent call last): +SyntaxError: augmented assignment to 2-argument dynamic-attribute expression not possible (<doctest test.test_syntax[34]>, line 1) +>>> x.('foo', 0) = 1 +Traceback (most recent call last): +SyntaxError: can't assign to 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[35]>, line 1) +>>> del x.('foo', 0) +Traceback (most recent call last): +SyntaxError: can't delete 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[36]>, line 1) """ import re diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Parser/Python.asdl Python-2.5/Parser/Python.asdl --- ORIG__Python-2.5/Parser/Python.asdl 2006-04-04 05:00:23.000000000 +0100 +++ Python-2.5/Parser/Python.asdl 2007-02-02 12:39:36.665151000 +0000 @@ -72,6 +72,7 @@ -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) + | DynamicAttribute(expr value, expr attr, expr? dflt, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ast.c Python-2.5/Python/ast.c --- ORIG__Python-2.5/Python/ast.c 2006-09-05 04:56:01.000000000 +0100 +++ Python-2.5/Python/ast.c 2007-02-02 13:54:51.388446000 +0000 @@ -353,6 +353,13 @@ } e->v.Attribute.ctx = ctx; break; + case DynamicAttribute_kind: + if ((ctx == Store || ctx == Del) + && e->v.DynamicAttribute.dflt) + expr_name = "2-argument form of dynamic-attribute expression"; + else + e->v.DynamicAttribute.ctx = ctx; + break; case Subscript_kind: e->v.Subscript.ctx = ctx; break; @@ -1427,7 +1434,7 @@ static expr_ty ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) { - /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME + /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] */ @@ -1440,8 +1447,30 @@ return ast_for_call(c, CHILD(n, 1), left_expr); } else if (TYPE(CHILD(n, 0)) == DOT ) { + if (TYPE(CHILD(n, 1)) == NAME) return Attribute(left_expr, NEW_IDENTIFIER(CHILD(n, 1)), Load, LINENO(n), n->n_col_offset, c->c_arena); + else { + expr_ty e_val, e_dflt; + REQ(CHILD(n, 1), LPAR); + if (!(e_val = ast_for_expr(c, CHILD(n, 2)))) + return NULL; + + if (TYPE(CHILD(n, 3)) == RPAR) { + assert(NCH(n) == 3); + e_dflt = NULL; + } else { + assert(NCH(n) == 5); + REQ(CHILD(n, 3), COMMA); + REQ(CHILD(n, 5), RPAR); + if (!(e_dflt = ast_for_expr(c, CHILD(n, 4)))) + return NULL; + } + + return DynamicAttribute(left_expr, e_val, e_dflt, Load, + LINENO(n), n->n_col_offset, + c->c_arena); + } } else { REQ(CHILD(n, 0), LSQB); @@ -1964,6 +1993,15 @@ case Attribute_kind: case Subscript_kind: break; + case DynamicAttribute_kind: + if (expr1->v.DynamicAttribute.dflt) { + ast_error(ch, "augmented assignment to " + "2-argument dynamic-attribute " + "expression not possible"); + return NULL; + } + break; + default: ast_error(ch, "illegal expression for augmented " "assignment"); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ceval.c Python-2.5/Python/ceval.c --- ORIG__Python-2.5/Python/ceval.c 2006-08-13 19:10:10.000000000 +0100 +++ Python-2.5/Python/ceval.c 2007-02-08 12:56:39.637479000 +0000 @@ -1787,6 +1787,17 @@ if (err == 0) continue; break; + case STORE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + u = POP(); + err = PyObject_SetAttr(v, w, u); /* v.w = u */ + Py_DECREF(w); + Py_DECREF(v); + Py_DECREF(u); + if (err == 0) continue; + break; + case DELETE_ATTR: w = GETITEM(names, oparg); v = POP(); @@ -1795,6 +1806,14 @@ Py_DECREF(v); break; + case DELETE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + err = PyObject_SetAttr(v, w, (PyObject *)NULL); + Py_DECREF(w); + Py_DECREF(v); + break; + case STORE_GLOBAL: w = GETITEM(names, oparg); v = POP(); @@ -1994,6 +2013,28 @@ if (x != NULL) continue; break; + case LOAD_DYNAMIC_ATTR: + if (oparg) + u = POP(); + else + u = NULL; + w = POP(); + v = TOP(); + x = PyObject_GetAttr(v, w); + if (x == NULL && u != NULL + && PyErr_ExceptionMatches(PyExc_AttributeError)) + { + PyErr_Clear(); + Py_INCREF(u); + x = u; + } + Py_DECREF(v); + Py_DECREF(w); + Py_XDECREF(u); /* This one may be NULL (if no default) */ + SET_TOP(x); + if (x != NULL) continue; + break; + case COMPARE_OP: w = POP(); v = TOP(); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/compile.c Python-2.5/Python/compile.c --- ORIG__Python-2.5/Python/compile.c 2006-08-12 02:45:47.000000000 +0100 +++ Python-2.5/Python/compile.c 2007-02-08 12:55:51.958809000 +0000 @@ -1357,6 +1357,13 @@ case SLICE+3: return -1; + case LOAD_DYNAMIC_ATTR: + return -1 - oparg; + case STORE_DYNAMIC_ATTR: + return -3; + case DELETE_DYNAMIC_ATTR: + return -2; + case STORE_SLICE+0: return -2; case STORE_SLICE+1: @@ -3641,6 +3648,41 @@ return 0; } break; + case DynamicAttribute_kind: + { + int has_default_p = (e->v.DynamicAttribute.dflt != NULL); + + if (e->v.DynamicAttribute.ctx != AugStore) { + VISIT(c, expr, e->v.DynamicAttribute.value); + VISIT(c, expr, e->v.DynamicAttribute.attr); + if (has_default_p) + VISIT(c, expr, e->v.DynamicAttribute.dflt); + } + switch (e->v.DynamicAttribute.ctx) { + case AugLoad: + assert(!has_default_p); + ADDOP_I(c, DUP_TOPX, 2); + /* Fall through to Load */ + case Load: + ADDOP_I(c, LOAD_DYNAMIC_ATTR, has_default_p); + break; + case AugStore: + ADDOP(c, ROT_THREE); + /* Fall through to Store */ + case Store: + assert(!has_default_p); + ADDOP(c, STORE_DYNAMIC_ATTR); + break; + case Del: + ADDOP(c, DELETE_DYNAMIC_ATTR); + break; + default: + PyErr_SetString(PyExc_SystemError, + "invalid context in dynamic-attribute expression"); + return 0; + } + break; + } case Subscript_kind: switch (e->v.Subscript.ctx) { case AugLoad: @@ -3700,6 +3742,18 @@ auge->v.Attribute.ctx = AugStore; VISIT(c, expr, auge); break; + case DynamicAttribute_kind: + assert(e->v.DynamicAttribute.dflt == NULL); + auge = DynamicAttribute(e->v.DynamicAttribute.value, e->v.DynamicAttribute.attr, NULL, + AugLoad, e->lineno, e->col_offset, c->c_arena); + if (auge == NULL) + return 0; + VISIT(c, expr, auge); + VISIT(c, expr, s->v.AugAssign.value); + ADDOP(c, inplace_binop(c, s->v.AugAssign.op)); + auge->v.DynamicAttribute.ctx = AugStore; + VISIT(c, expr, auge); + break; case Subscript_kind: auge = Subscript(e->v.Subscript.value, e->v.Subscript.slice, AugLoad, e->lineno, e->col_offset, c->c_arena); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/symtable.c Python-2.5/Python/symtable.c --- ORIG__Python-2.5/Python/symtable.c 2006-08-12 02:43:40.000000000 +0100 +++ Python-2.5/Python/symtable.c 2007-02-08 12:53:36.279608000 +0000 @@ -1192,6 +1192,12 @@ case Attribute_kind: VISIT(st, expr, e->v.Attribute.value); break; + case DynamicAttribute_kind: + VISIT(st, expr, e->v.DynamicAttribute.value); + VISIT(st, expr, e->v.DynamicAttribute.attr); + if (e->v.DynamicAttribute.dflt) + VISIT(st, expr, e->v.DynamicAttribute.dflt); + break; case Subscript_kind: VISIT(st, expr, e->v.Subscript.value); VISIT(st, slice, e->v.Subscript.slice);
I've thought of the same syntax. I think you should submit this to the PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- there's no benefit that I see of waiting until 3.0. --Guido On 2/9/07, Ben North <ben@redfrontdoor.org> wrote:
Hi,
I'd like to describe an addition I made to Python syntax which allows easier access to attributes where the attribute name is only known at run-time. For example:
setattr(self, method_name, getattr(self.metadata, method_name))
from Lib/distutils/dist.py could be rewritten
self.(method_name) = self.metadata.(method_name)
As noted in the PEP-style description below, I mostly did this for fun, but I thought it might be worth bringing to the attention of python-ideas. A quick search through prior postings and Google for this idea didn't come up with anything.
Ben.
- - - - 8< - - - -
PEP: XXX Title: Syntax For Dynamic Attribute Access Version: $Revision$ Last-Modified: $Date$ Author: Ben North <ben@redfrontdoor.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 29-Jan-2007 Post-History:
Abstract
Dynamic attribute access is currently possible using the "getattr" and "setattr" builtins. The present PEP suggests a new syntax to make such access easier, allowing the coder for example to write
x.('foo_%d' % n) += 1
z = y.('foo_%d' % n).('bar_%s' % s)
instead of
attr_name = 'foo_%d' % n setattr(x, attr_name, getattr(x, attr_name) + 1)
z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s)
Note
(I wrote this patch mostly to advance my own understanding of and experiment with the python language, but I've written it up in the style of a PEP in case it might be a useful idea.)
Rationale
Dictionary access and indexing both have a friendly invocation syntax: instead of x.__getitem__(12) the coder can write x[12]. This also allows the use of subscripted elements in an augmented assignment, as in "x[12] += 1". The present proposal brings this ease-of-use to dynamic attribute access too.
Attribute access is currently possible in two ways:
* When the attribute name is known at code-writing time, the ".NAME" trailer can be used, as in
x.foo = 42 y.bar += 100
* When the attribute name is computed dynamically at run-time, the "getattr" and "setattr" builtins must be used:
x = getattr(y, 'foo_%d' % n) setattr(z, 'bar_%s' % s, 99)
The "getattr" builtin also allows the coder to specify a default value to be returned in the event that the object does not have an attribute of the given name:
x = getattr(y, 'foo_%d' % n, 0)
This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in:
x = y.('foo_%d' % n, None)
This 2-argument form of dynamic attribute access is not permitted as the target of an (augmented or normal) assignment. Finally, the new syntax can be used with the "del" statement, as in
del x.(attr_name)
Impact On Existing Code
The proposed new syntax is not currently valid, so no existing well-formed programs have their meaning altered by this proposal.
Across all "*.py" files in the 2.5 distribution, there are around 600 uses of "getattr", "setattr" or "delattr". They break down as follows (figures have some room for error because they were arrived at by partially-manual inspection):
c.300 uses of plain "getattr(x, attr_name)", which could be replaced with the new syntax;
c.150 uses of the 3-argument form, i.e., with the default value; these could be replaced with the 2-argument form of the new syntax (the cases break down into c.125 cases where the attribute name is a literal string, and c.25 where it's only known at run-time);
c.5 uses of the 2-argument form with a literal string attribute name, which I think could be replaced with the standard "x.attribute" syntax;
c.120 uses of setattr, of which 15 use getattr to find the new value; all could be replaced with the new syntax, the 15 where getattr is also involved would show a particular increase in clarity;
c.5 uses which would have to stay as "getattr" because they are calls of a variable named "getattr" whose default value is the builtin "getattr";
c.5 uses of the 2-argument form, inside a try/except block which catches AttributeError and uses a default value instead; these could use 2-argument form of the new syntax;
c.10 uses of "delattr", which could use the new syntax.
As examples, the line
setattr(self, attr, change_root(self.root, getattr(self, attr)))
from Lib/distutils/command/install.py could be rewritten
self.(attr) = change_root(self.root, self.(attr))
and the line
setattr(self, method_name, getattr(self.metadata, method_name))
from Lib/distutils/dist.py could be rewritten
self.(method_name) = self.metadata.(method_name)
Alternative Syntax For The New Feature
Other syntaxes could be used, for example braces are currently invalid in a "trailer", so could be used here, giving
x{'foo_%d' % n} += 1
My personal preference is for the
x.('foo_%d' % n) += 1
syntax though: the presence of the dot shows there is attribute access going on; the parentheses have an analogous meaning to the mathematical "work this out first" meaning. This is also the syntax used in the language Matlab [1] for dynamic "field" access (where "field" is the Matlab term analogous to Python's "attribute").
Error Cases
Only strings are permitted as attribute names, so for instance the following error is produced:
>>> x.(99) = 8 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: attribute name must be string, not 'int'
This is handled by the existing PyObject_GetAttr function.
Draft Implementation
A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests.
References
[1] Using Dynamic Field Names :: Data Types (MATLAB Programming) http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.h...
Copyright
This document has been placed in the public domain.
[PAGE-BREAK GOES HERE BUT REMOVED FOR EMAIL] Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
- - - - 8< - - - - diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Grammar/Grammar Python-2.5/Grammar/Grammar --- ORIG__Python-2.5/Grammar/Grammar 2006-05-25 12:25:51.000000000 +0100 +++ Python-2.5/Grammar/Grammar 2007-02-01 18:07:04.133160000 +0000 @@ -119,7 +119,7 @@ listmaker: test ( list_for | (',' test)* [','] ) testlist_gexp: test ( gen_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test -trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME +trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] sliceop: ':' [test] diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Include/opcode.h Python-2.5/Include/opcode.h --- ORIG__Python-2.5/Include/opcode.h 2006-02-27 22:32:47.000000000 +0000 +++ Python-2.5/Include/opcode.h 2007-02-02 13:36:58.542848000 +0000 @@ -39,6 +39,10 @@ #define SLICE 30 /* Also uses 31-33 */
+/* LOAD_DYNAMIC_ATTR is below because it takes an argument. */ +#define STORE_DYNAMIC_ATTR 36 +#define DELETE_DYNAMIC_ATTR 37 + #define STORE_SLICE 40 /* Also uses 41-43 */
@@ -89,6 +93,9 @@ #define UNPACK_SEQUENCE 92 /* Number of sequence items */ #define FOR_ITER 93
f() += 1 Traceback (most recent call last): SyntaxError: illegal expression for augmented assignment (<doctest test.test_syntax[33]>, line 1)
+#define LOAD_DYNAMIC_ATTR 94 /* Whether default given; 0 = no, 1 = yes */ +/* STORE_DYNAMIC_ATTR, DELETE_DYNAMIC_ATTR are above; they take no argument. */ + #define STORE_ATTR 95 /* Index in name list */ #define DELETE_ATTR 96 /* "" */ #define STORE_GLOBAL 97 /* "" */ diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_dynattr.py Python-2.5/Lib/test/test_dynattr.py --- ORIG__Python-2.5/Lib/test/test_dynattr.py 1970-01-01 01:00:00.000000000 +0100 +++ Python-2.5/Lib/test/test_dynattr.py 2007-02-02 17:37:37.982390000 +0000 @@ -0,0 +1,91 @@ +import unittest +import warnings +import sys +from test import test_support + +class AttrHolder: + pass + +class TestDynAttr(unittest.TestCase): + + def test_simple_get(self): + a = AttrHolder() + a.foo = 100 + a.attr_42 = 1 + self.assertEqual(a.('foo'), 100) + self.assertEqual(a.('fo' + 'o'), 100) + self.assertEqual(a.('f' + 'o' + [('o' if True else 'obar')][0]), 100) + self.assertEqual(a.({'FOO': 'fo'}['FOO'] + 'o'), 100) + self.assertEqual(a.('fo%s' % 'o'), 100) + self.assertEqual(a.('attr_42'), 1) + self.assertEqual(a.('attr_%d' % 42), 1) + self.assertEqual(a.('foo' if True else 'attr_42'), 100) + + def test_nested_get(self): + a = AttrHolder() + a.b = AttrHolder() + a.b.c = 1 + attr_name_b = 'b' + attr_name_c = 'c' + self.assertEqual(a.(attr_name_b).(attr_name_c), 1) + + def test_defaulting_get(self): + a = AttrHolder() + a.foo = 100 + self.assertEqual(a.('foo', 99), 100) + self.assertEqual(a.('bar', 99), 99) + self.assertEqual(a.('baz', 99), 99) + self.assertEqual(a.('foo' if True else 'attr_42', 99), 100) + self.assertEqual(a.('foo' if False else 'attr_42', 99), 99) + + @staticmethod + def attempt_non_string_use(attr_name): + a = AttrHolder() + return a.(attr_name) + + def test_only_strings_allowed(self): + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 99) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, None) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, 1.0) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, AttrHolder) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, sys) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, ()) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, []) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, {}) + self.assertRaises(TypeError, TestDynAttr.attempt_non_string_use, (1, 2)) + + def test_augassign(self): + a = AttrHolder() + a.foo = 100 + a.('foo') += 10 + self.assertEqual(a.foo, 110) + self.assertEqual(a.('fo' + 'o'), 110) + a.('f' + 'o' + 'o') *= 10 + self.assertEqual(a.foo, 1100) + self.assertEqual(a.('fo' + 'o'), 1100) + a.('foobar'[:3]) /= 5 + self.assertEqual(a.foo, 220) + self.assertEqual(a.('fo' + 'o'), 220) + a.(['foo', 'bar', 'baz'][0]) -= 40 + self.assertEqual(a.foo, 180) + self.assertEqual(a.('fo' + 'o'), 180) + + def test_setattr(self): + a = AttrHolder() + a.('foo') = 99 + self.assertEqual(a.foo, 99) + a.('bar' + '_baz') = 100 + self.assertEqual(a.bar_baz, 100) + + def test_delattr(self): + a = AttrHolder() + a.foo = 99 + del a.('foo') + self.assertEqual(hasattr(a, 'foo'), False) + + +def test_main(): + test_support.run_unittest(TestDynAttr) + +if __name__ == "__main__": + test_main() diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Lib/test/test_syntax.py Python-2.5/Lib/test/test_syntax.py --- ORIG__Python-2.5/Lib/test/test_syntax.py 2006-05-19 07:43:50.000000000 +0100 +++ Python-2.5/Lib/test/test_syntax.py 2007-02-02 17:21:04.991119000 +0000 @@ -235,6 +235,18 @@ + +Outlawed uses of dynamic attributes: + +>>> x.('foo', 0) += 1 +Traceback (most recent call last): +SyntaxError: augmented assignment to 2-argument dynamic-attribute expression not possible (<doctest test.test_syntax[34]>, line 1) +>>> x.('foo', 0) = 1 +Traceback (most recent call last): +SyntaxError: can't assign to 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[35]>, line 1) +>>> del x.('foo', 0) +Traceback (most recent call last): +SyntaxError: can't delete 2-argument form of dynamic-attribute expression (<doctest test.test_syntax[36]>, line 1) """
import re diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Parser/Python.asdl Python-2.5/Parser/Python.asdl --- ORIG__Python-2.5/Parser/Python.asdl 2006-04-04 05:00:23.000000000 +0100 +++ Python-2.5/Parser/Python.asdl 2007-02-02 12:39:36.665151000 +0000 @@ -72,6 +72,7 @@
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) + | DynamicAttribute(expr value, expr attr, expr? dflt, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ast.c Python-2.5/Python/ast.c --- ORIG__Python-2.5/Python/ast.c 2006-09-05 04:56:01.000000000 +0100 +++ Python-2.5/Python/ast.c 2007-02-02 13:54:51.388446000 +0000 @@ -353,6 +353,13 @@ } e->v.Attribute.ctx = ctx; break; + case DynamicAttribute_kind: + if ((ctx == Store || ctx == Del) + && e->v.DynamicAttribute.dflt) + expr_name = "2-argument form of dynamic-attribute expression"; + else + e->v.DynamicAttribute.ctx = ctx; + break; case Subscript_kind: e->v.Subscript.ctx = ctx; break; @@ -1427,7 +1434,7 @@ static expr_ty ast_for_trailer(struct compiling *c, const node *n, expr_ty left_expr) { - /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME + /* trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '.' '(' test [',' test] ')' subscriptlist: subscript (',' subscript)* [','] subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] */ @@ -1440,8 +1447,30 @@ return ast_for_call(c, CHILD(n, 1), left_expr); } else if (TYPE(CHILD(n, 0)) == DOT ) { + if (TYPE(CHILD(n, 1)) == NAME) return Attribute(left_expr, NEW_IDENTIFIER(CHILD(n, 1)), Load, LINENO(n), n->n_col_offset, c->c_arena); + else { + expr_ty e_val, e_dflt; + REQ(CHILD(n, 1), LPAR); + if (!(e_val = ast_for_expr(c, CHILD(n, 2)))) + return NULL; + + if (TYPE(CHILD(n, 3)) == RPAR) { + assert(NCH(n) == 3); + e_dflt = NULL; + } else { + assert(NCH(n) == 5); + REQ(CHILD(n, 3), COMMA); + REQ(CHILD(n, 5), RPAR); + if (!(e_dflt = ast_for_expr(c, CHILD(n, 4)))) + return NULL; + } + + return DynamicAttribute(left_expr, e_val, e_dflt, Load, + LINENO(n), n->n_col_offset, + c->c_arena); + } } else { REQ(CHILD(n, 0), LSQB); @@ -1964,6 +1993,15 @@ case Attribute_kind: case Subscript_kind: break; + case DynamicAttribute_kind: + if (expr1->v.DynamicAttribute.dflt) { + ast_error(ch, "augmented assignment to " + "2-argument dynamic-attribute " + "expression not possible"); + return NULL; + } + break; + default: ast_error(ch, "illegal expression for augmented " "assignment"); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/ceval.c Python-2.5/Python/ceval.c --- ORIG__Python-2.5/Python/ceval.c 2006-08-13 19:10:10.000000000 +0100 +++ Python-2.5/Python/ceval.c 2007-02-08 12:56:39.637479000 +0000 @@ -1787,6 +1787,17 @@ if (err == 0) continue; break;
+ case STORE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + u = POP(); + err = PyObject_SetAttr(v, w, u); /* v.w = u */ + Py_DECREF(w); + Py_DECREF(v); + Py_DECREF(u); + if (err == 0) continue; + break; + case DELETE_ATTR: w = GETITEM(names, oparg); v = POP(); @@ -1795,6 +1806,14 @@ Py_DECREF(v); break;
+ case DELETE_DYNAMIC_ATTR: + w = POP(); + v = POP(); + err = PyObject_SetAttr(v, w, (PyObject *)NULL); + Py_DECREF(w); + Py_DECREF(v); + break; + case STORE_GLOBAL: w = GETITEM(names, oparg); v = POP(); @@ -1994,6 +2013,28 @@ if (x != NULL) continue; break;
+ case LOAD_DYNAMIC_ATTR: + if (oparg) + u = POP(); + else + u = NULL; + w = POP(); + v = TOP(); + x = PyObject_GetAttr(v, w); + if (x == NULL && u != NULL + && PyErr_ExceptionMatches(PyExc_AttributeError)) + { + PyErr_Clear(); + Py_INCREF(u); + x = u; + } + Py_DECREF(v); + Py_DECREF(w); + Py_XDECREF(u); /* This one may be NULL (if no default) */ + SET_TOP(x); + if (x != NULL) continue; + break; + case COMPARE_OP: w = POP(); v = TOP(); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/compile.c Python-2.5/Python/compile.c --- ORIG__Python-2.5/Python/compile.c 2006-08-12 02:45:47.000000000 +0100 +++ Python-2.5/Python/compile.c 2007-02-08 12:55:51.958809000 +0000 @@ -1357,6 +1357,13 @@ case SLICE+3: return -1;
+ case LOAD_DYNAMIC_ATTR: + return -1 - oparg; + case STORE_DYNAMIC_ATTR: + return -3; + case DELETE_DYNAMIC_ATTR: + return -2; + case STORE_SLICE+0: return -2; case STORE_SLICE+1: @@ -3641,6 +3648,41 @@ return 0; } break; + case DynamicAttribute_kind: + { + int has_default_p = (e->v.DynamicAttribute.dflt != NULL); + + if (e->v.DynamicAttribute.ctx != AugStore) { + VISIT(c, expr, e->v.DynamicAttribute.value); + VISIT(c, expr, e->v.DynamicAttribute.attr); + if (has_default_p) + VISIT(c, expr, e->v.DynamicAttribute.dflt); + } + switch (e->v.DynamicAttribute.ctx) { + case AugLoad: + assert(!has_default_p); + ADDOP_I(c, DUP_TOPX, 2); + /* Fall through to Load */ + case Load: + ADDOP_I(c, LOAD_DYNAMIC_ATTR, has_default_p); + break; + case AugStore: + ADDOP(c, ROT_THREE); + /* Fall through to Store */ + case Store: + assert(!has_default_p); + ADDOP(c, STORE_DYNAMIC_ATTR); + break; + case Del: + ADDOP(c, DELETE_DYNAMIC_ATTR); + break; + default: + PyErr_SetString(PyExc_SystemError, + "invalid context in dynamic-attribute expression"); + return 0; + } + break; + } case Subscript_kind: switch (e->v.Subscript.ctx) { case AugLoad: @@ -3700,6 +3742,18 @@ auge->v.Attribute.ctx = AugStore; VISIT(c, expr, auge); break; + case DynamicAttribute_kind: + assert(e->v.DynamicAttribute.dflt == NULL); + auge = DynamicAttribute(e->v.DynamicAttribute.value, e->v.DynamicAttribute.attr, NULL, + AugLoad, e->lineno, e->col_offset, c->c_arena); + if (auge == NULL) + return 0; + VISIT(c, expr, auge); + VISIT(c, expr, s->v.AugAssign.value); + ADDOP(c, inplace_binop(c, s->v.AugAssign.op)); + auge->v.DynamicAttribute.ctx = AugStore; + VISIT(c, expr, auge); + break; case Subscript_kind: auge = Subscript(e->v.Subscript.value, e->v.Subscript.slice, AugLoad, e->lineno, e->col_offset, c->c_arena); diff --exclude=graminit.c --exclude='Python-ast.[ch]' -Nwuar ORIG__Python-2.5/Python/symtable.c Python-2.5/Python/symtable.c --- ORIG__Python-2.5/Python/symtable.c 2006-08-12 02:43:40.000000000 +0100 +++ Python-2.5/Python/symtable.c 2007-02-08 12:53:36.279608000 +0000 @@ -1192,6 +1192,12 @@ case Attribute_kind: VISIT(st, expr, e->v.Attribute.value); break; + case DynamicAttribute_kind: + VISIT(st, expr, e->v.DynamicAttribute.value); + VISIT(st, expr, e->v.DynamicAttribute.attr); + if (e->v.DynamicAttribute.dflt) + VISIT(st, expr, e->v.DynamicAttribute.dflt); + break; case Subscript_kind: VISIT(st, expr, e->v.Subscript.value); VISIT(st, slice, e->v.Subscript.slice);
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 2/9/07, Ben North <ben@redfrontdoor.org> wrote:
Hi,
I'd like to describe an addition I made to Python syntax which allows easier access to attributes where the attribute name is only known at run-time. For example:
setattr(self, method_name, getattr(self.metadata, method_name))
from Lib/distutils/dist.py could be rewritten
self.(method_name) = self.metadata.(method_name)
Wow! I have to say this is a compelling idea. The syntax is a bit foreign looking, but obj.(expr) += 1 is just such a huge win over setattr(obj, expr, getattr(obj, expr) + 1) that I feel like I could learn to like it anyway. So a few thoughts from the peanut gallery. The problem with the syntax is that it looks a good bit like a function call. The dot can get easily lost when rendered in a proportional font. Because of this, I keep wanting to write the punctuation as something else, possibly .* , but I think that's my C++ bias shining through. (And as you point, your syntax does have the precedent of MATLAB.)
This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in: x = y.('foo_%d' % n, None)
This is the one bit I really don't like. y.('foobar') is arguably a natural extension to y.foobar, but y.('foobar', None) isn't analogous to any current syntax, and the multiple arguments make it look even more dangerously like a call. I think the goal should be to capture all of the readable syntax of attribute access, not to capture all of the expressiveness of getattr. In Python, you already have to be explicit when you're worried if the thing you're accessing might not be there. So you write either: x.foo += 1 or x.foo = getattr(x, foo, 0) + 1 and you write either: x[bar].append(1) or x.setdefault(bar, []).append(1) Even if x.(foo, bar) wasn't scary syntax, I don't think breaking tradition here is worth it. But in any event, it's a very interesting idea and patch. I'll definitely play around with it this weekend. Greg F
Ben North <ben@redfrontdoor.org> wrote:
Hi,
I'd like to describe an addition I made to Python syntax which allows easier access to attributes where the attribute name is only known at run-time. For example:
setattr(self, method_name, getattr(self.metadata, method_name))
from Lib/distutils/dist.py could be rewritten
self.(method_name) = self.metadata.(method_name)
As noted in the PEP-style description below, I mostly did this for fun, but I thought it might be worth bringing to the attention of python-ideas. A quick search through prior postings and Google for this idea didn't come up with anything.
My only concern with your propsed change is your draft implementation...
Draft Implementation
A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests.
Specifically, your changes to ceval.c and the compiler may have been easier to implement, but it may negatively affect general Python performance. Have you run a recent pystone before and after the changes? Since the *value* being used for the dynamic set, get, and load should be available, it may be possible to replace the 3 new opcodes with 1 new opcode that shifts the the value on the top of stack into the the code object co_names, which can then be accessed directly by the standard [LOAD|STORE|DEL]_ATTR opcodes. The major concern with such a change is that the co_name field would no longer be read-only, so wouldn't be sharable between threads (and would need to grow by at least 1 when such dynamic accesses were allowed, though growing by 2 could help in augmented assignment cases). We could probably get away with a single new attribute on the stack frame, adding an alternate GETITEM implementation... #ifndef Py_DEBUG #define GETITEM2(v, i, s) \ ((i) != -1) ? PyTuple_GET_ITEM((PyTupleObject *)(v), (i)) : (s)) #else #define GETITEM2(v, i, s) \ ((i) != -1) ? PyTuple_GetItem((v), (i)) : (s)) #endif With that change, dynamic attribute access would result in a single opcode DYNAMIC_ACCESS, which would copy/move the value on the top of the stack into some stack frame attribute, which is then automatically accessed in the [LOAD|STORE|DEL]_ATTR opcodes (the GETITEM2 macro not being used in any other opcodes). The problem with the alternate approach above is that the overhead of general [LOAD|STORE|DEL]_ATTR opcodes may become measurably larger (depending on the branch prediction efficiency of a processor). Something to think about. - Josiah
participants (4)
-
Ben North -
Greg Falcon -
Guido van Rossum -
Josiah Carlson