Allow function __globals__ to be arbitrary mapping not just dict
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict: py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping I propose to allow function.__globals__ to accept any mapping type. That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules. E.g. if I want to write a function with a custom namespace, I have to do something like this: ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d) which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this: ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d) # This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__) (By the way, ChainMap is only one possible example namespace.) -- Steven
What are restrictions to supported classes in global? I think MutableMapping ABC is good requirement. On Sun, Mar 18, 2012 at 5:27 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict:
py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping
I propose to allow function.__globals__ to accept any mapping type.
That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules.
E.g. if I want to write a function with a custom namespace, I have to do something like this:
ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d)
which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this:
ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d)
# This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__)
(By the way, ChainMap is only one possible example namespace.)
-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Thanks, Andrew Svetlov
On 3/18/2012 8:27 AM, Steven D'Aprano wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict:
py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping
The API of internal classes is intentionally not documented in the types module doc. I take this to mean that the api, if not the existence, of such classes, is implementation specific. Hence any such call is a 'cpython' rather than generic python call. Someone pointed out on python-list that FunctionType checks and rejects non-dict second args because the CPython ceval loop code for CPython LOAD_GLOBAL and STORE_GLOBAL opcodes directly access dicts rather than using the generic mapping interface.
I propose to allow function.__globals__ to accept any mapping type.
The important question is whether the current ceval code in merely a holdover from ancient days when dict was the only mapping type and the user function type was not accessible to users, or whether the direct dict access is an important speed optimization that affects essentially all code run on cpython. If not done already, experiments are needed to assess the degree of slowdown. Something to keep in mind: LOAD_GLOBAL is *also* used to access builtins that are not in globals() itself via globals['__builtins__']. So not just any mapping will work as a replacement. See 'Alternative proposal' below.
That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules.
E.g. if I want to write a function with a custom namespace, I have to do something like this:
ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d)
which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this:
ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d)
# This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__)
(By the way, ChainMap is only one possible example namespace.)
Alternative proposal: write a function to replace LOAD_GLOBAL and STORE_GLOBAL for non-builtin names with the opcodes to access a mapping passed in as an arg to the rewrite function or the function itself. The latter would be perhaps easier since the name of the replacement mapping would already be in the code object. def f(a, _globals = {}):pass f = reglobalize(f) # assume '_globals' is the replacement # now call f with whatever 'global' dict you want on a per-call basis. -- Terry Jan Reedy
On 3/18/12 5:26 PM, Terry Reedy wrote:
On 3/18/2012 8:27 AM, Steven D'Aprano wrote:
I propose to allow function.__globals__ to accept any mapping type.
The important question is whether the current ceval code in merely a holdover from ancient days when dict was the only mapping type and the user function type was not accessible to users, or whether the direct dict access is an important speed optimization that affects essentially all code run on cpython. If not done already, experiments are needed to assess the degree of slowdown.
At one time, both the locals and globals dictionaries used by exec and eval() were required to be true dicts, but this restriction was loosened for the locals mapping but not the globals. This distinction is documented in the language reference. You can probably find the original discussions for the reasons why the restriction on globals dicts remained; I don't remember the details, but it wasn't just an oversight. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Terry Reedy wrote:
On 3/18/2012 8:27 AM, Steven D'Aprano wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict: [...] The API of internal classes is intentionally not documented in the types module doc. I take this to mean that the api, if not the existence, of such classes, is implementation specific. Hence any such call is a 'cpython' rather than generic python call.
Are you sure that they are *intentionally* not documented, or merely haven't been due to lack of time and interest? The types themselves are documented as public, they aren't given _single underscore private names, and FunctionType has a minimal but useful doc string with nothing about it being implementation specific.
Someone pointed out on python-list that FunctionType checks and rejects non-dict second args because the CPython ceval loop code for CPython LOAD_GLOBAL and STORE_GLOBAL opcodes directly access dicts rather than using the generic mapping interface.
I expect that the existing optimization would be used whenever __globals__ is an actual dict, and the mapping interface would only be used when it is a subclass of dict or other mapping. So for normal functions, the only cost would be an extra type check to see whether __globals__ is a dict or not. [...]
Alternative proposal: write a function to replace LOAD_GLOBAL and STORE_GLOBAL for non-builtin names with the opcodes to access a mapping passed in as an arg to the rewrite function or the function itself.
Are you proposing that I hack the byte code of the function? Now that would be unsupported and implementation dependent! If not, I'm afraid I don't understand your suggestion. -- Steven
On 3/19/2012 7:51 PM, Steven D'Aprano wrote:
Terry Reedy wrote:
On 3/18/2012 8:27 AM, Steven D'Aprano wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict: [...] The API of internal classes is intentionally not documented in the types module doc. I take this to mean that the api, if not the existence, of such classes, is implementation specific. Hence any such call is a 'cpython' rather than generic python call.
Are you sure that they are *intentionally* not documented, or merely haven't been due to lack of time and interest?
100% absolutely sure, no. 90% fairly sure, yes. Less that half of the internal types even *are* Python-level callables with Python-object APIs: FunctionType, CodeType, MethodType, ModuleType. The rest are not. I think there once may have been a discussion of how much to expose them. While the last two are trivial, the help for CodeType says "Not for the faint of heart." I suspect that some of these accessible APIs are new since 1.3, or may have changed. It is hard to know since they are not documented ;-).
The types themselves are documented as public, they aren't given _single underscore private names,
The module says, "Typical use is for isinstance() or issubclass() checks." Exposing the internal types was done to do once and do correctly things that people did (and sometimes still do) in their code, such as isinstance(f, type(lambda: None)). The module has similar code to get the types. def _f(): pass FunctionType = type(_f) LambdaType = type(lambda: None) # Same as FunctionType CodeType = type(_f.__code__) So there is not way for the objects themselves to not be public. The names bound to them in types are not their .__name__ attributes. It is certainly intentional that they are not bound to their names in builtins. Getting traceback and frame types is trickier to get right: try: raise TypeError except TypeError: tb = sys.exc_info()[2] TracebackType = type(tb) FrameType = type(tb.tb_frame) tb = None; del tb
and FunctionType has a minimal but useful doc string with nothing about it being implementation specific.
Marking *anything* other than ref-counting and gc behavior as implementation specific in the docs is fairly recent. The same goes for cpython-only tests in the test suite. Some of this splitting has been prompted by questions from Iron-py and pypy developers. -- Terry Jan Reedy
On Sun, Mar 18, 2012 at 5:27 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict:
py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping
I propose to allow function.__globals__ to accept any mapping type.
That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules.
E.g. if I want to write a function with a custom namespace, I have to do something like this:
ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d)
which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this:
ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d)
# This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__)
(By the way, ChainMap is only one possible example namespace.)
A casual search of the archives identifies some similar discussion over the years [1]. If PyEval_EvalCodeEx were more fully exposed in Python things would be simpler (see function_call() [2]). I made a small effort to that effect last year, but it turned out to be relatively unnecessary. However, I didn't factor in the restriction on the type of globals... -eric [1] past discussion/efforts: http://www.python.org/download/releases/2.2/descrintro/#subclassing (optimization surrounding globals as dict) http://mail.python.org/pipermail/python-dev/2002-October/029752.html http://mail.python.org/pipermail/python-dev/2002-October/029753.html http://mail.python.org/pipermail/python-dev/2002-October/029761.html http://bugs.python.org/issue215126 http://mail.python.org/pipermail/python-list/2010-December/1262045.html [2] http://hg.python.org/cpython/file/default/Objects/funcobject.c#l592
On Sun, Mar 18, 2012 at 11:27:09PM +1100, Steven D'Aprano wrote:
I propose to allow function.__globals__ to accept any mapping type.
Jython already supports this behaviour. steve@runes:~$ jython [...] Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19) [OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_18 Type "help", "copyright", "credits" or "license" for more information.
class Mapping:
... def __getitem__(self, key): ... if key == 'a': return 1 ... else: raise KeyError(key) ...
from types import FunctionType f = lambda x: x + a g = FunctionType(f.func_code, Mapping(), 'g') g(2) 3
-- Steven
+1 on the idea. Would be interesting to play with frozendict (if it's accepted someday) On 2012-03-18, at 8:27 AM, Steven D'Aprano wrote:
Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict:
py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping
I propose to allow function.__globals__ to accept any mapping type.
That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules.
E.g. if I want to write a function with a custom namespace, I have to do something like this:
ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d)
which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this:
ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d)
# This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__)
(By the way, ChainMap is only one possible example namespace.)
-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
I propose to allow function.__globals__ to accept any mapping type.
I proposed a different but related change: support types other than dict for __builtins__ in the following issue. http://bugs.python.org/issue14385 Victor
participants (7)
-
Andrew Svetlov -
Eric Snow -
Robert Kern -
Steven D'Aprano -
Terry Reedy -
Victor Stinner -
Yury Selivanov