Namespace context managers
I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects. It would make closures possible in python with a pythonic syntax. a = 4 namespace = {} with namespace: a = 3 assert a == 4 assert namespace["a"] == 3
On Fri, Jul 26, 2019 at 10:13:45AM +0300, Batuhan Taskaya wrote:
I don't understand what you mean here. Closures are already possible in Python with a pythonic syntax.
I have long wanted namespaces in Python, but I don't like the above. I would prefer to see something like this: a = 4 with Namespace("ns") as ns: a = 3 print(a) # prints "3", not "4" def func(): return a # Closes on ns.a, not global a assert isinstance(ns, types.ModuleType) assert ns.name = "ns" assert ns.func() == 3 assert a == 4 Our standard namespace is the module, which is great for small libraries and scripts. When your needs are greater (too much code to comfortably co-exist in a single file), you can use a package. But when your needs are lower, and a seperate .py file is too heavyweight, we don't have anything convenient for a seperate namespace. You can build a module object by hand: from types import ModuleType ns = ModuleType("ns") ns.a = 3 def func(): return ns.a ns.func = func but it's not pretty code, its not convenient, and closures and name lookups don't work or look right. You can use a class instead, but again closures don't work right, and it is surprising to use a class object without instantiating it. I think a namespace context manager that created and populated a new module (or subclass of module) object would fit this use-case nicely. Use-case summary: You have a collection of classes, functions and variables which should live together in a namespace, seperate from the rest of your classes etc, but you don't want to push them out into a seperate .py file. -- Steven
This is a neat idea and I have wanted something similar myself in situations I do not want to instantiate class object or break code out to another module. Controversial opinion: it may even justify a keyword. But it wouldn't have to be a new one, def could work just fine: def ns: a = 3 Food goodly or for badly: something like this would allow one to write code similar to how you can in javascript (by simply putting it in curly braces in js), without going to the effort of creating a full fledged class: def a: x = 1 def b: x = 2 def func(): return x # prints 1 print(a.b.x) # prints 2 Final observation: I think one shortfall in python is that it nudges the user to use OOP a little bit too forcefully-- especially inexperienced users, after learning how to write a class the first time. I remember very well regretting using classes quite a bit too readily in my first couple years of coding (on year 6 now), when a module would have done much better. It's true you CAN create a module object and do all of this, but as a relatively "non expert user" IMO it is very opaque syntax for someone getting started.
On Jul 26, 2019, at 05:25, Ricky Teachey <ricky@teachey.org> wrote:
It’s not clear to me whether people want a module-like, class-like, or function-like namespace here, but I do think it’s probably exactly one of those three. And it’ll be a lot easier to define and implement—and for everyone to understand future Python code—if it is. I don’t think anyone cares about fast locals here, and I hope nobody cares about things like super magic. But the differences in how variables in the namespace interact with functions and classes (and new-kind-of-namespace-things) defined within the namespace do probably matter. And you’d probably want to know what to expect from eval/exec, locals/globals/vars, etc., because someone is going to do that and you want it to be obvious what it means, rather than having to dig through the spec or trial-and-error at the interactive prompt to figure it out. And people probably occasionally want to know how to do the same thing programmatically (just like people occasionally want to know how to call a metaclass explicitly). But the nesting is the key question. Meanwhile, do we even need to name the namespace? While it does allow the JS-style trick you gave for defining a “prototype” object without a class, do we actually want that in Python? And are there any other practical uses for names here? If not, there’s a really easy design and implementation, something like one of the following: A bare “def:” statement defines a function, doesn’t bind it to anything, calls it with no args, and discards both the function and the result. So locals, nonlocal, eval, etc. work exactly the same way as in any other def body. A bare “class:” statement defines a class namespace, doesn’t call the metaclass, and doesn’t bind anything to anything. Again, everything works exactly the same way as in any other class body. A bare “import:” statement defines a module namespace, doesn’t construct a module out of it, and doesn’t bind anything to anything.
Final observation: I think one shortfall in python is that it nudges the user to use OOP a little bit too forcefully
I think it’s a strength of Python that it uses OOP under the hood for all kinds of stuff, but usually doesn’t make you think in OOP terms while writing and reading the code. And I think your version of the proposal takes it farther in that direction, which is good. For example, nobody thinks of def as being syntactic sugar for constructing an object of type function that has a descriptor that returns an object of type method, and then binding it to a name. It just defines a method on a class, and it works. But, because that’s what it does under the covers, on the rare occasions where you need to go under the covers, you use the same OO style as with everything else. Compare that to the way reflection on methods works in, say, ObjC, where you end up calling a bunch of C functions, and passing them not actual classes and methods but pointers to opaque Class and Selector objects, and it feels more like you’re writing an ObjC interpreter than writing ObjC code.
On Fri, Jul 26, 2019 at 01:37:29PM -0700, Andrew Barnert wrote:
I'm not going to speak for "people", but for me, I think that the answer should be obvious: it is module-like. I'm not even sure what you mean by "function-like" or "class-like" namespaces. Functions aren't namespaces (except in the sense that they are objects with a __dict__ and so support attributes): you can't access a function locals from the outside. Classes are namespaces, but if you want a namespace that behaves like a class, use a class. For me, the idea is to (optionally) decouple module objects from .py files. Of course most of the time when you want a module object, it is most useful to split the code out into a seperate file, and that won't change. But there are times when you have a relatively small amount of code that cries to be split off into a seperate namespace, but shifting it into a seperate physical file is inconvenient. Going from one physical file to two physical files is a significant jump in project complexity that's not always worth the cost. So a namespace object that behaves like a module but doesn't need to split off into a seperate .py file would be helpful. The code in a namespace object ought to behave precisely the same as if it were copied into a seperate file. (*Almost* -- see below.) That is, given some module file containing code: # module.py code I should be able to copy the content of the file ("code"), and paste it into another file like this: # another.py with Namespace("module") as module: # paste and indent here code and the resulting namespace object bound to the name "module" should be (almost) identical to what you would have got from calling "import module" in the first place. A few possible differences: 1. It isn't clear what the namespace __file__ and __package__ attributes ought to contain, or if it should have them at all. 2. For introspection purposes, and to allow for possible future changes, it might be wise to use a subclass of ModuleType for namespace objects. 3. The repr will probably be different. One more difference needs a bit of explanation: The standard scoping rule used by Python is LEGB, which goes: - Local - Enclosing functions (non-local/closures) - Global (module) - Builtins with a slight variation on that for code inside classes. So when you run code in module.py, the global namespace it sees is module.py i.e. itself. But consider what happens when you copy the code from module.py and paste it into another file, into a namespace. It would be surprising if the code inside the namespace with block couldn't see module level globals. So we need a slight variation on the scoping rule: - Local - Enclosing functions (non-local/closures) - Namespace - Global (module) - Builtins (and an analogous change for when classes are involved). To make it more clear with a concrete example: a = 1 b = 2 c = 3 with Namespace("ns") as ns: a = 100 b = 200 def function(): a = 999 return a, b, c ns.function() ought to return (999, 200, 3). This implies a small difference in behaviour for code in a namespace versus a seperate physical file. If I copied that code block out of ns above, and pasted it into a physical file, then running function() would raise: NameError: name 'c' is not defined I think that this slight difference will be considered desirable, unsurprising (it would be surprising if namespaces didn't see their surrounding global scope, and even more surprising if external modules could pry into another module!) and hopefully uncontroversial.
I don’t think anyone cares about fast locals here, and I hope nobody cares about things like super magic.
A function shouldn't become slower just because you move it into a namespace. And classes ought to behave the same, including super, whether they were defined inside or outside a namespace.
Absolutely! There are a couple of tricky corner cases involving eval (and exec?) already, related to comprehensions. They will need to be ironed out. I've spent some time (unsuccessfully) trying to get this to work using a class statement: @namespace class ns: a = 100 def function(): print(a) but I couldn't get the function inside the class to see the surrounding "a". If Batuhan Taskaya has solved that problem using a with statement, it sounds like there's no technical barriers to starting with a third party library, iron out the corner cases, and then consider moving it into the std lib. Or even a keyword: namespace ns: code which would eliminate the need to repeat ourselves when naming the namespace.
Meanwhile, do we even need to name the namespace?
How do you refer to it without binding it to a name? Does it need an internal ns.__name__ attribute? I think that would be useful for the repr, introspection etc. It is sad that we have to repeat ourselves: with namespace("ns") as ns but that's a limitation of the language. We could solve it with a keyword, but let's get a working, usable prototype first before worrying about justifying a keyword. -- Steven
Let's say you do this or any of the variants suggested... What does this do? a = {"foo": 1} b = {} with a: with b: foo = 0 On Fri, Jul 26, 2019 at 3:20 AM Batuhan Taskaya <isidentical@gmail.com> wrote:
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 [image: https://red.ht/sig] <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Fri, Jul 26, 2019 at 10:52:46AM -0400, Calvin Spealman wrote:
Re-writing that to my suggested version: with namespace("a") as a: with namespace("b") as b: foo = 0 will create a namespace object (a subclass of ModuleType) and bind it to "a", containing a namespace object "b", containing a variable "foo": assert isinstance(a, NamespaceType) assert isinstance(a.b, NamespaceType) assert a.b.foo == 0 That would be analogous to this existing code: class a: class b: foo = 0 except classes have different scoping rules to modules. -- Steven
I feel like this could have some interesting side uses (typing on mobile so ignore the crappy indentation): with Namespace('a'): x = 1 y = 2 print(a.__dict__) # we now have a dict The GN build system flat-out has no dictionary / mapping type in favor of scopes like this. On Sun, Jul 28, 2019, 9:36 PM Steven D'Aprano <steve@pearwood.info> wrote:
The class statement works in this fashion.
I would argue there are at least 2 significant reasons why a separate namespace would be preferred over using class in this way: 1. using class in this way signals to the user that you are writing a class (ie, meant to be instantiated), and not a namespace; the author has to tell the reader it isn't actually intended to be a class in the docstring or in the class name ("eg, MyNamespace") 2. using class means that all of the "new style class" machinery is at work-- for good or for ill-- when working in your class-defined namespace. so if, for example, you create a descriptor and instantiate it inside a class namespace, the descriptor __get__ method will be invoked when accessing the descriptor via the class name. this could be very limiting in certain cases.
On 7/26/19 12:23 PM, Ricky Teachey wrote:
[...]
Flat is better than nested. Yes, at some point, flat becomes unweildy, and you have to do something. Or not. Simple is better than complex. With every one of these nesting constructs (e.g., comprehensions, classes, functions, methods, modules), there are possibly subtle scoping and name lookup rules. For every new such construct, there are new and possibly subtle such rules. At some point, the expressiveness of a language disappears into the noise of complexity (no, I don't have a citation, just lots of experience). There Is Only One Way To Do It. Enough said. I'm pretty sure I'm not a luddite, but I'm also pretty sure that new ways to capture ("close over") a value and/or to encapsulate state is a long way from your father's executable pseudo code. Yes, at some point, every little throwaway automation script takes on a life of its own, and you have to make a choice. Please choose the simplicity of small reusable building blocks (aka modules full of functions; or classes full of methods if you must) rather than yet another way to inline your encapsulation.
On 26/07/2019 08:13, Batuhan Taskaya wrote:
I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects.
Didn't we have this discussion recently? What do these __enter__() and __exit__() do? Please don't answer with an example, I want to understand what this mechanism is supposed to achieve. Also, I can't tell if you're proposing to add __enter__ and __exit__ to dict objects or have the "with" statement magically add them. Is it just dicts, or will any mapping type do?
Right, so it's in effect supplying a different dictionary for local variables. How do these stack? Innermost namespace takes priority? Are outer namespaces accessible using nonlocal? How does this interact with other sorts of namespaces? I don't think dicts are the right place to start. I think you want a specialised namespace object that supplies a mapping interface. That gives you a lot more flexibility when it comes to implementation, amongst other things. (I'm not actually very keen on namespaces like this, personally. Too much debugging of C++ code has made me very twitchy about unstructured names. That's besides the point, though.) -- Rhodri James *-* Kynesim Ltd
This proposal seems to be harking back to the meaning of 'with' in earlier languages, e.g. Pascal, where it makes the fields of a structure temporarily accessible without qualification. Before we got the current version of 'with' in Python, there were periodic proposals to add a similar kind of 'with', but the consensus was that it was better to just assign the object to a short local name and qualify with that. As far as I can see, the situation hasn't changed. -- Greg
I agree. This is why my preferred solution would be more something like I proposed, such as: def my_namespace: a = 1 def f(x): ... etc etc. I also do not see any benefit in being able to provide some arbitrary string as the module/namespace name, as others have proposed: with Namespace('ns') as ns: ... Modules are already named by the file name they reside in. Functions (also defined using "def") are named using the name given to the function. Same with a class. Creating a namespace/module type thing that receives its name as some argument would seem, to me, to be orthogonal to these other situations.
On Mon, Jul 29, 2019 at 05:58:02PM -0400, Ricky Teachey wrote:
This has the advantages of: 1. Avoiding the need to repeat the name twice. 2. Avoiding the need for a new keyword but two serious disadvantages: 1. It is currently illegal syntax, so it doesn't work without compiler support. That means we can't publish it as a module on PyPy, can't backport it to older versions, and you have to convince the core-devs that both the feature itself and the syntax are worth building into the language. 2. It looks like a function. That means that these two very similar looking defs do *radically* different things: def spam(): # creates a FUNCTION and binds to name "spam" def spam: # creates a NAMESPACE and binds to name "spam" That will lead to a lot of accidental "why is my namespace/function not working?" problems. As a prototype, it might be possible to wrap the machinary to create a namespace in a function call, but as permanant language syntax, it isn't a good idea.
It's not ideal because we are repeating ourselves, but if its good enough for namedtuples, its good enough for namespaces. If the core developers don't consider namedtuples important enough to get syntactic support, I doubt that namespaces will. If namespace objects are considered a kind of module or module-like object, they need a name for the repr and for introspection, and they need to be bound to a name (a variable). The name binding is automatically handled by the with-statement, but how else are you going to inform the namespace object of its internal name?
This doesn't "just happen", but because the interpreter does the work of extracting the name and feeding it to the constructors, which have these signatures: ModuleType(name[, doc]) FunctionType(code, globals[, name[, argdefs[, closure]]]) type(name, bases, dict) # Creates a new class.
How else are you going to inform the namespace object what name it should use internally, without interpreter support? -- Steven
If the core developers don't consider namedtuples important enough to get syntactic support, I doubt that namespaces will.
This by itself is pretty convincing. This doesn't "just happen", but because the interpreter does the work I understand that; my conception of the idea was to add this syntax to the interpreter, so that it is creating module objects. One could get around the "why is my function broken?" issue by recycling another already used keyword, if there could be one made to work (perhaps import?). But you made a pretty convincing case that this is very unlikely to happen (though I'd definitely argue for it, I think, if I thought it were actually a possibility...). But I ask: how is it possible, then, to create a with block that "gobbles up" things declared in it and adds those things to some namespace using existing python syntax...? Won't the syntax HAVE to be changed- or the way that syntax is currently interpreted be modified- to make any of the suggested versions of the idea possible? I currently can't create an object that allows this to work: with NameSpace('ns') as ns: a = 1 assert ns.a=1 That would require blessing a big change by the core devs to work. One could write a metaclass that does it, I suppose... but it doesn't look particularly great to me: import types class NameSpace(type): def __new__(mcls, name, bases, dct): if bases: raise TypeError("this is a namespace, not a class.") mod = types.ModuleType(dct.pop("__qualname__"), dct.pop("__doc__", None)) mod.__dict__.update(dct) return mod class ns(metaclass=NameSpace): class Desc: def __get__(self, obj, cls): return None d = Desc() assert ns.d # descriptor protocol broken as expected This seems to work. I'm sure there are tons of things wrong with it that could be fixed. But again, the code isn't very elegant.
On Mon, Jul 29, 2019 at 09:53:38PM -0400, Ricky Teachey wrote:
Neither can I, and I've been mucking about with this idea for about three years. (To be honest, I poke it with a stick about every six months, so its not like I've spend a lot of time on it.) I kinda-sorta got it to work with a class statement plus a decorator, not too different from your metaclass example, but couldn't get functions to work correctly. But if I understand the original post here: https://mail.python.org/archives/list/python-ideas@python.org/message/TAVHEK... Batuhan Taskaya has got it to work using a dict. If that's the case, that seems to be the hardest part done. (On the other hand, if this is just a hypothetical "wouldn't it be cool if this worked?" post, then we're actually no closer to a technical solution.) -- Steven
On Tue, Jul 30, 2019 at 12:13 PM Steven D'Aprano <steve@pearwood.info> wrote:
A class + decorator gets so almost there that it might be possible with just a tiny bit of language support. What if there could be a way to change a function's __globals__? Currently that's readonly according to funcobject.c, but by hacking that flag out, I can create the following example: def namespace(cls): """Decorate a class to turn it into a namespace""" ns = dict(cls.__dict__) for member in ns.values(): if hasattr(member, "__globals__"): member.__globals__ = ns return cls @namespace class ns: x = 1 def foo(y): return x + y def replace(new_x): global x x = new_x print(ns.foo(4)) # 5 ns.replace(100) print(ns.foo(4)) # 104 Due to CPython limitations, the new __globals__ MUST be a plain dict (not the mappingproxy that cls.__dict__ is, and not a ChainMap), but this is extremely close to what could actually be quite useful. Also, since the function objects get mutated by the decorator, calling a function during the namespace's construction will have the wrong behaviour. Maybe the only language support needed is some way to say "from now on, functions should be created with THIS value for their globals"? It might be possible to get even closer using a metaclass rather than a decorator. ChrisA
On Tue, Jul 30, 2019 at 12:54:12PM +1000, Chris Angelico wrote:
That's one of the problems I hit in my experiments. I don't think you need to hack funcobject.c if you are willing to disassemble the function object (all of the parts are accessible as attributes) and re-assemble it into a new function using the types.FunctionType constructor.
Due to CPython limitations, the new __globals__ MUST be a plain dict (not the mappingproxy that cls.__dict__ is, and not a ChainMap),
In my experiments, I think I overcame that by subclassing ChainMap and dict: class MyChainMap(ChainMap, dict): pass was, I think, enough to fool the function constructor. Don't quote me on this, it's been a while and I don't remember all the details. But I'm pretty sure nothing segfaulted, I'd remember it if it did :-) I'm pretty sure that this is doable, at least well enough to get a working prototype that people can experiment with. At worst, we might need AST or bytecode hacking to get the functions defined inside the namespace to work correctly.
Yes, there's that too. But then the namespace doesn't actually exist at that point, so this is rather analogous to what happens when you call a function inside a class during class construction. I can live with that limitation. -- Steven
Forgive me if this doesn't make sense, but what about a way to make a copy of a function's locals while it is being called? Maybe something like: @namespace def ns(): x = 1 def f(): ... Inside the decorator, you simply call the function, make a copy of the locals, and assign it to the function name as a module?
Forgive me if this doesn't make sense, but what about a way to make a copy of a function's locals while it is being called?
This can be easily accomplished if you simply return locals() at the end of creating the namespace, but this isn't all that elegant: def namespace(ns): ns_locals = ns() from types import ModuleType mod = ModuleType(ns.__name__) mod.__dict__.update(ns_locals) return mod @namespace def ns(): x=1 y=2 def f(): ... return locals()
On Mon, Jul 29, 2019 at 6:55 PM Ricky Teachey <ricky@teachey.org> wrote: [Steven]
If the core developers don't consider namedtuples important enough to
get syntactic support, I doubt that namespaces will.
[Ricky]
This by itself is pretty convincing.
Except it's wrong. The successor of namedtuple is typing.NamedTuple which supports this syntactically. From the [docs]( https://docs.python.org/3/library/typing.html#typing.NamedTuple): ``` class Employee(NamedTuple): name: str id: int ``` -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Except it's wrong. The successor of namedtuple is typing.NamedTuple which supports this syntactically.
Point taken. And I definitely don't mean to imply the core devs aren't reasonable people or are unwilling to make big changes, I'm just not experienced enough to be the guy making the very-thoroughly-well-thought-out-version of the case for a major syntax change.
On Mon, Jul 29, 2019 at 07:19:42PM -0700, Guido van Rossum wrote:
Oh that's nice! I thought that typing.NamedTuple was only for type annotations, I had no idea you could use it for actual named tuples. Should it be aliased in collections if we want people to consider using it instead of namedtuple? I would never go looking in typing for non-annotation-related uses, and I'm probably not the only one. In any case, that's still using existing syntax (class statement), not a dedicated new "NamedTuple" keyword. And if there's no NamedTuple keyword, we shouldn't expect a Namespace keyword either :-) -- Steven
On Thu, Aug 08, 2019 at 10:28:44AM +0100, Ivan Levkivskyi wrote:
Done: https://bugs.python.org/issue37809 -- Steven
On Fri, Jul 26, 2019 at 10:13:45AM +0300, Batuhan Taskaya wrote:
I don't understand what you mean here. Closures are already possible in Python with a pythonic syntax.
I have long wanted namespaces in Python, but I don't like the above. I would prefer to see something like this: a = 4 with Namespace("ns") as ns: a = 3 print(a) # prints "3", not "4" def func(): return a # Closes on ns.a, not global a assert isinstance(ns, types.ModuleType) assert ns.name = "ns" assert ns.func() == 3 assert a == 4 Our standard namespace is the module, which is great for small libraries and scripts. When your needs are greater (too much code to comfortably co-exist in a single file), you can use a package. But when your needs are lower, and a seperate .py file is too heavyweight, we don't have anything convenient for a seperate namespace. You can build a module object by hand: from types import ModuleType ns = ModuleType("ns") ns.a = 3 def func(): return ns.a ns.func = func but it's not pretty code, its not convenient, and closures and name lookups don't work or look right. You can use a class instead, but again closures don't work right, and it is surprising to use a class object without instantiating it. I think a namespace context manager that created and populated a new module (or subclass of module) object would fit this use-case nicely. Use-case summary: You have a collection of classes, functions and variables which should live together in a namespace, seperate from the rest of your classes etc, but you don't want to push them out into a seperate .py file. -- Steven
This is a neat idea and I have wanted something similar myself in situations I do not want to instantiate class object or break code out to another module. Controversial opinion: it may even justify a keyword. But it wouldn't have to be a new one, def could work just fine: def ns: a = 3 Food goodly or for badly: something like this would allow one to write code similar to how you can in javascript (by simply putting it in curly braces in js), without going to the effort of creating a full fledged class: def a: x = 1 def b: x = 2 def func(): return x # prints 1 print(a.b.x) # prints 2 Final observation: I think one shortfall in python is that it nudges the user to use OOP a little bit too forcefully-- especially inexperienced users, after learning how to write a class the first time. I remember very well regretting using classes quite a bit too readily in my first couple years of coding (on year 6 now), when a module would have done much better. It's true you CAN create a module object and do all of this, but as a relatively "non expert user" IMO it is very opaque syntax for someone getting started.
On Jul 26, 2019, at 05:25, Ricky Teachey <ricky@teachey.org> wrote:
It’s not clear to me whether people want a module-like, class-like, or function-like namespace here, but I do think it’s probably exactly one of those three. And it’ll be a lot easier to define and implement—and for everyone to understand future Python code—if it is. I don’t think anyone cares about fast locals here, and I hope nobody cares about things like super magic. But the differences in how variables in the namespace interact with functions and classes (and new-kind-of-namespace-things) defined within the namespace do probably matter. And you’d probably want to know what to expect from eval/exec, locals/globals/vars, etc., because someone is going to do that and you want it to be obvious what it means, rather than having to dig through the spec or trial-and-error at the interactive prompt to figure it out. And people probably occasionally want to know how to do the same thing programmatically (just like people occasionally want to know how to call a metaclass explicitly). But the nesting is the key question. Meanwhile, do we even need to name the namespace? While it does allow the JS-style trick you gave for defining a “prototype” object without a class, do we actually want that in Python? And are there any other practical uses for names here? If not, there’s a really easy design and implementation, something like one of the following: A bare “def:” statement defines a function, doesn’t bind it to anything, calls it with no args, and discards both the function and the result. So locals, nonlocal, eval, etc. work exactly the same way as in any other def body. A bare “class:” statement defines a class namespace, doesn’t call the metaclass, and doesn’t bind anything to anything. Again, everything works exactly the same way as in any other class body. A bare “import:” statement defines a module namespace, doesn’t construct a module out of it, and doesn’t bind anything to anything.
Final observation: I think one shortfall in python is that it nudges the user to use OOP a little bit too forcefully
I think it’s a strength of Python that it uses OOP under the hood for all kinds of stuff, but usually doesn’t make you think in OOP terms while writing and reading the code. And I think your version of the proposal takes it farther in that direction, which is good. For example, nobody thinks of def as being syntactic sugar for constructing an object of type function that has a descriptor that returns an object of type method, and then binding it to a name. It just defines a method on a class, and it works. But, because that’s what it does under the covers, on the rare occasions where you need to go under the covers, you use the same OO style as with everything else. Compare that to the way reflection on methods works in, say, ObjC, where you end up calling a bunch of C functions, and passing them not actual classes and methods but pointers to opaque Class and Selector objects, and it feels more like you’re writing an ObjC interpreter than writing ObjC code.
On Fri, Jul 26, 2019 at 01:37:29PM -0700, Andrew Barnert wrote:
I'm not going to speak for "people", but for me, I think that the answer should be obvious: it is module-like. I'm not even sure what you mean by "function-like" or "class-like" namespaces. Functions aren't namespaces (except in the sense that they are objects with a __dict__ and so support attributes): you can't access a function locals from the outside. Classes are namespaces, but if you want a namespace that behaves like a class, use a class. For me, the idea is to (optionally) decouple module objects from .py files. Of course most of the time when you want a module object, it is most useful to split the code out into a seperate file, and that won't change. But there are times when you have a relatively small amount of code that cries to be split off into a seperate namespace, but shifting it into a seperate physical file is inconvenient. Going from one physical file to two physical files is a significant jump in project complexity that's not always worth the cost. So a namespace object that behaves like a module but doesn't need to split off into a seperate .py file would be helpful. The code in a namespace object ought to behave precisely the same as if it were copied into a seperate file. (*Almost* -- see below.) That is, given some module file containing code: # module.py code I should be able to copy the content of the file ("code"), and paste it into another file like this: # another.py with Namespace("module") as module: # paste and indent here code and the resulting namespace object bound to the name "module" should be (almost) identical to what you would have got from calling "import module" in the first place. A few possible differences: 1. It isn't clear what the namespace __file__ and __package__ attributes ought to contain, or if it should have them at all. 2. For introspection purposes, and to allow for possible future changes, it might be wise to use a subclass of ModuleType for namespace objects. 3. The repr will probably be different. One more difference needs a bit of explanation: The standard scoping rule used by Python is LEGB, which goes: - Local - Enclosing functions (non-local/closures) - Global (module) - Builtins with a slight variation on that for code inside classes. So when you run code in module.py, the global namespace it sees is module.py i.e. itself. But consider what happens when you copy the code from module.py and paste it into another file, into a namespace. It would be surprising if the code inside the namespace with block couldn't see module level globals. So we need a slight variation on the scoping rule: - Local - Enclosing functions (non-local/closures) - Namespace - Global (module) - Builtins (and an analogous change for when classes are involved). To make it more clear with a concrete example: a = 1 b = 2 c = 3 with Namespace("ns") as ns: a = 100 b = 200 def function(): a = 999 return a, b, c ns.function() ought to return (999, 200, 3). This implies a small difference in behaviour for code in a namespace versus a seperate physical file. If I copied that code block out of ns above, and pasted it into a physical file, then running function() would raise: NameError: name 'c' is not defined I think that this slight difference will be considered desirable, unsurprising (it would be surprising if namespaces didn't see their surrounding global scope, and even more surprising if external modules could pry into another module!) and hopefully uncontroversial.
I don’t think anyone cares about fast locals here, and I hope nobody cares about things like super magic.
A function shouldn't become slower just because you move it into a namespace. And classes ought to behave the same, including super, whether they were defined inside or outside a namespace.
Absolutely! There are a couple of tricky corner cases involving eval (and exec?) already, related to comprehensions. They will need to be ironed out. I've spent some time (unsuccessfully) trying to get this to work using a class statement: @namespace class ns: a = 100 def function(): print(a) but I couldn't get the function inside the class to see the surrounding "a". If Batuhan Taskaya has solved that problem using a with statement, it sounds like there's no technical barriers to starting with a third party library, iron out the corner cases, and then consider moving it into the std lib. Or even a keyword: namespace ns: code which would eliminate the need to repeat ourselves when naming the namespace.
Meanwhile, do we even need to name the namespace?
How do you refer to it without binding it to a name? Does it need an internal ns.__name__ attribute? I think that would be useful for the repr, introspection etc. It is sad that we have to repeat ourselves: with namespace("ns") as ns but that's a limitation of the language. We could solve it with a keyword, but let's get a working, usable prototype first before worrying about justifying a keyword. -- Steven
Let's say you do this or any of the variants suggested... What does this do? a = {"foo": 1} b = {} with a: with b: foo = 0 On Fri, Jul 26, 2019 at 3:20 AM Batuhan Taskaya <isidentical@gmail.com> wrote:
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 [image: https://red.ht/sig] <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Fri, Jul 26, 2019 at 10:52:46AM -0400, Calvin Spealman wrote:
Re-writing that to my suggested version: with namespace("a") as a: with namespace("b") as b: foo = 0 will create a namespace object (a subclass of ModuleType) and bind it to "a", containing a namespace object "b", containing a variable "foo": assert isinstance(a, NamespaceType) assert isinstance(a.b, NamespaceType) assert a.b.foo == 0 That would be analogous to this existing code: class a: class b: foo = 0 except classes have different scoping rules to modules. -- Steven
I feel like this could have some interesting side uses (typing on mobile so ignore the crappy indentation): with Namespace('a'): x = 1 y = 2 print(a.__dict__) # we now have a dict The GN build system flat-out has no dictionary / mapping type in favor of scopes like this. On Sun, Jul 28, 2019, 9:36 PM Steven D'Aprano <steve@pearwood.info> wrote:
The class statement works in this fashion.
I would argue there are at least 2 significant reasons why a separate namespace would be preferred over using class in this way: 1. using class in this way signals to the user that you are writing a class (ie, meant to be instantiated), and not a namespace; the author has to tell the reader it isn't actually intended to be a class in the docstring or in the class name ("eg, MyNamespace") 2. using class means that all of the "new style class" machinery is at work-- for good or for ill-- when working in your class-defined namespace. so if, for example, you create a descriptor and instantiate it inside a class namespace, the descriptor __get__ method will be invoked when accessing the descriptor via the class name. this could be very limiting in certain cases.
On 7/26/19 12:23 PM, Ricky Teachey wrote:
[...]
Flat is better than nested. Yes, at some point, flat becomes unweildy, and you have to do something. Or not. Simple is better than complex. With every one of these nesting constructs (e.g., comprehensions, classes, functions, methods, modules), there are possibly subtle scoping and name lookup rules. For every new such construct, there are new and possibly subtle such rules. At some point, the expressiveness of a language disappears into the noise of complexity (no, I don't have a citation, just lots of experience). There Is Only One Way To Do It. Enough said. I'm pretty sure I'm not a luddite, but I'm also pretty sure that new ways to capture ("close over") a value and/or to encapsulate state is a long way from your father's executable pseudo code. Yes, at some point, every little throwaway automation script takes on a life of its own, and you have to make a choice. Please choose the simplicity of small reusable building blocks (aka modules full of functions; or classes full of methods if you must) rather than yet another way to inline your encapsulation.
On 26/07/2019 08:13, Batuhan Taskaya wrote:
I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects.
Didn't we have this discussion recently? What do these __enter__() and __exit__() do? Please don't answer with an example, I want to understand what this mechanism is supposed to achieve. Also, I can't tell if you're proposing to add __enter__ and __exit__ to dict objects or have the "with" statement magically add them. Is it just dicts, or will any mapping type do?
Right, so it's in effect supplying a different dictionary for local variables. How do these stack? Innermost namespace takes priority? Are outer namespaces accessible using nonlocal? How does this interact with other sorts of namespaces? I don't think dicts are the right place to start. I think you want a specialised namespace object that supplies a mapping interface. That gives you a lot more flexibility when it comes to implementation, amongst other things. (I'm not actually very keen on namespaces like this, personally. Too much debugging of C++ code has made me very twitchy about unstructured names. That's besides the point, though.) -- Rhodri James *-* Kynesim Ltd
This proposal seems to be harking back to the meaning of 'with' in earlier languages, e.g. Pascal, where it makes the fields of a structure temporarily accessible without qualification. Before we got the current version of 'with' in Python, there were periodic proposals to add a similar kind of 'with', but the consensus was that it was better to just assign the object to a short local name and qualify with that. As far as I can see, the situation hasn't changed. -- Greg
I agree. This is why my preferred solution would be more something like I proposed, such as: def my_namespace: a = 1 def f(x): ... etc etc. I also do not see any benefit in being able to provide some arbitrary string as the module/namespace name, as others have proposed: with Namespace('ns') as ns: ... Modules are already named by the file name they reside in. Functions (also defined using "def") are named using the name given to the function. Same with a class. Creating a namespace/module type thing that receives its name as some argument would seem, to me, to be orthogonal to these other situations.
On Mon, Jul 29, 2019 at 05:58:02PM -0400, Ricky Teachey wrote:
This has the advantages of: 1. Avoiding the need to repeat the name twice. 2. Avoiding the need for a new keyword but two serious disadvantages: 1. It is currently illegal syntax, so it doesn't work without compiler support. That means we can't publish it as a module on PyPy, can't backport it to older versions, and you have to convince the core-devs that both the feature itself and the syntax are worth building into the language. 2. It looks like a function. That means that these two very similar looking defs do *radically* different things: def spam(): # creates a FUNCTION and binds to name "spam" def spam: # creates a NAMESPACE and binds to name "spam" That will lead to a lot of accidental "why is my namespace/function not working?" problems. As a prototype, it might be possible to wrap the machinary to create a namespace in a function call, but as permanant language syntax, it isn't a good idea.
It's not ideal because we are repeating ourselves, but if its good enough for namedtuples, its good enough for namespaces. If the core developers don't consider namedtuples important enough to get syntactic support, I doubt that namespaces will. If namespace objects are considered a kind of module or module-like object, they need a name for the repr and for introspection, and they need to be bound to a name (a variable). The name binding is automatically handled by the with-statement, but how else are you going to inform the namespace object of its internal name?
This doesn't "just happen", but because the interpreter does the work of extracting the name and feeding it to the constructors, which have these signatures: ModuleType(name[, doc]) FunctionType(code, globals[, name[, argdefs[, closure]]]) type(name, bases, dict) # Creates a new class.
How else are you going to inform the namespace object what name it should use internally, without interpreter support? -- Steven
If the core developers don't consider namedtuples important enough to get syntactic support, I doubt that namespaces will.
This by itself is pretty convincing. This doesn't "just happen", but because the interpreter does the work I understand that; my conception of the idea was to add this syntax to the interpreter, so that it is creating module objects. One could get around the "why is my function broken?" issue by recycling another already used keyword, if there could be one made to work (perhaps import?). But you made a pretty convincing case that this is very unlikely to happen (though I'd definitely argue for it, I think, if I thought it were actually a possibility...). But I ask: how is it possible, then, to create a with block that "gobbles up" things declared in it and adds those things to some namespace using existing python syntax...? Won't the syntax HAVE to be changed- or the way that syntax is currently interpreted be modified- to make any of the suggested versions of the idea possible? I currently can't create an object that allows this to work: with NameSpace('ns') as ns: a = 1 assert ns.a=1 That would require blessing a big change by the core devs to work. One could write a metaclass that does it, I suppose... but it doesn't look particularly great to me: import types class NameSpace(type): def __new__(mcls, name, bases, dct): if bases: raise TypeError("this is a namespace, not a class.") mod = types.ModuleType(dct.pop("__qualname__"), dct.pop("__doc__", None)) mod.__dict__.update(dct) return mod class ns(metaclass=NameSpace): class Desc: def __get__(self, obj, cls): return None d = Desc() assert ns.d # descriptor protocol broken as expected This seems to work. I'm sure there are tons of things wrong with it that could be fixed. But again, the code isn't very elegant.
On Mon, Jul 29, 2019 at 09:53:38PM -0400, Ricky Teachey wrote:
Neither can I, and I've been mucking about with this idea for about three years. (To be honest, I poke it with a stick about every six months, so its not like I've spend a lot of time on it.) I kinda-sorta got it to work with a class statement plus a decorator, not too different from your metaclass example, but couldn't get functions to work correctly. But if I understand the original post here: https://mail.python.org/archives/list/python-ideas@python.org/message/TAVHEK... Batuhan Taskaya has got it to work using a dict. If that's the case, that seems to be the hardest part done. (On the other hand, if this is just a hypothetical "wouldn't it be cool if this worked?" post, then we're actually no closer to a technical solution.) -- Steven
On Tue, Jul 30, 2019 at 12:13 PM Steven D'Aprano <steve@pearwood.info> wrote:
A class + decorator gets so almost there that it might be possible with just a tiny bit of language support. What if there could be a way to change a function's __globals__? Currently that's readonly according to funcobject.c, but by hacking that flag out, I can create the following example: def namespace(cls): """Decorate a class to turn it into a namespace""" ns = dict(cls.__dict__) for member in ns.values(): if hasattr(member, "__globals__"): member.__globals__ = ns return cls @namespace class ns: x = 1 def foo(y): return x + y def replace(new_x): global x x = new_x print(ns.foo(4)) # 5 ns.replace(100) print(ns.foo(4)) # 104 Due to CPython limitations, the new __globals__ MUST be a plain dict (not the mappingproxy that cls.__dict__ is, and not a ChainMap), but this is extremely close to what could actually be quite useful. Also, since the function objects get mutated by the decorator, calling a function during the namespace's construction will have the wrong behaviour. Maybe the only language support needed is some way to say "from now on, functions should be created with THIS value for their globals"? It might be possible to get even closer using a metaclass rather than a decorator. ChrisA
On Tue, Jul 30, 2019 at 12:54:12PM +1000, Chris Angelico wrote:
That's one of the problems I hit in my experiments. I don't think you need to hack funcobject.c if you are willing to disassemble the function object (all of the parts are accessible as attributes) and re-assemble it into a new function using the types.FunctionType constructor.
Due to CPython limitations, the new __globals__ MUST be a plain dict (not the mappingproxy that cls.__dict__ is, and not a ChainMap),
In my experiments, I think I overcame that by subclassing ChainMap and dict: class MyChainMap(ChainMap, dict): pass was, I think, enough to fool the function constructor. Don't quote me on this, it's been a while and I don't remember all the details. But I'm pretty sure nothing segfaulted, I'd remember it if it did :-) I'm pretty sure that this is doable, at least well enough to get a working prototype that people can experiment with. At worst, we might need AST or bytecode hacking to get the functions defined inside the namespace to work correctly.
Yes, there's that too. But then the namespace doesn't actually exist at that point, so this is rather analogous to what happens when you call a function inside a class during class construction. I can live with that limitation. -- Steven
Forgive me if this doesn't make sense, but what about a way to make a copy of a function's locals while it is being called? Maybe something like: @namespace def ns(): x = 1 def f(): ... Inside the decorator, you simply call the function, make a copy of the locals, and assign it to the function name as a module?
Forgive me if this doesn't make sense, but what about a way to make a copy of a function's locals while it is being called?
This can be easily accomplished if you simply return locals() at the end of creating the namespace, but this isn't all that elegant: def namespace(ns): ns_locals = ns() from types import ModuleType mod = ModuleType(ns.__name__) mod.__dict__.update(ns_locals) return mod @namespace def ns(): x=1 y=2 def f(): ... return locals()
On Mon, Jul 29, 2019 at 6:55 PM Ricky Teachey <ricky@teachey.org> wrote: [Steven]
If the core developers don't consider namedtuples important enough to
get syntactic support, I doubt that namespaces will.
[Ricky]
This by itself is pretty convincing.
Except it's wrong. The successor of namedtuple is typing.NamedTuple which supports this syntactically. From the [docs]( https://docs.python.org/3/library/typing.html#typing.NamedTuple): ``` class Employee(NamedTuple): name: str id: int ``` -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Except it's wrong. The successor of namedtuple is typing.NamedTuple which supports this syntactically.
Point taken. And I definitely don't mean to imply the core devs aren't reasonable people or are unwilling to make big changes, I'm just not experienced enough to be the guy making the very-thoroughly-well-thought-out-version of the case for a major syntax change.
On Mon, Jul 29, 2019 at 07:19:42PM -0700, Guido van Rossum wrote:
Oh that's nice! I thought that typing.NamedTuple was only for type annotations, I had no idea you could use it for actual named tuples. Should it be aliased in collections if we want people to consider using it instead of namedtuple? I would never go looking in typing for non-annotation-related uses, and I'm probably not the only one. In any case, that's still using existing syntax (class statement), not a dedicated new "NamedTuple" keyword. And if there's no NamedTuple keyword, we shouldn't expect a Namespace keyword either :-) -- Steven
On Thu, Aug 08, 2019 at 10:28:44AM +0100, Ivan Levkivskyi wrote:
Done: https://bugs.python.org/issue37809 -- Steven
participants (15)
-
Anders Hovmöller -
Andrew Barnert -
Batuhan Taskaya -
Calvin Spealman -
Chris Angelico -
Dan Sommers -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
Ivan Levkivskyi -
Michael Selik -
Rhodri James -
Ricky Teachey -
Ryan Gonzalez -
Steven D'Aprano