Re: [Python-ideas] Trial balloon: adding variable type declarations in support of PEP 484
I can't avoid noting that there is an opportunity here to insert NamedTuple into the core language. The above example is almost there, except it's mutable and without convenient methods. But class Starship(tuple): damage: int = 0 captain: str = "Kirk" Is an obvious syntax for Starship = NamedTuple('Starship', [('damage', int), ('captain', str)]) Only much more available and intuitive to read, use, and of course - type check. (Of course, it does mean adding semantics to the declaration syntax in general) I'm not really suggesting to make this change now, but I believe it will be done, sooner or later. My brief experience with mypy convinced me that it must be the case. The new declaration syntax only makes it easier. ~Elazar
That's a very interesting idea and one that deserves pursuing (though I agree it's not a blocker for the PEP I'm hoping to write). I think the next step is to prototype this -- which can only happen once we have an implementation of the PEP. Though perhaps you could start by writing a prototype that works by having the user write the following: class Starship(PrototypeNamedTuple): damage = 0 captain = "Kirk" __annotations__ = dict(damage=int, captain=str) It could also benefit from PEP 520 (Preserving Class Attribute Definition Order). Who's game? --Guido On Mon, Aug 8, 2016 at 1:13 PM, אלעזר <elazarg@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
class PrototypeNamedTuple: cache = {} def __new__(cls, *args): P = PrototypeNamedTuple if cls not in P.cache: P.cache[cls] = typing.NamedTuple(cls.__name__, cls.__annotations__.items()) return P.cache[cls](*args) Works modulo ordering, though I'm not sure that's the right way to do it. The ordering part of namedtuple is orthogonal to the value-type/immutability part. So I would imagine making "Value" for the latter, "tuple" for the former, and namedtuple is mixing both (possibly given a convenient name, such as PrototypeNamedTuple). "Value" can also seen as mixing "Struct" and "Immutable", but that's overdoing it I guess. ~Elazar On Mon, Aug 8, 2016 at 11:25 PM Guido van Rossum <guido@python.org> wrote:
Hm, overlooking the ordering is kind of a big deal for something with "tuple" in its name. :-) Also it feels like it needs a metaclass instead of a cache. Maybe from this we can learn though that __anotations__ should be an OrderedDict? On Mon, Aug 8, 2016 at 1:58 PM, אלעזר <elazarg@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
Feels like named parameters are better off being an OrderedDict in the first place. NamedTuple pushes OrderedDict to become kind of builtin. On Tue, Aug 9, 2016 at 12:09 AM Guido van Rossum <guido@python.org> wrote:
On Mon, Aug 8, 2016 at 2:11 PM, אלעזר <elazarg@gmail.com> wrote:
Feels like named parameters are better off being an OrderedDict in the first place.
PEP 468.
NamedTuple pushes OrderedDict to become kind of builtin.
Why? Having both in the collections module is good enough.
-- --Guido van Rossum (python.org/~guido)
On Tue, Aug 9, 2016 at 12:29 AM Guido van Rossum <guido@python.org> wrote:
Sorry, I should have read this PEP before.
NamedTuple pushes OrderedDict to become kind of builtin.
Why? Having both in the collections module is good enough.
What I meant in becoming builtin is not the accessibility of the name, but the parallel incremental support of namedtuple, OrderedDict and (as I find out) order of **kwargs. In much the same way that class is made out of dict (and keeps one), namedtuple is an OrderedDict (and keeps one). Much like dict has a constructor `dict(a=1, b=2.0)` and a literal `{'a' : 1, 'b' : 2.0}`, OrderedDict has its OrderedDict(a=1, b=2.0) and should have the literal ('a': 1, 'b': 2.0). Replace 1 and 2 with int and float and you get a very natural syntax for a NamedTuple that is the type of a matching OrderedDict. And in my mind, the requirement for type names matches nicely the enforcement of immutability. Continuing this thought, the annotations for the class is actually its structural type. It opens the door for future requests for adding e.g. an operator for structural equivalence. This is all very far from the current language, so it's only thoughts and not really ideas; probably does not belong here. Sorry about that. ~Elazar
On Mon, Aug 8, 2016 at 3:08 PM, אלעזר <elazarg@gmail.com> wrote:
No problem, it's fine to participate before reading *everything*!
Well, they're all already in the stdlib, although namedtuple is written in Python, OrderedDict in C (recently), and **kwarg is ancient.
In much the same way that class is made out of dict (and keeps one), namedtuple is an OrderedDict (and keeps one).
I'm not sure if you're talking about the class or the instances. A class instance usually has a dict (unless if it has __slots__ and all its base classes have __slots__). But a namedtuple instance does not have a dict or OrderedDict -- it is a tuple at heart.
That's debatable at least, and probably there are better solutions.
-- --Guido van Rossum (python.org/~guido)
With PEP 520 accepted, would it be possible to iterate over __definition_order__? class PrototypeNamedTuple: cache = {} def __new__(cls, *args): P = PrototypeNamedTuple if cls not in P.cache: P.cache[cls] = typing.NamedTuple(cls.__name__, [(definition, cls.__annotations__[definition]) for definition in cls.__definition_order__] ) return P.cache[cls](*args) On Monday, August 8, 2016 at 5:09:50 PM UTC-4, Guido van Rossum wrote:
Sounds like you're thinking with your runtime hat on, not your type checker hat. :-) On Tue, Aug 9, 2016 at 9:46 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Wed, Aug 10, 2016 at 11:56 AM, Guido van Rossum <guido@python.org> wrote:
Still, it would be really nice to be able to introspect a class's instance attributes at run-time. A stdlib helper for that would be great, e.g. "inspect.get_inst_attrs(cls)". At one point a few years back I wrote something like that derived from the signature of cls.__init__() and in the spirit of inspect.signature(). It turned out to be quite useful. Relatedly, it may make sense to recommend in PEP 8 that all instance attribute declarations in a class definition be grouped together and to do so right before the methods (right before __new__/__init__?). (...or disallow instance attribute declarations in the stdlib for now.) -eric
On Wed, Aug 10, 2016 at 4:04 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote: [...]
Yes, the proposal will store variable annotations for globals and for classes in __annotations__ (one global, one per class). Just not for local variables.
Let's wait until we have some experience with how it's used before updating PEP 8... :-) -- --Guido van Rossum (python.org/~guido)
It's already possible to overload NamedTuple, in a way that will allow the following abuse of notation: @NamedTuple def Starship(damage:int, captain:str): pass The 'def' is unfortunate and potentially confusing (although it *is* a callable definition), and the ": pass" is meaningless. But I think it is clear and concise if you know what NamedTuple is. Introducing new keyword will of course solve both problems (if there's "async def", why not "type def"? :) ). case class Starship(damage:int, captain:str) Possible variations on the decorator theme: "unordered" namedtuple (note the *) @Value def Starship(*, damage:int, captain:str): pass self-describing (another level of notation abuse): @Type def Starship(damage:int, captain:str) -> NamedTuple: pass On Mon, Aug 8, 2016 at 11:25 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Aug 9, 2016 at 3:47 PM, אלעזר <elazarg@gmail.com> wrote:
If we're dealing with classes then we should be using the class syntax. There are a number of options here for identifying attributes in a definition and even auto-generating parts of the class (e.g. __init__). Let's look at several (with various objectives): # For the sake of demonstration, we ignore opportunities for type inference. # currently (with comments for type info) class Bee(namedtuple('Bee', 'name ancient_injury managerie')): """can a bee be said to be...""" # name: str # ancient_injury: bool # menagerie: bool def __new__(cls, name='Eric', ancient_injury=False, menagerie=False): return super().__new__(cls, name, ancient_injury, menagerie) def half_a(self): return self.ancient_injury or self.menagerie # using attribute annotations and a decorator (and PEP 520) @as_namedtuple class Bee: """...""" name: str = 'Eric' ancient_injury: bool = False menagerie: bool = False def half_a(self): ... # using attribute annotations and a metaclass (and PEP 520) class Bee(metaclass=NamedtupleMeta): """...""" name: str = 'Eric' ancient_injury: bool = False menagerie: bool = False def half_a(self): ... # using one class decorator and PEP 520 and comments for type info @as_namedtuple class Bee: name = 'Eric' # str ancient_injury = False # bool menagerie = False # bool def half_a(self): ... # using one class decorator and comments for type info @as_namedtuple('name ancient_injury managerie', name='Eric', ancient_injury=False, menagerie=False) class Bee: """...""" # name: str # ancient_injury: bool # menagerie: bool def half_a(self): ... # using one class decorator (and PEP 468) and comments for type info # similar to the original motivation for PEP 468 @as_namedtuple(name='Eric', ancient_injury=False, menagerie=False) class Bee: """...""" # name: str # ancient_injury: bool # menagerie: bool def half_a(self): ... # using a class decorator for each attribute @as_namedtuple('name ancient_injury managerie') @attr('name', str, 'Eric') @attr('ancient_injury', bool, False) @attr('menagerie', bool, False) class Bee: """...""" def half_a(self): ... Those are simple examples and we could certainly come up with others, all using the class syntax. For me, the key is finding the sweet spot between readability/communicating intent and packing too many roles into the class syntax. To be honest, the examples using attribute annotations seem fine to me. Even if you don't need attr type info (which you don't most of the time, particularly with default values), my favorite solution (a class decorator that leverages PEP 520) is still the most readable and in-line with the class syntax, at least for me. :) -eric p.s. The same approaches could also be applied to generating non-namedtuple classes, e.g. SimpleNamespace subclasses.
On Tue, Aug 9, 2016 at 5:32 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Another approach that I've used in the past (along with a derivative): # using a non-data descriptor @as_namedtuple class Bee: """...""" name = Attr(str, 'Eric', doc='the bee's name') ancient_injury = Attr(bool, False) menagerie = Attr(bool, False) def half_a(self): ... # using a non-data descriptor along with PEP 487 class Bee(Namedtuple): """...""" name = Attr(str, 'Eric', doc='the bee's name') ancient_injury = Attr(bool, False) menagerie = Attr(bool, False) def half_a(self): ... While the descriptor gives you docstrings (a la property), I expect it isn't as static-analysis-friendly as the proposed variable annotations. -eric
אלעזר wrote:
But the untyped version of that already has a meaning -- it's a tuple subclass with two extra class attributes that are unrelated to its indexable items. I thought that type annotations weren't meant to change runtime semantics? -- Greg
On Mon, Aug 8, 2016 at 11:17 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Correct, but we can invent a new base class that has these semantics. It's no different from Enum. Anyway, I think this will have to be an add-on to be designed after the basics are done. -- --Guido van Rossum (python.org/~guido)
That's a very interesting idea and one that deserves pursuing (though I agree it's not a blocker for the PEP I'm hoping to write). I think the next step is to prototype this -- which can only happen once we have an implementation of the PEP. Though perhaps you could start by writing a prototype that works by having the user write the following: class Starship(PrototypeNamedTuple): damage = 0 captain = "Kirk" __annotations__ = dict(damage=int, captain=str) It could also benefit from PEP 520 (Preserving Class Attribute Definition Order). Who's game? --Guido On Mon, Aug 8, 2016 at 1:13 PM, אלעזר <elazarg@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
class PrototypeNamedTuple: cache = {} def __new__(cls, *args): P = PrototypeNamedTuple if cls not in P.cache: P.cache[cls] = typing.NamedTuple(cls.__name__, cls.__annotations__.items()) return P.cache[cls](*args) Works modulo ordering, though I'm not sure that's the right way to do it. The ordering part of namedtuple is orthogonal to the value-type/immutability part. So I would imagine making "Value" for the latter, "tuple" for the former, and namedtuple is mixing both (possibly given a convenient name, such as PrototypeNamedTuple). "Value" can also seen as mixing "Struct" and "Immutable", but that's overdoing it I guess. ~Elazar On Mon, Aug 8, 2016 at 11:25 PM Guido van Rossum <guido@python.org> wrote:
Hm, overlooking the ordering is kind of a big deal for something with "tuple" in its name. :-) Also it feels like it needs a metaclass instead of a cache. Maybe from this we can learn though that __anotations__ should be an OrderedDict? On Mon, Aug 8, 2016 at 1:58 PM, אלעזר <elazarg@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
Feels like named parameters are better off being an OrderedDict in the first place. NamedTuple pushes OrderedDict to become kind of builtin. On Tue, Aug 9, 2016 at 12:09 AM Guido van Rossum <guido@python.org> wrote:
On Mon, Aug 8, 2016 at 2:11 PM, אלעזר <elazarg@gmail.com> wrote:
Feels like named parameters are better off being an OrderedDict in the first place.
PEP 468.
NamedTuple pushes OrderedDict to become kind of builtin.
Why? Having both in the collections module is good enough.
-- --Guido van Rossum (python.org/~guido)
On Tue, Aug 9, 2016 at 12:29 AM Guido van Rossum <guido@python.org> wrote:
Sorry, I should have read this PEP before.
NamedTuple pushes OrderedDict to become kind of builtin.
Why? Having both in the collections module is good enough.
What I meant in becoming builtin is not the accessibility of the name, but the parallel incremental support of namedtuple, OrderedDict and (as I find out) order of **kwargs. In much the same way that class is made out of dict (and keeps one), namedtuple is an OrderedDict (and keeps one). Much like dict has a constructor `dict(a=1, b=2.0)` and a literal `{'a' : 1, 'b' : 2.0}`, OrderedDict has its OrderedDict(a=1, b=2.0) and should have the literal ('a': 1, 'b': 2.0). Replace 1 and 2 with int and float and you get a very natural syntax for a NamedTuple that is the type of a matching OrderedDict. And in my mind, the requirement for type names matches nicely the enforcement of immutability. Continuing this thought, the annotations for the class is actually its structural type. It opens the door for future requests for adding e.g. an operator for structural equivalence. This is all very far from the current language, so it's only thoughts and not really ideas; probably does not belong here. Sorry about that. ~Elazar
On Mon, Aug 8, 2016 at 3:08 PM, אלעזר <elazarg@gmail.com> wrote:
No problem, it's fine to participate before reading *everything*!
Well, they're all already in the stdlib, although namedtuple is written in Python, OrderedDict in C (recently), and **kwarg is ancient.
In much the same way that class is made out of dict (and keeps one), namedtuple is an OrderedDict (and keeps one).
I'm not sure if you're talking about the class or the instances. A class instance usually has a dict (unless if it has __slots__ and all its base classes have __slots__). But a namedtuple instance does not have a dict or OrderedDict -- it is a tuple at heart.
That's debatable at least, and probably there are better solutions.
-- --Guido van Rossum (python.org/~guido)
With PEP 520 accepted, would it be possible to iterate over __definition_order__? class PrototypeNamedTuple: cache = {} def __new__(cls, *args): P = PrototypeNamedTuple if cls not in P.cache: P.cache[cls] = typing.NamedTuple(cls.__name__, [(definition, cls.__annotations__[definition]) for definition in cls.__definition_order__] ) return P.cache[cls](*args) On Monday, August 8, 2016 at 5:09:50 PM UTC-4, Guido van Rossum wrote:
Sounds like you're thinking with your runtime hat on, not your type checker hat. :-) On Tue, Aug 9, 2016 at 9:46 PM, Neil Girdhar <mistersheik@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Wed, Aug 10, 2016 at 11:56 AM, Guido van Rossum <guido@python.org> wrote:
Still, it would be really nice to be able to introspect a class's instance attributes at run-time. A stdlib helper for that would be great, e.g. "inspect.get_inst_attrs(cls)". At one point a few years back I wrote something like that derived from the signature of cls.__init__() and in the spirit of inspect.signature(). It turned out to be quite useful. Relatedly, it may make sense to recommend in PEP 8 that all instance attribute declarations in a class definition be grouped together and to do so right before the methods (right before __new__/__init__?). (...or disallow instance attribute declarations in the stdlib for now.) -eric
On Wed, Aug 10, 2016 at 4:04 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote: [...]
Yes, the proposal will store variable annotations for globals and for classes in __annotations__ (one global, one per class). Just not for local variables.
Let's wait until we have some experience with how it's used before updating PEP 8... :-) -- --Guido van Rossum (python.org/~guido)
It's already possible to overload NamedTuple, in a way that will allow the following abuse of notation: @NamedTuple def Starship(damage:int, captain:str): pass The 'def' is unfortunate and potentially confusing (although it *is* a callable definition), and the ": pass" is meaningless. But I think it is clear and concise if you know what NamedTuple is. Introducing new keyword will of course solve both problems (if there's "async def", why not "type def"? :) ). case class Starship(damage:int, captain:str) Possible variations on the decorator theme: "unordered" namedtuple (note the *) @Value def Starship(*, damage:int, captain:str): pass self-describing (another level of notation abuse): @Type def Starship(damage:int, captain:str) -> NamedTuple: pass On Mon, Aug 8, 2016 at 11:25 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Aug 9, 2016 at 3:47 PM, אלעזר <elazarg@gmail.com> wrote:
If we're dealing with classes then we should be using the class syntax. There are a number of options here for identifying attributes in a definition and even auto-generating parts of the class (e.g. __init__). Let's look at several (with various objectives): # For the sake of demonstration, we ignore opportunities for type inference. # currently (with comments for type info) class Bee(namedtuple('Bee', 'name ancient_injury managerie')): """can a bee be said to be...""" # name: str # ancient_injury: bool # menagerie: bool def __new__(cls, name='Eric', ancient_injury=False, menagerie=False): return super().__new__(cls, name, ancient_injury, menagerie) def half_a(self): return self.ancient_injury or self.menagerie # using attribute annotations and a decorator (and PEP 520) @as_namedtuple class Bee: """...""" name: str = 'Eric' ancient_injury: bool = False menagerie: bool = False def half_a(self): ... # using attribute annotations and a metaclass (and PEP 520) class Bee(metaclass=NamedtupleMeta): """...""" name: str = 'Eric' ancient_injury: bool = False menagerie: bool = False def half_a(self): ... # using one class decorator and PEP 520 and comments for type info @as_namedtuple class Bee: name = 'Eric' # str ancient_injury = False # bool menagerie = False # bool def half_a(self): ... # using one class decorator and comments for type info @as_namedtuple('name ancient_injury managerie', name='Eric', ancient_injury=False, menagerie=False) class Bee: """...""" # name: str # ancient_injury: bool # menagerie: bool def half_a(self): ... # using one class decorator (and PEP 468) and comments for type info # similar to the original motivation for PEP 468 @as_namedtuple(name='Eric', ancient_injury=False, menagerie=False) class Bee: """...""" # name: str # ancient_injury: bool # menagerie: bool def half_a(self): ... # using a class decorator for each attribute @as_namedtuple('name ancient_injury managerie') @attr('name', str, 'Eric') @attr('ancient_injury', bool, False) @attr('menagerie', bool, False) class Bee: """...""" def half_a(self): ... Those are simple examples and we could certainly come up with others, all using the class syntax. For me, the key is finding the sweet spot between readability/communicating intent and packing too many roles into the class syntax. To be honest, the examples using attribute annotations seem fine to me. Even if you don't need attr type info (which you don't most of the time, particularly with default values), my favorite solution (a class decorator that leverages PEP 520) is still the most readable and in-line with the class syntax, at least for me. :) -eric p.s. The same approaches could also be applied to generating non-namedtuple classes, e.g. SimpleNamespace subclasses.
On Tue, Aug 9, 2016 at 5:32 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Another approach that I've used in the past (along with a derivative): # using a non-data descriptor @as_namedtuple class Bee: """...""" name = Attr(str, 'Eric', doc='the bee's name') ancient_injury = Attr(bool, False) menagerie = Attr(bool, False) def half_a(self): ... # using a non-data descriptor along with PEP 487 class Bee(Namedtuple): """...""" name = Attr(str, 'Eric', doc='the bee's name') ancient_injury = Attr(bool, False) menagerie = Attr(bool, False) def half_a(self): ... While the descriptor gives you docstrings (a la property), I expect it isn't as static-analysis-friendly as the proposed variable annotations. -eric
אלעזר wrote:
But the untyped version of that already has a meaning -- it's a tuple subclass with two extra class attributes that are unrelated to its indexable items. I thought that type annotations weren't meant to change runtime semantics? -- Greg
On Mon, Aug 8, 2016 at 11:17 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Correct, but we can invent a new base class that has these semantics. It's no different from Enum. Anyway, I think this will have to be an add-on to be designed after the basics are done. -- --Guido van Rossum (python.org/~guido)
participants (5)
-
Eric Snow -
Greg Ewing -
Guido van Rossum -
Neil Girdhar -
אלעזר