Optional Static Typing -- the Python Way
After reading through the OST threads, and learning a bit more about what static typing means, and the difference between nominal and structural subclassing, I think I have a better appreciation for the knee-jerk "no way!" reaction, and also how to make what seems like a foreign-to-Python concept pythonistic. The most-feared problem seems to be over-specification (it is certainly mine). I think the answer has been alluded to a couple times already, but just to hopefully bring it front and center: Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme. In other words: def spamify(current: datetime.date, moved:int) -> bool: # ONE_DAY is a one day time delta, previously declared days = ONE_DAY * moved proposed_date = current + days return it_works(proposed_date) The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's __add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works with datetime.delta. I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than structural as it's only checking the methods and attributes actually used. On the minus side, there are times when we need *exactly* a datetime.date, so we would need a way to specify exact vs dynamic, but dynamic should be the default. -- ~Ethan~
On 19 August 2014 06:15, Ethan Furman <ethan@stoneleaf.us> wrote:
After reading through the OST threads, and learning a bit more about what static typing means, and the difference between nominal and structural subclassing, I think I have a better appreciation for the knee-jerk "no way!" reaction, and also how to make what seems like a foreign-to-Python concept pythonistic.
The most-feared problem seems to be over-specification (it is certainly mine). I think the answer has been alluded to a couple times already, but just to hopefully bring it front and center:
The exact same fear was raised when ABCs were added in PEP 3119 - that by formalising ABCs, we'd see a plethora of additional isinstance checks as people tried to make things "fail fast". That fear turned out to be spurious then, and I think it's spurious now (unless we allow builtins to be used for type annotations - we shouldn't do that, as I think it *would* cause problems when developers migrate to Python from languages like C, Java and C#). Most Python code should continue to be written without type annotations - the use of annotations (and type checkers) should be limited to projects (and organisations) that are large enough for the additional complexity to be worth the hassle. Anecdotally, my experience is that the split between proponents of static and dynamic typing tends to divide along "team size" lines. If someone thinks a team of 20 working on one project is "a lot of people", then they're likely to favour dynamic typing for the additional flexibility it offers. If they think such a team is small, then they're more likely to want compiler enforced assistance in ensuring that everyone is playing by the rules of the API. Myself, I think the kind of structural typechecking offered by "pylint -E" and the "optional type declarations" proposal starts looking far more attractive the first time you're stuck at work after hours debugging a production failure that happened due to a typo in an error handling path that wasn't covered by your test suite. However, if your code isn't in the category of "if this doesn't work, my company/project/organisation stops until it is fixed", then type assertions are unlikely to be worth the investment needed to write them in the first place :) Ultimately, my perspective is that Guido's proposal boils down to having a nice syntax where: def myfunc(a : KindX, b: KindY, c: KindZ): ... is the moral equivalent of: def myfunc(a, b, c): assert isinstance(a, KindX) assert isinstance(b, KindY) assert isinstance(c, KindZ) except done in a way that is more amenable to static analysis by looking at the compiled AST, rather than actually executing the code. Calling it "optional static typing" is probably a bad idea, since the proposal isn't to change the type system itself - that will be just as dynamic as it always has been. "optional type assertions" is probably the most accurate, but then people may think it is *actually* translated to runtime checks as I show above, which it won't be. "optional type hinting" is probably the most reassuring term that could be used, while still remaining accurate.
Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme.
In other words:
def spamify(current: datetime.date, moved:int) -> bool: # ONE_DAY is a one day time delta, previously declared days = ONE_DAY * moved proposed_date = current + days return it_works(proposed_date)
The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's __add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works with datetime.delta.
I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than structural as it's only checking the methods and attributes actually used.
ABCs already support ducktyping, and explicit registration, etc, etc. We don't need a completely new type system, we can just file the rough edges off the categories we have already defined, and fill in some of the missing pieces (like allowing union types, which I seem to recall being discussed back in the PEP 3119 time frame, but postponed until more concrete use cases presented themselves). We may even need to finally add String and BytesLike ABCs, but we've been muttering about doing that for years. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: [Agreeable musings about team size...] Ultimately, my perspective is that Guido's proposal boils down to
having a nice syntax where:
def myfunc(a : KindX, b: KindY, c: KindZ): ...
is the moral equivalent of:
def myfunc(a, b, c): assert isinstance(a, KindX) assert isinstance(b, KindY) assert isinstance(c, KindZ)
Please no. The asserts affect runtime. The type declarations are for linting, IDEs and docs. I don't want the difference to be swept under the rug. I want it to be the key feature of the proposal.
except done in a way that is more amenable to static analysis by looking at the compiled AST, rather than actually executing the code.
Ironically, mypy also understands isinstance() checks, using them as a kind of implied "typecase" (though I don't know if it understands assert -- it definitely understands "if isinstance(x, int): x = x**2".
Calling it "optional static typing" is probably a bad idea, since the proposal isn't to change the type system itself - that will be just as dynamic as it always has been. "optional type assertions" is probably the most accurate, but then people may think it is *actually* translated to runtime checks as I show above, which it won't be.
I don't know what you mean by "accurate".
"optional type hinting" is probably the most reassuring term that could be used, while still remaining accurate.
I like that.
Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme.
In other words:
def spamify(current: datetime.date, moved:int) -> bool: # ONE_DAY is a one day time delta, previously declared days = ONE_DAY * moved proposed_date = current + days return it_works(proposed_date)
The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's __add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works with datetime.delta.
I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than structural as it's only checking the methods and attributes actually used.
ABCs already support ducktyping, and explicit registration, etc, etc. We don't need a completely new type system, we can just file the rough edges off the categories we have already defined, and fill in some of the missing pieces (like allowing union types, which I seem to recall being discussed back in the PEP 3119 time frame, but postponed until more concrete use cases presented themselves).
Yeah, my response to Ethan's use case is that either he shouldn't bother with type annotations, or he should probably care enough to also declare his duck type for datetime.date as such, and an ABC regsitration sounds perfect for that.
We may even need to finally add String and BytesLike ABCs, but we've been muttering about doing that for years.
Mypy has some ideas here. -- --Guido van Rossum (python.org/~guido)
As others have said on the list, 'go away for a few days, and find a mega-thread in your inbox'. Yay. On Aug 19, 2014, at 1:01 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: [Agreeable musings about team size...]
Ultimately, my perspective is that Guido's proposal boils down to having a nice syntax where:
def myfunc(a : KindX, b: KindY, c: KindZ): ...
is the moral equivalent of:
def myfunc(a, b, c): assert isinstance(a, KindX) assert isinstance(b, KindY) assert isinstance(c, KindZ)
Please no. The asserts affect runtime. The type declarations are for linting, IDEs and docs. I don't want the difference to be swept under the rug. I want it to be the key feature of the proposal.
<SNIP> Going slightly sideways on this -- I think this is why we should use decorators instead of annotations; as has already been mentioned twice on the list, http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a good job pointing out that static and dynamic typing systems are separate but overlapping concepts. It also points out that both have their place. If we define decorators that can be turned on or off easily (command-line option? Environment variable? other?), then the end user can choose if if her or she is going to do static, dynamic, both, or none. This could be useful when trying to track down that annoying bug in long-running production code. Also, Types and Programming Languages by Dr. Pierce has been mentioned at least twice as well; would it be useful to ask him to join in the discussion? Thanks, Cem Karan
On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote:
Going slightly sideways on this -- I think this is why we should use decorators instead of annotations; as has already been mentioned twice on the list, http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a good job pointing out that static and dynamic typing systems are separate but overlapping concepts. It also points out that both have their place. If we define decorators that can be turned on or off easily (command-line option? Environment variable? other?), then the end user can choose if if her or she is going to do static, dynamic, both, or none. This could be useful when trying to track down that annoying bug in long-running production code.
This applies equally to annotations, and in fact that's exactly what Mypy already does. You can run Mypy to do static type checking, or not run it, and the annotations will be ignored. E.g. given a module program.py, you can: # Type check and then run the program: mypy program.py # Just run it, with no extra type checks: python3 program.py Guido's proposal is *not* to add static types to the CPython interpreter, at least not yet. It is just to standardise on annotations for type hinting, agree on a syntax for those type hints, and then allow third-part tools (linters, editors, IDEs, etc.) and alternative interpreters (like mypy) to actually use the type hints. Looking forward to the distant future, if CPython gains its own built-in type checker, it will probably come with a runtime switch to enable or disable such type checking. But that's possible regardless of whether we use decorators, annotations, or both.
Also, Types and Programming Languages by Dr. Pierce has been mentioned at least twice as well; would it be useful to ask him to join in the discussion?
Does he know anything about Python? Will he care? There are hundreds of programming languages, unless he has a particular interest in Python I can't see why he would care about this discussion. But if you are a colleague or friend of his, by all means invite him to join up, this is a public forum. -- Steven
On Aug 24, 2014, at 8:41 AM, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote:
Going slightly sideways on this -- I think this is why we should use decorators instead of annotations; as has already been mentioned twice on the list, http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a good job pointing out that static and dynamic typing systems are separate but overlapping concepts. It also points out that both have their place. If we define decorators that can be turned on or off easily (command-line option? Environment variable? other?), then the end user can choose if if her or she is going to do static, dynamic, both, or none. This could be useful when trying to track down that annoying bug in long-running production code.
This applies equally to annotations, and in fact that's exactly what Mypy already does. You can run Mypy to do static type checking, or not run it, and the annotations will be ignored.
E.g. given a module program.py, you can:
# Type check and then run the program: mypy program.py
# Just run it, with no extra type checks: python3 program.py
Guido's proposal is *not* to add static types to the CPython interpreter, at least not yet. It is just to standardise on annotations for type hinting, agree on a syntax for those type hints, and then allow third-part tools (linters, editors, IDEs, etc.) and alternative interpreters (like mypy) to actually use the type hints.
OK, I think I see what you're saying. Since the annotations will always be available, the static/dynamic nature doesn't matter, since we just choose the flags/interpreter/whatever, and it will check the annotations, correct? If so, then you're right, and we don't need decorators for this.
Looking forward to the distant future, if CPython gains its own built-in type checker, it will probably come with a runtime switch to enable or disable such type checking. But that's possible regardless of whether we use decorators, annotations, or both.
Also, Types and Programming Languages by Dr. Pierce has been mentioned at least twice as well; would it be useful to ask him to join in the discussion?
Does he know anything about Python? Will he care? There are hundreds of programming languages, unless he has a particular interest in Python I can't see why he would care about this discussion. But if you are a colleague or friend of his, by all means invite him to join up, this is a public forum.
I just invited him. Thanks, Cem Karan
On 8/19/2014 9:27 AM, Nick Coghlan wrote: What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for years. What I also like: shadow skeleton files (for the stdlib) written by people other than core developers, who obviously have the energy to do so, having already started with various syntaxes, and who just need a standard to combine their efforts. There should be one set that, as much as possible, fully allows duck-typing, which is to say, does not reject valid arguments. What I want to add: the advantage of separate skeleton files, aside from parallel development by other people, is that there can be more than one skeleton file for a given stdlib module. Linters typically allow users to set local standards by selecting options. A group could also set local standards by restrictive type hints in their local skeletons. A realistic example would be sum(it: Iterable[Number]). -- Terry Jan Reedy
Le 19/08/2014 18:50, Terry Reedy a écrit :
On 8/19/2014 9:27 AM, Nick Coghlan wrote:
What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for years.
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are). Besides, the fact that type descriptions must be parametrable, and therefore end up being used as instances, sounds like it kills the idea of making them regular (runtime) types as well. Unless we make them metaclasses? Regards Antoine.
On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 18:50, Terry Reedy a écrit :
On 8/19/2014 9:27 AM, Nick Coghlan wrote:
What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for years.
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Besides, the fact that type descriptions must be parametrable, and therefore end up being used as instances, sounds like it kills the idea of making them regular (runtime) types as well. Unless we make them metaclasses?
No, that doesn't follow at all. The concept of metaclasses means that classes *are* instances (of the metaclass) and it lets us define operations (in particular __getitem__ :-) on the classes. (I was going to show a working example, but I ran out of time, and I'm taking a vacation the rest of this week.) -- --Guido van Rossum (python.org/~guido)
On 20 Aug 2014 10:14, "Guido van Rossum" <guido@python.org> wrote:
On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine@python.org>
wrote:
Le 19/08/2014 18:50, Terry Reedy a écrit :
On 8/19/2014 9:27 AM, Nick Coghlan wrote:
What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for years.
Hmm, I've been saying this already, but my intuition is that it's a bad
idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
One of the ways I describe the explicit registration mechanism to people is as giving you the option of lying to the interpreter. If you use explicit registration, your registered type basically walks around with a sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and assumes it will be able to quack without actually checking in advance. It's really just a constrained version of normal duck typing, where the interpreter just assumes that *everything* may exhibit duck-like tendencies. Cheers, Nick.
Was that meant to support Antoine's position (ABCs are not type descriptions) or mine (they are)? On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 Aug 2014 10:14, "Guido van Rossum" <guido@python.org> wrote:
On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine@python.org>
Le 19/08/2014 18:50, Terry Reedy a écrit :
On 8/19/2014 9:27 AM, Nick Coghlan wrote:
What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for
years.
Hmm, I've been saying this already, but my intuition is that it's a bad
idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that
wrote: they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
One of the ways I describe the explicit registration mechanism to people is as giving you the option of lying to the interpreter. If you use explicit registration, your registered type basically walks around with a sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and assumes it will be able to quack without actually checking in advance. It's really just a constrained version of normal duck typing, where the interpreter just assumes that *everything* may exhibit duck-like tendencies.
Cheers, Nick.
-- --Guido van Rossum (python.org/~guido)
On 21 Aug 2014 01:53, "Guido van Rossum" <guido@python.org> wrote:
Was that meant to support Antoine's position (ABCs are not type
descriptions) or mine (they are)? Yours. The ability to register any type with any ABC means that even an explicit isinstance() ABC check at runtime is only advisory - developers are free to force a "true" answer if they choose to do so. You can even disable a check entirely by registering "object" with the relevant ABC (actually doing that would likely be a *bad idea*, but the interpreter allows it). That puts them squarely in the "descriptions of intent" category for me. A bit more rambling, since I've only been skimming (at best) the type hinting threads: The thing I appreciate getting out of linters is picking up the "this is almost certainly wrong" cases (like trying to call "indx" on something I have said I expect to be a sequence). This is especially useful when code is modified over time, but retains the original annotations documenting the interface assumptions - if you expand the function to cover new types, you'll need to change the annotation, which is also a good reminder that the docs will need updating. I'm not particularly worried if there are still cases where linters go "I have no idea what that object is, so I'll assume anything goes". After all, that's the baseline we start from when we don't run a linter at all. Cheers, Nick.
On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 Aug 2014 10:14, "Guido van Rossum" <guido@python.org> wrote:
On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine@python.org>
Le 19/08/2014 18:50, Terry Reedy a écrit :
On 8/19/2014 9:27 AM, Nick Coghlan wrote:
What I like: 'optional type hints' based on a fleshed-out ABC systems that collects all ABCs together on one module via import from current files. It seems that we have been slowly groping towards this for
years.
Hmm, I've been saying this already, but my intuition is that it's a
bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that
wrote: they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
One of the ways I describe the explicit registration mechanism to people
is as giving you the option of lying to the interpreter. If you use explicit registration, your registered type basically walks around with a sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and assumes it will be able to quack without actually checking in advance. It's really just a constrained version of normal duck typing, where the interpreter just assumes that *everything* may exhibit duck-like tendencies.
Cheers, Nick.
-- --Guido van Rossum (python.org/~guido)
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class. Regards Antoine.
On Wed Aug 20 2014 at 5:58:31 PM Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Isn't that the magic of __instancecheck__/__subclasscheck__? If typing.py does the right thing here then it shouldn't matter. You could use __init__ or __getitem__ to construct an object who upon introspection provides the details you want and do what you expect, and then override the appropriate methods so that isinstance() checks check against the ABC instead of the type class itself (heck we could make the base type class require that you inherit from an ABC to promote working with ABCs instead of concrete classes). I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =) P.S.: Off-topic for this email but something I don't think has been mentioned more than once, type hinting like we are proposing is exactly what Dart does and it is nice. Built-in API documentation of interfaces along with IDE support at the API makes all of this worth it.
Le 21/08/2014 10:30, Brett Cannon a écrit :
Isn't that the magic of __instancecheck__/__subclasscheck__? If typing.py does the right thing here then it shouldn't matter.
What is "the right thing"?
You could use __init__ or __getitem__ to construct an object who upon introspection provides the details you want and do what you expect, and then override the appropriate methods so that isinstance() checks check against the ABC instead of the type class itself (heck we could make the base type class require that you inherit from an ABC to promote working with ABCs instead of concrete classes).
So we agree that type descriptions should be separate things from ABCs, or not?
I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =)
Except that "x is a.Mapping" is False with a normal Python. Regards Antoine.
On Thu Aug 21 2014 at 10:55:47 AM Antoine Pitrou <antoine@python.org> wrote:
Le 21/08/2014 10:30, Brett Cannon a écrit :
Isn't that the magic of __instancecheck__/__subclasscheck__? If typing.py does the right thing here then it shouldn't matter.
What is "the right thing"?
I don't feel like scrolling all the way back through this thread to find your original email, but "the right thing" is "whatever it takes". =) My point is that I don't think there is any technical reason why we can't make ABCs work.
You could use __init__ or __getitem__ to construct an object who upon introspection provides the details you want and do what you expect, and then override the appropriate methods so that isinstance() checks check against the ABC instead of the type class itself (heck we could make the base type class require that you inherit from an ABC to promote working with ABCs instead of concrete classes).
So we agree that type descriptions should be separate things from ABCs, or not?
We don't agree. I'm fine with reusing ABCs. My only point is that if you really don't want to tweak ABCs to work directly, typing.py can still use ABCs as a subclass and then add whatever is necessary on top of them.
I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =)
Except that "x is a.Mapping" is False with a normal Python.
You're reading that too literally as code and not as a descriptive sentence. What I meant to convey is that code `def foo(x: a.Mapping)` can be read in your inner voice as "function foo has a parameter x which is a.Mapping", where "a.Mapping" somewhat reads like "a Mapping". -Brett
Regards
Antoine.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Le 22/08/2014 08:53, Brett Cannon a écrit :
> I have to also say that I like using ABCs because `from collections > import abc as a` leads to reading parameters like "x is a.Mapping" which > is linguistically quaint. =)
Except that "x is a.Mapping" is False with a normal Python.
You're reading that too literally as code and not as a descriptive sentence. What I meant to convey is that code `def foo(x: a.Mapping)` can be read in your inner voice as "function foo has a parameter x which is a.Mapping", where "a.Mapping" somewhat reads like "a Mapping".
So? That would be true with anything else that isn't an ABC, as long as you import the relevant module with the name "a" (which is silly, but hey :-)). def foo(x: a.List(a.Number)) -> a.Number Regards Antoine.
On Aug 21, 2014, at 7:30, Brett Cannon <brett@python.org> wrote:
I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =)
Cute, but it seems like it actually impedes the reading of "x is an Iterable". (Although I guess as compensation you get "x is an Integer" if you import numbers as n, and you get to yodel like Adam Ant when reading code about file types.) More seriously: I argued along the same lines as you as to the advantages of using the ABCs directly, but I understand the opposing viewpoint: although static type hints and runtime type constraints are _normally_ the same things, they're not _necessarily_ so, and forcing them to share a representation may preclude some of the fancy things people want to do with types beyond the simple generic model (e.g.; the PyConstraints sub-thread).
On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote:
P.S.: Off-topic for this email but something I don't think has been mentioned more than once, type hinting like we are proposing is exactly what Dart does and it is nice. Built-in API documentation of interfaces along with IDE support at the API makes all of this worth it.
For anyone not familiar with Dart, I think it is worth reading the Dart justification for static type hinting in a dynamic language: https://www.dartlang.org/articles/optional-types/ https://www.dartlang.org/articles/why-dart-types/ It's not quite the same as Guido's proposal, for example in Dart the type hints have no runtime effect at all, but I think it demonstrates that this is a proven approach and can work well with a dynamic language like Python without compromising the dynamic nature of the language. -- Steven
On Aug 24, 2014, at 4:59, Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote:
P.S.: Off-topic for this email but something I don't think has been mentioned more than once, type hinting like we are proposing is exactly what Dart does and it is nice. Built-in API documentation of interfaces along with IDE support at the API makes all of this worth it.
For anyone not familiar with Dart, I think it is worth reading the Dart justification for static type hinting in a dynamic language:
https://www.dartlang.org/articles/optional-types/
https://www.dartlang.org/articles/why-dart-types/
It's not quite the same as Guido's proposal, for example in Dart the type hints have no runtime effect at all, but I think it demonstrates that this is a proven approach and can work well with a dynamic language like Python without compromising the dynamic nature of the language.
Great reference. It's also worth looking at Boo (http://boo.codehaus.org), which was intentionally designed to read like Python, while also having static typing. So, it was designed in the opposite direction: given the CLR compile-time type system, modify Python syntax and semantics as little as possible to implement it. And it was also designed for completely different goals (mainly performance). But it does show how a type system pretty similar to the one Jukka and Guido are envisioning can fit into a language pretty similar to Python, which makes Boo code a useful example for "how readable could this be in real-life code?" questions.
On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping. This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong). Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6. Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime. I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
The use case I see is functions composition. I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid. 2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> :
On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Aug 21, 2014, at 9:43, Gregory Salvan <apieum@gmail.com> wrote:
I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
The use case I see is functions composition. I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid.
OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?) that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime. For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert@yahoo.com.dmarc.invalid>:
On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin,...
and, how often do you miss it ? sufficiently to have written my own tool, which I started to extract from a
True, explicit composition is actually uncommon practice in python, whereas there are lots of libs: toolz, funcy, fn.py, Pymonads.... Note that composition can be implicit: f(g(x)) Using mypy syntax was originally pushed with a presentation about what python can learn from FP. Personnally, I consider FP types have a different meaning than OO types, I feel them more as a way to describe programs flows (by func de/composition) and that's something I find really usefull in some cases (mainly for parsing and parallel computing). That's what I expect python would learn from FP and why I'm delighted about this proposition. (my opinion doesn't matter but +1000 for the proposition :) project few days before Guido announces his mypy proposal. (in development but usable: https://github.com/apieum/lawvere )
that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime.
I've not such cases in mind, I thought more of these ones: from functools import singledispatch @singledispatch def div1000(a: intexcept0) -> int: return 1000 / a @div1000.register def divby0(a: Just(0)): log('something')... In this case it's interesting to know if for ex. div1000 can be composed with sub500... when compiling and if it's 0 or not at runtime. (OK the example is simple and you can use a condition inside div1000 but you can have more complex cases of this type or want to use divby0 elsewhere)
For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
I'm not sure to understand what you are asking. I see Iterable[Number] as a kind of type constructor (like Just(0)) even if it's cached. I would probably check it at runtime with isinstance. 2014-08-21 20:36 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
On Aug 21, 2014, at 9:43, Gregory Salvan <apieum@gmail.com> wrote:
I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
The use case I see is functions composition. I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid.
OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?) that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime.
For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
2014-08-21 18:16 GMT+02:00 Andrew Barnert < abarnert@yahoo.com.dmarc.invalid>:
On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine@python.org> wrote:
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that
Le 19/08/2014 20:12, Guido van Rossum a écrit : the
ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Thursday, August 21, 2014 1:59 PM, Gregory Salvan <apieum@gmail.com> wrote:
OK, how about a _concrete_ use case?
I've not such cases in mind, I thought more of these ones:
from functools import singledispatch @singledispatch def div1000(a: intexcept0) -> int:
return 1000 / a
@div1000.register def divby0(a: Just(0)):
log('something')... In this case it's interesting to know if for ex. div1000 can be composed with sub500... when compiling and if it's 0 or not at runtime.
Antoine was against using the same types for runtime ABCs and compile-time type-checking, because "I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class." You want something that's more powerful than existing ABCs at runtime, and more powerful than existing typing.py classes at compile-time, but I don't see any reason why you can't write them as the _same_ something. In fact, your example seems to be doing exactly that. So, it seems like you're arguing for the exact same position as the one you disagreed with? What am I missing?
For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
I'm not sure to understand what you are asking. I see Iterable[Number] as a kind of type constructor (like Just(0)) even if it's cached. I would probably check it at runtime with isinstance.
But how would a runtime isinstance check on Iterable[Number] work? Either it has to ignore the generic part and just test Iterable (which is exactly what Guido's suggestion entails, which Antoine, and I thought you, didn't like), or it has to iterate all of the values and check them all, leaving you an empty iterable after checking (which may take infinite time, to boot).
2014-08-21 20:36 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
On Aug 21, 2014, at 9:43, Gregory Salvan <apieum@gmail.com> wrote:
I'll admit that I could be off on some of this. It's hard to discuss
anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some
realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
The use case I see is functions composition. I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid.
OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?) that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime.
For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert@yahoo.com.dmarc.invalid>:
On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine@python.org> wrote:
Le 19/08/2014 20:12, Guido van Rossum a écrit :
Hmm, I've been saying this already, but my intuition is that it's a bad idea to conflate *type descriptions* (what this proposal is about) and actual *runtime types* (what ABCs are).
But are they? I think the registration mechanism makes it clear that they aren't (necessarily) runtime types -- by linking a concrete type with an ABC through registration you are pretty clearly stating that the ABC *describes* (an aspect of) the concrete class without automatically adding any behavior from the ABC to it.
Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
participants (10)
-
Andrew Barnert -
Antoine Pitrou -
Brett Cannon -
Cem Karan -
Ethan Furman -
Gregory Salvan -
Guido van Rossum -
Nick Coghlan -
Steven D'Aprano -
Terry Reedy