Thanks Mark for posting this. I know some of us are uneasy about the pace of the typing train .... On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith <njs@pobox.com> wrote:

...

If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence.

You say this like it's a bad thing, but how is this avoidable, even in principle? Structural typing lets you check whether Foo is duck-shaped -- has appropriate attribute names, etc. But quacking like a duck is harder: you also have to implement the Sequence behavioral contract, and realistically the only way to know that is if the author of Foo tells you.

But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck. So yes, isinstance(obj, Sequence) is really the only way to know that obj is a Sequence in every important way -- but if you only need it to do one or two things like a Sequence, then you don't care. And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever). And a lot of the standard library works exactly this way. Two examples (chosen arbitrarily, I just happen to have thought about how they work): json.load() simply calls ``fp.read()``, and passes the result on down to json.loads(). That's it -- no checking of anything. If fp does not have a read() method, you get an AttributeError. If fp has a read() method, but it returns something other than a string, then you get some other Exception. And if it returns a string, but that string isn't valid JSON, you get yet another kind of error. In short, json.load(fp, ...) requires fp to have a read() method that returns a valid JSON string. But it doesn't check, nor does it need to, if it's getting an actual io.TextIOBase object. Is that the right one? I'm not totally sure, which I kind of think makes my point -- I've been using "file-like" objects for years (decades) without worrying about it. Example 2: The str.translate method takes: "a mapping of Unicode ordinals to Unicode ordinals, strings, or None" Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs go on to say: The table must implement lookup/indexing via __getitem__, for instance a dictionary or list. Ah -- so we don't need a Mapping -- we need anything indexable by an integer that contains "ordinals, strings, or None". What the heck ABC could we use for that? The ABCs do have an already complex hierarchy of containers, but there is no "Indexable", (quacks) and certainly no "indexable and returns these particular things. (quacks a certain way). (maybe there's something in the typing module that would work for static typing -- I have no idea). I'm pretty sure this particular API was designed to accommodate the old py2 str.translate, which took a length-256 sequence, while also accommodating full Unicode, which would have required a 2^32 length sequence to do the same thing :-) But again -- this is duck typing, built into the stdlib, and it works just fine. Granted, until PEP 563 (kind of) , there has been nothing that weakens or disallows such duck typing -- those of us that want to write fully duck-typed code can continue to do so. But there is the "culture" of Python -- and it has been very much shifting toward more typing -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there? And I've seen a lot more isinstance(Some_ABC) code lately as well. From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). Maybe it is -- but if the community is moving that way, we should be honest about it.

I'm not even sure that this *is* nominal typing. You could just as well argue that "the operation `isinstance(..., Sequence)` returns `True`" is just another of the behavioral constraints that are required to quack like a sequence.

I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this). The big distinction is whether we want to know if the object is a duck, or if we only need it to do one or two things like a duck. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

El mié, 21 abr 2021 a las 15:30, Paul Bryan (<pbryan@anode.ca>) escribió:

As demonstrated, protocols don't get us there because duck typing isn't a matter of having an object exhibit all of the attributes of a duck, but rather some subset of attributes to be used by the consumer. I want this duck to quack; someone else will want it to waddle. I don't see how type hints could reasonably support "file like object" in the duck type sense (unless the consumer were to specify the exact attributes of the duck it's interested in, which I fear would become a tedious type writing style). '

This approach (a Protocol that declares exactly what you need the file-like object to do) is in fact what we've been doing in typeshed, the repository that provides type hints for the standard library. For those unfamiliar, it would look something like this: from typing import Protocol class SupportsRead(Protocol): def read(self) -> bytes: ... def uses_a_file(f: SupportsRead) -> None: f.read() That's somewhat tedious, to be sure, but it is working duck typing.

I too have sensed static typing driving the typing development agenda in Python recently, causing other typing methods to take a back seat, so to speak. I add my voice to those requesting Python handle other typing methods.

Barring an innovation to allow a "subset" of a type to be declared in a type hint, I would conclude that static typing and duck typing are diametrically opposed. If we agree that both are valuable, developers could build consensus on that point, and work to ensure that one does not move forward at the expense of the other.

What would you suggest? Should the syntax for declaring Protocols be more concise?

Paul

On Wed, 2021-04-21 at 12:36 -0700, Christopher Barker wrote:

Thanks Mark for posting this. I know some of us are uneasy about the pace of the typing train ....

On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith <njs@pobox.com> wrote:

...

If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence.

You say this like it's a bad thing, but how is this avoidable, even in principle? Structural typing lets you check whether Foo is duck-shaped -- has appropriate attribute names, etc. But quacking like a duck is harder: you also have to implement the Sequence behavioral contract, and realistically the only way to know that is if the author of Foo tells you.

But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck.

So yes, isinstance(obj, Sequence) is really the only way to know that obj is a Sequence in every important way -- but if you only need it to do one or two things like a Sequence, then you don't care.

And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever).

And a lot of the standard library works exactly this way. Two examples (chosen arbitrarily, I just happen to have thought about how they work):

json.load() simply calls ``fp.read()``, and passes the result on down to json.loads(). That's it -- no checking of anything.

If fp does not have a read() method, you get an AttributeError. If fp has a read() method, but it returns something other than a string, then you get some other Exception. And if it returns a string, but that string isn't valid JSON, you get yet another kind of error.

In short, json.load(fp, ...) requires fp to have a read() method that returns a valid JSON string. But it doesn't check, nor does it need to, if it's getting an actual io.TextIOBase object. Is that the right one? I'm not totally sure, which I kind of think makes my point -- I've been using "file-like" objects for years (decades) without worrying about it.

Example 2:

The str.translate method takes:

"a mapping of Unicode ordinals to Unicode ordinals, strings, or None"

Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs go on to say:

The table must implement lookup/indexing via __getitem__, for instance a dictionary or list.

Ah -- so we don't need a Mapping -- we need anything indexable by an integer that contains "ordinals, strings, or None". What the heck ABC could we use for that?

The ABCs do have an already complex hierarchy of containers, but there is no "Indexable", (quacks) and certainly no "indexable and returns these particular things. (quacks a certain way). (maybe there's something in the typing module that would work for static typing -- I have no idea).

I'm pretty sure this particular API was designed to accommodate the old py2 str.translate, which took a length-256 sequence, while also accommodating full Unicode, which would have required a 2^32 length sequence to do the same thing :-)

But again -- this is duck typing, built into the stdlib, and it works just fine.

Granted, until PEP 563 (kind of) , there has been nothing that weakens or disallows such duck typing -- those of us that want to write fully duck-typed code can continue to do so.

But there is the "culture" of Python -- and it has been very much shifting toward more typing -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there?

And I've seen a lot more isinstance(Some_ABC) code lately as well.

From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). Maybe it is -- but if the community is moving that way, we should be honest about it.

I'm not even sure that this *is* nominal typing. You could just as well argue that "the operation `isinstance(..., Sequence)` returns `True`" is just another of the behavioral constraints that are required to quack like a sequence.

I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this).

The big distinction is whether we want to know if the object is a duck, or if we only need it to do one or two things like a duck.

-CHB

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZXI3RTBV... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PDW6UUK2... Code of Conduct: http://python.org/psf/codeofconduct/

My personal motivating example for PEP 637 was shorthand for protocols. `x: Protocol[foo=int, bar=Callable[str, int]]` could say x has attribute foo which is an int and method bar from str to int. On Wed, Apr 21, 2021 at 4:23 PM Paul Bryan <pbryan@anode.ca> wrote:

I agree, that's duck typing with a protocol, and precisely the tedious type style I would want to avoid.

I don't know what would be a good suggestion. Something where you reference a "fully equipped" type and cherry pick the attributes you want? Something like `Protocol[Duck, ("quack", "waddle")]`?

Paul

On Wed, 2021-04-21 at 16:13 -0700, Jelle Zijlstra wrote:

El mié, 21 abr 2021 a las 15:30, Paul Bryan (<pbryan@anode.ca>) escribió:

As demonstrated, protocols don't get us there because duck typing isn't a matter of having an object exhibit all of the attributes of a duck, but rather some subset of attributes to be used by the consumer. I want this duck to quack; someone else will want it to waddle. I don't see how type hints could reasonably support "file like object" in the duck type sense (unless the consumer were to specify the exact attributes of the duck it's interested in, which I fear would become a tedious type writing style). '

This approach (a Protocol that declares exactly what you need the file-like object to do) is in fact what we've been doing in typeshed, the repository that provides type hints for the standard library. For those unfamiliar, it would look something like this:

from typing import Protocol

class SupportsRead(Protocol): def read(self) -> bytes: ...

def uses_a_file(f: SupportsRead) -> None: f.read()

That's somewhat tedious, to be sure, but it is working duck typing.

I too have sensed static typing driving the typing development agenda in Python recently, causing other typing methods to take a back seat, so to speak. I add my voice to those requesting Python handle other typing methods.

Barring an innovation to allow a "subset" of a type to be declared in a type hint, I would conclude that static typing and duck typing are diametrically opposed. If we agree that both are valuable, developers could build consensus on that point, and work to ensure that one does not move forward at the expense of the other.

What would you suggest? Should the syntax for declaring Protocols be more concise?

Paul

On Wed, 2021-04-21 at 12:36 -0700, Christopher Barker wrote:

Thanks Mark for posting this. I know some of us are uneasy about the pace of the typing train ....

On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith <njs@pobox.com> wrote:

...

If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence.

You say this like it's a bad thing, but how is this avoidable, even in principle? Structural typing lets you check whether Foo is duck-shaped -- has appropriate attribute names, etc. But quacking like a duck is harder: you also have to implement the Sequence behavioral contract, and realistically the only way to know that is if the author of Foo tells you.

But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck.

So yes, isinstance(obj, Sequence) is really the only way to know that obj is a Sequence in every important way -- but if you only need it to do one or two things like a Sequence, then you don't care.

And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever).

And a lot of the standard library works exactly this way. Two examples (chosen arbitrarily, I just happen to have thought about how they work):

json.load() simply calls ``fp.read()``, and passes the result on down to json.loads(). That's it -- no checking of anything.

If fp does not have a read() method, you get an AttributeError. If fp has a read() method, but it returns something other than a string, then you get some other Exception. And if it returns a string, but that string isn't valid JSON, you get yet another kind of error.

In short, json.load(fp, ...) requires fp to have a read() method that returns a valid JSON string. But it doesn't check, nor does it need to, if it's getting an actual io.TextIOBase object. Is that the right one? I'm not totally sure, which I kind of think makes my point -- I've been using "file-like" objects for years (decades) without worrying about it.

Example 2:

The str.translate method takes:

"a mapping of Unicode ordinals to Unicode ordinals, strings, or None"

Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs go on to say:

The table must implement lookup/indexing via __getitem__, for instance a dictionary or list.

Ah -- so we don't need a Mapping -- we need anything indexable by an integer that contains "ordinals, strings, or None". What the heck ABC could we use for that?

The ABCs do have an already complex hierarchy of containers, but there is no "Indexable", (quacks) and certainly no "indexable and returns these particular things. (quacks a certain way). (maybe there's something in the typing module that would work for static typing -- I have no idea).

I'm pretty sure this particular API was designed to accommodate the old py2 str.translate, which took a length-256 sequence, while also accommodating full Unicode, which would have required a 2^32 length sequence to do the same thing :-)

But again -- this is duck typing, built into the stdlib, and it works just fine.

Granted, until PEP 563 (kind of) , there has been nothing that weakens or disallows such duck typing -- those of us that want to write fully duck-typed code can continue to do so.

But there is the "culture" of Python -- and it has been very much shifting toward more typing -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there?

And I've seen a lot more isinstance(Some_ABC) code lately as well.

From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). Maybe it is -- but if the community is moving that way, we should be honest about it.

I'm not even sure that this *is* nominal typing. You could just as well argue that "the operation `isinstance(..., Sequence)` returns `True`" is just another of the behavioral constraints that are required to quack like a sequence.

I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this).

The big distinction is whether we want to know if the object is a duck, or if we only need it to do one or two things like a duck.

-CHB

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZXI3RTBV... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PDW6UUK2... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HY6XET25... Code of Conduct: http://python.org/psf/codeofconduct/

On Apr 21, 2021, 5:29 PM -0500, Paul Bryan <pbryan@anode.ca>, wrote:

As demonstrated, protocols don't get us there because duck typing isn't a matter of having an object exhibit all of the attributes of a duck, but rather some subset of attributes to be used by the consumer. I want this duck to quack; someone else will want it to waddle. I don't see how type hints could reasonably support "file like object" in the duck type sense (unless the consumer were to specify the exact attributes of the duck it's interested in, which I fear would become a tedious type writing style).

I'd argue that, if you frequently have cases where functions use a relatively small subset of a much larger interface, it's simply that your interfaces are too large. You're always free to define your own, smaller protocols that just implement the subset of the interfaces you need.

I too have sensed static typing driving the typing development agenda in Python recently, causing other typing methods to take a back seat, so to speak. I add my voice to those requesting Python handle other typing methods.

Barring an innovation to allow a "subset" of a type to be declared in a type hint,

I mentioned above that defining smaller protocols is an option. If the main concern is due to wanting to define them in the type signature, or wanting to ensure the types match the base interface, then that's not new: TypeScript does them both (well, at least you could accomplish the latter roughly using intersection types).

Paul

On Wed, 2021-04-21 at 12:36 -0700, Christopher Barker wrote:

...

Thanks Mark for posting this. I know some of us are uneasy about the pace of the typing train ....

On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith <njs@pobox.com> wrote:

...

...

If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence.

You say this like it's a bad thing, but how is this avoidable, even in principle? Structural typing lets you check whether Foo is duck-shaped -- has appropriate attribute names, etc. But quacking like a duck is harder: you also have to implement the Sequence behavioral contract, and realistically the only way to know that is if the author of Foo tells you.

But that's not what duck typing is (at least to me :-) ) For a given function, I need the passed in object to quack (and yes, I need that quack to sound like a duck) -- but I usually don't care whether that object waddles like a duck.

So yes, isinstance(obj, Sequence) is really the only way to know that obj is a Sequence in every important way -- but if you only need it to do one or two things like a Sequence, then you don't care.

And this is not uncommon -- I suspect it's very rare for a single function to use most of the methods of a given ABC (or protocol, or whatever).

And a lot of the standard library works exactly this way. Two examples (chosen arbitrarily, I just happen to have thought about how they work):

json.load() simply calls ``fp.read()``, and passes the result on down to json.loads(). That's it -- no checking of anything.

If fp does not have a read() method, you get an AttributeError. If fp has a read() method, but it returns something other than a string, then you get some other Exception. And if it returns a string, but that string isn't valid JSON, you get yet another kind of error.

In short, json.load(fp, ...) requires fp to have a read() method that returns a valid JSON string. But it doesn't check, nor does it need to, if it's getting an actual io.TextIOBase object. Is that the right one? I'm not totally sure, which I kind of think makes my point -- I've been using "file-like" objects for years (decades) without worrying about it.

Example 2:

The str.translate method takes:

"a mapping of Unicode ordinals to Unicode ordinals, strings, or None"

Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs go on to say:

The table must implement lookup/indexing via __getitem__, for instance a dictionary or list.

Ah -- so we don't need a Mapping -- we need anything indexable by an integer that contains "ordinals, strings, or None". What the heck ABC could we use for that?

The ABCs do have an already complex hierarchy of containers, but there is no "Indexable", (quacks) and certainly no "indexable and returns these particular things. (quacks a certain way). (maybe there's something in the typing module that would work for static typing -- I have no idea).

I'm pretty sure this particular API was designed to accommodate the old py2 str.translate, which took a length-256 sequence, while also accommodating full Unicode, which would have required a 2^32 length sequence to do the same thing :-)

But again -- this is duck typing, built into the stdlib, and it works just fine.

Granted, until PEP 563 (kind of) , there has been nothing that weakens or disallows such duck typing -- those of us that want to write fully duck-typed code can continue to do so.

But there is the "culture" of Python -- and it has been very much shifting toward more typing -- A recent publication (sorry can't find it now -- my google fu is failing me) examined code on PyPi and found a lot of type hints -- many of which were apparently not being used with a static type checker. So why were they there?

And I've seen a lot more isinstance(Some_ABC) code lately as well.

From looking at the work of my beginning students, I can tell that they are seeing examples out there that use more typing, to the point that they think it's a best practice (or maybe even required?). Maybe it is -- but if the community is moving that way, we should be honest about it.

...

I'm not even sure that this *is* nominal typing. You could just as well argue that "the operation `isinstance(..., Sequence)` returns `True`" is just another of the behavioral constraints that are required to quack like a sequence.

I'm not sure of the definition of "nominal" typing -- but it absolutely is NOT duck typing (As Luciano pointed out, Alex Martelli coined the term Goose Typing for this).

The big distinction is whether we want to know if the object is a duck, or if we only need it to do one or two things like a duck.

-CHB

_______________________________________________ Python-Dev mailing list -- python-dev@python.orgTo unsubscribe send an email to python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZXI3RTBV... of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PDW6UUK2... Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, 22 Apr 2021 at 09:46, Chris Angelico <rosuav@gmail.com> wrote:

Maybe in some cases, the "smaller protocols" option is practical, but it would need to have a useful name. For instance, if it needs to be readable, iterable, closeable, and autocloseable via __enter__/__exit__, that's ... uhh.... a readable, iterable, closeable context manager? Not an improvement over "file-like object".

Note: I've not used protocols myself, so this is speculation. Is the name of the protocol important? Specifically, if I do, in my code class X(Protocol): def read(self): ... def __iter__(self): ... def close(self): ... def __enter__(self): ... def __exit__(self, exc_type, exc_val, exc_tb): ... def my_fn(fileobj: X) -> None: # my stuff would that not work? An argument is checked to see if it conforms with a protocol by confirming it has the right methods, not by name, inheritance or registration. And if you want, you can just call X "FileLike", it's only a local name so it won't clash with whatever other people (or you, in a different module) want to say is "file-like". Of course, that's incredibly verbose and messy, and it would result in a huge proliferation of throw-away protocol classes, which is probably not a good practice that we'd want to encourage, but it works. IMO, the problem isn't that *technically* static typing excludes the more traditional duck typing, but rather that the design, approach and as a result the emerging "best practices" are focused around inheritance based (is that what people mean by "nominal"?) models, to the point where duck typing feels like an afterthought that you have to work to include. I wonder whether type checkers could handle a "magic" type (let's call it DuckTyped for now :-)) which basically means "infer a protocol based on usage in this function". So if I do: def my_fn(f: DuckTyped): with f: data = f.read() for line in f: print(line) f.close() then the type checker would automatically build a protocol type like the one I defined above and use that as the type of f? That would make it much easier to include duck typed arguments in function signatures while keeping the benefits of static type checking. I will say that at the moment, this doesn't bother me much personally. On the larger projects where I've used typing, we've been fine with class-based typing and haven't really needed anything more complex like protocols. On smaller projects, I just don't use typing at all. Whether this will change if I decide to introduce typing in more places, I don't know at the moment. I've also not really used typing for public APIs, where an over-restrictive type would be more of an issue. Paul

On Thu, 22 Apr 2021 at 11:06, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Apr 22, 2021 at 7:53 PM Paul Moore <p.f.moore@gmail.com> wrote:

...

I wonder whether type checkers could handle a "magic" type (let's call it DuckTyped for now :-)) which basically means "infer a protocol based on usage in this function". So if I do:

def my_fn(f: DuckTyped): with f: data = f.read() for line in f: print(line) f.close()

then the type checker would automatically build a protocol type like the one I defined above and use that as the type of f? That would make it much easier to include duck typed arguments in function signatures while keeping the benefits of static type checking.

Someone will likely correct me if this is inaccurate, but my understanding is that that's exactly what you get if you just don't give a type hint. The point of type hints is to give more information to the type checker when it's unable to simply infer from usage and context.

Hmm, I sort of wondered about that as I wrote it. But in which case, what's the problem here? My understanding was that people were concerned that static typing was somehow in conflict with duck typing, but if the static checkers enforce the inferred duck type on untyped arguments, then that doesn't seem to be the case. Having said that, I thought that untyped arguments were treated as if they had a type of "Any", which means "don't type check". So I guess the point here is that either the typing community/documentation isn't doing a very good job of explaining how duck types and static types work together, or that people who like duck typed interfaces¹ aren't reading the available documentation in type checkers explaining how to do that with static typing :-) Paul ¹ And I include myself in that - maybe I need to go and read the mypy docs properly rather than just learning what I need by following examples in existing code...

According to PEP 484 all missing annotations in checked functions should be handled as Any. Any is compatible with all types. I think from a technical standpoint it should be possible to infer protocols for arguments for most functions, but there are some edge cases where this would not be possible, making it impractical to make this the default behavior. Having an annotation to make a type checker infer a protocol would be interesting though. For example: def f(x: int): ... def g(x: str): ... def main(t): if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1]) You could statically type t as Union[Tuple[Literal['version'], int], Tuple[Literal['name'], str]], but inferring a Protocol for this would be either very hard or even impossible, especially with even more complex conditions. Adrian Freund On April 22, 2021 1:04:11 PM GMT+02:00, Paul Moore <p.f.moore@gmail.com> wrote:

On Thu, 22 Apr 2021 at 11:21, Paul Moore <p.f.moore@gmail.com> wrote:

...

On Thu, 22 Apr 2021 at 11:06, Chris Angelico <rosuav@gmail.com> wrote:

...

Someone will likely correct me if this is inaccurate, but my understanding is that that's exactly what you get if you just don't give a type hint. The point of type hints is to give more information to the type checker when it's unable to simply infer from usage and context.

Hmm, I sort of wondered about that as I wrote it. But in which case, what's the problem here? My understanding was that people were concerned that static typing was somehow in conflict with duck typing, but if the static checkers enforce the inferred duck type on untyped arguments, then that doesn't seem to be the case. Having said that, I thought that untyped arguments were treated as if they had a type of "Any", which means "don't type check".

Looks like it doesn't:

...

cat .\test.py def example(f) -> None: f.close()

import sys example(12) example(sys.stdin) PS 12:00 00:00.009 C:\Work\Scratch\typing

...

mypy .\test.py Success: no issues found in 1 source file

What I was after was something that gave an error on the first call, but not on the second. Compare this:

...

cat .\test.py from typing import Protocol

class X(Protocol): def close(self): ...

def example(f: X) -> None: f.close()

import sys example(12) example(sys.stdin) PS 12:03 00:00.015 C:\Work\Scratch\typing

...

mypy .\test.py test.py:10: error: Argument 1 to "example" has incompatible type "int"; expected "X" Found 1 error in 1 file (checked 1 source file)

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/54C6G2JL... Code of Conduct: http://python.org/psf/codeofconduct/

On April 22, 2021 3:15:27 PM GMT+02:00, Paul Moore <p.f.moore@gmail.com> wrote:

On Thu, 22 Apr 2021 at 13:23, Adrian Freund <mail@freundtech.com> wrote:

...

According to PEP 484 all missing annotations in checked functions should be handled as Any. Any is compatible with all types.

Yep, that's what I understood to be the case.

...

I think from a technical standpoint it should be possible to infer protocols for arguments for most functions, but there are some edge cases where this would not be possible, making it impractical to make this the default behavior. Having an annotation to make a type checker infer a protocol would be interesting though.

Absolutely, I see no problem with "use duck typing for this argument" being opt-in.

...

For example:

def f(x: int): ... def g(x: str): ...

def main(t): if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1])

You could statically type t as Union[Tuple[Literal['version'], int], Tuple[Literal['name'], str]], but inferring a Protocol for this would be either very hard or even impossible, especially with even more complex conditions.

Yes, but that's inferred static typing which is *not* what I was proposing. I think I understood what you were proposing, but my example might have been less than ideal. Sorry for that. I mixed some static types in there to simplify it. The union wasn't meant at what it should infer but was meant as a comparison to what we would to currently, with static, nominal typing.

Let me try again without static types. def file(x): print(x.read()) # x has to have .read(): object def string(x): print(str(x)) # x has to have .__str__(self): object def main(t): if t[0] == 'file': file(t[1]) elif t[0] == 'string': string(t[1]) Here we can infer that t has to have a __getitem__(self, idx: int), but we can't infer it's return type

I was suggesting that the checker could easily infer that t must have a __getitem__ method, and nothing more. So the protocol to infer is

class TypeOfT(Protocol): def __getitem__(self, idx): ...

It would be nice to go one step further and infer

class TypeOfT(Protocol): def __getitem__(self, idx: int): ...

but that's *absolutely* as far as I'd want to go. Note in particular that I don't want to constrain the return value The problem is that this isn't enough to have a type safe program. You need to also constrain the return type to make sure the returned value can be safely passed to other functions. If you don't do this large parts of your codebase will either need explicit annotations or will be unchecked. - we've no way to know what type it might have in the general case. IMO, inferring anything else would over-constrain t - there's nothing in the available information, for example, that says t must be a tuple, or a list, or that t[3] should have any particular type, or anything like that. You can infer the return type of a function by looking at all the returns it contains, and inferring the types of the returned expressions. That isn't too hard and pytype for example already does it.

You can infer the return type a protocol function should have by looking at all the places it's result are used. If you have inferred return types then constraining return types using inferred protocols would be practical in my opinion.

My instinct is that working out that t needs to have a __getitem__ that takes an int is pretty straightforward, as all you have to do is look at where t is used in the function. Four places, all followed by [] with a literal integer in the brackets. That's it. I fully appreciate that writing *code* to do that can be a lot harder than it looks, but that's an implementation question, not a matter of whether it's reasonable as a proposal in theory.

This feels like *precisely* where there seems to be a failure of communication between the static typing and the duck typing worlds. I have no idea what I said that would make you think that I wanted anything like that Union type you quoted above. And yet obviously, you somehow got that message from what I did say.

Like I said above the Union. Was just meant as an example of that we would do with static, nominal typing, not what we want with duck typing. Sorry for the misunderstanding.

Anyway, as I said this is just an interesting idea as far as I'm concerned. I've no actual need for it right now, so I'm happy to leave it to the mypy developers whether they want to do anything with it.

Paul

Please let's not try to make Python a "typesafe" language. The success of Python owes a lot to the fact that duck typing is approachable, flexible and powerful. Even if you advocate static typing, I think it's a very good idea to limit the ambition of the type system if you want to keep most users happy—despite the opinion of language lawyers. Go is by far the most successful statically typed language created so far in the 21st century, and I believe a lot of that success is due to the simplicity of its type system. It's way more successful than Rust, possibly for this very reason. I find it admirable that Go was released without generics, which were seriously considered only after its runaway success. Generics will appear in Go 1.17, scheduled for August, 2021—9 years after Go 1.0. Please let us heed the wise words of Brian W. Kernighan and Alan A. A. Donovan in the introduction of "The Go Programming Language" book, a masterpiece (even if you don't like the language, the book is one of the best introduction to any language that I've ever read): """ Go has enough of a type system to avoid most of the careless mistakes that plague programmers in dynamic languages, but it has a simpler type system than comparable typed languages. This approach can sometimes lead to isolated pockets of ‘‘untyped’’ programming within a broader framework of types, and Go programmers do not go to the lengths that C++ or Haskell programmers do to express safety properties as type-based proofs. But in practice Go gives programmers much of the safety and run-time performance benefits of a relatively strong type system without the burden of a complex one. """ I also consider the support of static duck typing and goose typing (via runtime type assertions and type switches) key factors that make Go an excellent language to "get stuff done"—the best compliment Dave Beazley traditionally has for Python. Cheers, Luciano On Thu, Apr 22, 2021 at 12:05 PM Paul Moore <p.f.moore@gmail.com> wrote:

On Thu, 22 Apr 2021 at 15:22, Adrian Freund <mail@freundtech.com> wrote:

...

On April 22, 2021 3:15:27 PM GMT+02:00, Paul Moore <p.f.moore@gmail.com> wrote:

...

...

but that's *absolutely* as far as I'd want to go. Note in particular that I don't want to constrain the return value The problem is that this isn't enough to have a type safe program. You need to also constrain the return type to make sure the returned value can be safely passed to other functions.

But I don't want a type safe program. At least not in an absolute sense. All I want is for mypy to catch the occasional error I make where I pass the wrong parameter. For me, that's the "gradual" in "gradual typing" - it's not a lifestyle, just a convenience. You seem to be implying that it's "all or nothing".

...

If you don't do this large parts of your codebase will either need explicit annotations or will be unchecked.

That's just not true. (And even if it were true, you're assuming that I care - I've already said that my goal is much more relaxed than complete type safety).

I repeat, all I'm proposing is that

def f(x: int): ... def g(x: str): ...

def main(t: DuckTyped) -> None: if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1])

gets interpreted *exactly* the same as if I'd written

class TType(Protocol): def __getitem__(self, int): ...

def f(x: int): ... def g(x: str): ...

def main(t: TType) -> None: if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1])

How can you claim that the second example requires that " large parts of your codebase will either need explicit annotations or will be unchecked"? And if the second example doesn't require that, nor does the first because it's equivalent.

Honestly, this conversation is just reinforcing my suspicion that people invested in type annotations have a blind spot when it comes to dealing with people and use cases that don't need to go "all in" with typing :-(

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CUZNKEA3... Code of Conduct: http://python.org/psf/codeofconduct/

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

On 4/22/21 5:00 PM, Paul Moore wrote:

On Thu, 22 Apr 2021 at 15:22, Adrian Freund <mail@freundtech.com> wrote:

...

On April 22, 2021 3:15:27 PM GMT+02:00, Paul Moore <p.f.moore@gmail.com> wrote:

...

but that's *absolutely* as far as I'd want to go. Note in particular that I don't want to constrain the return value The problem is that this isn't enough to have a type safe program. You need to also constrain the return type to make sure the returned value can be safely passed to other functions. But I don't want a type safe program. At least not in an absolute sense. All I want is for mypy to catch the occasional error I make where I pass the wrong parameter. For me, that's the "gradual" in "gradual typing" - it's not a lifestyle, just a convenience. You seem to be implying that it's "all or nothing".

I don't think that inferring the required return type breaks gradual typing, but it is required for people who want type safety. If I understand correctly your concerns with inferring return types for inferred protocols are that it might be to restrictive and prevent gradual typing. Here are some examples to show how gradual typing would still work. If you have any concrete examples where inferring the return type would break gradual typing let me know and I'll have a look at them. def foo(x: DuckType):  # x has to have a .bar(self) method. # The return type of which is inferred as Any, as it isn't used     x.bar() def bar(x):     x.bar() def foo(x: DuckType):  # x has to have a .read(self) method. # The return type of which ist inferred as  Any, as the parameter to bar isn't typed.     bar(x.read()) Contrast that with def bar(x: DuckType):  # x has to have a .bar(self) method. # The return type of which is inferred as Any.     x.bar() def foo(x: DuckType):  # x has to have a .read(self) method that returns something with a .bar(self) method. # If we don't infer the return type our call to bar() might be unsafe despite both foo and bar being typed.     bar(x.read())

I repeat, all I'm proposing is that

def f(x: int): ... def g(x: str): ...

def main(t: DuckTyped) -> None: if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1])

gets interpreted *exactly* the same as if I'd written

class TType(Protocol): def __getitem__(self, int): ...

def f(x: int): ... def g(x: str): ...

def main(t: TType) -> None: if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1])

How can you claim that the second example requires that " large parts of your codebase will either need explicit annotations or will be unchecked"? And if the second example doesn't require that, nor does the first because it's equivalent.

Both examples don't check the calls to f and g despite f and g both being typed functions and being called from typed functions. In a real codebase this will lead to a lot more instances of this happening. It would happen every time you do anything with something returned from a method on an inferred protocol

Honestly, this conversation is just reinforcing my suspicion that people invested in type annotations have a blind spot when it comes to dealing with people and use cases that don't need to go "all in" with typing :-(

I don't think this is an all in or nothing. You can infer return types of inferred protocols and still use gradual typing.  It's just that not inferring return types causes problems for both full and gradual typing. Adrian Freund

On 4/22/2021 9:15 AM, Paul Moore wrote:

Absolutely, I see no problem with "use duck typing for this argument" being opt-in.

As in x: 'duck'? or x: '!', where '!' means 'infer it!', or from typing import Infer ... x: Infer ? Ditto for -> <infer indicator> ? -- Terry Jan Reedy

On Thu, Apr 22, 2021, at 1:01 PM, Brett Cannon wrote:

On Thu, Apr 22, 2021 at 4:11 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

On Thu, 22 Apr 2021 at 11:21, Paul Moore <p.f.moore@gmail.com> wrote:

...

On Thu, 22 Apr 2021 at 11:06, Chris Angelico <rosuav@gmail.com> wrote:

...

Someone will likely correct me if this is inaccurate, but my understanding is that that's exactly what you get if you just don't give a type hint. The point of type hints is to give more information to the type checker when it's unable to simply infer from usage and context.

Hmm, I sort of wondered about that as I wrote it. But in which case, what's the problem here? My understanding was that people were concerned that static typing was somehow in conflict with duck typing, but if the static checkers enforce the inferred duck type on untyped arguments, then that doesn't seem to be the case. Having said that, I thought that untyped arguments were treated as if they had a type of "Any", which means "don't type check".

Looks like it doesn't:

...

cat .\test.py def example(f) -> None: f.close()

import sys example(12) example(sys.stdin) PS 12:00 00:00.009 C:\Work\Scratch\typing

...

mypy .\test.py Success: no issues found in 1 source file

What I was after was something that gave an error on the first call, but not on the second. Compare this:

In PyCharm, the above code will result in it highlighting the number `12` with the following warning: "Type 'int' doesn't have expected attribute 'close'" Similarly, for: def f(x: int): ... def g(x: str): ... def main(t: DuckTyped) -> None: if t[0] == 'version': f(t[1]) elif t[0] == 'name': g(t[1]) If you replace `f(t[1])` or `g(t[1])` with just `t.` and activate auto-completion, it'll show `__getitem__` as an option (but it won't show any additional int/str methods). If instead you add an `elif isinstance(t, str):`, under that condition it'll auto-complete `t.` with all string properties/methods.

On Thu, Apr 22, 2021 at 5:43 AM Chris Angelico <rosuav@gmail.com> wrote:

File-like objects are used VERY frequently in the stdlib, and actual open file objects have quite a large interface. The use-case is a pretty real one: "if I were to create a simulant file object to pass to json.load(), what methods do I need?".

My experience with so-called "file-like objects" is that the interface required most of the time consists of a single method: read()

Maybe in some cases, the "smaller protocols" option is practical, but it would need to have a useful name.

The authors of the typing module already came up with an excellent convention. For the narrow protocol I mentioned, the conventional name would be "SupportsRead". Maybe "SupportsRead[str]" and "SupportsRead[bytes]".

For instance, if it needs to be readable, iterable, closeable, and autocloseable via __enter__/__exit__, that's ... uhh.... a readable, iterable, closeable context manager? Not an improvement over "file-like object".

Yes, file-like objects can and do have lots of methods. Often you don't need more than read() Cheers, Luciano On Thu, Apr 22, 2021 at 7:04 AM Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Apr 22, 2021 at 7:53 PM Paul Moore <p.f.moore@gmail.com> wrote:

...

I wonder whether type checkers could handle a "magic" type (let's call it DuckTyped for now :-)) which basically means "infer a protocol based on usage in this function". So if I do:

def my_fn(f: DuckTyped): with f: data = f.read() for line in f: print(line) f.close()

then the type checker would automatically build a protocol type like the one I defined above and use that as the type of f? That would make it much easier to include duck typed arguments in function signatures while keeping the benefits of static type checking.

Someone will likely correct me if this is inaccurate, but my understanding is that that's exactly what you get if you just don't give a type hint. The point of type hints is to give more information to the type checker when it's unable to simply infer from usage and context.

ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RW5ACSLJ... Code of Conduct: http://python.org/psf/codeofconduct/

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

Am 22.04.21 um 10:42 schrieb Chris Angelico:

File-like objects are used VERY frequently in the stdlib, and actual open file objects have quite a large interface. The use-case is a pretty real one: "if I were to create a simulant file object to pass to json.load(), what methods do I need?".

Maybe in some cases, the "smaller protocols" option is practical, but it would need to have a useful name. For instance, if it needs to be readable, iterable, closeable, and autocloseable via __enter__/__exit__, that's ... uhh.... a readable, iterable, closeable context manager? Not an improvement over "file-like object".

Experience from typeshed shows that many functions in the stdlib and third-party libraries only use one or very few methods, very often just read() or write(). From a practical standpoint, small protocols seem quite feasible. These are quite an improvement over "file-like" objects, where no one knows what that actually means.  - Sebastian

Luciano Ramalho writes:

A HUGE insight I learned studying Go is that Protocols should be defined near the code that CONSUMES it, and not near the code that PROVIDES it. That's exactly the opposite of how we use ABCs, or Java folks use interfaces (most of the time).

I don't see how that works for "public" protocols or ABCs in a standard library (maybe I drank too much C in my tea?) Yes, I write wrapper classes or factory functions that deal with the niggling details, even if they're only called once. And of course that's near where they're used. For example, Python's "default" 'shift_jis' codec uses an ancient version of the coded character set which errors on about half the Shift JIS files I get nowadays, so I wrote: def open_sjis(f, **kwargs): return open(f, encoding='shift_jisx0213', **kwargs) because I never remember the exact name, and Ireally don't care about the Shift JIS version, only that it's Japanese and not UTF-8. When called, the wrapper function expresses what's going on more clearly than the builtin call would. But that seems more like good programming workflow, not a protocol. Continuing the example, 'open_sjis' ended up in a small personal "tool box". Then it quickly became not a function at all when I realized that I wasn't going to ever add more "fixed" arguments, and I needed the string in many other places: SJIS = 'shift_jisx0213' *Now* that is a personal protocol, serving the same function of telling me "this code deals with a legacy Japanese encoding" and implementing it behind the scenes. But I don't see how that can be "defined close to its consumers", which are all over the place, including interactive sessions. What am I missing, and how might that be applied to Python? Steve

Paul Moore writes:

What you're missing, I think, is that we're talking about typing.Protocol - see here: https://docs.python.org/3/library/typing.html#typing.Protocol

I understand that we're talking about typing. What I don't understand is how any general facility provided in a standard library can be "defined close to the consumer". By definition, these facilities are far from the provider, since they're abstract (whether they're defined ABCs, "small" protocols, or duck typing). The point of my example is that indeed, my tiny "SJIS protocol" *arose* near the original consumer, but it was refined and is now *defined* "far" from any consumer. (The fact that it's self- providing is an accident of the fact that it's a concrete protocol.) I think that this is a useful process for software development (ie, design APIs from the point of view of the consumers and then make sure the backends provide them) but I don't see how this applies to structuring the Python stdlib or typing.*. Steve

I am not taking sides now, but I want to share with you a useful diagram to reason about typing support in Python. I struggled to explain what Python offers until I came up with this diagram: https://standupdev.com/wiki/doku.php?id=python_protocols#the_typing_map The Typing Map has two orthogonal axis: - when are types checked: -- runtime checking -- static checking - how are type checked: -- structural types -- nominal types The quadrants are informally labeled with the terms in ALL CAPS below. Traditionally, mainstream languages supported one of two diagonally opposite quadrants: STATIC TYPING and DUCK TYPING. Now the situation is more complicated. - Java supports only STATIC TYPING: static checking of nominal types; Python started supporting nominal types with PEP 484 - Before ABCs, Python supported only DUCK TYPING: runtime checking of structural types; - With ABCs, Python started supporting GOOSE TYPING (a term invented by Alex Martelli, in cc because I just quoted him): runtime checking of nominal types (with subclass hook which is a backdoor to support explicit checks on structural types as well); - With PEP 544, Python started supporting STATIC DUCK TYPING: static checking of structural types; There are languages that support multiple quadrants: - TypeScript, like Python, supports all four quadrants. - Go supports STATIC TYPING, but it also famously popularized STATIC DUCK TYPING, and even supports GOOSE TYPING with features like type assertions and type switches [1] designed for explicit runtime checking of nominal or structural types. [1] https://tour.golang.org/methods/16 The Typing Map will be featured in my upcoming PyCon US talk [2] [2] https://us.pycon.org/2021/schedule/presentation/80/ Cheers, Luciano PS. If you are aware of other languages that support more than one of these quadrants, please let me know! On Tue, Apr 20, 2021 at 6:53 PM Eric Casteleijn <thisfred@gmail.com> wrote:

On Tue, Apr 20, 2021 at 10:03 AM Mark Shannon <mark@hotpy.org> wrote:

...

... PEP 544 supports structural typing, but to declare a structural type you must inherit from Protocol. That smells a lot like nominal typing to me.

Note that to implement a protocol you do not have to inherit from anything. You create a structural type that subclasses Protocol, but then any object that satisfies that protocol can be passed where that type is expected, without having to inherit anything, so I would argue that this really is structural typing.

-- - eric casteleijn (he/him) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R3VP4KOR... Code of Conduct: http://python.org/psf/codeofconduct/

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

Now, given the terms I used in the last e-mail, it does worry me that the push towards supporting STATIC TYPING in Python sometimes ignores the needs of people who use the other typing disciplines. For example, Pydantic is an example of using type hints for GOOSE TYPING. And it alwo worries me that many of those not concerned with STATIC TYPING are not following closely the decisions being made in the 19 (and counting) typing PEPs., some of which affect everyone who uses Python. Here are two examples: PEP 484 DISMISSES THE NUMBERS ABCs Direct quote from section "The Numeric Tower" [0]: "There are some issues with these ABCs". [0] https://www.python.org/dev/peps/pep-0484/#the-numeric-tower No further explanation is given. The PEP then proposes ad-hoc rules to support int, float and complex, leaving users of the Numbers ABCs, Fraction, Decimal, and the huge NumPy community in the dark as to how to handle other numeric types. Now, I understand that the Numeric tower, as defined, cannot be used for STATIC TYPING because the root class—Number—has no methods. Also, there are deeper arguments regarding the soundness of that hierarchy. See for example "Bad Ideas In Type Theory" [1]. [1] https://www.yodaiken.com/2017/09/15/bad-ideas-in-type-theory/ However, I think a package in the standard library, and more importantly, the users of that package, deserve more attention. If PEP 484 et. al. wants to serve all the Python community, it must be part of its mission to address the problem of the Numbers ABCs, either fixing it, or deprecating it while proposing something better, but not creating an ad-hoc rule for three built-in types, ignoring all others. PEP 563 EMBRACES THEN DEPRECATES ALL OTHER USES OF ANNOTATIONS Section "Non-typing usage of annotations" [2] of PEP 563 starts with: "While annotations are still available for arbitrary use besides type checking..." but ends in a very different tone: "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated." [2] https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations I think if more eyes beyond the STATIC TYPING community were looking at the PEP, that inconsistency would have been flagged and dealt with in some way. OUR PROBLEM There is a rift that need to work to close in the Python community. Some people invested in STATIC TYPING are involved with million-line codebases where performance is paramount and the cost of bugs may be very high, and that creates a big motivation to use Python in a more constrained way. They are not paying attention to other uses of type hints—which are no deviations at all, because PEP 3107 explicitly encouraged people to experiment with the annotations. If that is deprecated, more people should be part of the conversation—as we are seeing now. On the other hand, most people who were attracted to Python *because* of the flexibility of DUCK TYPING and GOOSE TYPING perhaps don't follow the typing PEPs so closely—there are many, and most are pretty hard to read, by the nature of the subject matter. A BEAUTIFUL BRIDGE PEP 544 was a *huge* step forward to bridge the gap between static typing and duck typing in the language. Thank you very much for all who proposed and implemented it. That's the kind of bridge we need! (The `SupportsFloat` and similar protocols introduced with PEP 544 are a way to test number types when they are not limited to float, int and complex. But they expose some inconsistencies, and we still should do something about the Numbers ABCs, IMHO) GOING FORWARD I think the PEP 563 v. PyDantic et. al. case is a very good opportunity for us to think of ways to engage in dialog with the different parts of the Python community on how they use types in the language, considering STATIC TYPING, DUCK TYPING, STATIC DUCK TYPING, GOOSE TYPING, without dismissing any of these styles. If typing.get_type_hints is offered as a bridge for runtime uses of type hints, it must work well and its shortcomings must be documented as part of the official Python documentation, so that everyone knows how reliable they are. And in the future, if people come up with some new way of handling types, every new addition must be considered in light of existing typing styles—after a good, broad conversation involving different user bases. Cheers, Luciano On Tue, Apr 20, 2021 at 7:35 PM Luciano Ramalho <luciano@ramalho.org> wrote:

I am not taking sides now, but I want to share with you a useful diagram to reason about typing support in Python.

I struggled to explain what Python offers until I came up with this diagram:

https://standupdev.com/wiki/doku.php?id=python_protocols#the_typing_map

The Typing Map has two orthogonal axis:

- when are types checked: -- runtime checking -- static checking

- how are type checked: -- structural types -- nominal types

The quadrants are informally labeled with the terms in ALL CAPS below.

Traditionally, mainstream languages supported one of two diagonally opposite quadrants: STATIC TYPING and DUCK TYPING.

Now the situation is more complicated.

- Java supports only STATIC TYPING: static checking of nominal types; Python started supporting nominal types with PEP 484

- Before ABCs, Python supported only DUCK TYPING: runtime checking of structural types;

- With ABCs, Python started supporting GOOSE TYPING (a term invented by Alex Martelli, in cc because I just quoted him): runtime checking of nominal types (with subclass hook which is a backdoor to support explicit checks on structural types as well);

- With PEP 544, Python started supporting STATIC DUCK TYPING: static checking of structural types;

There are languages that support multiple quadrants:

- TypeScript, like Python, supports all four quadrants.

- Go supports STATIC TYPING, but it also famously popularized STATIC DUCK TYPING, and even supports GOOSE TYPING with features like type assertions and type switches [1] designed for explicit runtime checking of nominal or structural types.

[1] https://tour.golang.org/methods/16

The Typing Map will be featured in my upcoming PyCon US talk [2]

[2] https://us.pycon.org/2021/schedule/presentation/80/

Cheers,

Luciano

PS. If you are aware of other languages that support more than one of these quadrants, please let me know!

On Tue, Apr 20, 2021 at 6:53 PM Eric Casteleijn <thisfred@gmail.com> wrote:

...

On Tue, Apr 20, 2021 at 10:03 AM Mark Shannon <mark@hotpy.org> wrote:

...

... PEP 544 supports structural typing, but to declare a structural type you must inherit from Protocol. That smells a lot like nominal typing to me.

Note that to implement a protocol you do not have to inherit from anything. You create a structural type that subclasses Protocol, but then any object that satisfies that protocol can be passed where that type is expected, without having to inherit anything, so I would argue that this really is structural typing.

-- - eric casteleijn (he/him) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R3VP4KOR... Code of Conduct: http://python.org/psf/codeofconduct/

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

On Fri, Apr 23, 2021 at 10:33 AM Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Apr 23, 2021 at 11:22 AM Larry Hastings <larry@hastings.org> wrote:

...

On 4/20/21 10:03 AM, Mark Shannon wrote:

If you guarded your code with `isinstance(foo, Sequence)` then I could not use it with my `Foo` even if my `Foo` quacked like a sequence. I was forced to use nominal typing; inheriting from Sequence, or explicitly registering as a Sequence.

If I'm reading the library correctly, this is correct--but, perhaps, it could be remedied by adding a __subclasshook__ to Sequence that looked for an __iter__ attribute. That technique might also apply to other ABCs in collections.abc, Mapping for example. Would that work, or am I missing an critical detail?

How would you distinguish between a Sequence and a Mapping? Both have __iter__ and __len__. Without actually calling those methods, how would the subclass hook tell them apart?

ChrisA

We can add .keys() to Mapping to distinguish Mapping and Sequence. But it is breaking change, of course. We shouldn't change it. I think using ABC to distinguish sequence or mapping is a bad idea. There are three policies: a) Use duck-typing; just us it as sequence. No type check at all. b) Use strict type checking; isinstance(x, list) / isinstance(x, (list, tuple)). c) Use ABC. But (c) is broken by design. It is not fixable. IMHO, We should chose (a) or (b) and reject any idea relying on Sequence ABC. Regards, -- Inada Naoki <songofacandy@gmail.com>

On Fri, 23 Apr 2021, 12:34 pm Inada Naoki, <songofacandy@gmail.com> wrote:

I think using ABC to distinguish sequence or mapping is a bad idea.

There are three policies:

a) Use duck-typing; just us it as sequence. No type check at all. b) Use strict type checking; isinstance(x, list) / isinstance(x, (list, tuple)). c) Use ABC.

But (c) is broken by design. It is not fixable. IMHO, We should chose (a) or (b) and reject any idea relying on Sequence ABC.

That ship sailed long ago, since distinguishing sequences from mappings was one of the original motivating use cases for ABCs (see the last sentence in https://www.python.org/dev/peps/pep-3119/#abcs-vs-duck-typing ). One of the important things to remember about ABCs is that passing a *runtime* isinstance check is a matter of calling "the_abc.register(my_type)". Hence the comment earlier in the thread that with ABCs, passing "isinstance(obj, the_abc)" becomes just another criterion for quacking like a duck. Cheers, Nick.

Practically speaking, one issue I have is how easy it is to write isinstance or issubclass checks. It has historically been much more difficult to write and maintain a check that something looks like a duck.

`if hasattr(foo, 'close') and hasattr(foo, 'seek') and hasattr(foo, 'read'):`

Just does not roll off the figurative tongue and that is a relatively simple example of what is required for a duck check.

To prevent isinstance use when a duck check would be better,

I'm going to chime in briefly then return to lurking on this topic, trying to figure out all the changes to typing while I wasn't paying attention. Back in ancient times I recall "look before you leap" as the description of either of the above styles of checks, no matter which was easier to type. At the time, I thought the general recommendation was to document what attributes you expected objects to provide and just make the relevant unguarded references. I no longer recall what the tongue-in-cheek description of that style was (just "leap"?) Is that more simple usage more akin to classic "duck typing" than always guarding accesses? I assume that will still have a place in the pantheon of Python type variants. Skip

On Sat, Apr 24, 2021 at 10:14 AM Nick Coghlan <ncoghlan@gmail.com> wrote:

On Sat, 24 Apr 2021, 10:02 am Skip Montanaro, <skip.montanaro@gmail.com> wrote:

...

...

Practically speaking, one issue I have is how easy it is to write isinstance or issubclass checks. It has historically been much more difficult to write and maintain a check that something looks like a duck.

`if hasattr(foo, 'close') and hasattr(foo, 'seek') and hasattr(foo, 'read'):`

Just does not roll off the figurative tongue and that is a relatively simple example of what is required for a duck check.

To prevent isinstance use when a duck check would be better,

I'm going to chime in briefly then return to lurking on this topic, trying to figure out all the changes to typing while I wasn't paying attention. Back in ancient times I recall "look before you leap" as the description of either of the above styles of checks, no matter which was easier to type. At the time, I thought the general recommendation was to document what attributes you expected objects to provide and just make the relevant unguarded references. I no longer recall what the tongue-in-cheek description of that style was (just "leap"?) Is that more simple usage more akin to classic "duck typing" than always guarding accesses? I assume that will still have a place in the pantheon of Python type variants.

LBYL: Look before you leap (check then use) EAFP: Easier to ask forgiveness than permission (use, then handle any exceptions or let them escape)

(for accessing external resources rather than satisfying internal type consistency, you usually need the latter, as the external state may change between checking and usage)

Duck typing usually falls squarely into the EAFP category.

Isn't it orthogonal? LBYL means checking first, EAFP means going ahead and doing things, and coping with failure. Duck typing means caring about it having a specific method/attribute, nominal typing means caring about it "being a list". You can test either form in advance, and you can certainly duck-type in EAFP style, so there's only one quadrant that might not be possible (and even that, I'm sure, would be possible somehow). ChrisA

Am 24.04.2021 um 01:26 schrieb Gregory P. Smith:

Practically speaking, one issue I have is how easy it is to write isinstance or issubclass checks. It has historically been much more difficult to write and maintain a check that something looks like a duck.

 `if hasattr(foo, 'close') and hasattr(foo, 'seek') and hasattr(foo, 'read'):`

Just does not roll off the figurative tongue and that is a relatively simple example of what is required for a duck check.

To prevent isinstance use when a duck check would be better, we're missing an easy builtin elevated to the isinstance() availability level behaving as lookslikeaduck() that does matches against a (set of) declared typing.Protocol shape(s). An implementation of this exists - https://www.python.org/dev/peps/pep-0544/#runtime-checkable-decorator-and-na... <https://www.python.org/dev/peps/pep-0544/#runtime-checkable-decorator-and-na...> - but it requires the protocols to declare runtime checkability and has them work with isinstance similar to ABCs...  technically accurate /BUT via isinstance/? Doh!  It promotes the use of isinstance when it really isn't about class hierarchy at all...

Edit: Maybe that's okay, isinstance can be read leniently to mean "is an instance of something that one of these things over here says it matches" rather than meaning "a parent class type is..."?  From a past experience user perspective I don't read "isinstance" as "looks like a duck" when I read code.  I assume I'm not alone.

As Nathaniel indicated, how deep do we want to go down this rabbit hole of checking?  just names? signatures and types on those?  What about exceptions (something our type system has no way to declare at all)?  and infinite side effects?  At the end of the day we're required to trust the result of whatever check we use and any implementation may not conform to our desires no matter how much checking we do. Unless we solve the halting problem. :P I think a PEP to discuss these questions would be appropriate. Things

I'm using isinstance from time to time, mostly to satisfy the type checker in some cases. I think having a "hasshape()" builtin would be a huge win. In addition it would be useful for type checkers to better support hasattr() for distinguishing between separate types. For example, mypy has a problem with the following:     class A: pass     class B:         x: int     def foo(x: A | B) -> None:         if hasattr(x, "b"):             accepts_b(x)         else:             accepts_a(x) like also checking types could also be optional.

Not quite.  A Protocol is merely a way to describe a structural type.  You do not /need/ to have your /implementations/ of anything inherit from typing.Protocol.  I'd /personally/ advise people /do not inherit/ from Protocol in their implementation.

+1

Luciano notes that it is preferred to define your protocols as narrow and define them in places *where they're used*, to follow a golang interface practice.  My thinking aligns with that.

My background is with TypeScript, not go. But TypeScript uses "interfaces" extensively to describe the shape of expected objects. I agree mostly with you and Luciano here, but it can make sense to define some protocols in a more visible location. Examples are the protocols in collections.abc. In typeshed we collected a few more common protocols in the type-check only module _typeshed. (https://github.com/python/typeshed/tree/master/stdlib/_typeshed) These are a bit experimental, but could eventually find their way into the standard library. A bit off-topic: But intersection types (as discussed here: https://github.com/python/typing/issues/213) could also make a nice addition to quickly compose ad-hoc protocols from pre-made protocols:     def read_stuff(f: HasRead & HasSeek) -> None: ...

That inheritance is used in the /declaration/ of the protocol is an implementation detail because our language has never had a syntax for declaring an interface. 544 fit within our existing language syntax.

Jukka Lehtosalo proposed a new "type" keyword over at typing-sig: https://mail.python.org/archives/list/typing-sig@python.org/thread/LV22PX454... This could also be used to define protocols a bit more succintly, and with additional syntax constraints, making protocols feel a bit more "first class" than they feel now.  - Sebastian