Per-field syntax for defining potentially missing fields in TypedDict
I believe I was suggested to post this here (instead of at https://github.com/python/mypy/issues/9867).
TLDR; Allow `Omittable[...]` annotation, to mark fields that may not exist. Current way of doing this, with `total=False`, is verbose and forces to break the natural grouping of data.
As I understand, `total=False` was discussed/added in https://github.com/python/mypy/issues/2632. There's not much debate about the syntax there - the examples/suggestions there are `MayExist[...]`, `NotRequired[...]`, `Checked[...]`, `OptionalItem[...]` or (the other way around) `Required[...]`. However, none of these were considered good enough, so `total=False` got chosen instead.
How about `Omittable[...]`? It's only +1 char to `Optional`, which would be perfect, but of course cannot be used here.
Longer explanation for why we should need this below, or view the https://github.com/python/mypy/issues/9867 which has Markdown formatting. Though, Guido already said that "There is not much disagreement on the problem".
---
We can define optional fields for `TypedDict` as in:
class MyDictBase(TypedDict): required: str
class MyDict(MyDict): optional: str
Compare this to some other languages (like Typescript):
interface Data { required: string; optional?: string; }
The problem is with syntax here - the Python version is less intuitive, more verbose (for short declarations) and it breaks the "flow" of fields.
**1. Less intuitive**
When you think about:
{ "required1": "...", "optional1": "...", "required2": "..." }
Why do you need to split up the required and optional fields of one single structure into 2 classes? It's just... not nice.
**2. More verbose**
While I understand that Python syntax saves characters when there are lots of optional fields, which would be represented like this:
class MyDict(TypedDict): optional1: Checked[str] optional2: Checked[str] optional3: Checked[str] ...
Omitting `Checked[]` we save 9 chars per field and the result looks nicer as:
class MyDict(TypedDict, total=False): optional1: str optional2: str optional3: str ...
However, there is certain limit, when this starts to make sense. If you only have a few fields, the drawback of having to break the data into two classes and splitting up fields from their context (see point 3 below) outweighs the benefit of saving a few characters.
Furthermore, the problem of verbosity is oftentimes solved by `?` or similar in other languages:
interface Data { required: string; optional1?: string; optional2?: string; optional3?: string; ... }
This adds only single character per field, and is completely acceptable.
Like we are now getting `|` for `Union`, maybe we'll get `?` some day, and it will solve the problem of verbosity. Even without, I would be glad to add some visual clutter to avoid having to split up my structure into two different sections.
**3. Breaks the "flow" of fields**
While the order of items in a dictionary-like structure is insignificant to a machine (though we got ordered dicts by default in 3.6+), it often has meaning to a human. For example, a structure in our application looks something like:
{ "schema": "some-schema", "v": 1,
"msg_id": "...", "flow_id": "(O) ...",
"category": "...", "type": "(O) ...",
"created_on": "...", "created_by": "...", "updated_on": "(O) ...", "updated_by": "(O) ..." }
As you can see, there are sort of field "groups" there. This grouping/ordering must be broken when defining a TypedDict for the structure:
class StructBase(TypedDict): schema: str v: int
msg_id: str
category: str
created_on: str created_by: str
class Struct(StructBase, total=False): flow_id: str
type: str
updated_on: str updated_by: str
The result is much more verbose, and more difficult to follow.
On Mon, Jan 4, 2021 at 7:37 AM Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
I believe I was suggested to post this here (instead of at https://github.com/python/mypy/issues/9867).
TLDR; Allow `Omittable[...]` annotation, to mark fields that may not exist. Current way of doing this, with `total=False`, is verbose and forces to break the natural grouping of data.
As I understand, `total=False` was discussed/added in https://github.com/python/mypy/issues/2632. There's not much debate about the syntax there - the examples/suggestions there are `MayExist[...]`, `NotRequired[...]`, `Checked[...]`, `OptionalItem[...]` or (the other way around) `Required[...]`. However, none of these were considered good enough, so `total=False` got chosen instead.
How about `Omittable[...]`? It's only +1 char to `Optional`, which would be perfect, but of course cannot be used here.
Longer explanation for why we should need this below, or view the https://github.com/python/mypy/issues/9867 which has Markdown formatting. Though, Guido already said that "There is not much disagreement on the problem".
I'm not crazy about Omittable, because it starts with the same letter as Optional and my poor eyesight is likely to confuse these. So here's a proposed bikeshed color: ``` class MyThing(TypedDict, total=False): req1: Required[int] opt1: str req2: Required[float] ``` In a TypedDict subclass with total=True, all fields are Required and Required is redundant.
That's a potential workaround as `Required` is pretty self-explanatory indeed. Though, you would expect the same to work in reverse:
class MyThing(TypedDict, total=True): req1: int opt1: NotRequired[str] # or whatever we call it req2: float
Having support for just the one way around would feel insufficient...
1) How about:
Droppable Potential Open
2) I'm actually wondering where the "optionality" here should be indicated. In typescript, this is marked next to the variable and not the type:
interface Data { opt?: string; }
So should we instead:
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value
That would maybe allow to re-use `Optional` as it would exist in different context (don't know how confusing that would be). But this probably doesn't fit the design and the parser?
3) One more suggestion:
class MyThing(TypedDict, optional_types_may_be_missing=True): req1: int opt1: Optional[str]
This would allow `Optional` fields to be also missing. It would not work when you want a field, that can be missing, but if present, is not allowed to have `None` as value. I would expect that to be rare (but still a requirement for some).
Obviously a better name would be needed for the parameter. Maybe `allow_missing`, `missing`, `strict=False` etc.
Sorry, none of your proposals (Droppable, Potential, NotRequired, Open, or another metaclass flag) sound right to me. We don't have an inverse of Optional -- why would we need an inverse of Required?
The only objection to Required I can think of is actually that it *sounds* like the inverse of Optional, which it isn't. Then again, Optional should have been named Nullable from the beginning, but it's too late to change it now.
On Tue, Jan 5, 2021 at 12:36 AM Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
One more suggestion:
class MyThing(TypedDict, optional_types_may_be_missing=True): req1: int opt1: Optional[str]
This would allow `Optional` fields to be also missing. It would not work when you want a field, that can be missing, but if present, is not allowed to have `None` as value. I would expect that to be rare (but still a requirement for some).
Obviously a better name would be needed for the parameter. Maybe `allow_missing`, `missing`, `strict=False` etc. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
I agree with Guido that none of these proposals sounds quite right. I like the `Required` annotation proposal.
If this becomes a common use case, we could make it terser in the future by leveraging one of the existing unary operators (`__pos__`, `__neg__` and `__invert__`). This could be done without modifying the grammar. For example, by defining an `__invert__` operator on `type`, the syntax would look like the following:
```python class MyThing(TypedDict, total=False): req1: int opt1: ~str req2: float ```
I recommend first releasing support for `Required` before attempting to make it more terse. This is the same approach that was used for `Union`, which can now be specified using the `__or__` operator.
I don't disagree with you. But the fact is that the perfect word, "optional", has already been taken and a compromise would be needed (assuming there is a feature worth debating here).
We don't have an inverse of Optional -- why would we need an inverse of Required? The only objection to Required I can think of is actually that it sounds like the inverse of Optional, which it isn't.
Maybe it was confusing when I wrote "the other way around". If we consider:
- Required = always there - Opposite of it = never there ("missing"?) - Optional = sometimes there
I didn't mean the opposite/inverse, I meant "sometimes there".
Anyway, I don't think it's a problem. Required/optional is common terminology, and pretty self-explanatory even to non-native English speakers.
Theoretically, `Missing` would be needed if you wanted to declare a Typescript-like interface:
class MyThing(TypedDict): [key: str]: int denied: Missing
...which would allow arbitrary key pairs but not field with `denied` as key. But, this is highly theoretical, I don't think there's use case. So yes, I agree that inverse of "required" is not needed.
Why I am after something else than your earlier proposal of:
class MyThing(TypedDict, total=False): req1: Required[int] opt1: str req2: Required[float]
...is that people _expect_ it to work the other way around. It feels more natural if the stuff you define is expected to exist, and you then mark up what differs from that. That's how ~everything else already works and I would say that's how our brain are already tuned.
opt1: ~str
I think this would read as "inverse of string" (strictly) or "not string" (in a more loose sense), rather than as "perhaps not there"?
A1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote:
Why I am after something else than your earlier proposal of:
class MyThing(TypedDict, total=False): req1: Required[int] opt1: str req2: Required[float]...is that people _expect_ it to work the other way around. It feels
more natural if the stuff you define is expected to exist, and you then mark up what differs from that. That's how ~everything else already works and I would say that's how our brain are already tuned.
Agreed. I would think "required" would be assumed to be the default, and "not-required" would be the special case that needs marking.
B1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote:
opt1: NotRequired[str] # or whatever we call it
I actually like NotRequired.
However I can also see some objections: * It feels a bit odd to bless a term that starts with "not", in negative form rather than some positive form. * A one-word term seems preferable over a two-word term like NotRequired.
B2. On 1/5/21 12:03 AM, Tuukka Mustonen wrote:
Droppable
Mmm. The Rustacean in me thinks this means "deallocatable" which isn't correct here. -1
Potential
Mmm. Feels too vague. -1
Open
Mmm. Feels like you could apply "open" to an entire class/structure, but not to an individual member. -1
B3. I will also contribute a few bikeshed colors:
* OptionalKey * MissingOk
C1. On 1/5/21 12:03 AM, Tuukka Mustonen wrote:
So should we instead:
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value
is string
opt2: Optional[str] # always exists, but may have null valueThat would maybe allow to re-use `Optional` as it would exist in
different context (don't know how confusing that would be). But this probably doesn't fit the design and the parser?
Interesting. I like how this looks, and it doesn't require us to introduce a new (key-only) synonym for Optional.
But as you say it's not clear to me how it would be parsed exactly.
Also, we already have precedent from Final[] and Annotated[] that suggests that properties of a variable (as opposed to properties of the value shape) still be put into the type hint that wraps the other component of the type that contains the value shape.
C2. If we *do* wrap the right-hand-side rather than the left-hand-side, any spelling we have for "the key for this value can be missing" must also support interacting with the existing Optional spelling without looking too strange. For example:
class MyThing(TypedDict): opt: OptionalKey[Optional[str]]
class MyThing(TypedDict): opt: MissingOk[Optional[str]]
class MyThing(TypedDict): opt: NotRequired[Optional[str]]
The last example above in particular looks pretty strange to me.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On Thu, Jan 7, 2021 at 11:35 PM David Foster davidfstr@gmail.com wrote:
A1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote:
Why I am after something else than your earlier proposal of:
class MyThing(TypedDict, total=False): req1: Required[int] opt1: str req2: Required[float]...is that people _expect_ it to work the other way around. It feels
more natural if the stuff you define is expected to exist, and you then mark up what differs from that. That's how ~everything else already works and I would say that's how our brain are already tuned.
Agreed. I would think "required" would be assumed to be the default, and "not-required" would be the special case that needs marking.
But "required" *is* the default -- you have to specify `total=False` to make fields non-required.
B1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote:
opt1: NotRequired[str] # or whatever we call itI actually like NotRequired.
It's pretty ugly (though there's a precedent, `NoReturn` for functions that don't return). It's also long and difficult to type due to the camelcase.
If there were a better word I'd be okay with doing it this way around, but NotRequired just doesn't cut it for me. Hence my proposal.
However I can also see some objections:
- It feels a bit odd to bless a term that starts with "not", in negative
form rather than some positive form.
- A one-word term seems preferable over a two-word term like NotRequired.
What I said. :-)
B2. On 1/5/21 12:03 AM, Tuukka Mustonen wrote:
DroppableMmm. The Rustacean in me thinks this means "deallocatable" which isn't correct here. -1
PotentialMmm. Feels too vague. -1
OpenMmm. Feels like you could apply "open" to an entire class/structure, but not to an individual member. -1
B3. I will also contribute a few bikeshed colors:
- OptionalKey
- MissingOk
C1. On 1/5/21 12:03 AM, Tuukka Mustonen wrote:
So should we instead:
class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, valueis string
opt2: Optional[str] # always exists, but may have null valueThat would maybe allow to re-use `Optional` as it would exist in
different context (don't know how confusing that would be). But this probably doesn't fit the design and the parser?
Interesting. I like how this looks, and it doesn't require us to introduce a new (key-only) synonym for Optional.
But as you say it's not clear to me how it would be parsed exactly.
Also, we already have precedent from Final[] and Annotated[] that suggests that properties of a variable (as opposed to properties of the value shape) still be put into the type hint that wraps the other component of the type that contains the value shape.
C2. If we *do* wrap the right-hand-side rather than the left-hand-side, any spelling we have for "the key for this value can be missing" must also support interacting with the existing Optional spelling without looking too strange. For example:
class MyThing(TypedDict): opt: OptionalKey[Optional[str]]
class MyThing(TypedDict): opt: MissingOk[Optional[str]]
class MyThing(TypedDict): opt: NotRequired[Optional[str]]
The last example above in particular looks pretty strange to me.
Let's just not put funny syntax before the colon.
But "required" is the default -- you have to specify total=False to make fields non-required.
Yeah, required is the default. But the convenience syntax that we're discussing here wouldn't work with that. Having to specify `total=False` and then `Required` for required fields is not intuitive. It would be a good addition and a workaround, but it doesn't fulfill the need here.
It's pretty ugly (though there's a precedent, NoReturn for functions that don't return). It's also long and difficult to type due to the camelcase.
I also think most proposals look ugly. However, I'm not sure if it makes sense to reject a feature just because perfect wording cannot be found? People who can't live with the name could create an alias anyway, like:
O = NotRequired
That's probably what I would do - not to avoid ugliness but simply to make the syntax less verbose.
Sure, I could do the same with `R = Required` but it doesn't fix the problem of having to think about it "the other way around".
Let's just not put funny syntax before the colon.
Is it silly? If we consider typescript-like syntax again:
interface Data { opt?: str opt: str? }
The first one reads as "opt may be there?" while the second one reads as "type may be string?". Doing `NotRequired[str]` reads as "type may be string but doesn't have to be" or perhaps as "type is not required but may be string".
Sorry if I add to bikeshedding, but would it make sense to have `Missing` special value that's used by Union? i.e. you'd write:
class Person(TypeDict): name: str date_birth: date date_death: date|Missing
Even from an internal typechecker standpoint I think it would make formal sense to say that {"name": "Alex"}["not_a_name"] has type Missing, and many rules on unions would work seamlessly and coherently.
On Sat, 9 Jan 2021 at 16:49, Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
But "required" is the default -- you have to specify total=False to make fields non-required.
Yeah, required is the default. But the convenience syntax that we're discussing here wouldn't work with that. Having to specify `total=False` and then `Required` for required fields is not intuitive. It would be a good addition and a workaround, but it doesn't fulfill the need here.
It's pretty ugly (though there's a precedent, NoReturn for functions that don't return). It's also long and difficult to type due to the
camelcase.
I also think most proposals look ugly. However, I'm not sure if it makes sense to reject a feature just because perfect wording cannot be found? People who can't live with the name could create an alias anyway, like:
O = NotRequiredThat's probably what I would do - not to avoid ugliness but simply to make the syntax less verbose.
Sure, I could do the same with `R = Required` but it doesn't fix the problem of having to think about it "the other way around".
Let's just not put funny syntax before the colon.
Is it silly? If we consider typescript-like syntax again:
interface Data { opt?: str opt: str? }The first one reads as "opt may be there?" while the second one reads as "type may be string?". Doing `NotRequired[str]` reads as "type may be string but doesn't have to be" or perhaps as "type is not required but may be string". _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
El lun, 11 ene 2021 a las 10:32, Daniel Moisset (dfmoisset@gmail.com) escribió:
Sorry if I add to bikeshedding, but would it make sense to have `Missing` special value that's used by Union? i.e. you'd write:
class Person(TypeDict): name: str date_birth: date date_death: date|Missing
Even from an internal typechecker standpoint I think it would make formal sense to say that {"name": "Alex"}["not_a_name"] has type Missing, and many rules on unions would work seamlessly and coherently.
Perhaps `typing.NoReturn` could be used for this purpose too.
On Sat, 9 Jan 2021 at 16:49, Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
But "required" is the default -- you have to specify total=False to make fields non-required.
Yeah, required is the default. But the convenience syntax that we're discussing here wouldn't work with that. Having to specify `total=False` and then `Required` for required fields is not intuitive. It would be a good addition and a workaround, but it doesn't fulfill the need here.
It's pretty ugly (though there's a precedent, NoReturn for functions that don't return). It's also long and difficult to type due to the
camelcase.
I also think most proposals look ugly. However, I'm not sure if it makes sense to reject a feature just because perfect wording cannot be found? People who can't live with the name could create an alias anyway, like:
O = NotRequiredThat's probably what I would do - not to avoid ugliness but simply to make the syntax less verbose.
Sure, I could do the same with `R = Required` but it doesn't fix the problem of having to think about it "the other way around".
Let's just not put funny syntax before the colon.
Is it silly? If we consider typescript-like syntax again:
interface Data { opt?: str opt: str? }The first one reads as "opt may be there?" while the second one reads as "type may be string?". Doing `NotRequired[str]` reads as "type may be string but doesn't have to be" or perhaps as "type is not required but may be string". _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Honestly I'm not keen on this type of solution. No matter what we do, unless we introduce new *syntax* (e.g. `age?: int`) we're already making a statement about the *variable* in the form of a statement about the *type*.
We have precedents for that, but they are things like `Final[...]` and `ClassVar[...]`, and I think it would be better to continue along that line. We can make a simple rule that "Required" (for example) may only appear at the top level in a typed dict -- this seems simpler than a rule that says "one branch of a union can be a magic cookie, but only at the top level".
In terms of perceived economy, regardless of whether we reuse an existing keyword or add a new one, the behavior for this particular case has to be learned and implemented separately, so I don't think we're saving much by avoiding a new keyword. (If we could reuse it would be a little different, but we clearly can't.)
On Mon, Jan 11, 2021 at 10:40 AM Jelle Zijlstra jelle.zijlstra@gmail.com wrote:
El lun, 11 ene 2021 a las 10:32, Daniel Moisset (dfmoisset@gmail.com) escribió:
Sorry if I add to bikeshedding, but would it make sense to have `Missing` special value that's used by Union? i.e. you'd write:
class Person(TypeDict): name: str date_birth: date date_death: date|Missing
Even from an internal typechecker standpoint I think it would make formal sense to say that {"name": "Alex"}["not_a_name"] has type Missing, and many rules on unions would work seamlessly and coherently.
Perhaps `typing.NoReturn` could be used for this purpose too.
On Sat, 9 Jan 2021 at 16:49, Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
But "required" is the default -- you have to specify total=False to make fields non-required.
Yeah, required is the default. But the convenience syntax that we're discussing here wouldn't work with that. Having to specify `total=False` and then `Required` for required fields is not intuitive. It would be a good addition and a workaround, but it doesn't fulfill the need here.
It's pretty ugly (though there's a precedent, NoReturn for functions that don't return). It's also long and difficult to type due to the
camelcase.
I also think most proposals look ugly. However, I'm not sure if it makes sense to reject a feature just because perfect wording cannot be found? People who can't live with the name could create an alias anyway, like:
O = NotRequiredThat's probably what I would do - not to avoid ugliness but simply to make the syntax less verbose.
Sure, I could do the same with `R = Required` but it doesn't fix the problem of having to think about it "the other way around".
Let's just not put funny syntax before the colon.
Is it silly? If we consider typescript-like syntax again:
interface Data { opt?: str opt: str? }The first one reads as "opt may be there?" while the second one reads as "type may be string?". Doing `NotRequired[str]` reads as "type may be string but doesn't have to be" or perhaps as "type is not required but may be string". _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
Catching up...
On 1/8/21 11:51 AM, Guido van Rossum wrote:> On Thu, Jan 7, 2021 at 11:35 PM David Foster <davidfstr@gmail.com
mailto:davidfstr@gmail.com> wrote:
A1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote: > Why I am after something else than your earlier proposal of: > > class MyThing(TypedDict, total=False): > req1: Required[int] > opt1: str > req2: Required[float] > > ...is that people _expect_ it to work the other way around.
It feels
> more natural if the stuff you define is expected to exist,
and you then
> mark up what differs from that. That's how ~everything else
already
> works and I would say that's how our brain are already tuned. Agreed. I would think "required" would be assumed to be the default, and "not-required" would be the special case that needs marking.But "required" *is* the default -- you have to specify `total=False` to make fields non-required.
I think I've finally wrapped my head around this comment. In long-form:
* If you have a TypedDict with total=True (the default), then all keys are required. * If you have a TypedDict with total=False, then all keys become not-required by default. * Therefore introducing a Required[...] form is a backward-compatible way of marking certain keys of a total=False TypedDict as *required*, which currently isn't possible to do.
It bends my brain a bit: If I want to create an entirely new TypedDict with some keys required and some keys not-required then I must: (1) declare a total=False TypedDict and (2) explicitly mark the keys I want required with Required[...] and (3) remember that in this context all unmarked keys are NOT-required.
(This is backwards from TypeScript which essentially always declares TypedDicts as total=True but then only marks certain keys as NOT-required using the "name?: type" syntax.)
I suppose total=False TypedDicts bend my brain a bit anyway: It strikes me as odd that a single keyword at the top can flip the default required-ness of every key in the dictionary. However the total=False syntax is here now and I don't think it's going away, so we might as well work with it.
Long story short: I think Required[...] on a total=False TypedDict would be reasonable, given that the total=False syntax already exists. +0.5
Also, Required[...] satisfies: ✅ Single word. Easy to type. ✅ Is in positive form, rather than a "not X" or "no X" negative form. ✅ Does not "put funny syntax before the colon"
If we want the marker to go on the not-required keys, a possible singe-word positive bikeshed is Excludable
class MyThing(TypedDict, total=True): req1: int opt1: Excludable[str] req2: float
On Sat, 16 Jan 2021 at 19:48, David Foster davidfstr@gmail.com wrote:
Catching up...
On 1/8/21 11:51 AM, Guido van Rossum wrote:> On Thu, Jan 7, 2021 at 11:35 PM David Foster <davidfstr@gmail.com
mailto:davidfstr@gmail.com> wrote:
A1. On 1/6/21 3:54 AM, Tuukka Mustonen wrote: > Why I am after something else than your earlier proposal of: > > class MyThing(TypedDict, total=False): > req1: Required[int] > opt1: str > req2: Required[float] > > ...is that people _expect_ it to work the other way around.It feels
> more natural if the stuff you define is expected to exist,and you then
> mark up what differs from that. That's how ~everything elsealready
> works and I would say that's how our brain are already tuned. Agreed. I would think "required" would be assumed to be the default, and "not-required" would be the special case that needs marking.But "required" *is* the default -- you have to specify `total=False` to make fields non-required.
I think I've finally wrapped my head around this comment. In long-form:
- If you have a TypedDict with total=True (the default), then all keys
are required.
- If you have a TypedDict with total=False, then all keys become
not-required by default.
- Therefore introducing a Required[...] form is a backward-compatible
way of marking certain keys of a total=False TypedDict as *required*, which currently isn't possible to do.
It bends my brain a bit: If I want to create an entirely new TypedDict with some keys required and some keys not-required then I must: (1) declare a total=False TypedDict and (2) explicitly mark the keys I want required with Required[...] and (3) remember that in this context all unmarked keys are NOT-required.
(This is backwards from TypeScript which essentially always declares TypedDicts as total=True but then only marks certain keys as NOT-required using the "name?: type" syntax.)
I suppose total=False TypedDicts bend my brain a bit anyway: It strikes me as odd that a single keyword at the top can flip the default required-ness of every key in the dictionary. However the total=False syntax is here now and I don't think it's going away, so we might as well work with it.
Long story short: I think Required[...] on a total=False TypedDict would be reasonable, given that the total=False syntax already exists. +0.5
Also, Required[...] satisfies: ✅ Single word. Easy to type. ✅ Is in positive form, rather than a "not X" or "no X" negative form. ✅ Does not "put funny syntax before the colon"
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
On 1/16/21 7:48 PM, David Foster wrote:
Long story short: I think Required[...] on a total=False TypedDict would be reasonable, given that the total=False syntax already exists. +0.5
Also, Required[...] satisfies: ✅ Single word. Easy to type. ✅ Is in positive form, rather than a "not X" or "no X" negative form. ✅ Does not "put funny syntax before the colon"
Let me play devil's advocate against myself for a moment here...
Let's say we use the form Required[...] only on total=False TypedDict definitions. You could encounter code like:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[Optional[int]] baz: Optional[float] qed: bool
Now to someone not intimately familiar with the differences between Required[...] and Optional[...], what kinds of questions or confusion could arise here?
* "It looks like `foo` and `bar` are required keys, and `baz` is an optional key, but what is the `qed` key at the end? Is it required? Is it optional?"
* "But wait a minute! The `bar` key is marked as both required AND as optional! What's going on here?"
A big problem you can see here is that it's very easy to conflate the concept of an "optional key" (no explicit spelling) with the concept of "nullable" (currently spelled Optional[...]).
It seems to me that *any* great spelling for "optional key" or "required key" (like Required[...]) is going to be significantly confusing so long as the current Optional[...] spelling (1) exists and (2) means "nullable" instead of "optional key".
So let me open a new can of worms: What if we deprecated the spelling Optional[...], replacing it with something like Nullable[...] instead?
My expectation is that PEP 645 ("Allow writing optional types as x?") will eventually phase out in-practice use of Optional[...] anyway, so deprecating the long-form spelling Optional[...] might not be so disruptive.
Now, if Optional[...] is deprecated it is less likely to show up in new user code, so my original code example would become:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int?] baz: float? qed: bool
Here I think the user confusion in the original example goes away. "Required[...]" and "TYPE?" don't even *look* like the same kind of thing and therefore I expect would be hard to (mistakenly) see as opposites of each other.
(In case folks are wondering, after Optional[...] is deprecated and maybe even removed, I'm NOT presently advocating for eventually bringing it back with the new meaning of "optional key" rather than its current meaning of "nullable". There's probably a lot of content on the internet that already assigns the old meaning to Optional[...], so we can't change it's interpretation without potentially confusing a lot of people who are reading old help resources.)
I don't think it's feasible to deprecate `Optional[x]`. It's too ubiquitous. However, I predict that it its use will fade over time in favor of `x | None` as specified in [PEP 604](https://www.python.org/dev/peps/pep-0604/).
One thing to note I suppose is that `name?` does not mean `T | null` in TS; it means something akin to `T | undefined`. `Required[T?]` might appear contradictory to someone coming from TS and I would personally prefer that the community converges on `T | None` - it's short and clear enough.
Best
D
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, January 18, 2021 6:30 PM, David Foster davidfstr@gmail.com wrote:
On 1/16/21 7:48 PM, David Foster wrote:
Long story short: I think Required[...] on a total=False TypedDict would be reasonable, given that the total=False syntax already exists. +0.5 Also, Required[...] satisfies: ✅ Single word. Easy to type. ✅ Is in positive form, rather than a "not X" or "no X" negative form. ✅ Does not "put funny syntax before the colon"
Let me play devil's advocate against myself for a moment here...
Let's say we use the form Required[...] only on total=False TypedDict definitions. You could encounter code like:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[Optional[int]] baz: Optional[float] qed: bool
Now to someone not intimately familiar with the differences between Required[...] and Optional[...], what kinds of questions or confusion could arise here?
"It looks like `foo` and `bar` are required keys, and `baz` is an optional key, but what is the `qed` key at the end? Is it required? Is it optional?"
"But wait a minute! The `bar` key is marked as both required AND as optional! What's going on here?"
A big problem you can see here is that it's very easy to conflate the concept of an "optional key" (no explicit spelling) with the concept of "nullable" (currently spelled Optional[...]).
It seems to me that any great spelling for "optional key" or "required key" (like Required[...]) is going to be significantly confusing so long as the current Optional[...] spelling (1) exists and (2) means "nullable" instead of "optional key".
So let me open a new can of worms: What if we deprecated the spelling Optional[...], replacing it with something like Nullable[...] instead?
My expectation is that PEP 645 ("Allow writing optional types as x?") will eventually phase out in-practice use of Optional[...] anyway, so deprecating the long-form spelling Optional[...] might not be so disruptive.
Now, if Optional[...] is deprecated it is less likely to show up in new user code, so my original code example would become:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int?] baz: float? qed: bool
Here I think the user confusion in the original example goes away. "Required[...]" and "TYPE?" don't even look like the same kind of thing and therefore I expect would be hard to (mistakenly) see as opposites of each other.
(In case folks are wondering, after Optional[...] is deprecated and maybe even removed, I'm NOT presently advocating for eventually bringing it back with the new meaning of "optional key" rather than its current meaning of "nullable". There's probably a lot of content on the internet that already assigns the old meaning to Optional[...], so we can't change it's interpretation without potentially confusing a lot of people who are reading old help resources.)
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: layday@protonmail.com
I think one of the considerations that hasn't been brought up here is accessibility for people who only occasionally write Python or people who have to switch forth and back between languages (in our case between Hack/Flow and Python). From that perspective there's a strong case to be made for working towards syntax changes that other languages seem to agree on rather than introducing yet more clunky overhead.
It also makes the argument in favor of `T | None` for optionals more difficult to make. Sure, it's short. But it's also difficult to explain to anyone outside of our community why we're using the value `None` to stand for the type of the value. It's impossible for people new to the language to reason about this from first principles.
On Mon, Jan 18, 2021 at 10:35 AM layday via Typing-sig typing-sig@python.org wrote:
One thing to note I suppose is that `name?` does not mean `T | null` in TS; it means something akin to `T | undefined`. `Required[T?]` might appear contradictory to someone coming from TS and I would personally prefer that the community converges on `T | None` - it's short and clear enough.
Best
D
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, January 18, 2021 6:30 PM, David Foster davidfstr@gmail.com wrote:
On 1/16/21 7:48 PM, David Foster wrote:
Long story short: I think Required[...] on a total=False TypedDict would be reasonable, given that the total=False syntax already exists. +0.5 Also, Required[...] satisfies: ✅ Single word. Easy to type. ✅ Is in positive form, rather than a "not X" or "no X" negative form. ✅ Does not "put funny syntax before the colon"
Let me play devil's advocate against myself for a moment here...
Let's say we use the form Required[...] only on total=False TypedDict definitions. You could encounter code like:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[Optional[int]] baz: Optional[float] qed: bool
Now to someone not intimately familiar with the differences between Required[...] and Optional[...], what kinds of questions or confusion could arise here?
"It looks like `foo` and `bar` are required keys, and `baz` is an optional key, but what is the `qed` key at the end? Is it required? Is it optional?"
"But wait a minute! The `bar` key is marked as both required AND as optional! What's going on here?"
A big problem you can see here is that it's very easy to conflate the concept of an "optional key" (no explicit spelling) with the concept of "nullable" (currently spelled Optional[...]).
It seems to me that any great spelling for "optional key" or "required key" (like Required[...]) is going to be significantly confusing so long as the current Optional[...] spelling (1) exists and (2) means "nullable" instead of "optional key".
So let me open a new can of worms: What if we deprecated the spelling Optional[...], replacing it with something like Nullable[...] instead?
My expectation is that PEP 645 ("Allow writing optional types as x?") will eventually phase out in-practice use of Optional[...] anyway, so deprecating the long-form spelling Optional[...] might not be so disruptive.
Now, if Optional[...] is deprecated it is less likely to show up in new user code, so my original code example would become:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int?] baz: float? qed: bool
Here I think the user confusion in the original example goes away. "Required[...]" and "TYPE?" don't even look like the same kind of thing and therefore I expect would be hard to (mistakenly) see as opposites of each other.
(In case folks are wondering, after Optional[...] is deprecated and maybe even removed, I'm NOT presently advocating for eventually bringing it back with the new meaning of "optional key" rather than its current meaning of "nullable". There's probably a lot of content on the internet that already assigns the old meaning to Optional[...], so we can't change it's interpretation without potentially confusing a lot of people who are reading old help resources.)
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: layday@protonmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dkgispam@gmail.com
On Mon, Jan 18, 2021 at 12:10 PM Dominik Gabi dmnkgb@gmail.com wrote:
I think one of the considerations that hasn't been brought up here is accessibility for people who only occasionally write Python or people who have to switch forth and back between languages (in our case between Hack/Flow and Python). From that perspective there's a strong case to be made for working towards syntax changes that other languages seem to agree on rather than introducing yet more clunky overhead.
It also makes the argument in favor of `T | None` for optionals more difficult to make. Sure, it's short. But it's also difficult to explain to anyone outside of our community why we're using the value `None` to stand for the type of the value. It's impossible for people new to the language to reason about this from first principles.
Using None is ubiquitous in typed Python code (e.g. any function that doesn't return anything has `-> None`) so this is just something people will have to get used to. I am strongly in favor of evolving usage to T | None instead of Optional[T] since it leverages two things commonly used in other ways (None and unions) without introducing any new syntax or semantics, and it avoids the confusion between different meanings of "optional" in English (and in Python, whether you're talking about function arguments or dict keys).
The syntax x? is quite controversial and unlikely to be accepted anytime soon (and requires parser changes so invites much more scrutiny). In TypeScript it has a special meaning, whereas T | null and T | undefined in TypeScript appears to mean similar things to T | None in Python (except for the endless shades of difference in meaning between null and undefined, and the fact that TypeScript, like JavaScript, always lets you leave out function parameters and keys, defaulting to undefined unless you look real careful).
On 1/18/21 9:19 AM, Eric Traut wrote:
I don't think it's feasible to deprecate `Optional[x]`. It's too
ubiquitous. However, I predict that it its use will fade over time in favor of `x | None` as specified in [PEP 604](https://www.python.org/dev/peps/pep-0604/).
On 1/18/21 12:55 PM, Guido van Rossum wrote:
The syntax x? is quite controversial and unlikely to be accepted anytime soon (and requires parser changes so invites much more scrutiny).
Points taken. I'll restate my last argument using this new information:
Let's say we use the form Required[...] only on total=False TypedDict definitions. You could encounter code like:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: bool
This code seems fairly straightforward to me. So long as we discourage in documentation the use of Optional[...] in the presence of Required[...] -- perhaps explicitly in favor of the `T|None` form -- it looks to me like Required[...] would be readable even in a more complex (real-world) case like this definition shows.
Any major objections to the Required[...] syntax (in combination with total=False) being the solution to defining the optionality/requiredness of individual fields on TypedDicts? I'm feeling like the discussion and decisions so far are ripe to make into an actual PEP.
I'm all for proceeding to the PEP stage.
Maybe you can even consider a PR for mypy?
On Sun, Jan 24, 2021 at 8:15 PM David Foster davidfstr@gmail.com wrote:
On 1/18/21 9:19 AM, Eric Traut wrote:
I don't think it's feasible to deprecate `Optional[x]`. It's too
ubiquitous. However, I predict that it its use will fade over time in favor of `x | None` as specified in [PEP 604](https://www.python.org/dev/peps/pep-0604/).
On 1/18/21 12:55 PM, Guido van Rossum wrote:
The syntax x? is quite controversial and unlikely to be accepted anytime soon (and requires parser changes so invites much more scrutiny).
Points taken. I'll restate my last argument using this new information:
Let's say we use the form Required[...] only on total=False TypedDict definitions. You could encounter code like:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: boolThis code seems fairly straightforward to me. So long as we discourage in documentation the use of Optional[...] in the presence of Required[...] -- perhaps explicitly in favor of the `T|None` form -- it looks to me like Required[...] would be readable even in a more complex (real-world) case like this definition shows.
Any major objections to the Required[...] syntax (in combination with total=False) being the solution to defining the optionality/requiredness of individual fields on TypedDicts? I'm feeling like the discussion and decisions so far are ripe to make into an actual PEP.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 1/24/21 8:58 PM, Guido van Rossum wrote:
I'm all for proceeding to the PEP stage.
Cool. I seem to be in a design mood this month, so I'd be happy to shepherd the drafting of a PEP for this feature.
Maybe you can even consider a PR for mypy?
Perhaps: we'll see if I have the bandwidth. I believe the extensions to mypy to support Required[...] wouldn't be that difficult, but I'm hoping to shepherd my work on TypeForm (both PEP and implementation) through first, and that I expect will be much more involved to implement.
Just want to note that while this would be a good addition:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: bool
...it's still "the wrong way around". This is what people usually want:
class MyTypedDict(TypedDict): foo: str bar: int|None baz: float|None|Missing qed: bool|Missing
Were there actually any comments on Daniel Moisset's suggestion of `| Missing`? Guido replied but maybe I don't understand the difference - if `Required[str]`, which defines the existence of the field (left side) attached to the type (right side), is considered acceptable approach, what's bad with `x|Missing`, which simply does the same?
On 1/25/21 12:09 AM, Tuukka Mustonen wrote:
Just want to note that while this would be a good addition:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: bool...it's still "the wrong way around". This is what people usually want:
class MyTypedDict(TypedDict): foo: str bar: int|None baz: float|None|Missing qed: bool|Missing
I agree that it would be preferable to introduce a spelling for "optional key" rather than "required key", but there just doesn't seem to be a way to get a reasonable spelling:
* We can't easily introduce any new single-word wrapper like Missing[...] because any good name that means "optional key" will necessarily be too close to Optional[...] and will be confusing when used in combination with Optional[...]. For example Optional[Missing[...]] is confusing.
* So that leaves us with spelling the *opposite* of "optional key", which is a "required key". There's no name collision with any existing typing construct by just using Required[...], which is the obvious name for a "required key" wrapper.
Were there actually any comments on Daniel Moisset's suggestion of `| Missing`? Guido replied but maybe I don't understand the difference - if `Required[str]`, which defines the existence of the field (left side) attached to the type (right side), is considered acceptable approach, what's bad with `x|Missing`, which simply does the same?
The syntax `x|Missing` implies the introduction of a new constant (`Missing`) that's like `None`. Using such a constant as a magic flag is unusual. We typically use wrappers like Final[...], Annotated[...], etc instead.
Thanks for the clarification. So it's not that something couldn't be done, it's rather that too dramatic measures would need to be taken.
One more alternative that I suggested earlier, to which I failed to see any comments, was this:
class MyThing(TypedDict, optional_types_may_be_missing=True): req1: int opt1: Optional[str]
(Obviously, with some shorter name for `optional_types_may_be_missing` argument.)
It would make no distinction between "null" and "missing", but that's what one normally wants anyway. It wouldn't work to define a PATCH HTTP API endpoint, where sending out "null" (clear the value in DB) has different meaning from not sending the field at all (no change), but it would be sufficient for most cases. It's not perfect, but perhaps sufficient (like introducing the Required key).
On Wed, Jan 27, 2021 at 2:31 AM Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
Thanks for the clarification. So it's not that something couldn't be done, it's rather that too dramatic measures would need to be taken.
One more alternative that I suggested earlier, to which I failed to see any comments, was this:
class MyThing(TypedDict, optional_types_may_be_missing=True): req1: int opt1: Optional[str](Obviously, with some shorter name for `optional_types_may_be_missing` argument.)
It would make no distinction between "null" and "missing", but that's what one normally wants anyway. It wouldn't work to define a PATCH HTTP API endpoint, where sending out "null" (clear the value in DB) has different meaning from not sending the field at all (no change), but it would be sufficient for most cases. It's not perfect, but perhaps sufficient (like introducing the Required key).
This would add more confusion for users because it would mean that in *some* contexts the meaning of Optional[] is different than in other contexts.
Can you please just get behind the Required[] proposal?
I think that something in the proposal I sent was missed. The key of the proposal wasn't as much of "add a special value" (which it does), but mostly "use a Union[]" (I wrote it with the new | syntax, but I think this should be the same with explicit Union[Foo, Missing].
I stated this because the behaviour of "holes" in dictionaries already has Union semantics. Not just in the abstract sense of "there is more than one possible behaviour when evaluating mydict['mykey'] ", but in the general properties of Unions. For example consider this TypeDict declarations:
class A(TypedDict): x: int y: int|str z: int|Missing
class B(TypedDict): x: str z: int
value: A|B
Currently the type of value["x"] is int|str (the union of the types for x in both types). What happens for "holes" then?
value["z"] has type int|Missing. That is the Union[int|Missing, int] which again are the types on both sides (note that we use here that Unions can be flattened and int|int == int). value["y"] is a bit more trickier, but we can consider that there's an implicit "y: Missing" in B (for every name except "x" and "z" ) so the type of value["y"] is int|str|Missing. value["doesnotexist"] has type Missing|Missing, which due to union rules, is equivalent to Missing.
Adding some special form like "Required[int|str]" will need to define what does it mean to have Union[Required[T1], T2] and similar interactions, and in the end you will end up recreating the definition and algebraic rules for a Union, because what we have conceptually here is a Union. I know some people have expressed their concern that these standalone values are unusual (but not unheard of, we already have NoReturn), but for me this emerges from the type system.
In fact, I think we might get away defining Missing as an alias for NoReturn, because right now I see no much of a semantic difference . For example, any code after value["foo"] should be ignored in the same way as code after sys.exit() (a function with result type NoReturn). Anyway, this is an extra idea and if I am wrong this paragraph can be ignored without affecting the previous argument
Hope the example drives the idea more clearly
Best, D.
On Mon, 25 Jan 2021 at 08:10, Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
Just want to note that while this would be a good addition:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: bool...it's still "the wrong way around". This is what people usually want:
class MyTypedDict(TypedDict): foo: str bar: int|None baz: float|None|Missing qed: bool|MissingWere there actually any comments on Daniel Moisset's suggestion of `| Missing`? Guido replied but maybe I don't understand the difference - if `Required[str]`, which defines the existence of the field (left side) attached to the type (right side), is considered acceptable approach, what's bad with `x|Missing`, which simply does the same? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
Just as an extra paragraph, note that these semantics make it about the values and not the variables as Guido said earlier
On Wed, 27 Jan 2021 at 19:45, Daniel Moisset dfmoisset@gmail.com wrote:
I think that something in the proposal I sent was missed. The key of the proposal wasn't as much of "add a special value" (which it does), but mostly "use a Union[]" (I wrote it with the new | syntax, but I think this should be the same with explicit Union[Foo, Missing].
I stated this because the behaviour of "holes" in dictionaries already has Union semantics. Not just in the abstract sense of "there is more than one possible behaviour when evaluating mydict['mykey'] ", but in the general properties of Unions. For example consider this TypeDict declarations:
class A(TypedDict): x: int y: int|str z: int|Missing
class B(TypedDict): x: str z: int
value: A|B
Currently the type of value["x"] is int|str (the union of the types for x in both types). What happens for "holes" then?
value["z"] has type int|Missing. That is the Union[int|Missing, int] which again are the types on both sides (note that we use here that Unions can be flattened and int|int == int). value["y"] is a bit more trickier, but we can consider that there's an implicit "y: Missing" in B (for every name except "x" and "z" ) so the type of value["y"] is int|str|Missing. value["doesnotexist"] has type Missing|Missing, which due to union rules, is equivalent to Missing.
Adding some special form like "Required[int|str]" will need to define what does it mean to have Union[Required[T1], T2] and similar interactions, and in the end you will end up recreating the definition and algebraic rules for a Union, because what we have conceptually here is a Union. I know some people have expressed their concern that these standalone values are unusual (but not unheard of, we already have NoReturn), but for me this emerges from the type system.
In fact, I think we might get away defining Missing as an alias for NoReturn, because right now I see no much of a semantic difference . For example, any code after value["foo"] should be ignored in the same way as code after sys.exit() (a function with result type NoReturn). Anyway, this is an extra idea and if I am wrong this paragraph can be ignored without affecting the previous argument
Hope the example drives the idea more clearly
Best, D.
On Mon, 25 Jan 2021 at 08:10, Tuukka Mustonen tuukka.mustonen@gmail.com wrote:
Just want to note that while this would be a good addition:
class MyTypedDict(TypedDict, total=False): foo: Required[str] bar: Required[int|None] baz: float|None qed: bool...it's still "the wrong way around". This is what people usually want:
class MyTypedDict(TypedDict): foo: str bar: int|None baz: float|None|Missing qed: bool|MissingWere there actually any comments on Daniel Moisset's suggestion of `| Missing`? Guido replied but maybe I don't understand the difference - if `Required[str]`, which defines the existence of the field (left side) attached to the type (right side), is considered acceptable approach, what's bad with `x|Missing`, which simply does the same? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
On Wed, Jan 27, 2021 at 11:45 AM Daniel Moisset dfmoisset@gmail.com wrote:
I think that something in the proposal I sent was missed. The key of the proposal wasn't as much of "add a special value" (which it does), but mostly "use a Union[]" (I wrote it with the new | syntax, but I think this should be the same with explicit Union[Foo, Missing].
I stated this because the behaviour of "holes" in dictionaries already has Union semantics. Not just in the abstract sense of "there is more than one possible behaviour when evaluating mydict['mykey'] ", but in the general properties of Unions. For example consider this TypeDict declarations:
class A(TypedDict): x: int y: int|str z: int|Missing
class B(TypedDict): x: str z: int
value: A|B
Currently the type of value["x"] is int|str (the union of the types for x in both types). What happens for "holes" then?
value["z"] has type int|Missing. That is the Union[int|Missing, int] which again are the types on both sides (note that we use here that Unions can be flattened and int|int == int). value["y"] is a bit more trickier, but we can consider that there's an implicit "y: Missing" in B (for every name except "x" and "z" ) so the type of value["y"] is int|str|Missing. value["doesnotexist"] has type Missing|Missing, which due to union rules, is equivalent to Missing.
Adding some special form like "Required[int|str]" will need to define what does it mean to have Union[Required[T1], T2] and similar interactions, and in the end you will end up recreating the definition and algebraic rules for a Union, because what we have conceptually here is a Union.
But Required does not add anything new. TypedDict already supports having some mandatory arguments and some that are not mandatory (by combining classes with total=False/True through inheritance).
Adding a special type (`Missing`) to a union to indicate that a field is not required would be inconsistent with every other use of `Union`. It breaks the mental model users have for `Union`, and it would require type checkers to introduce a bunch of special-case logic in multiple places where unions are used. Please don't do this.
The proposed `Required` annotation is consistent with existing annotations like `Final`, so it will be familiar to users and will work nicely with existing type checker implementations.
I'll also note that `NoReturn` within a union makes no sense. The `NoReturn` type is semantically equivalent to an empty union. It should never appear within a union along with other types. Pyright contains special logic to remove it from a union if it is combined with other types.
-Eric
-- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
Ok, thanks for the detailed explanation. I stand corrected
On Thu, 28 Jan 2021, 03:44 Eric Traut, eric@traut.com wrote:
Adding a special type (`Missing`) to a union to indicate that a field is not required would be inconsistent with every other use of `Union`. It breaks the mental model users have for `Union`, and it would require type checkers to introduce a bunch of special-case logic in multiple places where unions are used. Please don't do this.
The proposed `Required` annotation is consistent with existing annotations like `Final`, so it will be familiar to users and will work nicely with existing type checker implementations.
I'll also note that `NoReturn` within a union makes no sense. The `NoReturn` type is semantically equivalent to an empty union. It should never appear within a union along with other types. Pyright contains special logic to remove it from a union if it is combined with other types.
-Eric
-- Eric Traut Contributor to Pyright and Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
I think this is ready to be PEPped. Who wants to be the author? I can be sponsor.
Or perhaps you want to start with a prototype implementation? Some additions to typeshed, typing_extensions and typing are also needed, I can help with some pointers.
On Fri, Jan 29, 2021 at 04:05 Daniel Moisset dfmoisset@gmail.com wrote:
Ok, thanks for the detailed explanation. I stand corrected
On Thu, 28 Jan 2021, 03:44 Eric Traut, eric@traut.com wrote:
Adding a special type (`Missing`) to a union to indicate that a field is not required would be inconsistent with every other use of `Union`. It breaks the mental model users have for `Union`, and it would require type checkers to introduce a bunch of special-case logic in multiple places where unions are used. Please don't do this.
The proposed `Required` annotation is consistent with existing annotations like `Final`, so it will be familiar to users and will work nicely with existing type checker implementations.
I'll also note that `NoReturn` within a union makes no sense. The `NoReturn` type is semantically equivalent to an empty union. It should never appear within a union along with other types. Pyright contains special logic to remove it from a union if it is combined with other types.
-Eric
-- Eric Traut Contributor to Pyright and Pylance Microsoft Corp. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: dfmoisset@gmail.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
On 1/29/21 7:32 AM, Guido van Rossum wrote:
I think this is ready to be PEPped. Who wants to be the author? I can be sponsor.
I'm game to draft this PEP. I feel like I've internalized the conversation fairly well. I'll start on that tonight.
I expect this feature will be a lot easier to implement than the TypeForm syntax I'm designing in a different thread. ;)
+1
On Sat, Jan 30, 2021 at 9:33 PM David Foster davidfstr@gmail.com wrote:
On 1/29/21 7:32 AM, Guido van Rossum wrote:
I think this is ready to be PEPped. Who wants to be the author? I can be sponsor.
I'm game to draft this PEP. I feel like I've internalized the conversation fairly well. I'll start on that tonight.
I expect this feature will be a lot easier to implement than the TypeForm syntax I'm designing in a different thread. ;)
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
Hi. Sorry if this is a little bit of a digression, but can I get the chance of a new PEP to ask for a more "strict" check of the optional/required keys of a TypedDict? Looking at the PEP589 I see:
Type checkers may allow reading an item using d['x'] even if the key 'x' is not required, instead of requiring the use of d.get('x') or an explicit 'x' in d check. The rationale is that tracking the existence of keys is difficult to implement in full generality, and that disallowing this could require many changes to existing code.
In fact, I see no error actually raised by current mypy on this snipped of code:
from typing import TypedDict
class _ARequired(TypedDict): required: int
class A(_ARequired, total=False): optional: int
A(required=123).get('required', 456) # no way the fallback value is used A(required=123)['optional'] # KeyError
I'm adding TypedDict in my project as a way to "describe" the response of some external API I consume, so I would really appreciate if I could be "warned" against "dead code" or, worse, risky one.
I'm not sure about the meaning of the rationale I quoted - the problem is in the user codebase checked? Or are internal difficulties of the type checker? In the first case - at least for me - I think that having stricter checks could help showing latent bugs.
FWIW I also think that "total=False" + "Required[...]" is the best syntax, given the backcompatibility with the actual semantic of TypedDict
The rationale is what the quoted paragraph says. Type checkers don’t track the *value* of expressions, so they don’t have the information needed for the check you desire. Some type checkers may go beyond this and keep track of values, but this is usually not as easy as it would seem from your example, and most code won’t benefit that much, so I don’t blame type checkers that don’t do this.
—Guido
On Fri, Feb 5, 2021 at 08:03 Vito De Tullio vito.detullio@gmail.com wrote:
Hi. Sorry if this is a little bit of a digression, but can I get the chance of a new PEP to ask for a more "strict" check of the optional/required keys of a TypedDict? Looking at the PEP589 I see:
Type checkers may allow reading an item using d['x'] even if the key
'x' is not required, instead of requiring the use of d.get('x') or an explicit 'x' in d check. The rationale is that tracking the existence of keys is difficult to implement in full generality, and that disallowing this could require many changes to existing code.
In fact, I see no error actually raised by current mypy on this snipped of code:
from typing import TypedDict class _ARequired(TypedDict): required: int class A(_ARequired, total=False): optional: int A(required=123).get('required', 456) # no way the fallback value isused A(required=123)['optional'] # KeyError
I'm adding TypedDict in my project as a way to "describe" the response of some external API I consume, so I would really appreciate if I could be "warned" against "dead code" or, worse, risky one.
I'm not sure about the meaning of the rationale I quoted - the problem is in the user codebase checked? Or are internal difficulties of the type checker? In the first case - at least for me - I think that having stricter checks could help showing latent bugs.
FWIW I also think that "total=False" + "Required[...]" is the best syntax, given the backcompatibility with the actual semantic of TypedDict _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
Hi.
Sorry for the misunderstanding. I understand that keep track of the value of each instance can be cumbersone (and it's not exactly in the scope of a *type* checker). Let me focus on the "type definition" of a TypedDict. Currently with TypedDict you can define all the 'required' or, with `total=False`, 'possible' keys a dict can have.
I can understand how can be difficult to deal with current "non total" TypedDict, and how the simple flag takes to a "too coarse" definition, exacerbated in "non total" TypedDict with a lot of keys that can take to a lot of false positives.
Still, now we are talking about giving a more "precise" definition of what keys are required, what are possible, and what are simply not present. How are the implementors suppose to use this additional information?
What I mean is: let's say that now we use the `Required` annotation:
from typing import TypedDict, Required
class C(TypedDict, total=False): required: Required[int] optional: int
def h(c: C, fallback: int) -> None: getitem_required = c['required'] getitem_optional = c['optional'] getitem_nonexistent = c['nonexistent']
get_required = c.get('required, fallback) get_optional = c.get('optional', fallback) get_existent = c.get('nonexistent', fallback)
I think it's a "waste" to not use the additional explicit annotation and mark as error, in addition to the `getitem_nonexistent` and `get_existent` examples, also the `getitem_optional` and the `get_required` rows.
Both the cases are code smell: in the first case is an hidden bug; the second one is basically dead code.
You are tying two unrelated issues together. You're welcome to file bugs in the mypy tracker (or in the tracker of whatever checker you use) to ask for such checking as a feature. However, that feature does not require a PEP and this PEP should not be involved in requesting the feature.
On Sun, Feb 7, 2021 at 8:37 AM Vito De Tullio vito.detullio@gmail.com wrote:
Hi.
Sorry for the misunderstanding. I understand that keep track of the value of each instance can be cumbersone (and it's not exactly in the scope of a *type* checker). Let me focus on the "type definition" of a TypedDict. Currently with TypedDict you can define all the 'required' or, with `total=False`, 'possible' keys a dict can have.
I can understand how can be difficult to deal with current "non total" TypedDict, and how the simple flag takes to a "too coarse" definition, exacerbated in "non total" TypedDict with a lot of keys that can take to a lot of false positives.
Still, now we are talking about giving a more "precise" definition of what keys are required, what are possible, and what are simply not present. How are the implementors suppose to use this additional information?
What I mean is: let's say that now we use the `Required` annotation:
from typing import TypedDict, Required class C(TypedDict, total=False): required: Required[int] optional: int def h(c: C, fallback: int) -> None: getitem_required = c['required'] getitem_optional = c['optional'] getitem_nonexistent = c['nonexistent'] get_required = c.get('required, fallback) get_optional = c.get('optional', fallback) get_existent = c.get('nonexistent', fallback)I think it's a "waste" to not use the additional explicit annotation and mark as error, in addition to the `getitem_nonexistent` and `get_existent` examples, also the `getitem_optional` and the `get_required` rows.
Both the cases are code smell: in the first case is an hidden bug; the second one is basically dead code. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
participants (10)
-
Daniel Moisset -
David Foster -
Dominik Gabi -
Eric Traut -
Guido van Rossum -
Jelle Zijlstra -
layday -
Shantanu Jain -
Tuukka Mustonen -
Vito De Tullio