Remove typing from the stdlib
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
I'm strongly in favor on such move.
My experience with asyncio in the stdlib is that users expect changes faster than the very slow release process of the stdlib (a release every 18 months in average).
I saw many PEPs and discussion on the typing design (meta-classes vs regular classes), as if the typing is not stable enough to be part of the stdlib.
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
Victor
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
# If typing has not been imported, then it's impossible for # any annotation to be a ClassVar. So, only look for ClassVar # if typing has been imported. typing = sys.modules.get('typing') if typing is not None: # This test uses a typing internal class, but it's the best # way to test if this is a ClassVar. if type(a_type) is typing._ClassVar: # This field is a ClassVar. Ignore it. continue
See also https://github.com/ericvsmith/dataclasses/issues/14
Eric.
On Fri, 3 Nov 2017 12:46:33 -0400 "Eric V. Smith" eric@trueblade.com wrote:
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
I don't think other modules should start relying on the typing module at runtime. The dataclasses module can define its own "ClassVar" thing and then I suspect it's easy to map it to typing._ClassVar. It seems we should be careful not to blur the distinction between declarations that have an effect on actual code, and typing declarations which only affect type-checking tools.
Regards
Antoine.
On 3 November 2017 at 17:47, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 3 Nov 2017 12:46:33 -0400 "Eric V. Smith" eric@trueblade.com wrote:
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
I don't think other modules should start relying on the typing module at runtime. The dataclasses module can define its own "ClassVar" thing and then I suspect it's easy to map it to typing._ClassVar. It seems we should be careful not to blur the distinction between declarations that have an effect on actual code, and typing declarations which only affect type-checking tools.
I'm looking forward to the dataclasses module, and I'm perfectly OK with the way that it uses type annotations to declare attributes. I also don't have a problem with it relying on the typing module - but *only* if the typing module is in the stdlib. I don't think it's good if a standard feature needs an external library for some of its functionality.
So I guess the point is, if we're considering moving typing out of the stdlib, then what's the impact on PEP 557?
Personally, I don't use type annotations myself yet, but I've used code that does and I'm considering looking into them - for a variety of reasons, documentation, IDE support, and the ability to type check my code via mypy. If typing moves out of the stdlib, I'd be much less inclined to do so - adding a runtime dependency is a non-trivial cost in terms of admin for deployment, handling within my (peculiar, if you want to debate workflow) development workflow, etc. Working out how to add type annotations *without* them being a runtime dependency (just at test-time) is too much work. So I am concerned that if we move typing out of the stdlib, it'll reduce adoption rates.
Paul
Also, as for dataclasses specifically, since it may become as pervasive as namedtuples, we probably want it to be as light-weight as possible. Which will be harder to achieve if its *API* depends on the typing machinery.
Regards
Antoine.
Le 03/11/2017 à 19:04, Paul Moore a écrit :
On 3 November 2017 at 17:47, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 3 Nov 2017 12:46:33 -0400 "Eric V. Smith" eric@trueblade.com wrote:
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
I don't think other modules should start relying on the typing module at runtime. The dataclasses module can define its own "ClassVar" thing and then I suspect it's easy to map it to typing._ClassVar. It seems we should be careful not to blur the distinction between declarations that have an effect on actual code, and typing declarations which only affect type-checking tools.
I'm looking forward to the dataclasses module, and I'm perfectly OK with the way that it uses type annotations to declare attributes. I also don't have a problem with it relying on the typing module - but *only* if the typing module is in the stdlib. I don't think it's good if a standard feature needs an external library for some of its functionality.
So I guess the point is, if we're considering moving typing out of the stdlib, then what's the impact on PEP 557?
Personally, I don't use type annotations myself yet, but I've used code that does and I'm considering looking into them - for a variety of reasons, documentation, IDE support, and the ability to type check my code via mypy. If typing moves out of the stdlib, I'd be much less inclined to do so - adding a runtime dependency is a non-trivial cost in terms of admin for deployment, handling within my (peculiar, if you want to debate workflow) development workflow, etc. Working out how to add type annotations *without* them being a runtime dependency (just at test-time) is too much work. So I am concerned that if we move typing out of the stdlib, it'll reduce adoption rates.
Paul
On Fri, Nov 3, 2017 at 6:47 PM, Antoine Pitrou solipsis@pitrou.net wrote:
On Fri, 3 Nov 2017 12:46:33 -0400 "Eric V. Smith" eric@trueblade.com wrote:
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
I don't think other modules should start relying on the typing module at runtime.
They already do, as I've noted in my message about dependency injection, and I find this quite elegant.
Previously, similar projects used decorators, or less elegant constructs.
S.
Le 03/11/2017 à 19:34, Stéfane Fermigier a écrit :
On Fri, Nov 3, 2017 at 6:47 PM, Antoine Pitrou <solipsis@pitrou.net mailto:solipsis@pitrou.net> wrote:
I don't think other modules should start relying on the typing module at runtime.They already do, as I've noted in my message about dependency injection, and I find this quite elegant.
Third-party libraries do what they want, but we are talking about the stdlib here.
Regards
Antoine.
On 4 November 2017 at 02:46, Eric V. Smith eric@trueblade.com wrote:
On 11/3/2017 12:15 PM, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I'm planning on using it for PEP 557: https://www.python.org/dev/peps/pep-0557/#class-variables
The way the code currently checks for this should still work if typing is not in the stdlib, although of course it's assuming that the name "typing" really is the "official" typing library.
# If typing has not been imported, then it's impossible for # any annotation to be a ClassVar. So, only look for ClassVar # if typing has been imported. typing = sys.modules.get('typing') if typing is not None: # This test uses a typing internal class, but it's the best # way to test if this is a ClassVar. if type(a_type) is typing._ClassVar: # This field is a ClassVar. Ignore it. continueSee also https://github.com/ericvsmith/dataclasses/issues/14
That particular dependency could also be avoided by defining an "is_class_var(annotation)" generic function and a "ClassVar" helper object in the dataclasses module. For example:
class _ClassVar: def __init__(self, annotation): self.annotation = annotation
class _MakeClassVar: def __getitem__(self, key): return _ClassVar(key)
ClassVar = _MakeClassVar()
@functools.singledispatch def is_class_var(annotation): return isinstance(annotation, _ClassVar)
It would put the burden on static analysers and the typing module to understand that `dataclasses.ClassVar` meant the same thing conceptually as `typing.ClassVar`, but I think that's OK.
Cheers, Nick.
On 6 November 2017 at 12:18, Nick Coghlan ncoghlan@gmail.com wrote:
That particular dependency could also be avoided by defining an "is_class_var(annotation)" generic function and a "ClassVar" helper object in the dataclasses module. For example:
class _ClassVar: def __init__(self, annotation): self.annotation = annotation class _MakeClassVar: def __getitem__(self, key): return _ClassVar(key) ClassVar = _MakeClassVar() @functools.singledispatch def is_class_var(annotation): return isinstance(annotation, _ClassVar)It would put the burden on static analysers and the typing module to understand that `dataclasses.ClassVar` meant the same thing conceptually as `typing.ClassVar`, but I think that's OK.
Eric filed a new issue for this idea here: https://github.com/ericvsmith/dataclasses/issues/61
As I indicated in my comment there, I'm now wondering if there might be an opportunity here whereby we could use the *dataclasses* module to define a stable non-provisional syntactically compatible subset of the typing module, and require folks to start explicitly depending on the typing module (as a regular PyPI dependency) if they want anything more sophisticated than that (Unions, Generics, TypeVars, etc).
Unlike the typing module, dataclasses *wouldn't* attempt to give annotations any meaningful runtime semantics, they'd just be ordinary class instances (hence no complex metaclasses required, hence a relatively fast import time).
Beyond the API already proposed in PEP 557, this would mean adding:
* dataclasses.ClassVar (as proposed above) * dataclasses.Any (probably just set to the literal string "dataclasses.Any") * dataclasses.NamedTuple (as a replacement for typing.NamedTuple) * potentially dataclasses.is_class_var (instead of dataclasses implicitly snooping in sys.modules looking for a "typing" module)
In effect, where we now have "typing" and "typing_extensions", we'd instead have "dataclasses" in the standard library (with the subset of annotations needed to define data record classes), and "typing" as an independently versioned module on PyPI.
I think such an approach could address a few problems:
- the useful-at-runtime concepts (in dataclasses) would be clearly separated from the static-type-analysis enablers (in typing) - type hints would still have a clear presence in the regular standard library, but the more complex parts would be opt-in (similar to statistics vs SciPy, secrets vs cryptography, etc) - as various concepts in typing matured and became genuinely stable, they could potentially "graduate" into the standard library's dataclasses module
Cheers, Nick.
Beyond the API already proposed in PEP 557, this would mean adding:
- dataclasses.ClassVar (as proposed above)
- dataclasses.Any (probably just set to the literal string
"dataclasses.Any")
- dataclasses.NamedTuple (as a replacement for typing.NamedTuple)
- potentially dataclasses.is_class_var (instead of dataclasses
implicitly snooping in sys.modules looking for a "typing" module)
In effect, where we now have "typing" and "typing_extensions", we'd instead have "dataclasses" in the standard library (with the subset of annotations needed to define data record classes), and "typing" as an independently versioned module on PyPI.
I'm not so keen on this because I think some things in typing (such as NamedTuple) probably deserve to be in the collections module. And some of the ABCs could probably also be merged with collections.abc but doing this correctly and not all at once would be quite difficult. I do think the typing concepts should be better integrated into the standard library. However, a fair amount of the clases you list (such as NamedTuple) are in of themselves dependent on parts of typing.
Cheers, Ethan
I think such an approach could address a few problems:
- the useful-at-runtime concepts (in dataclasses) would be clearly
separated from the static-type-analysis enablers (in typing)
- type hints would still have a clear presence in the regular standard
library, but the more complex parts would be opt-in (similar to statistics vs SciPy, secrets vs cryptography, etc)
- as various concepts in typing matured and became genuinely stable,
they could potentially "graduate" into the standard library's dataclasses module
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ ethan%40ethanhs.me
On 7 November 2017 at 05:20, Ethan Smith ethan@ethanhs.me wrote:
I'm not so keen on this because I think some things in typing (such as NamedTuple) probably deserve to be in the collections module. And some of the ABCs could probably also be merged with collections.abc but doing this correctly and not all at once would be quite difficult. I do think the typing concepts should be better integrated into the standard library. However, a fair amount of the clases you list (such as NamedTuple) are in of themselves dependent on parts of typing.
This is a good point. To what extent is it true that the stdlib *already* uses the typing module internally, it's just that the usage is hidden by the fact that the examples of this are in the typing module - not because they "should" be there but simply because it isolates the use of typing to that one module?
Paul
On Nov 6, 2017, at 8:01 PM, Nick Coghlan ncoghlan@gmail.com wrote:
As I indicated in my comment there, I'm now wondering if there might be an opportunity here whereby we could use the *dataclasses* module to define a stable non-provisional syntactically compatible subset of the typing module, and require folks to start explicitly depending on the typing module (as a regular PyPI dependency) if they want anything more sophisticated than that (Unions, Generics, TypeVars, etc).
I have an issue open about essentially this idea: https://github.com/python/typing/issues/496
The consensus is that this is too expensive to do this in time for Python 3.7. On the other hand, not doing anything is terrible in terms of usability: now users will be forced to use both `typing` (a frozen non-provisional version in the stdlib) and `typing_extensions` (new features that we couldn't add directly to `typing`).
This is the reason why Guido asked about moving `typing` out of the standard library. Currently it seems that the best thing that we can do is to ship an upgradeable version of `typing` with Python 3.7.0. There are many details to be figured out here:
1. Does that mean `typing` is going to be provisional forever? The answer is surprisingly "yes" since it makes sense to keep updating the bundled `typing` in 3.7.1, 3.7.2, etc. We need to update PEP 411 to talk about this new case.
2. Does that mean `typing` can now become backwards incompatible randomly? The answer should be "no" and already is "no". If you look at typing.py, it already ships in Python 3.6 with some crazy dances that makes it work even on Python 3.3.
3. Does that mean that Debian is going to rip it out and make people install a `python-typing` .deb? Sadly, probably yes. We need to figure out what that means for us.
4. How do we even version this library then? Probably like this: 3.7.0.0, 3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other questions above.
5. What will happen to typing_extensions and mypy_extensions? They can probably be folded into typing 3.7.0.0. This is a huge win for usability.
6. Where will we track issues for it? Which repo will be the source of truth? Fortunately, we already have https://github.com/python/typing/ https://github.com/python/typing/
There's more things to figure out for sure. And the clock is ticking. But this is a great case study to figure out since more packages in the standard library could use a release model like this, and maybe in the future this will finally lead to unbundling of the standard library from the runtime. Yeah, it's a long shot and the dependency graph between libraries in the standard library is rather extensive. But this is the first step.
- Ł
On Nov 7, 2017, at 10:41, Lukasz Langa lukasz@langa.pl wrote:
- Does that mean that Debian is going to rip it out and make people install a `python-typing` .deb? Sadly, probably yes. We need to figure out what that means for us.
Maybe. Full disclosure, I’ve recently scaled back my contributions to Debian, so I won’t be doing this work, but if I was, I’d probably handle it very similarly to other replaceable external dependencies (e.g. pip).
There is a small loophole in policy to allow for the building and use of wheels for just this *limited* purpose. So roughly I would propose packaging the external python-typing package as a separately installable deb, but also to build this into a wheel that we can pull in at python3.7 interpreter package build time.
It’s fairly easy to do, and all the infrastructure is already there. What would be useful is for upstream CPython to make it easy to import an externally built and installed wheel, from some location outside of its own installed tree (/usr/share/python-wheels).
Cheers, -Barry
On 8 November 2017 at 04:41, Lukasz Langa lukasz@langa.pl wrote:
- How do we even version this library then? Probably like this: 3.7.0.0,
3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other questions above.
Something you may want to consider is switching to CalVer for typing itself, such that we end up saying something like "Python 3.7.0 includes typing 2017.12.1".
My experience has been that CalVer is just *better* for interoperability specifications, since it inherently conveys information about the age of the specification. Saying "We target typing 2017.12.1" in 2018 immediately lets people know they're going to need some pretty up to date software to run a project that has that caveat on it. By contrast, saying the same thing in 2021 means most things released in the past 3 years should be able to handle it.
Such an approach also avoids future confusion if the final version of Python 3.7 were to start bundling Python 3.8's version of typing.
Cheers, Nick.
On Nov 7, 2017, at 2:33 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 8 November 2017 at 04:41, Lukasz Langa lukasz@langa.pl wrote:
- How do we even version this library then? Probably like this: 3.7.0.0,
3.7.0.1, 3.7.1.0, and so on. But that depends on answers to the other questions above.
Something you may want to consider is switching to CalVer for typing itself, such that we end up saying something like "Python 3.7.0 includes typing 2017.12.1".
You don't need to sell me on CalVer, all my private packages use this versioning scheme (just with the shorthand 17) :-)
And yes, this is a good suggestion.
- Ł
Let me just cut this short. typing.py will stay, and it will stay provisional.
On 03Nov2017 0915, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
I'm strongly in favor on such move.
I'm torn.
If not having typing installed means you can't use "Any" or "Optional" in an annotation, that basically kills the whole thing. Some primitives need to be there.
If annotations become glorified strings (which IMHO they should) and typing gains a function to parse those into type hints (which IMHO it should), then I'm in favour of splitting typing out. (Personally, if making every type hint a literal 'string' meant that I could avoid importing typing then I'd do it.)
However, if typing is split out then its API (specifically, the contents of typing.__all__ and their semantic meaning (e.g. "Iterable" means something with an "__iter__" method)) needs to stay in PEPs.
Static analysers using type hints encode much more information about these types than can be inferred statically from typing.py, which means the definitions should not change faster than Python x.*y*. Ideally, they would not change at all once released.
For example, my static analyser has an existing object representing iterables, since we've been inferring iterables for years. When I parse a type annotation and see "typing.Iterable", I'm going to just substitute my own implementation - the definition in typing.py isn't going to be used (or be useful). And because it has to map across languages, it has to be a hard-coded mapping that can't rely on typing.py at all.
Since the contents of typing.py will likely be completely ignored by my analyser, which means that I can't treat "whatever version of typing is installed" as ground truth. It needs to move slower or be purely additive. Being in the standard library is a nice easy way to ensure this - moving it out is a risk.
That said, because I don't care about the contents of the file, all the heavy execution stuff can totally be moved out. If typing in the stdlib became very trivial definitions or just a set of names to support searching/editors/forward-refs, and typing out of the stdlib had the ability to convert annotations into an object model that provides rich runtime introspection, I'd also be fine. At least then the interface is highly stable, even if the implementation (for those who use it) changes.
Cheers, Steve
A side note (I'm reading all responses but staying out of the discussion):
No static checker should depend on the *contents* of typing.py, since it's just a bunch of runtime gymnastics to allow types to be evaluated at runtime without errors, with a secondary goal of making them introspectable (some folks don't even agree with the latter, e.g. Mark Shannon).
Static analyzers should be able to make strong *assumptions* about what things defined there mean -- in mypy such assumptions are all over the place, based on the full name of things -- it never reads typing.py. (It reads typing.pyi from typeshed, but what's there is ignored in many cases too.)
On Fri, Nov 3, 2017 at 10:00 AM, Steve Dower steve.dower@python.org wrote:
On 03Nov2017 0915, Victor Stinner wrote:
Hi,
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
I'm strongly in favor on such move.
I'm torn.
If not having typing installed means you can't use "Any" or "Optional" in an annotation, that basically kills the whole thing. Some primitives need to be there.
If annotations become glorified strings (which IMHO they should) and typing gains a function to parse those into type hints (which IMHO it should), then I'm in favour of splitting typing out. (Personally, if making every type hint a literal 'string' meant that I could avoid importing typing then I'd do it.)
However, if typing is split out then its API (specifically, the contents of typing.__all__ and their semantic meaning (e.g. "Iterable" means something with an "__iter__" method)) needs to stay in PEPs.
Static analysers using type hints encode much more information about these types than can be inferred statically from typing.py, which means the definitions should not change faster than Python x.*y*. Ideally, they would not change at all once released.
For example, my static analyser has an existing object representing iterables, since we've been inferring iterables for years. When I parse a type annotation and see "typing.Iterable", I'm going to just substitute my own implementation - the definition in typing.py isn't going to be used (or be useful). And because it has to map across languages, it has to be a hard-coded mapping that can't rely on typing.py at all.
Since the contents of typing.py will likely be completely ignored by my analyser, which means that I can't treat "whatever version of typing is installed" as ground truth. It needs to move slower or be purely additive. Being in the standard library is a nice easy way to ensure this - moving it out is a risk.
That said, because I don't care about the contents of the file, all the heavy execution stuff can totally be moved out. If typing in the stdlib became very trivial definitions or just a set of names to support searching/editors/forward-refs, and typing out of the stdlib had the ability to convert annotations into an object model that provides rich runtime introspection, I'd also be fine. At least then the interface is highly stable, even if the implementation (for those who use it) changes.
Cheers, Steve
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org
On 03Nov2017 1124, Guido van Rossum wrote:
A side note (I'm reading all responses but staying out of the discussion):
No static checker should depend on the *contents* of typing.py, since it's just a bunch of runtime gymnastics to allow types to be evaluated at runtime without errors, with a secondary goal of making them introspectable (some folks don't even agree with the latter, e.g. Mark Shannon).
Static analyzers should be able to make strong *assumptions* about what things defined there mean -- in mypy such assumptions are all over the place, based on the full name of things -- it never reads typing.py. (It reads typing.pyi from typeshed, but what's there is ignored in many cases too.)
Thank you. Very glad to hear I understood it correctly.
Cheers, Steve
On 4 November 2017 at 04:24, Guido van Rossum guido@python.org wrote:
A side note (I'm reading all responses but staying out of the discussion):
No static checker should depend on the *contents* of typing.py, since it's just a bunch of runtime gymnastics to allow types to be evaluated at runtime without errors, with a secondary goal of making them introspectable (some folks don't even agree with the latter, e.g. Mark Shannon).
Static analyzers should be able to make strong *assumptions* about what things defined there mean -- in mypy such assumptions are all over the place, based on the full name of things -- it never reads typing.py. (It reads typing.pyi from typeshed, but what's there is ignored in many cases too.)
If I understand correctly, a lot of the complexity in the current typing.py implementation is there to make isinstance and issubclass do something "useful" at runtime, and to allow generics to be used as base classes.
If it wasn't for those design goals, then "typing.List[int]" could just return a lightweight instance of a regular class rather than a usable Python class definition.
If I'm right about that, then PEP 560's proposal to allow types to implement a hook that says "Replace me with this other object for runtime subclassing purposes" may be enough to let you delete most of the current code in the typing module - you'd just need to have isinstance and issubclass respect that new hook as well, and define the hooks as returning the relevant ABCs.
Cheers, Nick.
On 4 November 2017 at 03:53, Nick Coghlan ncoghlan@gmail.com wrote:
If I understand correctly, a lot of the complexity in the current typing.py implementation is there to make isinstance and issubclass do something "useful" at runtime, and to allow generics to be used as base classes.
If it wasn't for those design goals, then "typing.List[int]" could just return a lightweight instance of a regular class rather than a usable Python class definition.
+1 to this.
If I'm right about that, then PEP 560's proposal to allow types to implement a hook that says "Replace me with this other object for runtime subclassing purposes" may be enough to let you delete most of the current code in the typing module - you'd just need to have isinstance and issubclass respect that new hook as well, and define the hooks as returning the relevant ABCs.
That would seem ideal to me.
Lukasz Langa said:
So, the difference is in perceived usability. It's psychological.
Please, let's not start the "not in the stdlib isn't an issue" debate again. If I concede it's a psychological issue, will you concede that the fact that it's psychological doesn't mean that it's not a real, difficult to solve, problem for some people? I'm also willing to concede that it's a *minority* problem, if that helps. But can we stop dismissing it as a non-existent problem?
Paul
On 4 Nov, 2017, at 3:39 AM, Paul Moore p.f.moore@gmail.com wrote:
Lukasz Langa said:
So, the difference is in perceived usability. It's psychological.
Please, let's not start the "not in the stdlib isn't an issue" debate again. If I concede it's a psychological issue, will you concede that the fact that it's psychological doesn't mean that it's not a real, difficult to solve, problem for some people? I'm also willing to concede that it's a *minority* problem, if that helps. But can we stop dismissing it as a non-existent problem?
Paul, if you read the words I wrote in my e-mail verbatim, you will note that I am not saying it's not real or it's not important. Quite the opposite. Can you elaborate what made you think that my assertion that the issue is psychological made you think I'm being dismissive? To me it looks like you're aggressively agreeing with me, so I'd like to understand what caused your reaction.
- Ł
On 6 November 2017 at 03:41, Lukasz Langa lukasz@langa.pl wrote:
On 4 Nov, 2017, at 3:39 AM, Paul Moore p.f.moore@gmail.com wrote:
Lukasz Langa said:
So, the difference is in perceived usability. It's psychological.
Please, let's not start the "not in the stdlib isn't an issue" debate again. If I concede it's a psychological issue, will you concede that the fact that it's psychological doesn't mean that it's not a real, difficult to solve, problem for some people? I'm also willing to concede that it's a *minority* problem, if that helps. But can we stop dismissing it as a non-existent problem?
Paul, if you read the words I wrote in my e-mail verbatim, you will note that I am not saying it's not real or it's not important. Quite the opposite. Can you elaborate what made you think that my assertion that the issue is psychological made you think I'm being dismissive? To me it looks like you're aggressively agreeing with me, so I'd like to understand what caused your reaction.
Apologies. On rereading your email, I can see how you meant that. However, it didn't come across that way to me on first reading. There have been a few other threads on various lists I've been involved in recently where it's been really hard to get anyone to see that there's any practical issues for people if a module is on PyPI rather than being in the stdlib. So I'm afraid I'd got pretty tired of arguing the same points over and over again, and over-reacted to what I thought you said.
To explain my actual point a little more clearly:
1. Without typing available, some programs using type annotations won't run. That is, using type annotations (a test-time/development-time feature) introduces a runtime dependency on typing, and hence introduces an extra deployment concern (unlike other development-type features like test frameworks). 2. For some people, if something isn't in the Python standard library (technically, in a standard install), it's not available (without significant effort, or possibly not at all). For those people, a runtime dependency on a non-stdlib typing module means "no use of type annotations allowed". 3. Virtual environments typically only include the stdlib, and "use system site-packages" has affects more than just a single module, so bundling still has issues for virtualenvs - and in the packaging tutorials, we're actively encouraging people to use virtualenvs. We (python-dev) can work around this issue for venv by auto-installing typing, but that still leaves virtualenv (which is critically short of resources, and needs to support older versions of Python, so a major change like bundling typing is going to be a struggle to get implemented).
None of these problems are really addressed by simply saying "pip install typing". That's not for psychological reasons, there are genuine barriers to having that work. However, it's not at all clear how many people are affected by these issues. My personal feeling is that the group of people participating in open source development is biased towards environments where it's not a problem, but that's more gut feeling than hard facts. The place where barriers are perceived/psychological is when we try to discuss workarounds or solutions - because there's a lot of guesswork about what people's environments are like, there's a tendency to assume scenarios that support our preferred solution, rather than those that don't.
Personally, I like to think that my arguments are "giving a voice to people in constrained corporate environments like mine, who are under-represented in open source environments". But it's very easy for a view like that to turn into "banging on about a minor problem as if it were a crisis", and I'm probably guilty of doing that. Worse still, that results in me getting frustrated and then over-reacting further - which is where we came in.
So again, apologies. I misunderstood what you were saying, but more as a result of my personal biases than because you weren't sufficiently clear.
Paul
On Nov 6, 2017, at 4:35 AM, Paul Moore p.f.moore@gmail.com wrote:
- Without typing available, some programs using type annotations
won't run. That is, using type annotations (a test-time/development-time feature) introduces a runtime dependency on typing, and hence introduces an extra deployment concern (unlike other development-type features like test frameworks). 2. For some people, if something isn't in the Python standard library (technically, in a standard install), it's not available (without significant effort, or possibly not at all). For those people, a runtime dependency on a non-stdlib typing module means "no use of type annotations allowed". 3. Virtual environments typically only include the stdlib, and "use system site-packages" has affects more than just a single module, so bundling still has issues for virtualenvs - and in the packaging tutorials, we're actively encouraging people to use virtualenvs. We (python-dev) can work around this issue for venv by auto-installing typing, but that still leaves virtualenv (which is critically short of resources, and needs to support older versions of Python, so a major change like bundling typing is going to be a struggle to get implemented).
Maybe we just need to fully flesh out the idea of a "Python Core" (What exists now as “Python”) and a “Python Platform” (Python Core + A select set of preinstalled libraries). Then typing can just be part of the Python Platform, and gets installed as part of your typical installation, but is otherwise an independent piece of code.
On 7 November 2017 at 00:35, Donald Stufft donald@stufft.io wrote:
Maybe we just need to fully flesh out the idea of a "Python Core" (What exists now as “Python”) and a “Python Platform” (Python Core + A select set of preinstalled libraries). Then typing can just be part of the Python Platform, and gets installed as part of your typical installation, but is otherwise an independent piece of code.
I quite like this idea, but at a minimum it would mean that the Windows installers for Python should include "core" and "platform" builds, with "platform" being the version that new users are directed to. And I'm not sure python-dev would be comfortable with agreeing and distributing what would essentially be a curated subset of PyPI packages. The most recent discussion on "a recommended set of packages" didn't get much further than 3 or 4 candidates, which hardly constitutes a "platform".
Conversely, something like Anaconda could be considered as the "Python Platform" (assuming they bundle typing...), but I'm not really comfortable with python-dev aligning themselves that strongly with a single distributor. (We're talking about a recommendation along the lines of "people wanting to gain, for example, corporate approval for "Python" should request approval for the Anaconda distribution rather than the python.org distribution").
Paul
On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft donald@stufft.io wrote: [..]
Maybe we just need to fully flesh out the idea of a "Python Core" (What exists now as “Python”) and a “Python Platform” (Python Core + A select set of preinstalled libraries). Then typing can just be part of the Python Platform, and gets installed as part of your typical installation, but is otherwise an independent piece of code.
Given that (type and other) annotations have been promoted as an optional
feature of the language it seems unfair and perhaps unwise to add a dependency specifically to support them to the stdlib and therefore the Python core. Since type annotations are, as Paul pointed out, development-time features, it would appear to behoove those wishing to use them to separate them in such a way that the software can be installed without annotations, and therefore without the need for the typing module. Assuming they would like to see the widest possible distribution, of course. For selected audiences I am sure typing will be *de rigeur*,
In this scenario, surely the most "typical" installation would be a virtualenv. Do all virtualenvs rely on the same Platform? Who decides which additional libraries are required for the Platform? Doesn't this just add another "type of installation" distinction to confuse the unwary? How do I maintain the Platform separately from the Core? Does my sysadmin maintain either or both? I'd like to see a little more clarity about the benefits such a schism would offer.
regards Steve
On 7 November 2017 at 10:18, Steve Holden steve@holdenweb.com wrote:
On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft donald@stufft.io wrote: [..]
Maybe we just need to fully flesh out the idea of a "Python Core" (What exists now as “Python”) and a “Python Platform” (Python Core + A select set of preinstalled libraries). Then typing can just be part of the Python Platform, and gets installed as part of your typical installation, but is otherwise an independent piece of code.
Given that (type and other) annotations have been promoted as an optional feature of the language it seems unfair and perhaps unwise to add a dependency specifically to support them to the stdlib and therefore the Python core. Since type annotations are, as Paul pointed out, development-time features, it would appear to behoove those wishing to use them to separate them in such a way that the software can be installed without annotations, and therefore without the need for the typing module. Assuming they would like to see the widest possible distribution, of course. For selected audiences I am sure typing will be de rigeur,
From my point of view, I take the same premise and come to the
opposite conclusion.
Because type annotations are a development-time feature, they should *not* require a dependency in the final deployment (apart from Python itself). However, because they are a language syntax feature they are of necessity written in the application source. And type specification of anything more complex than basic types (for example, List[int]) requires classes defined in the typing module. Therefore, typing must be in the stdlib so that use of type annotations by the developer doesn't impose a runtime dependency on the end user.
If there were a way of including type annotations that had no runtime effect on the final deployed program, things would be different. But the decision to make annotations part of the language syntax precludes that. In particular, "it would appear to behoove those wishing to use them to separate them" - there's no way of doing that *precisely* because they are a language syntax feature.
Paul
On Tue, 7 Nov 2017 at 03:34 Paul Moore p.f.moore@gmail.com wrote:
On 7 November 2017 at 10:18, Steve Holden steve@holdenweb.com wrote:
On Tue, Nov 7, 2017 at 12:35 AM, Donald Stufft donald@stufft.io wrote: [..]
Maybe we just need to fully flesh out the idea of a "Python Core" (What exists now as “Python”) and a “Python Platform” (Python Core + A select
set
of preinstalled libraries). Then typing can just be part of the Python Platform, and gets installed as part of your typical installation, but
is
otherwise an independent piece of code.
Given that (type and other) annotations have been promoted as an optional feature of the language it seems unfair and perhaps unwise to add a dependency specifically to support them to the stdlib and therefore the Python core. Since type annotations are, as Paul pointed out, development-time
features,
it would appear to behoove those wishing to use them to separate them in such a way that the software can be installed without annotations, and therefore without the need for the typing module. Assuming they would
like
to see the widest possible distribution, of course. For selected
audiences I
am sure typing will be de rigeur,
From my point of view, I take the same premise and come to the opposite conclusion.
Because type annotations are a development-time feature, they should *not* require a dependency in the final deployment (apart from Python itself). However, because they are a language syntax feature they are of necessity written in the application source. And type specification of anything more complex than basic types (for example, List[int]) requires classes defined in the typing module. Therefore, typing must be in the stdlib so that use of type annotations by the developer doesn't impose a runtime dependency on the end user.
That's not *necessarily* true if Lukasz's PEP lands and annotations become strings. The dependency would only need to be installed if someone chose to introspect the annotations and then "instantiate" them into actual objects. And that only comes up if someone does it from outside by a 3rd-party, who would then need to install the type annotation dependencies themselves.
I fully admit that could get messy if introspection from 3rd-parties happens at all regularly and trying to manage that kind of situation. In this instance I would argue that any code that is to facilitate creating an object from an annotation exist outside of the stdlib if 'typing' gets removed to prevent this sort of situation without the user of such code being fully aware of what they are up against.
-Brett
If there were a way of including type annotations that had no runtime effect on the final deployed program, things would be different. But the decision to make annotations part of the language syntax precludes that. In particular, "it would appear to behoove those wishing to use them to separate them" - there's no way of doing that *precisely* because they are a language syntax feature.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Nov 7, 2017, at 9:39 AM, Brett Cannon brett@python.org wrote:
On Tue, 7 Nov 2017 at 03:34 Paul Moore <p.f.moore@gmail.com mailto:p.f.moore@gmail.com> wrote:
Because type annotations are a development-time feature, they should *not* require a dependency in the final deployment (apart from Python itself). However, because they are a language syntax feature they are of necessity written in the application source. And type specification of anything more complex than basic types (for example, List[int]) requires classes defined in the typing module. Therefore, typing must be in the stdlib so that use of type annotations by the developer doesn't impose a runtime dependency on the end user.
That's not necessarily true if Lukasz's PEP lands and annotations become strings. The dependency would only need to be installed if someone chose to introspect the annotations and then "instantiate" them into actual objects. And that only comes up if someone does it from outside by a 3rd-party, who would then need to install the type annotation dependencies themselves.
PEP 563 is the first step there but it's not enough. For this idea to work, `typing.TYPE_CHECKING` would need to be moved to the Python runtime, so that you can do
if TYPE_CHECKING: from typing import *
More importantly, you would put type aliases, type variables and new-types in this `if` block, too.
But even if all this is true, there's still two remaining features that require runtime availability:
1. `cast()`; and 2. creating generic classes by subclassing `Generic[T]` or any of its subclasses. And soon enough, Protocols.
I hope I'm not forgetting any other cases. For `cast()` I have an idea how to move it to a variable annotation. For generic classes, there's no magic, you need `typing` at runtime. Fortunately, that latter case is (I hope?) relatively unlikely.
All the above is summarized here: https://github.com/python/typing/issues/496 https://github.com/python/typing/issues/496
- Ł
On Nov 3, 2017, at 10:00, Steve Dower steve.dower@python.org wrote:
On 03Nov2017 0915, Victor Stinner wrote:
Hi, 2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
I'm strongly in favor on such move.
I'm torn.
Me too. We’re seeing much greater adoption of type annotations, and it’s becoming one of the killer features for adopting Python 3. I’d be hesitant to accept anything that slows that adoption down. While it’s been technically provisional, a lot of code is beginning to depend on it and it would be a shame to break that code as we also start to adopt Python 3.7. But I can appreciate the need to iterate on its API faster.
I don’t know if a middle ground is feasible. What core functionality and stable-enough APIs can be kept in stdlib typing, and can we provide an extension or override mechanism if you want the latest and greatest?
Cheers, -Barry
On 3 November 2017 at 17:00, Steve Dower steve.dower@python.org wrote:
I'm torn.
If not having typing installed means you can't use "Any" or "Optional" in an annotation, that basically kills the whole thing. Some primitives need to be there.
Thinking some more about this, I think that it's now established that the annotation syntax is for types - any debate over other uses for them is now past. As a result, though, I think it's important that the language (and/or the standard library) should include support for *expressing* those types. The typing module is what allows users to express types like "list of integers", or "optional string", or "iterable", and if we move typing out of the stdlib, we make it impossible for people who want to use the language feature to do so from within the core language.
Consider someone who's downloaded Python and PyCharm (or Visual Studio). They want to get the benefit of the IDE code completion facilities, so they declare their argument as List[int], following the information they've found on how to declare lists of integers. And now their code won't run, until they install typing from PyPI. And there's no workaround, because you can't express List[int] in the core language/stdlib. That's not a very good start for a newcomer to Python.
I'm fine with the "advanced" bits of typing being removed from the stdlib, but I think we need to include in the stdlib at least enough to express the basic types of the language (including common combinations such as Optional and Union).
Paul
PS Apologies if I've misunderstood any of the technical aspects of typing - I'm happy to be corrected. As I said in another email, I've not actually used type annotations in my own code yet, although I'm thinking of starting to - precisely because I've been using PyCharm recently and the IDE support when you declare types is quite nice.
On 03.11.2017 21:34, Paul Moore wrote:
Consider someone who's downloaded Python and PyCharm (or Visual Studio). They want to get the benefit of the IDE code completion facilities, so they declare their argument as List[int], following the information they've found on how to declare lists of integers.
The PyCharm I know is capable of detecting such simple types on its own, without type hints.
I for one like the idea of a faster evolution of typing.py.
Cheers, Sven
PS: pip is pretty standard these days, so I don't think it's much of an issue for guys who really needs it installed.
2017-11-03 18:00 GMT+01:00 Steve Dower steve.dower@python.org:
If not having typing installed means you can't use "Any" or "Optional" in an annotation, that basically kills the whole thing. Some primitives need to be there.
I'm not sure that I understand you.
The question is if you would only need <Python 3.7> or <Python 3.7 + pip install typing>.
If typing is removed from the stdlib, you can still use it in your application. It's "just" another dependency no? Which major (non trivial) application or Python module has zero external dependency nowaday?
The only drawback is that we cannot use typing "anymore" in the stdlib itself, since we don't allow external dependency in the stdlib for practical reasons. But as I wrote, we don't use typing currently in stdlib. I'm perfectly fine with the current status of having annotations on the stdlib in a third party project.
Victor
On 3 Nov, 2017, at 4:01 PM, Victor Stinner victor.stinner@gmail.com wrote:
The question is if you would only need <Python 3.7> or <Python 3.7 + pip install typing>.
If typing is removed from the stdlib, you can still use it in your application. It's "just" another dependency no?
The ideal situation is that something is built-in and just works, examples: dicts, lists, sorted().
So, if you have to import it to use it, it's still great but less seamless, current example: regular expressions. Let's say Guido suggests we should import sorted, dict, and list before use. Not a big deal, right? I mean, how many applications do you know that don't use any other imports?
Finally, if you have to find a third-party package, add it to requirements.txt and manage the dependency forward, that's even less seamless. The standard library has a pretty conservative approach to backwards compatibility. On the other hand, third-party libraries often don't. Sure, there's noble exceptions but the general feel is that you need to be more careful with dependencies from PyPI. If somebody suggested that regular expressions or dictionaries should be moved to PyPI, in my book that would suggest strange things will start happening in the future.
So, the difference is in perceived usability. It's psychological.
- Ł
On Fri, 3 Nov 2017 at 17:59 Lukasz Langa lukasz@langa.pl wrote:
On 3 Nov, 2017, at 4:01 PM, Victor Stinner victor.stinner@gmail.com
wrote:
The question is if you would only need <Python 3.7> or <Python 3.7 + pip install typing>.
If typing is removed from the stdlib, you can still use it in your application. It's "just" another dependency no?
The ideal situation is that something is built-in and just works, examples: dicts, lists, sorted().
So, if you have to import it to use it, it's still great but less seamless, current example: regular expressions. Let's say Guido suggests we should import sorted, dict, and list before use. Not a big deal, right? I mean, how many applications do you know that don't use any other imports?
Finally, if you have to find a third-party package, add it to requirements.txt and manage the dependency forward, that's even less seamless. The standard library has a pretty conservative approach to backwards compatibility. On the other hand, third-party libraries often don't. Sure, there's noble exceptions but the general feel is that you need to be more careful with dependencies from PyPI. If somebody suggested that regular expressions or dictionaries should be moved to PyPI, in my book that would suggest strange things will start happening in the future.
So, the difference is in perceived usability. It's psychological.
I think another psychological side-effect of removing 'typing' is how strongly people will view the guidance that annotations are generally expected to be type hints. With 'typing' in the stdlib, it made that assumption a bit strong, especially if the stdlib ever started to use type hints itself. But by saying "we expect annotations will be used for type hints, but we actually don't do that in the stdlib nor provide a mechanism for actually specifying type hints", that weakens the message. It starts to feel like a "do as I say, not as I do" bit of guidance (which to me is different than the worry that the usage of type hints will decrease).
But that might be fine; I personally don't know. But my suspicion is we will see an uptick of alternative annotation uses that aren't type hints if we take 'typing' out, and that's something we should be aware of and okay with.
On Nov 3, 2017, at 9:15 AM, Victor Stinner victor.stinner@gmail.com wrote:
2017-11-03 15:36 GMT+01:00 Guido van Rossum guido@python.org:
Maybe we should remove typing from the stdlib? https://github.com/python/typing/issues/495
I'm strongly in favor on such move.
My experience with asyncio in the stdlib is that users expect changes faster than the very slow release process of the stdlib (a release every 18 months in average).
I saw many PEPs and discussion on the typing design (meta-classes vs regular classes), as if the typing is not stable enough to be part of the stdlib.
The typing module is not used yet in the stdlib, so there is no technically reason to keep typing part of the stdlib. IMHO it's perfectly fine to keep typing and annotations out of the stdlib, since the venv & pip tooling is now rock solid ;-)
I concur with Victor on every point. In particular, many of the good reasons that typeshed is external to the standard library will also apply to typing.py.
It would also be nice to not have typing.py vary with each version of CPython's release cycle. Not only would typing benefit from more frequent updates, it would be nice to have updates that aren't tied to a specific version of CPython -- that would help folks who have to maintain code that works across multiple CPython versions (i.e. the same benefit that we get by always installing the most up-to-date versions of requests, typeshed, jinja2, etc).
Already, we've have updates to typing.py in the point releases of Python because those updates were considered so useful and important.
Raymond
participants (17)
-
Antoine Pitrou -
Antoine Pitrou -
Barry Warsaw -
Brett Cannon -
Donald Stufft -
Eric V. Smith -
Ethan Smith -
Guido van Rossum -
Lukasz Langa -
Nick Coghlan -
Paul Moore -
Raymond Hettinger -
Steve Dower -
Steve Holden -
Stéfane Fermigier -
Sven R. Kunze -
Victor Stinner