Validating type completeness for py.typed packages
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
I can think of a few solutions here: 1. For py.typed packages, type checkers should always apply PEP 484 import rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. 2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
-Eric
--- Eric Traut Contributor to Pyright and Pylance Microsoft
Would it be possible to look at the origin of the imported name? If a 3rd party package imports something from the stdlib, I would venture that in 99.99% of the cases it is not meant for export. OTOH if a `__init__.py` file imports something from `.` it could well be meant for export (this is a pretty common pattern, though I doubt universal). When importing from another package it may vary. (A special case is `pytest`, whose `__init__.py` imports lots of stuff from `_pytest`, all of which is for export.)
Another heuristic is to look at `__all__`. If it's present, you have a good idea of what's meant for export (though there are some packages that define additional symbols that must be imported explicitly, while `__all__` is used to guide `from ... import *`).
I assume there aren't that many py.typed packages yet. Maybe you can somehow find them on GitHub and see how your simpler rule works out? (This list doesn't reach most typing users.)
On Fri, Sep 25, 2020 at 4:59 PM Eric Traut eric@traut.com wrote:
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
I can think of a few solutions here:
- For py.typed packages, type checkers should always apply PEP 484 import
rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. 2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
-Eric
Eric Traut Contributor to Pyright and Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
+1 for using __all__ as a heuristic to find public symbols. E.g, doing so would cover the aforementioned __init__.py of pytest.
There's a website called grep.app that lets you search a subset of Github; I've found it very useful. It was a little tricky to get it to work here, since py.typed is usually empty, but https://grep.app/search?q=.%3F%C2%AEexp=true&filter%5Bpath.pattern%5D%5B... seems to do what we want. Datapoints here would be good!
My intuition is that Rule 1 would require a lot of changes, but that almost all of the discrepancy between public symbol intention and Rule 1 interpretation would be from __init__.py's where people didn't use stub-style explicit reexports (which IME are quite uncommon in .py in the wild) and didn't use __all__ or used a dynamic __all__. For an example, see https://github.com/numpy/numpy/blob/master/numpy/__init__.py (numpy has a py.typed).
It might be worth special casing __init__.py to have "implicit reexport" (potentially only for same-package symbols). I believe this is pretty principled as far as special cases go: reexporting in __init__.py lets you skip a layer of nesting and so is used specially in this context.
We could also push to ensure that type checkers understand a couple common __all__ idioms. For instance, we'd be basically able to accurately reflect numpy's top level public symbols by understanding `__all__.extend(<list of strings literal or some_other_module.__all__>)`.
On Fri, 25 Sep 2020 at 17:55, Guido van Rossum guido@python.org wrote:
Would it be possible to look at the origin of the imported name? If a 3rd party package imports something from the stdlib, I would venture that in 99.99% of the cases it is not meant for export. OTOH if a `__init__.py` file imports something from `.` it could well be meant for export (this is a pretty common pattern, though I doubt universal). When importing from another package it may vary. (A special case is `pytest`, whose `__init__.py` imports lots of stuff from `_pytest`, all of which is for export.)
Another heuristic is to look at `__all__`. If it's present, you have a good idea of what's meant for export (though there are some packages that define additional symbols that must be imported explicitly, while `__all__` is used to guide `from ... import *`).
I assume there aren't that many py.typed packages yet. Maybe you can somehow find them on GitHub and see how your simpler rule works out? (This list doesn't reach most typing users.)
On Fri, Sep 25, 2020 at 4:59 PM Eric Traut eric@traut.com wrote:
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
I can think of a few solutions here:
- For py.typed packages, type checkers should always apply PEP 484
import rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. 2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
-Eric
Eric Traut Contributor to Pyright and Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/ _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
On Fri, Sep 25, 2020 at 4:59 PM Eric Traut eric@traut.com wrote:
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
One problem with the `from x import y as z` heuristic in .py files is it's sometimes used to either avoid shadowing a similar name or to give a shorter, more obvious name.
For instance, I have code where I do `from . import builtins as debuiltins`. This is specifically to avoid confusion over the actual 'builtins' module in Python (and the naming of the module in my package is on purpose as it is meant to act as a reimplementation of part of the 'builtins' module). So this heuristic would break for me. I have also used `from x import y as z` in other cases like `from very.long.package.name import util as very_util` to disambiguate with other 'util' modules being imported.
I can think of a few solutions here:
- For py.typed packages, type checkers should always apply PEP 484 import
rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. 2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
Guido already brought it up, but __all__ is a good heuristic to go by when provided.
I would also say that for the typical __init__.py that re-exports, often they are nothing but comments, docstrings, and import statements. So if you can detect that a file is essentially just import statements that would be another reasonable heuristic to say those names are meant to be re-exported.
-Brett
-Eric
Eric Traut Contributor to Pyright and Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: brett@python.org
My goal is to come up with a simple, straightforward set of rules — not a bunch of complex, loosely-defined, or hard-to-understand heuristics.
I've been exploring the suggestions from Guido and Shantanu, and those ideas are looking pretty good. Here's an updated version of my proposed rule that incorporates this feedback:
For py.typed packages, type checkers should always apply PEP 484 import rules for “.py” files. If a symbol x is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. If a “.py” file defines a module-level `__all__` symbol, the names it includes are considered public exports, overriding all other rules to the contrary.
This modified rule doesn't cover 100% of existing py.typed packages, but it covers the vast majority, and it should be straightforward to update the small percentage that are not handled. As Guido pointed out, the number of "py.typed" packages is still relatively small, so now is a good time to standardize this.
It looks like the following idioms for manipulating `__all__` cover the vast majority of the cases: * `__all__ = (‘a', ‘b', ‘c')` * `__all__ = [‘a', ‘b', ‘c']` * `__all__ += [‘a’, ’b', ‘c']` * `__all__.extend([‘a', ‘b', ‘c'])` * `__all__.extend(lib.__all__)` * `__all__.remove(‘a')`
Brett, if we adopt this set of rules, you would simply need to change your usage from `from . import builtins as debuiltins` to something like `from . import builtins as _builtins`. By adding the underscore, the symbol will no longer be interpreted as a public export.
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
Do we have an indication that others are interpreting it that way? If not, maybe we should amend PEP 484 to clarify this.
—Guido
Here's some datapoints on typeshed for the relevant use of `[from ...] import X as Y`. Seems like a number of them are intended as re-exports.
I personally like `import X as X` and so would be happy to volunteer to change existing typeshed definitions, if it allows us to more cleanly translate concepts to .py / correctly interface Brett's code.
In [*6*]: *import* *ast*
...: *from* *pathlib* *import* Path
...:
...: *def* find_realiases(filename):
...: *with* open(filename) *as* f:
...: module = ast.parse(f.read())
...: aliases = [node *for* node *in* ast.walk(module) *if* isinstance(node, ast.alias)]
...: realiased = [a *for* a *in* aliases *if* a.asname *and* a.asname != a.name.split(".")[-1] *and* *not* a.asname.startswith("_")]
...: *for* alias *in* realiased:
...: relpath = filename.relative_to(Path.home() / "dev/typeshed")
...: print(f"*{*str(relpath)*:*60*}* *{*alias.name*}* as *{* alias.asname*}*")
...:
...:
...: *for* p *in* sorted((Path.home() / "dev/typeshed").glob("**/*.pyi" )):
...: find_realiases(p)
...:
stdlib/2/collections.pyi AbstractSet as Set
stdlib/2/future_builtins.pyi ifilter as filter
stdlib/2/future_builtins.pyi imap as map
stdlib/2/future_builtins.pyi izip as zip
stdlib/2/md5.pyi md5 as new
stdlib/2/os/__init__.pyi OSError as error
stdlib/2/os/__init__.pyi _PathLike as PathLike
stdlib/2and3/_dummy_threading.pyi _ExceptHookArgs as ExceptHookArgs
stdlib/2and3/contextlib.pyi ContextManager as AbstractContextManager
stdlib/2and3/contextlib.pyi AsyncContextManager as AbstractAsyncContextManager
stdlib/2and3/plistlib.pyi Dict as DictT
stdlib/2and3/threading.pyi _ExceptHookArgs as ExceptHookArgs
stdlib/3/collections/__init__.pyi AbstractSet as Set
stdlib/3/os/__init__.pyi OSError as error
stdlib/3/os/__init__.pyi _PathLike as PathLike
stdlib/3/platform.pyi devnull as DEV_NULL
third_party/2/six/__init__.pyi StringIO as BytesIO
third_party/2/six/moves/__init__.pyi __builtin__ as builtins
third_party/2/six/moves/__init__.pyi raw_input as input
third_party/2/six/moves/__init__.pyi reload as reload_module
third_party/2/six/moves/__init__.pyi xrange as range
third_party/2/six/moves/__init__.pyi StringIO as cStringIO
third_party/2/six/moves/__init__.pyi ifilter as filter
third_party/2/six/moves/__init__.pyi ifilterfalse as filterfalse
third_party/2/six/moves/__init__.pyi imap as map
third_party/2/six/moves/__init__.pyi izip as zip
third_party/2/six/moves/__init__.pyi izip_longest as zip_longest
third_party/2/six/moves/__init__.pyi getcwd as getcwdb
third_party/2/six/moves/__init__.pyi getcwdu as getcwd
third_party/2/six/moves/__init__.pyi quote as shlex_quote
third_party/2/six/moves/urllib/parse.pyi unquote as unquote_to_bytes
third_party/2and3/Crypto/Protocol/KDF.pyi SHA as SHA1
third_party/2and3/boto/compat.pyi encodestring as encodebytes
third_party/2and3/click/types.pyi _ParamType as ParamType
third_party/2and3/dateutil/rrule.pyi weekday as weekdaybase
third_party/2and3/flask/templating.pyi Environment as BaseEnvironment
third_party/2and3/flask/wrappers.pyi Request as RequestBase
third_party/2and3/flask/wrappers.pyi Response as ResponseBase
third_party/2and3/google/protobuf/map_unittest_pb2.pyi ForeignMessage as ForeignMessage1
third_party/2and3/jinja2/_compat.pyi quote_from_bytes as url_quote
third_party/2and3/jinja2/_compat.pyi quote as url_quote
third_party/2and3/jinja2/compiler.pyi iskeyword as is_python_keyword
third_party/2and3/jinja2/defaults.pyi FILTERS as DEFAULT_FILTERS
third_party/2and3/jinja2/defaults.pyi TESTS as DEFAULT_TESTS
third_party/2and3/requests/adapters.pyi exceptions as urllib3_exceptions
third_party/2and3/requests/exceptions.pyi HTTPError as BaseHTTPError
third_party/2and3/requests/models.pyi exceptions as urllib3_exceptions
third_party/2and3/werkzeug/_compat.pyi StringIO as BytesIO
third_party/2and3/werkzeug/_compat.pyi StringIO as BytesIO
third_party/2and3/werkzeug/contrib/securecookie.pyi new as hmac
third_party/2and3/werkzeug/debug/__init__.pyi BaseRequest as Request
third_party/2and3/werkzeug/debug/__init__.pyi BaseResponse as Response
third_party/2and3/werkzeug/test.pyi Request as U2Request
third_party/2and3/werkzeug/test.pyi Request as U2Request
third_party/2and3/werkzeug/testapp.pyi BaseRequest as Request
third_party/2and3/werkzeug/testapp.pyi BaseResponse as Response
third_party/3/aiofiles/__init__.pyi _os as os
third_party/3/six/moves/__init__.pyi range as xrange
third_party/3/six/moves/__init__.pyi reload as reload_module
third_party/3/six/moves/__init__.pyi StringIO as cStringIO
third_party/3/six/moves/__init__.pyi quote as shlex_quote
third_party/3/waitress/compat.pyi parse as urlparse
On Mon, 28 Sep 2020 at 14:10, Guido van Rossum guido@python.org wrote:
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
Do we have an indication that others are interpreting it that way? If not, maybe we should amend PEP 484 to clarify this.
—Guido
--Guido (mobile) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
Yeah, six.moves is a big one there, and seems to have spawned some imitations... (FWIW I am wondering whether urlparse in waitress/compat.pyi should be there -- I can't find in their GitHub repo.)
I guess the alternative syntax that will work (in the absence of `__all__`) is this: ``` # Do not use this from foo import X as Y # where Y != X
# Do use this from foo import X as _X Y = _X ```
On Mon, Sep 28, 2020 at 3:42 PM Shantanu Jain hauntsaninja@gmail.com wrote:
Here's some datapoints on typeshed for the relevant use of `[from ...] import X as Y`. Seems like a number of them are intended as re-exports.
I personally like `import X as X` and so would be happy to volunteer to change existing typeshed definitions, if it allows us to more cleanly translate concepts to .py / correctly interface Brett's code.
In [*6*]: *import* *ast*
...: *from* *pathlib* *import* Path
...:
...: *def* find_realiases(filename):
...: *with* open(filename) *as* f:
...: module = ast.parse(f.read())
...: aliases = [node *for* node *in* ast.walk(module) *if* isinstance(node, ast.alias)]
...: realiased = [a *for* a *in* aliases *if* a.asname *and* a.asname != a.name.split(".")[-1] *and* *not* a.asname.startswith("_")]
...: *for* alias *in* realiased:
...: relpath = filename.relative_to(Path.home() / "dev/typeshed")
...: print(f"*{*str(relpath)*:*60*}* *{*alias.name*}* as *{* alias.asname*}*")
...:
...:
...: *for* p *in* sorted((Path.home() / "dev/typeshed").glob("**/*.pyi" )):
...: find_realiases(p)
...:
stdlib/2/collections.pyi AbstractSet as Set
stdlib/2/future_builtins.pyi ifilter as filter
stdlib/2/future_builtins.pyi imap as map
stdlib/2/future_builtins.pyi izip as zip
stdlib/2/md5.pyi md5 as new
stdlib/2/os/__init__.pyi OSError as error
stdlib/2/os/__init__.pyi _PathLike as PathLike
stdlib/2and3/_dummy_threading.pyi _ExceptHookArgs as ExceptHookArgs
stdlib/2and3/contextlib.pyi ContextManager as AbstractContextManager
stdlib/2and3/contextlib.pyi AsyncContextManager as AbstractAsyncContextManager
stdlib/2and3/plistlib.pyi Dict as DictT
stdlib/2and3/threading.pyi _ExceptHookArgs as ExceptHookArgs
stdlib/3/collections/__init__.pyi AbstractSet as Set
stdlib/3/os/__init__.pyi OSError as error
stdlib/3/os/__init__.pyi _PathLike as PathLike
stdlib/3/platform.pyi devnull as DEV_NULL
third_party/2/six/__init__.pyi StringIO as BytesIO
third_party/2/six/moves/__init__.pyi __builtin__ as builtins
third_party/2/six/moves/__init__.pyi raw_input as input
third_party/2/six/moves/__init__.pyi reload as reload_module
third_party/2/six/moves/__init__.pyi xrange as range
third_party/2/six/moves/__init__.pyi StringIO as cStringIO
third_party/2/six/moves/__init__.pyi ifilter as filter
third_party/2/six/moves/__init__.pyi ifilterfalse as filterfalse
third_party/2/six/moves/__init__.pyi imap as map
third_party/2/six/moves/__init__.pyi izip as zip
third_party/2/six/moves/__init__.pyi izip_longest as zip_longest
third_party/2/six/moves/__init__.pyi getcwd as getcwdb
third_party/2/six/moves/__init__.pyi getcwdu as getcwd
third_party/2/six/moves/__init__.pyi quote as shlex_quote
third_party/2/six/moves/urllib/parse.pyi unquote as unquote_to_bytes
third_party/2and3/Crypto/Protocol/KDF.pyi SHA as SHA1
third_party/2and3/boto/compat.pyi encodestring as encodebytes
third_party/2and3/click/types.pyi _ParamType as ParamType
third_party/2and3/dateutil/rrule.pyi weekday as weekdaybase
third_party/2and3/flask/templating.pyi Environment as BaseEnvironment
third_party/2and3/flask/wrappers.pyi Request as RequestBase
third_party/2and3/flask/wrappers.pyi Response as ResponseBase
third_party/2and3/google/protobuf/map_unittest_pb2.pyi ForeignMessage as ForeignMessage1
third_party/2and3/jinja2/_compat.pyi quote_from_bytes as url_quote
third_party/2and3/jinja2/_compat.pyi quote as url_quote
third_party/2and3/jinja2/compiler.pyi iskeyword as is_python_keyword
third_party/2and3/jinja2/defaults.pyi FILTERS as DEFAULT_FILTERS
third_party/2and3/jinja2/defaults.pyi TESTS as DEFAULT_TESTS
third_party/2and3/requests/adapters.pyi exceptions as urllib3_exceptions
third_party/2and3/requests/exceptions.pyi HTTPError as BaseHTTPError
third_party/2and3/requests/models.pyi exceptions as urllib3_exceptions
third_party/2and3/werkzeug/_compat.pyi StringIO as BytesIO
third_party/2and3/werkzeug/_compat.pyi StringIO as BytesIO
third_party/2and3/werkzeug/contrib/securecookie.pyi new as hmac
third_party/2and3/werkzeug/debug/__init__.pyi BaseRequest as Request
third_party/2and3/werkzeug/debug/__init__.pyi BaseResponse as Response
third_party/2and3/werkzeug/test.pyi Request as U2Request
third_party/2and3/werkzeug/test.pyi Request as U2Request
third_party/2and3/werkzeug/testapp.pyi BaseRequest as Request
third_party/2and3/werkzeug/testapp.pyi BaseResponse as Response
third_party/3/aiofiles/__init__.pyi _os as os
third_party/3/six/moves/__init__.pyi range as xrange
third_party/3/six/moves/__init__.pyi reload as reload_module
third_party/3/six/moves/__init__.pyi StringIO as cStringIO
third_party/3/six/moves/__init__.pyi quote as shlex_quote
third_party/3/waitress/compat.pyi parse as urlparse
On Mon, 28 Sep 2020 at 14:10, Guido van Rossum guido@python.org wrote:
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
Do we have an indication that others are interpreting it that way? If not, maybe we should amend PEP 484 to clarify this.
—Guido
--Guido (mobile) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
On Mon, Sep 28, 2020 at 2:10 PM Guido van Rossum guido@python.org wrote:
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
I think the key difference in view between the two interpretations is whether you view the `as` clause as part of the import statement or a convenient/implied assignment that's done post-import.
So if the semantics were `as` clauses that lead to a name lacking a leading underscore were consider exposed, that trips my brain up because now because _some_ forms of import dictate my exporting via a special rule that I have to avoid while other don't:
- `import X`; not exported - `from X import Y`; not exported - `from X import Y as Z`; exported; brain goes, "why is this form of import special when it comes to naming conventions and not the other forms?"
The `from X import Y as Y` is an explicit action where I'm opting to inline the exporting of the symbol makes more sense to me since it is not a normal thing to do and I have to actively think of the opt-in. Otherwise I assume people can unwrap the naming to explicitly export and have the tools support that as well:
from X import Y Z = Y
-Brett
Do we have an indication that others are interpreting it that way? If not, maybe we should amend PEP 484 to clarify this.
—Guido
--Guido (mobile) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: brett@python.org
On Tue, Sep 29, 2020 at 12:19 PM Brett Cannon brett@python.org wrote:
On Mon, Sep 28, 2020 at 2:10 PM Guido van Rossum guido@python.org wrote:
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
I think the key difference in view between the two interpretations is whether you view the `as` clause as part of the import statement or a convenient/implied assignment that's done post-import.
So if the semantics were `as` clauses that lead to a name lacking a leading underscore were consider exposed, that trips my brain up because now because _some_ forms of import dictate my exporting via a special rule that I have to avoid while other don't:
- `import X`; not exported
- `from X import Y`; not exported
- `from X import Y as Z`; exported; brain goes, "why is this form of
import special when it comes to naming conventions and not the other forms?"
The `from X import Y as Y` is an explicit action where I'm opting to inline the exporting of the symbol makes more sense to me since it is not a normal thing to do and I have to actively think of the opt-in. Otherwise I assume people can unwrap the naming to explicitly export and have the tools support that as well:
from X import Y Z = Y
Right. The key observation is that you wouldn't write "from X import Y as Y", because it's redundant, so we can give it a different meaning (if only for tooling).
So do you think we can just update PEP 484? It currently says ``` * Modules and variables imported into the stub are not considered exported from the stub unless the import uses the ``import ... as ...`` form or the equivalent ``from ... import ... as ...`` form. ``` We could amend this (as we've done in a few other places) by adding the following: ``` (*UPDATE:* To clarify, the intention here is that only names using the form ``X as X`` will be exported, i.e. the name before and after ``as`` must be the same.) ``` That still doesn't say anything about non-stub modules, but at least it states the rule for stubs more clearly.
Unfortunately we'd have to fix mypy and typeshed, since mypy apparently implemented the wider rule that any use of `as ...` constitutes an export. (I never knew this!)
As mentioned before, I'm in favour of the amendment. If we go through with it, https://github.com/python/typeshed/pull/4586 is most of the needed typeshed changes and I can update mypy as well.
On Tue, 29 Sep 2020 at 14:30, Guido van Rossum guido@python.org wrote:
On Tue, Sep 29, 2020 at 12:19 PM Brett Cannon brett@python.org wrote:
On Mon, Sep 28, 2020 at 2:10 PM Guido van Rossum guido@python.org wrote:
I just realize there is an ambiguity in PEP 484 regarding the ‘import ... as ...‘ syntax (and the other form).
I’ve always interpreted (and intended) this to mean ‘[from ...] import X as X’ (i.e. a redundant ‘as’ clause) and that is how it’s used in typeshed.
But it seems you are interpreting it as anything that uses an ‘as’ clause (as in Brett’s ‘dabuiltins’ example). And the text can indeed be read that way.
I think the key difference in view between the two interpretations is whether you view the `as` clause as part of the import statement or a convenient/implied assignment that's done post-import.
So if the semantics were `as` clauses that lead to a name lacking a leading underscore were consider exposed, that trips my brain up because now because _some_ forms of import dictate my exporting via a special rule that I have to avoid while other don't:
- `import X`; not exported
- `from X import Y`; not exported
- `from X import Y as Z`; exported; brain goes, "why is this form of
import special when it comes to naming conventions and not the other forms?"
The `from X import Y as Y` is an explicit action where I'm opting to inline the exporting of the symbol makes more sense to me since it is not a normal thing to do and I have to actively think of the opt-in. Otherwise I assume people can unwrap the naming to explicitly export and have the tools support that as well:
from X import Y Z = Y
Right. The key observation is that you wouldn't write "from X import Y as Y", because it's redundant, so we can give it a different meaning (if only for tooling).
So do you think we can just update PEP 484? It currently says
* Modules and variables imported into the stub are not considered exported from the stub unless the import uses the ``import ... as ...`` form or the equivalent ``from ... import ... as ...`` form.We could amend this (as we've done in a few other places) by adding the following:
(*UPDATE:* To clarify, the intention here is that only names using the form ``X as X`` will be exported, i.e. the name before and after ``as`` must be the same.)That still doesn't say anything about non-stub modules, but at least it states the rule for stubs more clearly.
Unfortunately we'd have to fix mypy and typeshed, since mypy apparently implemented the wider rule that any use of `as ...` constitutes an export. (I never knew this!)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/ _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
I'm in favor of this amendment too. It makes sense to use only the redundant form to have this special meaning.
I'll need to update pyright as well, but the change is straightforward. I'll wait until the updates go into typeshed. (Thanks Shantanu!)
This will represent a "breaking change" that could impact other type stubs in the wild, but I don't think the impact will be significant.
-Eric
--- Eric Traut Microsoft
I'm also in favour of the amendment if that matters. 😉
On Tue, Sep 29, 2020 at 3:00 PM Eric Traut eric@traut.com wrote:
I'm in favor of this amendment too. It makes sense to use only the redundant form to have this special meaning.
I'll need to update pyright as well, but the change is straightforward. I'll wait until the updates go into typeshed. (Thanks Shantanu!)
This will represent a "breaking change" that could impact other type stubs in the wild, but I don't think the impact will be significant.
-Eric
Eric Traut Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: brett@python.org
Okay, I sent a PR: https://github.com/python/peps/pull/1630
On Wed, Sep 30, 2020 at 10:50 AM Brett Cannon brett@python.org wrote:
I'm also in favour of the amendment if that matters. 😉
On Tue, Sep 29, 2020 at 3:00 PM Eric Traut eric@traut.com wrote:
I'm in favor of this amendment too. It makes sense to use only the redundant form to have this special meaning.
I'll need to update pyright as well, but the change is straightforward. I'll wait until the updates go into typeshed. (Thanks Shantanu!)
This will represent a "breaking change" that could impact other type stubs in the wild, but I don't think the impact will be significant.
Thanks everyone for the feedback on this thread.
I've published a new version of pyright that supports a new "--verifytypes" mode. It analyzes a py.typed package, enumerates all of the symbols that comprise its "public interface" and determines which of these symbols are fully typed. In cases where a symbol is untyped or partially typed, a detailed diagnostic message is provided to help the library maintainer address the omission.
It also reports the percentage of symbols that are completely typed. Here's the current "type completeness score" for several popular py.typed packages:
* yarl: 100% * attrs: 100% * pyrsistent: 88% * aiohttp: 79% * packaging: 68% * pydantic: 65% * starlette: 60% * ask_sdk_core: 49% * fastapi: 45% * pytest: 43% * torch: 38% * tornado: 36%
Detailed documentation can be found [here](https://github.com/microsoft/pyright/blob/master/docs/typed-libraries.md)
Feedback is welcome.
-Eric
-- Eric Traut Contributor to Pyright and Pylance Microsoft
participants (4)
-
Brett Cannon -
Eric Traut -
Guido van Rossum -
Shantanu Jain