type hints for C extension modules?
Hello, I'm looking to understand how to best use type hints in a mixed C/Cython/Python2/Python3 code base (pygame). *[0] Such that: - Tools like mypy would work. - Sphinx (the doc tool we use) would work for documentation of the types (Perhaps using: https://github.com/agronholm/sphinx-autodoc-typehints) Note it uses typing.get_type_hints (and therefore __annotations__). - We could maintain it inside our code base (either eventually or from the beginning). - Python2 would be supported for type checking. (it's not a hard requirement). - typing.get_type_hints would work inside a python repl. - There are not lots of extra files would be nice. - No, or very little extra load time. This is important for end users not doing typechecking or documentation generation. (ie, people using apps made using pygame). - Would prefer a clean way if possible, even if it's not implemented yet. It seems that C extensions are currently only supported well by using type stubs? *[1] However typing.get_type_hints does not seem to use stubs. Can we define types in stubs, and load them into python at runtime somehow? Such that the types are stored in __annotations__. Failing that, being able to load them somehow so that sphinx can use them for documentation would be useful. I see retype *[3] uses typed_ast *[4] for parsing stub files. Using typed_ast.ast3 on python 3, and typed_ast.ast27 on python2 seems appropriate. This could perhaps be one path towards allowing annotations defined in stub files to be used by sphinx. Any pointers or ideas would be appreciated :) cheers! [0] Related pygame issue: https://github.com/pygame/pygame/issues/668 [1] https://www.python.org/dev/peps/pep-0561/ [2] issue that spawned 0561 https://github.com/python/typing/issues/84 [3] retype https://github.com/ambv/retype/blob/master/retype.py#L162 [4] typed_ast https://github.com/python/typed_ast
On 8 Dec 2018, at 3:52 am, René Dudfield <renesd@gmail.com<mailto:renesd@gmail.com>> wrote: Hello, I'm looking to understand how to best use type hints in a mixed C/Cython/Python2/Python3 code base (pygame). *[0] Such that: * Tools like mypy would work. * Sphinx (the doc tool we use) would work for documentation of the types (Perhaps using: https://github.com/agronholm/sphinx-autodoc-typehints) Note it uses typing.get_type_hints (and therefore __annotations__). * We could maintain it inside our code base (either eventually or from the beginning). * Python2 would be supported for type checking. (it's not a hard requirement). * typing.get_type_hints would work inside a python repl. * There are not lots of extra files would be nice. * No, or very little extra load time. This is important for end users not doing typechecking or documentation generation. (ie, people using apps made using pygame). * Would prefer a clean way if possible, even if it's not implemented yet. It seems that C extensions are currently only supported well by using type stubs? *[1] However typing.get_type_hints does not seem to use stubs. Can we define types in stubs, and load them into python at runtime somehow? Such that the types are stored in __annotations__. Failing that, being able to load them somehow so that sphinx can use them for documentation would be useful. I see retype *[3] uses typed_ast *[4] for parsing stub files. Using typed_ast.ast3 on python 3, and typed_ast.ast27 on python2 seems appropriate. This could perhaps be one path towards allowing annotations defined in stub files to be used by sphinx. Any pointers or ideas would be appreciated :) cheers! [0] Related pygame issue: https://github.com/pygame/pygame/issues/668 [1] https://www.python.org/dev/peps/pep-0561/ [2] issue that spawned 0561 https://github.com/python/typing/issues/84 [3] retype https://github.com/ambv/retype/blob/master/retype.py#L162 [4] typed_ast https://github.com/python/typed_ast The problem with C/Cython modules is that the builtin-function type doesn't have space for the annotations dict, so there's no room to store that. However for Cython you can enable the "binding" compiler directive to make it use its own function type instead, which does store the annotations. It might cause a slight performance decrease, since it's not the normal function type it'll miss out on some interpreter special-casing speedups. But you can use annotations exactly like normal Python code. Cython itself does have rudimentary support for type hints, so annotating a parameter as float will convert it to a machine double for example. Pure C modules will have to use type stubs. I don't think there's any project with a simple stubs parser other than inside MyPy for example, you might have to write that and upgrade the Sphinx extension to use it.
On Fri, 7 Dec 2018 at 17:51, René Dudfield <renesd@gmail.com> wrote:
Hello,
I'm looking to understand how to best use type hints in a mixed C/Cython/Python2/Python3 code base (pygame). *[0]
Such that:
- Tools like mypy would work. - Sphinx (the doc tool we use) would work for documentation of the types (Perhaps using: https://github.com/agronholm/sphinx-autodoc-typehints) Note it uses typing.get_type_hints (and therefore __annotations__). - We could maintain it inside our code base (either eventually or from the beginning). - Python2 would be supported for type checking. (it's not a hard requirement). - typing.get_type_hints would work inside a python repl. - There are not lots of extra files would be nice. - No, or very little extra load time. This is important for end users not doing typechecking or documentation generation. (ie, people using apps made using pygame). - Would prefer a clean way if possible, even if it's not implemented yet.
It seems that C extensions are currently only supported well by using type stubs? *[1] However typing.get_type_hints does not seem to use stubs.
Can we define types in stubs, and load them into python at runtime somehow? Such that the types are stored in __annotations__. Failing that, being able to load them somehow so that sphinx can use them for documentation would be useful. I see retype *[3] uses typed_ast *[4] for parsing stub files. Using typed_ast.ast3 on python 3, and typed_ast.ast27 on python2 seems appropriate. This could perhaps be one path towards allowing annotations defined in stub files to be used by sphinx.
It looks like using stub files is the best option in your situation. Stub files can be loaded in Python runtime only if you escape forward references with quotes "". Even if you don't do this (which is a standard practice, since stubs aren't supported to be loaded at runtime), they are still syntactically valid Python and can be parsed even by the native Python ast module. The typed_ast is only used to support type comments, the latter are rarely used in stub files (even on Python 2) exactly because stub files aren't loaded at runtime. -- Ivan
participants (3)
-
Ivan Levkivskyi
-
René Dudfield
-
Spencer Brown