On Sat, Nov 25, 2017 at 7:21 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:

A question of perhaps broader scope than what you were asking for, and more out of curiosity than anything else, but can one mix type annotations with others? E.g., in astropy, we have a decorator that looks for units in the annotations (not dissimilar from dtype, I guess). Could one mix annotations or does one have to stick with one purpose?

Hi Marten, I took a look at Astropy's units decorator: http://docs.astropy.org/en/stable/api/astropy.units.quantity_input.html Annotations for return values that "coerce" units would be hard to make compatible with typing, because type annotations are used to check programs, not change runtime semantics. But in principle, I think you could even make a physical units library that relies entirely on static type checking for correctness, imposing almost no run-time overhead at all. There are several examples for Haskell: https://wiki.haskell.org/Physical_units I don't see any obvious way to support to mixing of annotations for typing and runtime effects in the same function, though doing so in the same program might be possible. My guess is that the preferred way to do this would be to use decorators for runtime changes to arguments, and keep annotations for typing. The Python community seems to be standardizing on using annotations for typing: https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations Cheers, Stephan

Thoughts on basing this on a more generic Array type rather than the np.ndarray? I can imagine other nd-array libraries (XArray, Tensorflow, Dask.array) wanting to reuse this work. For dask.array in particular we would want to copy this entirely, but we probably can't specify that dask.arrays are np.ndarrays. It would be nice to ensure that the container type was swappable. On Sat, Nov 25, 2017 at 6:31 PM, Matthew Rocklin wrote:

Can you make a case for the usefulness numpy annotations? What benefits to you want to achieve and how will annotation aid in getting there.

1. Error checking on large codebases with systems like MyPy 2. Hinting and error checking at code-writing time with systems like Jedi "Hey, this function expects a 2-d square array but you just passed in a 3d array with irregular sizes" 3. Supporting systems like the Cython compiler with type information, allowing them to speedup pure-python code without switching to the Cython language

On Sat, Nov 25, 2017 at 6:12 PM, Stephan Hoyer wrote:

...

On Sat, Nov 25, 2017 at 7:21 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:

...

A question of perhaps broader scope than what you were asking for, and more out of curiosity than anything else, but can one mix type annotations with others? E.g., in astropy, we have a decorator that looks for units in the annotations (not dissimilar from dtype, I guess). Could one mix annotations or does one have to stick with one purpose?

Hi Marten,

I took a look at Astropy's units decorator: http://docs.astropy.org/en/stable/api/astropy.units.quantity_input.html

Annotations for return values that "coerce" units would be hard to make compatible with typing, because type annotations are used to check programs, not change runtime semantics. But in principle, I think you could even make a physical units library that relies entirely on static type checking for correctness, imposing almost no run-time overhead at all. There are several examples for Haskell: https://wiki.haskell.org/Physical_units

I don't see any obvious way to support to mixing of annotations for typing and runtime effects in the same function, though doing so in the same program might be possible. My guess is that the preferred way to do this would be to use decorators for runtime changes to arguments, and keep annotations for typing. The Python community seems to be standardizing on using annotations for typing: https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations

Cheers, Stephan

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

I'm strongly in support of this proposal. Type annotations have really helped me write more correct code. I started working on numpy type stubs a few months ago. I needed a mypy plugin to support shape-aware functions. Those whole thing is pretty tricky. Still very WIP, but I'll clean them up a little bit and opensource it shortly. -Robert On Sun, Nov 26, 2017 at 1:58 PM, Stephan Hoyer wrote:

On Sat, Nov 25, 2017 at 3:34 PM Matthew Rocklin wrote:

...

Thoughts on basing this on a more generic Array type rather than the np.ndarray? I can imagine other nd-array libraries (XArray, Tensorflow, Dask.array) wanting to reuse this work. For dask.array in particular we would want to copy this entirely, but we probably can't specify that dask.arrays are np.ndarrays. It would be nice to ensure that the container type was swappable.

Yes, absolutely. I do briefly mention this in my longer doc (see the "Syntax" section). This is also one of my personal goals for this project.

This will be most relevant when we start working on typing support for array shapes and broadcasting: details like data types can be more library specific, and can probably be expressed with the existing generics system in the typing module.

After we do some experimentation to figure out appropriate syntax and semantics for array shape typing, I would like to standardize the rules for typing multi-dimensional arrays in Python. This will probably entail writing a PEP, so we can add appropriate base classes in the typing module. I view this as the natural complement to existing standard library features that make it easier to interchange between multiple multi-dimensional array libraries, such as memory views and the buffer protocol.

...

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

-- -Robert

Here's the code: https://github.com/rmcgibbo/numpy-mypy. It's not 100% working yet, but it can do simple stuff, like inferring the shape of arrays created from np.zeros(literal_tuple), and fixing out the shape of the result of an indexing operation (i.e. https://github.com/rmcgibbo/numpy-mypy/blob/master/tests/test_indexing.py). To implement it, I have the beginnings of the stubs that you'd expect, borrowed from https://github.com/machinalis/mypy-data and then revised. Then, on top of that, I wrote some special type-level functions that are implemented inside of a mypy plugin. So, for example, the stub's signature for np.sum is def sum(a: ndarray[_S, _D], axis: AxesType=None, dtype: DtypeType=None, out: ndarray=None, keepdims: bool=False) -> ndarray[_InferDtypeWithDefault[_S], _InferNdimsReduction[_D]]: ... When the stub is applied, the resut's dtype is determined application of the _InferDtypeWithDefault type function, which defaults, as expected, to the dtype of the input array but checks of that was overridden dtype=None kwarg as well. And the _InferNdimsReduction type function has to check the axis and keepdims arguments as well. It's by no means ready for real users, but I hope this is a useful place to build from. Any feedback or contributions would be appreciated. -Robert On Tue, Nov 28, 2017 at 2:04 PM, Stephan Hoyer wrote:

On Tue, Nov 28, 2017 at 5:11 PM Robert T. McGibbon wrote:

...

I'm strongly in support of this proposal. Type annotations have really helped me write more correct code.

I started working on numpy type stubs a few months ago. I needed a mypy plugin to support shape-aware functions. Those whole thing is pretty tricky. Still very WIP, but I'll clean them up a little bit and opensource it shortly.

Great to hear -- I'd love to see what this looks like, or hear any lessons you learned from the experience!

Actual experience using and writing such a type checker gives you a valuable perspective to share, as opposed to my speculation.

Cheers, Stephan

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

-- -Robert

On Sat, Nov 25, 2017 at 3:09 PM, Juan Nunez-Iglesias wrote:

This is a complete outsider’s perspective but

(a) it would be good if NumPy type annotations could include an “array_like” type that allows lists, tuples, etc.

I'm sure this will exist.

(b) I’ve always thought (since PEP561) that it would be cool for type annotations to replace compiler type annotations for e.g. Cython and Numba. Is this in the realm of possibility for the future?

It turns out that the PEP 484 type system is *mostly* not useful for this. They're really designed for checking consistency across a large code-base, not for enabling compiler speedups. For example, if you annotate something as an int, that means "this object is a subclass of int". This is enough to let mypy catch your mistake if you accidentally pass in a float instead, but it's not enough to tell you anything at all about the object's behavior -- you could make a wacky int subclass that acts like a string or something. Probably there are some benefits that compilers can get from PEP 484 annotations, but you should think of them as largely an orthogonal thing. -n -- Nathaniel J. Smith -- https://vorpus.org

Hi! 2017-11-26 4:31 GMT+03:00 Juan Nunez-Iglesias :

On 26 Nov 2017, 12:27 PM +1100, Nathaniel Smith , wrote:

It turns out that the PEP 484 type system is *mostly* not useful for this. They're really designed for checking consistency across a large code-base, not for enabling compiler speedups. For example, if you annotate something as an int, that means "this object is a subclass of int". This is enough to let mypy catch your mistake if you accidentally pass in a float instead, but it's not enough to tell you anything at all about the object's behavior -- you could make a wacky int subclass that acts like a string or something.

I have subscribed to many lists, although I am not an active participant in them. Nevertheless this topic of using the type annotation in their projects was discussed several times on all Cython-like channels (and it becomes much more acute now days). "Misconceptions" arise both for ordinary users and developers, but I have never seen anyone to write clearly why the application of type annotation in Cython (and similar projects) is impossible or not reasonable. Maybe someone close to the topic has the time and energy to sum up and write a brief summary of how to perceive them and why they should be viewed "orthogonal"? Maybe I'm looking too superficially at this topic. But both Mypy and Cython perform type checking. From the Cython point of view I do not see any pitfalls, type checking and type conversions are what Cython is doing right now during compilation (and looks at types as strictly as necessary).

From Mypy's point of view, it's possible that it can delegate all this stuff, using a certain option, on a project's related type checker (which can be much stricter in its assumptions)

With kind regards, -gdg

(a) it would be good if NumPy type annotations could include an “array_like” type that allows lists, tuples, etc. I think that would be a sequence — already supported by the Typing system. (b) I’ve always thought (since PEP561) that it would be cool for type annotations to replace compiler type annotations for e.g. Cython and Numba. Is this in the realm of possibility for the future? Well, this was brought up early in the Typing discussion, and it was made clear that these kinds of truly static types, as needed by Cython, was a non-goal of the project. That being said, perhaps it could be made to work with a bunch of additional type objects. And we should lol lol to Cython for ideas about how to type numpy arrays. One note: in addition to shape (rank) and types, there is contiguous and C or F order. That may want to be considered. -CHB

On Tue, Dec 5, 2017 at 10:04 AM, Stephan Hoyer wrote:

This discussion has died down, but I don't want to lose momentum .

It sounds like there is at least strong interest from a subset of our community in type annotations. Are there any objections to the first part of my plan, to start developing type stubs for NumPy in separate repository?

I think there's been plenty of time for folks to object to this if they wanted, so we can assume consensus until we hear otherwise. -n -- Nathaniel J. Smith -- https://vorpus.org

OK, in that case let's get to work over in https://github.com/numpy/numpy_stubs! On Tue, Dec 5, 2017 at 2:43 PM Fernando Perez wrote:

On Tue, Dec 5, 2017 at 2:19 PM, Nathaniel Smith wrote:

...

...

This discussion has died down, but I don't want to lose momentum .

It sounds like there is at least strong interest from a subset of our community in type annotations. Are there any objections to the first

On Tue, Dec 5, 2017 at 10:04 AM, Stephan Hoyer wrote: part of

...

my plan, to start developing type stubs for NumPy in separate repository?

I think there's been plenty of time for folks to object to this if they wanted, so we can assume consensus until we hear otherwise.

peanut_gallery.throw('+1', thanks=True) # very happy to see this moving forward _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion