There's been growing interest in supporting PEP-484 style type annotations
in NumPy: https://github.com/numpy/numpy/issues/7370
This would allow NumPy users to add type-annotations to their code that
uses NumPy, which they could check with mypy, pycharm or pytype. For
def f(x: np.ndarray) -> np.ndarray:
"""Identity function on a NumPy array."""
Eventually, we could include data types and potentially array shapes as
part of the type. This gets quite a bit more complicated, and to do in a
really satisfying way would require new features in Python's typing system.
To help guide discussion, I wrote a doc describing use-cases and needs for
typing array shapes in more detail:
Nathaniel Smith and I recently met with group in San Francisco interested
in this topic, including several mypy/typeshed developers (Jelle Zijlstra
and Ethan Smith). We discussed and came up with a plan for moving forward:
1. Release basic type stubs for numpy.ndarray without dtypes or shapes, as
separate "numpy_stubs" package on PyPI per PEP 561. This will let us
iterate rapidly on (experimental) type annotations without coupling to
NumPy's release cycle.
2. Add support for dtypes in ndarray type-annotations. This might be as
simple as writing np.ndarray[np.float64], but will need a decision about
appropriate syntax for shape typing to ensure that this is forwards
compatible with typing shapes. Note: this will likely require minor changes
to NumPy itself, e.g., to add __class_getitem__ per PEP 560.
3. Add support for shapes in ndarray type-annotations, and define a broader
standard for typing array shapes. This will require collaboration with
type-checker developers on the required typing features (for details, see
my doc above). Eventually, this may entail writing a PEP.
I'm writing to gauge support for this general plan, and specifically to get
support for step 1.
I wondered if the move to python3-only starting with numpy 1.17 would
be a good reason to act on what we all seem to agree: that the matrix
class was a bad idea, with its overriding of multiplication and lack
of support for stacks of matrices. For 1.17, minimum python supposedly
is >=3.5, so we will be guaranteed to have the matrix multiply
operator @ available, and hence there is arguably even less of a case
for keeping the matrix class; removing it would allow taking out quite
a bit of accumulated special-casing (the immediate reasons for writing
this were gh-10123 and 10132).
What do people think? If we do go in this direction, we might want to
add PendingDeprecationWarning for 1.15 (maybe DeprecationWarning for
python3; for python2 matrix would never disappear).
All the best,
On Wed, 2017-11-29 at 12:00 -0500, numpy-discussion-request(a)python.org
> Date: Wed, 29 Nov 2017 14:56:28 +0000 (UTC)
> From: "ZHUO QL (KDr2)" <zhuoql(a)yahoo.com>
> To: Discussion of Numerical Python <numpy-discussion(a)python.org>
> Subject: [Numpy-discussion] Is there a way that indexing a matrix of
> data with a matrix of indices?
> Message-ID: <360382279.3966234.1511967388050(a)mail.yahoo.com>
> Content-Type: text/plain; charset="utf-8"
> Hi, all
> - D, is the data matrix, its shape is? M x N- I, is the indices
> matrix, its shape is M x K,? K<=N
> Is there a efficient way to get a Matrix R with the same shape of I
> so that R[x,y] = D[x, I[x,y]] ?
> A nested for-loop or list-comprehension is too slow for me.??
I don't know if this will be substantially faster, but you can try the
I += np.array(range(M))[:, np.newaxis] * N
R = D.ravel()[I.ravel()].reshape((M, K))
> ZHUO QL (KDr2) http://kdr2.com
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/numpy-
- D, is the data matrix, its shape is M x N- I, is the indices matrix, its shape is M x K, K<=N
Is there a efficient way to get a Matrix R with the same shape of I so that R[x,y] = D[x, I[x,y]] ?
A nested for-loop or list-comprehension is too slow for me.
ZHUO QL (KDr2) http://kdr2.com
See Thomas's reply quoted below (it was rejected by the mailing list since
he's not subscribed):
On Nov 9, 2017 01:24, "Thomas Kluyver" <thomas(a)kluyver.me.uk> wrote:
On Thu, Nov 9, 2017, at 08:52 AM, Nathaniel Smith wrote:
On Nov 8, 2017 23:59, "Ralf Gommers" <ralf.gommers(a)gmail.com> wrote:
Regarding http://www.python3statement.org/: I'd say that as long as there
are people who want to spend their energy on the LTS release (contributors
*and* enough maintainer power to review/merge/release), we should not
actively prevent them from doing that.
Yeah, agreed. I don't feel like this is incompatible with the spirit of
python3statement.org, though looking at the text I can see how it's not
clear. My guess is they'd be happy to adjust the text, especially if it
lets them add numpy :-). CC'ing Thomas and Matthias.
Thanks Nathaniel. We have (IMO) left a degree of deliberate ambiguity
around precisely what 'drop support' means, because it's not going to be
the same for all projects. The nature of open source also means that there
can be ambiguity over what 'support' entails and who is considered part of
I would say that the idea of the statement is compatible with an LTS
release series receiving critical bugfixes beyond 2020, while the main
energy of the project is focused on Py3-only feature releases.
[If numpy-discussion doesn't allow non-member posts, feel free to pass this
on or quote it in on-list messages]
I need to make use of the limited numpy API access Pybind11 gives, in order
to add a feature to it. It seems to give access to functions from
numpy_api.py . I need to use PyArray_GETITEM and PyArray_SETITEM in
order to get and set array elements , these functions / macros are not
exposed via numpy_api.py, but are in `numpy/ndarraytypes.h`.
We were wondering why aren't PyArray_GETITEM and PyArray_SETITEM exposed
like the rest of numpy API? Is it possible to replicate the behavior using
the members exposed in numpy_api.py ? Any help would be appreciated.
Hi NumPy and SciPy developers,
Apparently there is some work afoot to update the BLAS standard, with
a working document here:
This seems like something where we might want to get involved in, so
that the new standard works for us, and James Demmel (the first author
on that proposal and a professor here at Berkeley) suggested they'd be
interested to hear our thoughts.
I'm not sure exactly what the process is here -- apparently there have
been some workshops, and there was going to be a BoF today at
Supercomputing, but I don't know what the schedule is or how they'll
be making decisions. It's possible for anyone interested to click on
that google doc above and make "suggestions", but it seems like maybe
it would be useful for the NumPy/SciPy teams to come up with some sort
of shared document on what we want?
I'm really, really not the biggest linear algebra expert on these
lists, so I'm hoping those with more experience will jump in, but to
get started here are some initial ideas for things we might want to
- Support for arbitrary strided memory layout
- Replacing xerbla with proper error codes (already in that proposal)
- There's some discussion about NaN handling where I think we might
have opinions. (Am I remember right that currently we have to check
for NaNs ourselves all the time because there are libraries that blow
up if we don't, and we don't know which ones those are?)
- Where the spec ends up giving implementors flexibility, some way to
detect at compile time what options they chose.
Nathaniel J. Smith -- https://vorpus.org
Thought I'd toss this out there. I'm tending towards better sooner than
later in dropping Python 2.7 support as we are starting to run up against
places where we would like to use Python 3 features. That is particularly
true on Windows where the 2.7 compiler is really old and lacks C99
compatibility. In any case, the timeline I've been playing with is to keep
Python 2.7 support through 2018, which given our current pace, would be for
NumPy 1.15 and 1.16. After that 1.16 would become a long term support
release with backports of critical bug fixes up until the time that Python
2.7 support officially ends. In that timeline, NumPy 1.17 would drop
support for 2.7. That proposed schedule is subject to change pending
developments and feed back.
The main task I think is needed before dropping 2.7 is better handling of
unicode strings and bytes. There is the #4208
<https://github.com/numpy/numpy/pull/4208> PR that makes a start on that.
If there are other things that folks think are essential, please mention
them here. If nothing else, we can begin planning for the transition even
if the schedule changes.
I wanted to give everyone an update on what's going on with the NumPy
grant . As you may have noticed, things have been moving a bit
slower than originally hoped -- unfortunately my health is improving
but has continued to be rocky .
Fortunately, I have awesome co-workers, and BIDS has an institutional
interest/mandate for figuring out how to make these things happen, so
after thinking it over we've decided to reorganize how we're doing
things internally and split up the work to let me focus on the core
technical/community aspects without getting overloaded. Specifically,
Fernando Pérez and Jonathan Dugan  are taking on PI/administration
duties, Stéfan van der Walt will focus on handling day-to-day
management of the incoming hires, and Nelle Varoquaux & Jarrod Millman
will also be joining the team (exact details TBD).
This shouldn't really affect any of you, except that you might see
some familiar faces with @berkeley.edu emails becoming more engaged.
I'm still leading the Berkeley effort, and in any case it's still
ultimately the community and NumPy steering council who will be making
decisions about the project – this is just some internal details about
how we're planning to manage our contributions. But in the interest of
full transparency I figured I'd let you know what's happening.
In other news, the job ad to start the official hiring process has now
been submitted for HR review, so it should hopefully be up soon --
depending on how efficient the bureaucracy is. I'll definitely let
everyone know as soon as its posted.
I'll also be giving a lunch talk at BIDS tomorrow to let folks locally
know about what's going on, which I think will be recorded – I'll send
around a link after in case others are interested.
Nathaniel J. Smith -- https://vorpus.org