[Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

[Irwin Zaid]

Hello everyone,

Mark and I thought it would be good to weigh in here and also be explicitly
around to discuss DyND. To be clear, neither of us has strong feelings on
what NumPy *should* do -- we are both long-time NumPy users and we both see
NumPy being around for a while. But, as Francesc mentioned, there is also
the open question of where the community should be implementing new
features. It would certainly be nice to not have duplication of effort, but
a decision like that can only arise naturally from a broad consensus.

Travis covered DyND's history and it's relationship with Continuum pretty
well, so what's really missing here is what DyND is, where it is going, and
how long we think it'll take to get there. We'll try to stick to those topics.

We designed DyND to fill what we saw as fundamental gaps in NumPy. These are
not only missing features, but also limitations of its architecture. Many of
these gaps have been mentioned several times before in this thread and
elsewhere, but a brief list would include: better support for missing
values, variable-length strings, GPUs, more extensible types, categoricals,
more datetime features, ... Some of these were indeed on Nathaniel's list
and many of them are already working (albeit sometimes partially) in DyND.

And, yes, we strongly feel that NumPy's fundamental dependence on Python
itself is a limitation. Why should we not take the fantastic success of
NumPy and generalize it across other languages?

So, we see DyND is having a twofold purpose. The first is to expand upon the
kinds of data that NumPy can represent and do computations upon. The second
is to provide a standard array package that can cross the language barrier
and easily interoperate between C++, Python, or whatever you want.

DyND, at the moment, is quite functional in some areas and lacking a bit in
others. There is no doubt that it is still "experimental" and a bit
unstable. But, it has advanced by a lot recently, and we are steadily
working towards something like a version 1.0. In fact, DyND's internal C++
architecture stabilized some time ago -- what's missing now is really solid
coverage of some common use cases, alongside up-to-date Python bindings and
an easy installation process. All of these are in progress and advancing as
quick as we can make them.

On the other hand, we are also building out some other features. To give
just one example that might excite people, DyND now has Numba
interoperability -- one can write DyND's equivalent of a ufunc in Python
and, with a single decorator, have a broadcasting or reduction callable that
gets JITed or (soon) ahead-of-time compiled.

Over the next few months, we are hopeful that we can get DyND into a state
where it is largely usable by those familiar with NumPy semantics. The
reason why we can be a bit more aggressive in our timeline now is because of
the great support we are getting from Continuum.

With all that said, we are happy to be a part of of any broader conversation
involving NumPy and the community.

All the best,

Irwin and Mark