I thought I'd add a little more specifically about the kind of graphics/point cloud work I'm doing right now at Thinkbox, and how it relates. To echo Francesc's point about NumPy already being an industry standard, within the VFX/graphics industry there is a reference platform definition on Linux, and the most recent iteration of that specifies a version of NumPy. It also includes a bunch of other open source libraries worth taking a look at if you haven't seen them before: http://www.vfxplatform.com/ Point cloud/particle system data, mesh geometry, numerical grids (both dense and sparse), and many other primitive components in graphics are built out of arrays. What NumPy represents for that kind of data is amazing. The extra baggage of an API tied to the CPython GIL can be a hard pill to swallow, though, and this is one of the reasons I'm hopeful that as DyND continues maturing, it can make inroads into places NumPy hasn't been able to. Thanks, Mark On Wed, Aug 26, 2015 at 9:45 AM, Irwin Zaid <izaid@continuum.io> wrote:
Hello everyone,
Mark and I thought it would be good to weigh in here and also be explicitly around to discuss DyND. To be clear, neither of us has strong feelings on what NumPy *should* do -- we are both long-time NumPy users and we both see NumPy being around for a while. But, as Francesc mentioned, there is also the open question of where the community should be implementing new features. It would certainly be nice to not have duplication of effort, but a decision like that can only arise naturally from a broad consensus.
Travis covered DyND's history and it's relationship with Continuum pretty well, so what's really missing here is what DyND is, where it is going, and how long we think it'll take to get there. We'll try to stick to those topics.
We designed DyND to fill what we saw as fundamental gaps in NumPy. These are not only missing features, but also limitations of its architecture. Many of these gaps have been mentioned several times before in this thread and elsewhere, but a brief list would include: better support for missing values, variable-length strings, GPUs, more extensible types, categoricals, more datetime features, ... Some of these were indeed on Nathaniel's list and many of them are already working (albeit sometimes partially) in DyND.
And, yes, we strongly feel that NumPy's fundamental dependence on Python itself is a limitation. Why should we not take the fantastic success of NumPy and generalize it across other languages?
So, we see DyND is having a twofold purpose. The first is to expand upon the kinds of data that NumPy can represent and do computations upon. The second is to provide a standard array package that can cross the language barrier and easily interoperate between C++, Python, or whatever you want.
DyND, at the moment, is quite functional in some areas and lacking a bit in others. There is no doubt that it is still "experimental" and a bit unstable. But, it has advanced by a lot recently, and we are steadily working towards something like a version 1.0. In fact, DyND's internal C++ architecture stabilized some time ago -- what's missing now is really solid coverage of some common use cases, alongside up-to-date Python bindings and an easy installation process. All of these are in progress and advancing as quick as we can make them.
On the other hand, we are also building out some other features. To give just one example that might excite people, DyND now has Numba interoperability -- one can write DyND's equivalent of a ufunc in Python and, with a single decorator, have a broadcasting or reduction callable that gets JITed or (soon) ahead-of-time compiled.
Over the next few months, we are hopeful that we can get DyND into a state where it is largely usable by those familiar with NumPy semantics. The reason why we can be a bit more aggressive in our timeline now is because of the great support we are getting from Continuum.
With all that said, we are happy to be a part of of any broader conversation involving NumPy and the community.
All the best,
Irwin and Mark
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion