<div dir="ltr"><div><div><div><div>Travis,<br><br></div>Thank you for your perspective on this issue. Such input is always valuable in helping us see where we came from and where we might go.<br><br></div>My perspective on NumPy is fairly different, having come into Python right after the whole Numeric/NumArray transition to NumPy. One of the things that really sold me on NumPy was not only just how simple it was for me to use out of the box, but how easy it was to explicitly state that something needed to be of one type or another. The dtype notation was fairly simple and straight-forward.<br><br>-- We should not underestimate the value of simple to write and simple to read notations in Python --<br><br></div>We can go ahead and put as many bells and whistles into the underlaying infrastructure as you want, but if we can't design a simple notation language to utilize it, then it will never catch on. This isn't criticism of the work being done in dynd or the or the other projects, rather, it is a call for innovation. I don't know how I would design such a notation language, but we need that "ah-ha!" moment from *somebody*.<br><br></div>I expressed this back at the NumPy BoF this summer. I would love an improved notation system that Matplotlib could take advantage of that would facilitate the plotting of more complicated graphs. But I am also not really interested in seeing NumPy turn into Pandas. Nothing wrong with Pandas; I just like the idea of modularity and I think it has suited the community well. Striking the right balance is going to be extremely important.<br><br>Cheers!<br>Ben Root<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 23, 2014 at 9:34 AM, Travis Oliphant <span dir="ltr"><<a href="mailto:travis@continuum.io" target="_blank">travis@continuum.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Sun, Sep 21, 2014 at 6:50 PM, Stephan Hoyer <span dir="ltr"><<a href="mailto:shoyer@gmail.com" target="_blank">shoyer@gmail.com</a>></span> wrote:<br></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet.<div><div><br></div><div>Most of these look like numpy arrays but with custom dtypes and type specific methods/properties. But clearly nobody is particularly excited about writing the the C necessary to implement custom dtypes [3]. Nor is do we need the ndarray ABI.</div><div><br></div><div>In many cases, writing C may not actually even be necessary for performance reasons, e.g., categorical can be fast enough just by wrapping an integer ndarray for the internal storage and using vectorized operations. And even if it is necessary, I think we'd all rather write Cython than C.</div><div><br></div><div>It's great for pandas to write its own ndarray-like wrappers (*not* subclasses) that work with pandas, but it's a shame that there isn't a standard interface like the ndarray to make these arrays useable for the rest of the scientific Python ecosystem. For example, pandas has loads of fixes for np.datetime64, but nobody seems to be up for porting them to numpy (I doubt it would be easy).</div><div><br></div><div>I know these sort of concerns are not new, but I wish I had a sense of what the solution looks like. Is anyone actively working on these issues? Does the fix belong in numpy, pandas, blaze or a new project? I'd love to get a sense of where things stand and how I could help -- without writing any C :).</div><div><div><br></div></div></div></div></blockquote><div><br></div></div></div><div>Hey Stephan, </div><div><br></div><div>There are not easy answers to your questions. The reason is that NumPy's dtype system is not extensible enough with its fixed set of "builtin" data-types and its bolted-on "user-defined" datatypes. The implementation was adapted from the *descriptor* notion that was in Numeric (written almost 20 years ago). While a significant improvement over Numeric, the dtype system in NumPy still has several limitations:</div><div><br></div><div> 1) it was not designed to add new fundamental data-types without breaking the ABI (most of the ABI breakage between 1.3 and 1.7 due to the addition of np.datetime has been pushed to a small corner but it is still there). </div><div><br></div><div> 2) The user-defined data-type system which is present is not well tested and likely incomplete: it was the best I could come up with at the time NumPy first came out with a bit of input from people like Fernando Perez and Francesc Alted. </div><div> </div><div> 3) It is far easier than in Numeric to add new data-types (that was a big part of the effort of NumPy), but it is still not as easy as one would like to add new data-types (either fundamental ones requiring recompilation of NumPy or 'user-defined' data-types requiring C-code. </div><div><br></div><div>I believe this system has served us well, but it needs to be replaced eventually. I think it can be replaced fairly seamlessly in a largely backward compatible way (though requiring re-compilation of dependencies). Fixing the dtype system is a fundamental effort behind several projects we are working on at Continuum: datashape, dynd, and numba. These projects are addressing fundamental limitations in a way that can lead to a significantly improved framework for scientific and tabular computing in Python. </div><div><br></div><div>In the mean-time, NumPy can continue to improve in small ways and in orthogonal ways (like the new __numpy_ufunc__ mechanism which allows ufuncs to work more seamlessly with different kinds of array-like objects). This kind of effort as well as the improved buffer protocol in Python, mean that multiple array-like objects can co-exist and use each-other's data. Right now, I think that is the best current way to address the data-type limitations of NumPy. </div><div><br></div><div>Another small project is possible today --- one could today use Numba or Cython to generate user-defined data-types for existing NumPy. That would be an interesting project and would certainly help to understand the limitations of the user-defined data-type framework without making people write C-code. You could use a meta-class and some code-generation techniques so that by defining a particular class you end-up with a user-defined data-type for NumPy. </div><div><br></div><div>Even while we have been addressing the fundamental limitations of NumPy with our new tools at Continuum, replacing NumPy is a big undertaking because of its large user-base. While I personally think that NumPy could be replaced for new users as early as next year with a combination of dynd and numba, the big install base of NumPy means that many people (including the company I work with, Continuum) will be supporting NumPy 1.X and Pandas and the rest of the NumPy-Stack for many years to come. </div><div><br></div><div>So, even if you see me working and advocating new technology, that should never be construed as somehow ignoring or abandoning the current technology base. I remain deeply interested in the success of the scientific computing community --- even though I am not currently contributing a lot of code directly myself. As dynd and numba mature, I think it will be clear to more people how to proceed. </div><div><br></div><div>For example, just recently the thought emerged that because dynd addresses some of the major needs that Pandas has, it may be possible very soon for dynd to replace NumPy as the foundational container for Pandas data-frames. Because Pandas use of the NumPy API is limited, this is an easier undertaking than having dynd replace NumPy itself. And given that the new data-types of dynd: missing-data, categorical types, variable-length strings, etc. are some of the key areas that Pandas has work-arounds for, it may be a straight-forward project. </div><div><br></div><div>For those not aware: dynd is cython code that wraps the C++ library libdynd. Currently libdynd is not complete, so working on dynd may require some improvements / fixes to libdynd. However, the dynd layer should be accessible to many people. The libdynd layer is also fairly straightforward C++. I strongly believe that the combination of libdynd and dynd is a much easier foundation to work on and maintain than the NumPy code base. I say this after having personally spent over a decade on the Numeric code-base and then the NumPy code base. The NumPy "C" code-base has been improved since I left it by the excellent work of several patient developers --- but it is not easy to transmit the knowledge necessary to understand the code-base sufficient to maintain it without creating backward compatibility issues. </div><div><br></div><div>So, while I continue to support the NumPy code base and its extensions (personally, through Numfocus, and through Continuum) and believe it will be relevant for many years, I also believe the future lies in renewing the NumPy code base with a combination of dynd and numba with more emphasis on the high-level APIs like pandas and blaze. The good news is that this means: 1) a lot more code in Python or Cython, 2) compatibility with the PyPy world as part of a long term effort to heal the rift that exists between scientific-use of Python and "web-use" of Python. </div><div><br></div><div>In the end, all of this is good news for Python and scientific computing. More and better tools will continue to be written with better interop between them. There are many places to jump in and help: dynd, libdynd, datashape, blaze, numba, scipy, scikits, numpy, pandas, and even a new project you create that enhances some aspect of any of these or does something like use Cython or Numba to create NumPy user-defined data-types from a Python class-specification. </div><div><br></div><div>I agree it can be hard to know where things will eventually end up and so therefore where to spend your effort. All I can tell you is what I've decided and where I am pushing and promoting. </div><div><br></div><div>Best,</div><div><br></div><div>-Travis</div><div><br></div></div><div dir="ltr"></div>
</div></div>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div>