Mailman 3 July 2011 - NumPy-Discussion

ANN: EPD 7.1 released
by Ilan Schnell 08 Jul '11

08 Jul '11

Hello, I am pleased to announce that EPD (Enthought Python Distribution) version 7.1 has been released. The most significant change is the addition of an "EPD Free" version, which has its own very liberal license, and can be downloaded and used free of any charge by anyone (not only academics). "EPD Free" includes a subset of the packages included in the full EPD. The highlights of this subset are: numpy, scipy, matplotlib, traits and chaco. To see which libraries are included in the free vs. full version, please see: http://www.enthought.com/products/epdlibraries.php In addition we have opened our PyPI build mirror for everyone. This means that one can type "enpkg xyz" for 10,000+ packages. However, there are still benefits to becoming an EPD subscriber. http://www.enthought.com/products/getepd.php Apart from the addition of "EPD Free", this release includes updates to over 30 packages, including numpy, scipy, ipython and ETS. We have also added PySide, Qt and MDP to this release. Please find the complete list of additions, updates and bug fixes in the change log: http://www.enthought.com/products/changelog.php About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python programming language, including over 90 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, and many other tools. EPD is currently available as a single-click installer for Windows XP, Vista and 7, MacOSX (10.5 and 10.6), RedHat 3, 4 and 5, as well as Solaris 10 (x86 and x86_64/amd64 on all platforms). All versions of EPD (32 and 64-bit) are free for academic use. An annual subscription including installation support is available for individual and commercial use. Additional support options, including customization, bug fixes and training classes are also available: http://www.enthought.com/products/epd_sublevels.php - Ilan

2 2

Re: [Numpy-discussion] Compiling numpy on Red-Hat Import Error with lapack_lite.so
by Jeffrey Spencer 08 Jul '11

08 Jul '11

That actually makes sense because I am not sure the gnu that it was compiled with but I think it is different. I have since compiled gcc myself, then python, and atlas libraries. Then I tried to install numpy. It go tthrough the install no worries and found the correct libraries. It stuffed when I tried to import it with this error: >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jspender/lib/python2.6/site-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/home/jspender/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/home/jspender/lib/python2.6/site-packages/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/home/jspender/lib/python2.6/site-packages/numpy/lib/polynomial.py", line 17, in <module> from numpy.linalg import eigvals, lstsq File "/home/jspender/lib/python2.6/site-packages/numpy/linalg/__init__.py", line 48, in <module> from linalg import * File "/home/jspender/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 23, in <module> from numpy.linalg import lapack_lite ImportError: /home/jspender/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_concat_string >>> Any ideas??? Cheers, Jeff On Fri, Jul 8, 2011 at 12:37 AM, Bruce Southey <bsouthey(a)gmail.com> wrote: > On 07/07/2011 05:23 AM, Jeffrey Spencer wrote: > > The error is below: > > > > creating build/temp.linux-x86_64-2.6/numpy/core/blasdot > > compile options: '-DATLAS_INFO="\"None\"" -Inumpy/core/blasdot > > -Inumpy/core/include > > -Ibuild/src.linux-x86_64-2.6/numpy/core/include/numpy > > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > > -Inumpy/core/src/umath -Inumpy/core/include > > -I/home/jspender/include/python2.6 > > -Ibuild/src.linux-x86_64-2.6/numpy/core/src/multiarray > > -Ibuild/src.linux-x86_64-2.6/numpy/core/src/umath -c' > > gcc: numpy/core/blasdot/_dotblas.c > > numpy/core/blasdot/_dotblas.c: In function ‘dotblas_matrixproduct’: > > numpy/core/blasdot/_dotblas.c:239: warning: comparison of distinct > > pointer types lacks a cast > > numpy/core/blasdot/_dotblas.c:257: warning: passing argument 3 of > > ‘*(PyArray_API + 2240u)’ from incompatible pointer type > > numpy/core/blasdot/_dotblas.c:292: warning: passing argument 3 of > > ‘*(PyArray_API + 2240u)’ from incompatible pointer type > > gcc -pthread -shared > > build/temp.linux-x86_64-2.6/numpy/core/blasdot/_dotblas.o > > -L/usr/local/lib -Lbuild/temp.linux-x86_64-2.6 -lf77blas -lcblas -latlas > > -o build/lib.linux-x86_64-2.6/numpy/core/_dotblas.so > > /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when > > searching for -lf77blas > > /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when > > searching for -lf77blas > > /usr/bin/ld: cannot find -lf77blas > > collect2: ld returned 1 exit status > > /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when > > searching for -lf77blas > > /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when > > searching for -lf77blas > > /usr/bin/ld: cannot find -lf77blas > > collect2: ld returned 1 exit status > > error: Command "gcc -pthread -shared > > build/temp.linux-x86_64-2.6/numpy/core/blasdot/_dotblas.o > > -L/usr/local/lib -Lbuild/temp.linux-x86_64-2.6 -lf77blas -lcblas -latlas > > -o build/lib.linux-x86_64-2.6/numpy/core/_dotblas.so" failed with exit > > status 1 > > > > Any help would be appreciated. > > > Python is looking for a 64-bit library as the one in /usr/local/lib/ is > either 32-bit or built with a different compiler version. If you have > the correct library in another location then you need to point numpy to > it or just build everything with the same compiler. > > Bruce > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 1

Build failure for NumPy's HEAD in Git
by Dirk Ullrich 07 Jul '11

07 Jul '11

Hi, the current HEAD of NumPy fails to build. To be more precise: compilation of `numpy/core/src/multiarray/multiarraymodule_onefile.c' fails. It looks like that is caused by splitting the `nditer.c.src' stuff in the same directory into `nditer_api.c', `nditer_constr.c' and `nditer_templ.c.src': If you #include the three new `.c' files in `multiarraymodule_onefile.c' instead of `nditer.c' the problem seems to be fixed. Sorry for posting this to this mailing list instead of filing a bug. But I am new to NumPy and have now account yet for Trac (yet). Dirk

2 1

Missing Data development plan
by Mark Wiebe 07 Jul '11

07 Jul '11

It's been a day less than two weeks since I posted my first feedback request on a masked array implementation of missing data. I'd like to thank everyone that contributed to the discussion, and that continues to contribute. I believe my design is very solid thanks to all the feedback, and I understand at the same time there are still concerns that people have about the design. I sincerely hope that those concerns are further discussed and made more clear just as I have spent a lot of effort making sure my ideas are clear and understood by everyone in the discussion. Travis has directed me to for the moment focus a majority of my attention on the implementation. He will post further thoughts on the design issues in the next few days when he has enough of a break in his schedule. With the short time available for this implementation, my plan is as follows: 1) Implement the masked implementation of NA nearly to completion. This is the quickest way to get something that people can provide hands-on feedback with, and the NA dtype in my design uses the machinery of the masked implementation for all the computational kernels. 2) Assuming there is enough time left, implement the NA[] parameterized dtype in concert with a derived[] dtype and cleanups of the datetime64[] dtype, with the goal of creating some good structure for the possibility of creating more parameterized dtypes in the future. The derived[] dtype idea is based on an idea Travis had which he called computed columns, but generalized to apply in more contexts. When the time comes, I will post a proposal for feedback on this idea as well. Thanks once again for all the great feedback, and I look forward to getting a prototype into your hands to test as quickly as possible! -Mark

1 0

Compiling 2.0.0.dev-3071eab version on Red-Hat Error with -lf77blas
by Jeffrey Spencer 07 Jul '11

07 Jul '11

The error is below: creating build/temp.linux-x86_64-2.6/numpy/core/blasdot compile options: '-DATLAS_INFO="\"None\"" -Inumpy/core/blasdot -Inumpy/core/include -Ibuild/src.linux-x86_64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/home/jspender/include/python2.6 -Ibuild/src.linux-x86_64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-2.6/numpy/core/src/umath -c' gcc: numpy/core/blasdot/_dotblas.c numpy/core/blasdot/_dotblas.c: In function ‘dotblas_matrixproduct’: numpy/core/blasdot/_dotblas.c:239: warning: comparison of distinct pointer types lacks a cast numpy/core/blasdot/_dotblas.c:257: warning: passing argument 3 of ‘*(PyArray_API + 2240u)’ from incompatible pointer type numpy/core/blasdot/_dotblas.c:292: warning: passing argument 3 of ‘*(PyArray_API + 2240u)’ from incompatible pointer type gcc -pthread -shared build/temp.linux-x86_64-2.6/numpy/core/blasdot/_dotblas.o -L/usr/local/lib -Lbuild/temp.linux-x86_64-2.6 -lf77blas -lcblas -latlas -o build/lib.linux-x86_64-2.6/numpy/core/_dotblas.so /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when searching for -lf77blas /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when searching for -lf77blas /usr/bin/ld: cannot find -lf77blas collect2: ld returned 1 exit status /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when searching for -lf77blas /usr/bin/ld: skipping incompatible /usr/local/lib/libf77blas.a when searching for -lf77blas /usr/bin/ld: cannot find -lf77blas collect2: ld returned 1 exit status error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.6/numpy/core/blasdot/_dotblas.o -L/usr/local/lib -Lbuild/temp.linux-x86_64-2.6 -lf77blas -lcblas -latlas -o build/lib.linux-x86_64-2.6/numpy/core/_dotblas.so" failed with exit status 1 Any help would be appreciated. -- ________________________ Jeffrey Spencer jeffspencerd(a)gmail.com

2 1

random numbers from arbitrary distribution
by Wolfgang Kerzendorf 07 Jul '11

07 Jul '11

Hi all, Is there an way to get random numbers from an arbitrary distribution already built-in to numpy. I am interested to do that for a black body distribution Thanks Wolfgang

2 1

reading in files with fixed with format
by Wolfgang Kerzendorf 07 Jul '11

07 Jul '11

Dear all, I have a couple of data files that were written with fortran at a fixed with. That means its tabular data which might not have spaces (it is just specified how many characters each field has and what type it is). Is there anything to read that with scipy and or numpy? Cheers Wolfgang

2 2

using the same vocabulary for missing value ideas
by Mark Wiebe 07 Jul '11

07 Jul '11

It appears to me that one of the biggest reason some of us have been talking past each other in the discussions is that different people have different definitions for the terms being used. Until this is thoroughly cleared up, I feel the design process is tilting at windmills. In the interests of clarity in our discussions, here is a starting point which is consistent with the NEP. These definitions have been added in a glossary within the NEP. If there are any ideas for amendments to these definitions that we can agree on, I will update the NEP with those amendments. Also, if I missed any important terms which need to be added, please propose definitions for them. NA (Not Available) A placeholder for a value which is unknown to computations. That value may be temporarily hidden with a mask, may have been lost due to hard drive corruption, or gone for any number of reasons. This is the same as NA in the R project. IGNORE (Skip/Ignore) A placeholder which should be treated by computations as if no value does or could exist there. For sums, this means act as if the value were zero, and for products, this means act as if the value were one. It's as if the array were compressed in some fashion to not include that element. bitpattern A technique for implementing either NA or IGNORE, where a particular set of bit patterns are chosen from all the possible bit patterns of the value's data type to signal that the element is NA or IGNORE. mask A technique for implementing either NA or IGNORE, where a boolean or enum array parallel to the data array is used to signal which elements are NA or IGNORE. numpy.ma The existing implementation of a particular form of masked arrays, which is part of the NumPy codebase. The most important distinctions I'm trying to draw are: 1) NA vs IGNORE and bitpattern vs mask are completely independent. Any combination of NA as bitpattern, NA as mask, IGNORE as bitpattern, and IGNORE as mask are reasonable. 2) The idea of masking and the numpy.ma implementation are different. The numpy.ma object makes particular choices about how to interpret the mask, but while backwards compatibility is important, a fresh evaluation of all the design choices going into a mask implementation is worthwhile. Thanks, Mark

12 26

miniNEP 2: NA support via special dtypes
by Nathaniel Smith 06 Jul '11

06 Jul '11

Well, everyone seems to like my first attempt at this so far, so I guess I'll really stick my foot in it now... here's my second miniNEP, which lays out a plan for handling dtype/bit-pattern-style NAs. I've stolen bits of text from both the NEP and the alterNEP for this, but since the focus is on nailing down the details, most of the content is new. There are many FIXME's noted, where some decisions or more work is needed... the idea here is to lay out some specifics, so we can figure out if the idea will work and get the details right. So feedback is *very* welcome! Master version: https://gist.github.com/1068264 Current version for commenting: ####################################### miniNEP 2: NA support via special dtypes ####################################### To try and make more progress on the whole missing values/masked arrays/... debate, it seems useful to have a more technical discussion of the pieces which we *can* agree on. This is the second, which attempts to nail down the details of how NAs can be implemented using special dtype's. ***************** Table of contents ***************** .. contents:: ********* Rationale ********* An ordinary value is something like an integer or a floating point number. A missing value is a placeholder for an ordinary value that is for some reason unavailable. For example, in working with statistical data, we often build tables in which each row represents one item, and each column represents properties of that item. For instance, we might take a group of people and for each one record height, age, education level, and income, and then stick these values into a table. But then we discover that our research assistant screwed up and forgot to record the age of one of our individuals. We could throw out the rest of their data as well, but this would be wasteful; even such an incomplete row is still perfectly usable for some analyses (e.g., we can compute the correlation of height and income). The traditional way to handle this would be to stick some particular meaningless value in for the missing data, e.g., recording this person's age as 0. But this is very error prone; we may later forget about these special values while running other analyses, and discover to our surprise that babies have higher incomes than teenagers. (In this case, the solution would be to just leave out all the items where we have no age recorded, but this isn't a general solution; many analyses require something more clever to handle missing values.) So instead of using an ordinary value like 0, we define a special "missing" value, written "NA" for "not available". There are several possible ways to represent such a value in memory. For instance, we could reserve a specific value (like 0, or a particular NaN, or the smallest negative integer) and then ensure that this value is treated specially by all arithmetic and other operations on our array. Another option would be to add an additional mask array next to our main array, use this to indicate which values should be treated as NA, and then extend our array operations to check this mask array whenever performing computations. Each implementation approach has various strengths and weaknesses, but here we focus on the former (value-based) approach exclusively and leave the possible addition of the latter to future discussion. The core advantages of this approach are (1) it adds no additional memory overhead, (2) it is straightforward to store and retrieve such arrays to disk using existing file storage formats, (3) it allows binary compatibility with R arrays including NA values, (4) it is compatible with the common practice of using NaN to indicate missingness when working with floating point numbers, (5) the dtype is already a place where `weird things can happen' -- there are a wide variety of dtypes that don't act like ordinary numbers (including structs, Python objects, fixed-length strings, ...), so code that accepts arbitrary numpy arrays already has to be prepared to handle these (even if only by checking for them and raising an error). Therefore adding yet more new dtypes has less impact on extension authors than if we change the ndarray object itself. The basic semantics of NA values are as follows. Like any other value, they must be supported by your array's dtype -- you can't store a floating point number in an array with dtype=int32, and you can't store an NA in it either. You need an array with dtype=NAint32 or something (exact syntax to be determined). Otherwise, NA values act exactly like any other values. In particular, you can apply arithmetic functions and so forth to them. By default, any function which takes an NA as an argument always returns an NA as well, regardless of the values of the other arguments. This ensures that if we try to compute the correlation of income with age, we will get "NA", meaning "given that some of the entries could be anything, the answer could be anything as well". This reminds us to spend a moment thinking about how we should rephrase our question to be more meaningful. And as a convenience for those times when you do decide that you just want the correlation between the known ages and income, then you can enable this behavior by adding a single argument to your function call. For floating point computations, NAs and NaNs have (almost?) identical behavior. But they represent different things -- NaN an invalid computation like 0/0, NA a value that is not available -- and distinguishing between these things is useful because in some situations they should be treated differently. (For example, an imputation procedure should replace NAs with imputed values, but probably should leave NaNs alone.) And anyway, we can't use NaNs for integers, or strings, or booleans, so we need NA anyway, and once we have NA support for all these types, we might as well support it for floating point too for consistency. **************** General strategy **************** Numpy already has a general mechanism for defining new dtypes and slotting them in so that they're supported by ndarrays, by the casting machinery, by ufuncs, and so on. In principle, we could implement NA-dtypes just using these existing interfaces. But we don't want to do that, because defining all those new ufunc loops etc. from scratch would be a huge hassle, especially since the basic functionality needed is the same in all cases. So we need some generic functionality for NAs -- but it would be better not to bake this in as a single set of special "NA types", since users may well want to define new custom dtypes that have their own NA values, and have them integrate well the rest of the NA machinery. Our strategy, therefore, is to avoid the `mid-layer mistake`_ by exposing some code for generic NA handling in different situations, which dtypes can selectively use or not as they choose. .. _mid-layer mistake: https://lwn.net/Articles/336262/ Some example use cases: 1. We want to define a dtype that acts exactly like an int32, except that the most negative value is treated as NA. 2. We want to define a parametrized dtype to represent `categorical data`_, and the bit-pattern to be used for NA depends on the number of categories defined, so our code needs to play an active role handling it rather than simply deferring to the standard machinery. 3. We want to define a dtype that acts like an length-10 string and supports NAs. Since our string may hold arbitrary binary values, we want to actually allocate 11 bytes for it, with the first byte a flag indicating whether this string is NA and the rest containing the string content. 4. We want to define a dtype that allows multiple different types of NA data, which print differently and can be distinguished by the new ufunc that we define called ``is_na_of_type(...)``, but otherwise takes advantage of the generic NA machinery for most operations. .. _categorical data: http://mail.scipy.org/pipermail/numpy-discussion/2010-August/052401.html **************************** dtype C-level API extensions **************************** The `PyArray_Descr`_ struct gains the following new fields:: void * NA_value; PyArray_Descr * NA_extends; int NA_extends_offset; .. _PyArray_Descr: http://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html#P… The following new flag values are defined:: NPY_NA_SUPPORTED NPY_NA_AUTO_ARRFUNCS NPY_NA_AUTO_CAST NPY_NA_AUTO_UFUNC NPY_NA_AUTO_ALL /* the above flags OR'ed together */ The `PyArray_ArrFuncs`_ struct gains the following new fields:: void (*isna)(void * src, void * dst, npy_intp n, void * arr); void (*clearna)(void * data, npy_intp n, void * arr); .. _PyArray_ArrFuncs: http://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html#P… The general idea is that anywhere where we used to call a dtype-specific function pointer, the code will be modified to instead: 1. Check for whether the relevant NPY_NA_AUTO_... bit is enabled, the NA_extends field is non-NULL, and the function pointer we wanted to call is NULL. 2. If these conditions are met, then use ``isna`` to identify which entries in the array are NA, and handle them appropriately. Then look up whatever function we were *going* to call using this dtype on the ``NA_extends`` dtype instead, and use that to handle the non-NA elements. For more specifics, see following sections. Note that if ``NA_extends`` points to a parametrized dtype, then the dtype object it points to must be fully specified. For example, if it is a string dtype, it must have a non-zero ``elsize`` field. In order to handle the case where the NA information is stored in a field next to the `real' data, the ``NA_extends_offset`` field is set to a non-zero value; it must point to the location within each element of this dtype where some data of the ``NA_extends`` dtype is found. For example, if we have are storing 10-byte strings with an NA indicator byte at the beginning, then we have:: elsize == 11 NA_extends_offset == 1 NA_extends->elsize == 10 When delegating to the ``NA_extends`` dtype, we offset our data pointer by ``NA_extends_offset`` (while keeping our strides the same) so that it sees an array of data of the expected type (plus some superfluous padding). This is basically the same mechanism that record dtypes use, IIUC, so it should be pretty well-tested. When delegating to a function that cannot handle `misbehaved' source data (see the ``PyArray_ArrFuncs`` documentation for details), then we need to check for alignment issues before delegating (especially with a non-zero ``NA_extends_offset``). If there's a problem, when we need to `clean up' the source data first, using the usual mechanisms for handling misaligned data. (Of course, we should usually set up our dtypes so that there aren't any alignment issues, but someone screws that up, or decides that reduced memory usage is more important to them then fast inner loops, then we should still handle that gracefully, as we do now.) The ``NA_value`` and ``clearna`` fields are used for various sorts of casting. ``NA_value`` is a bit-pattern to be used when, for example, assigning from np.NA. ``clearna`` can be a no-op if ``elsize`` and ``NA_extends->elsize`` are the same, but if they aren't then it should clear whatever auxiliary NA storage this dtype uses, so that none of the specified array elements are NA. -------------------- Core dtype functions -------------------- The following functions are defined in ``PyArray_ArrFuncs``. The special behavior described here is enabled by the NPY_NA_AUTO_ARRFUNCS bit in the dtype flags, and only enabled if the given function field is *not* filled in. ``getitem``: Calls ``isna``. If ``isna`` returns true, returns np.NA. Otherwise, delegates to the ``NA_extends`` dtype. ``setitem``: If the input object is ``np.NA``, then runs ``memcpy(self->NA_value, data, arr->dtype->elsize);''. Otherwise, calls ``clearna``, and then delegates to the ``NA_extends`` dtype. ``copyswapn``, ``copyswap``: FIXME: Not sure whether there's any special handling to use for these? ``compare``: FIXME: how should this handle NAs? R's sort function *discards* NAs, which doesn't seem like a good option. ``argmax``: FIXME: what is this used for? If it's the underlying implementation for np.max, then it really needs some way to get a skipna argument. If not, then the appropriate semantics depends on what it's supposed to accomplish... ``dotfunc``: QUESTION: is it actually guaranteed that everything has the same dtype? FIXME: same issues as for ``argmax``. ``scanfunc``: This one's ugly. We may have to explicitly override it in all of our special dtypes, because assuming that we want the option of, say, having the token "NA" represent an NA value in a text file, we need some way to check whether that's there before delegating. But ``ungetc`` is only guaranteed to let us put back 1 character, and we need 2 (or maybe 3 if we actually check for "NA "). The other option would be to read to the next delimiter, check whether we have an NA, and if not then delegate to ``fromstr`` instead of ``scanfunc``, but according to the current API, each dtype might in principle use a totally different rule for defining `the next delimiter'. So... any ideas? (FIXME) ``fromstr``: Easy -- check for "NA ", if present then assign ``NA_value``, otherwise call ``clearna`` and delegate. ``nonzero``: FIXME: again, what is this used for? (It seems redundant with using the casting machinery to cast to bool.) Probably it needs to be modified so that it can return NA, though... ``fill``: Use ``isna`` to check if either of the first two values is NA. If so, then fill the rest of the array with ``NA_value``. Otherwise, call ``clearna`` and then delegate. ``fillwithvalue``: Guess this can just delegate? ``sort``, ``argsort``: These should probably arrange to sort NAs to a particular place in the array (either the front or the back -- any opinions?) ``scalarkind``: FIXME: I have no idea what this does. ``castdict``, ``cancastscalarkindto``, ``cancastto``: See section on casting below. ------- Casting ------- FIXME: this really needs attention from an expert on numpy's casting rules. But I can't seem to find the docs that explain how casting loops are looked up and decided between (e.g., if you're casting from dtype A to dtype B, which dtype's loops are used?), so I can't go into details. But those details are tricky and they matter... But the general idea is, if you have a dtype with ``NPY_NA_AUTO_CAST`` set, then the following conversions are automatically allowed: * Casting from the underlying type to the NA-type: this is performed by the usual ``clearna`` + potentially-strided copy dance. Also, ``isna`` is called to check that none of the regular values have been accidentally converted into NA; if so, then an error is raised. * Casting from the NA-type to the underlying type: allowed in principle, but if ``isna`` returns true for any of the values that are to be converted, then again, an error is raised. (If you want to get around this, use ``np.view(array_with_NAs, dtype=float)``.) * Casting between the NA-type and other types that do not support NA: this is allowed if the underlying type is allowed to cast to the other type, and is performed by combining a cast to or from the underlying type (using the above rules) with a cast to or from the other type (using the underlying type's rules). * Casting between the NA-type and other types that do support NA: if the other type has NPY_NA_AUTO_CAST set, then we use the above rules plus the usual dance with ``isna`` on one array being converted to ``NA_value`` elements in the other. If only one of the arrays has NPY_NA_AUTO_CAST set, then it's assumed that that dtype knows what it's doing, and we don't do any magic. (But this is one of the things that I'm not sure makes sense, as per my caveat above.) ------ Ufuncs ------ All ufuncs gain an additional optional keyword argument, ``skipNA=``, which defaults to False. If ``skipNA == True``, then the ufunc machinery *unconditionally* calls ``isna``, and then acts as if any values for which isna returns True were masked out in the ``where=`` argument (see miniNEP 1 for the behavior of ``where=``). If a ``where=`` argument is also given, then it acts as if the ``isna`` values had be ANDed out of the ``where=`` mask, though it does not actually modify the mask. Unlike the other changes below, this is performed *unconditionally* for any dtype which has an ``isna`` function defined; the NPY_NA_AUTO_UFUNC flag is *not* checked. If NPY_NA_AUTO_UFUNC is set, then ufunc loop lookup is modified so that whenever it checks for the existence of a loop on the current dtype, and does not find one, then it also checks for a loop on the ``NA_extends`` dtype. If that loop is found, then it uses it in the normal way, with the exceptions that (1) it is only called for values which are not NA according to ``isna``, (2) if the output array has NPY_NA_AUTO_UFUNC set, then ``clearna`` is called on it before calling the ufunc loop, (3) pointer offsets are adjusted by ``NA_extends_offset`` before calling the ufunc loop. FIXME: We should go into more detail here about how NPY_NA_AUTO_UFUNC works when there are multiple input arrays, of which potentially some have the flag set and some do not. -------- Printing -------- FIXME: There should be some sort of mechanism by which values which are NA are automatically repr'ed as NA, but I don't really understand how numpy printing works, so I'll let someone else fill in this section. -------- Indexing -------- Scalar indexing like ``a[12]`` goes via the ``getitem`` function, so according to the proposal as described above, if a dtype delegates ``getitem``, then scalar indexing on NAs will return the object ``np.NA``. (If it doesn't delegate ``getitem``, of course, then it can return whatever it wants.) This seems like the simplest approach, but an alternative would be to add a special case to scalar indexing, where if an ``NPY_NA_AUTO_INDEX`` flag were set, then it would call ``isna`` on the specified element. If this returned false, it would call ``getitem`` as usual; otherwise, it would return a 0-d array containing the specified element. The problem with this is that it breaks expressions like ``if a[i] is np.NA: ...``. (Of course, there is nothing nearly so convenient as that for NaN values now, but then, NaN values don't have their own global singleton.) So for now we stick to scalar indexing just returning ``np.NA``, but this can be revisited if anyone objects. ********************************* Python API for generic NA support ********************************* NumPy will gain a global singleton called numpy.NA, similar to None, but with semantics reflecting its status as a missing value. In particular, trying to treat it as a boolean will raise an exception, and comparisons with it will produce numpy.NA instead of True or False. These basics are adopted from the behavior of the NA value in the R project. To dig deeper into the ideas, http://en.wikipedia.org/wiki/Ternary_logic#Kleene_logic provides a starting point. Most operations on ``np.NA`` (e.g., ``__add__``, ``__mul__``) are overridden to unconditionally return ``np.NA``. The automagic dtype detection used for expressions like ``np.asarray([1, 2, 3])``, ``np.asarray([1.0, 2.0. 3.0])`` will be extended to recognize the ``np.NA`` value, and use it to automatically switch to a built-in NA-enabled dtype (which one being determined by the other elements in the array). A simple ``np.asarray([np.NA])`` will use an NA-enabled float64 dtype (which is analogous to what you get from ``np.asarray([])``). Note that this means that expressions like ``np.log(np.NA)`` will work: first ``np.NA`` will be coerced to a 0-d NA-float array, and then ``np.log`` will be called on that. Python-level dtype objects gain the following new fields:: NA_supported NA_value ``NA_supported`` is a boolean which simply exposes the value of the ``NPY_NA_SUPPORTED`` flag; it should be true if this dtype allows for NAs, false otherwise. [FIXME: would it be better to just key this off the existence of the ``isna`` function? Even if a dtype decides to implement all other NA handling itself, it still has to define ``isna`` in order to make ``skipNA=`` work correctly.] ``NA_value`` is a 0-d array of the given dtype, and its sole element contains the same bit-pattern as the dtype's underlying ``NA_value`` field. This makes it possible to determine the default bit-pattern for NA values for this type (e.g., with ``np.view(mydtype.NA_value, dtype=int8)``). We *do not* expose the ``NA_extends`` and ``NA_extends_offset`` values at the Python level, at least for now; they're considered an implementation detail (and it's easier to expose them later if they're needed then unexpose them if they aren't). Two new ufuncs are defined: ``np.isNA`` returns a logical array, with true values where-ever the dtype's ``isna`` function returned true. ``np.isnumber`` is only defined for numeric dtypes, and returns True for all elements which are not NA, and for which ``np.isfinite`` would return True. ***************** Builtin NA dtypes ***************** The above describes the generic machinery for NA support in dtypes. It's flexible enough to handle all sorts of situations, but we also want to define a few generally useful NA-supporting dtypes that are available by default. For each built-in dtype, we define an associated NA-supporting dtype, as follows:: floats: the associated dtype uses a specific NaN bit-pattern to indicate NA (chosen for R compatibility) complex: we do whatever R does (FIXME: look this up -- two NA floats, probably?) signed integers: the most-negative signed value is used as NA (chosen for R compatibility) unsigned integers: the most-positive value is used as NA (no R compatibility possible). strings: the first byte (or, in the case of unicode strings, first 4 bytes) is used as a flag to indicate NA, and the rest of the data gives the actual string. (no R compatibility possible) objects: Two options (FIXME): either we don't include an NA-ful version, or we use np.NA as the NA bit pattern. boolean: we do whatever R does (FIXME: look this up -- 0 == FALSE, 1 == TRUE, 2 == NA?) Each of these dtypes is trivially defined using the above machinery, and are what are automatically used by the automagic type inference machinery (for ``np.asarray([True, np.NA, False])``, etc.). They can also be accessed via a new function ``np.withNA``, which takes a regular dtype (or an object that can be coerced to a dtype, like 'float') and returns one of the above dtypes. Ideally ``withNA`` should also take some optional arguments that let you describe which values you want to count as NA, etc., but I'll leave that for a future draft (FIXME). FIXME: If ``d`` is one of the above dtypes, then should ``d.type`` return? The NEP also contains a proposal for a somewhat elaborate domain-specific-language for describing NA dtypes. I'm not sure how great an idea that is. (I have a bias against using strings as data structures, and find the already existing strings confusing enough as it is -- also, apparently the NEP version of numpy uses strings like 'f8' when printing dtypes, while my numpy uses object names like 'float64', so I'm not sure what's going on there. ``withNA(float64, arg1=value1)`` seems like a more pleasant way to print a dtype than "NA[f8,value1]", at least to me.) But if people want it, then cool. -------------- Type hierarchy -------------- FIXME: how should we do subtype checks, etc., for NA dtypes? What does ``issubdtype(withNA(float), float)`` return? How about ``issubdtype(withNA(float), np.floating)``? ------------- Serialization ------------- FIXME: How are dtypes stored in .npz or pickle files? Do we need to do anything special here?

2 5

NA/Missing Data Conference Call Summary
by Christopher Jordan-Squire 06 Jul '11

06 Jul '11

Here's a short-ish summary of the topics discussed in the conference call this afternoon. WARNING: I try to give examples for everything discussed to make it as concrete as possible. However, most of the examples were not explicitly discussed during the conference. I apologize in advance if I mischaracterize anyone's arguments, and please jump in to correct me if I did. Participants: Travis Oliphant, Mark Wiebe, Matthew Brett, Nathaniel Smith, Pierre GM, Ben Root, Chuck Harris, Wes McKinney, Chris Jordan-Squire First, areas of broad agreement: *There should be more functionality for missing data *There should be dtypes which support missing data ('parameterized dtypes' in the current NEP) *Adding a 'where' semantic to ufuncs *Have the same data with different sets of missing elements in different views *Easy for non-expert numpy users Since we only have Mark is only around Austin until early August, there's also broad agreement that we need to get something done quickly. However, the numpy community (and Travis in particular) are balancing this against the possibility of a sub-optimal solution which can't be taken back. BIT PATTERN & MASK IMPLEMENTATIONS FOR NA ------------------------------------------------------------------------------------------ The current NEP proposes both mask and bit pattern implementations for missing data. I use the terms bit pattern and parameterized dtype interchangeably, since the parameterized dtype will use a bit pattern for its implementation. The two implementations will support the same functionality with respect to NA, and the implementation details will be largely invisible to the user. Their differences are in the 'extra' features each supports. Two common questions were: 1. Why make two implementations of missing data: one with masks and the other with parameterized dtypes? 2. Why does the implementation using masks have higher priority? The answers are: 1. The mask implementation is more general and easier to implement and maintain. The bit pattern implementation saves memory, makes interoperability easier, and makes ABI (Application Binary Interface) compatibility easier. Since each has different strengths, the argument is both should be implemented. 2. The implementation for the parameterized dtypes will rely on the implementation using a mask. NA VS. IGNORE --------------------------------- A lot of discussion centered on IGNORE vs. NA types. We take IGNORE in aNEP sense and NA in NEP sense. With NA, there is a clear notion of how NA propagates through all basic numpy operations. (e.g., 3+NA=NA and log(NA) = NA, while NA | True = True.) IGNORE is separate from NA, with different interpretations depending on the use case. IGNORE could mean: 1. Data that is being temporarily ignored. e.g., a possible outlier that is temporarily being removed from consideration. 2. Data that cannot exist. e.g., a matrix representing a grid of water depths for a lake. Since the lake isn't square, some entries will represent land, and so depth will be a meaningless concept for those entries. 3. Using IGNORE to signal a jagged array. e.g., [ [1, 2, IGNORE], [IGNORE, 3, 4] ] should behave exactly the same as [ [1 , 2] , [3 , 4] ]. Though this leaves open how [1, 2, IGNORE] + [3 , 4] should behave. Because of these different uses of IGNORE, it doesn't have as clear a theoretical interpretation as NA. (For instance, what is IGNORE+3, IGNORE*3, or IGNORE | True?) But several of the discussants thought the use cases for IGNORE were very compelling. Specifically, they wanted to be able to use IGNORE's and NA's simultaneously while still being able to differentiate between them. So, for example, being able to designate some data as IGNORE while still able to determine which data was NA but not IGNORE. The current NEP does not allow for this directly. Although in some cases it can be indirectly done via views. (By taking a view of the original data, expanding the values which are considered NA in the view, and then comparing with the original data to see if the NA is in the original or not.) Since both are possible in this sense, Mark's NEP makes it so IGNORE is allowed but isn't the default. Another important point from the current NEP is that not being able to access values considered missing, even if the implementation of missingness is via a mask, is a feature and not a bug. It is a feature because if the data is missing then, conceptually, neither the user nor any function the user calls should be able to obtain that data. This is precisely why the indirect route, via views of the original data, is required to access data that a different view says is missing. The current NEP treats all NA's the same. The reasoning is that, regardless of where the NA originated, the functions the numpy array is fed in to will either ignore all NA's or propagate them (i.e. not ignore them). These two different behaviors are chosen when passed into a ufunc by setting the skipna ufunc parameter to True or False. Since the NA's are treated the same, their source is irrelevant. Though this could be argued against if there are compelling cases where the IGNORE and NA are treated differently. A possible solution to the above desires for an IGNORE notion of missingness is to allow for multiple types of missing values. For example, the mask underlying the missing data could have int types, and different ints mean different missing. E.g. 0 is present, 1 is NA, 2 is IGNORE. However, this was only discussed briefly at the end of the conference call, and should be discussed further. HOW DOES THIS RELATE TO THE CURRENT MASKED ARRAY? ---------------------------------------------------------------------------------------------------- Everyone seems to agree they'd love it if this could encompass all current use cases of the numpy.ma arrays, so numpy.ma arrays could be deprecated. (However they wouldn't be eliminated for several years, even in the most optimistic scenarios.) IMPLEMENTATION DETAILS ----------------------------------------------------- *Under the hood, the parameterized dtypes will use buffered masks when performing operations. This can be a source of confusion when discussing their behavior, since there is no true mask, hence no extra memory, but a mask is created on the fly. *The iterator will be given a new 'masked' mode, triggered by a flag, which will use or ignore data based on a boolean array. *Currently won't allow shared masks. But Pierre GM suggests that's just as well since they easily lead to buggy code. I hope this summary roughly captures what was said. Please chime in with additional comments/corrections. -Chris Jordan-Squire

11 22