Mailman 3 June 2011 - NumPy-Discussion

fast numpy i/o
by Neal Becker 27 Jun '11

27 Jun '11

I'm wondering what are good choices for fast numpy array serialization? mmap: fast, but I guess not self-describing? hdf5: ? pickle: self-describing, but maybe not fast? others?

5 11

data type specification when using numpy.genfromtxt
by Chao YUE 26 Jun '11

26 Jun '11

Dear all numpy users, I want to read a csv file with many (49) columns, the first column is string and remaning can be float. how can I avoid type in like data=numpy.genfromtxt('data.csv',delimiter=';',names=True, dtype=(S10, float, float, ......)) Can I just specify the type of first cloumn is tring and the remaing float? how can I do that? Thanks a lot, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 77 30; Fax:01.69.08.77.16 ************************************************************************************

3 3

Concepts for masked/missing data
by Nathaniel Smith 26 Jun '11

26 Jun '11

So obviously there's a lot of interest in this question, but I'm losing track of all the different issues that've being raised in the 150-post thread of doom. I think I'll find this easier if we start by putting aside the questions about implementation and such and focus for now on the *conceptual model* that we want. Maybe I'm not the only one? So as far as I can tell, there are three different ways of thinking about masked/missing data that people have been using in the other thread: 1) Missingness is part of the data. Some data is missing, some isn't, this might change through computation on the data (just like some data might change from a 3 to a 6 when we apply some transformation, NA | True could be True, instead of NA), but we can't just "decide" that some data is no longer missing. It makes no sense to ask what value is "really" there underneath the missingness. And It's critical that we keep track of this through all operations, because otherwise we may silently give incorrect answers -- exactly like it's critical that we keep track of the difference between 3 and 6. 2) All the data exists, at least in some sense, but we don't always want to look at all of it. We lay a mask over our data to view and manipulate only parts of it at a time. We might want to use different masks at different times, mutate the mask as we go, etc. The most important thing is to provide convenient ways to do complex manipulations -- preserve masks through indexing operations, overlay the mask from one array on top of another array, etc. When it comes to other sorts of operations then we'd rather just silently skip the masked values -- we know there are values that are masked, that's the whole point, to work with the unmasked subset of the data, so if sum returned NA then that would just be a stupid hassle. 3) The "all things to all people" approach: implement every feature implied by either (1) or (2), and switch back and forth between these conceptual frameworks whenever necessary to make sense of the resulting code. The advantage of deciding up front what our model is is that it makes a lot of other questions easier. E.g., someone asked in the other thread whether, after setting an array element to NA, it would be possible to get back the original value. If we follow (1), the answer is obviously "no", if we follow (2), the answer is obviously "yes", and if we follow (3), the answer is obviously "yes, probably, well, maybe you better check the docs?". My personal opinions on these are: (1): This is a real problem I face, and there isn't any good solution now. Support for this in numpy would be awesome. (2): This feels more like a convenience feature to me; we already have lots of ways to work with subsets of data. I probably wouldn't bother using it, but that's fine -- I don't use np.matrix either, but some people like it. (3): Well, it's a bit of a mess, but I guess it might be better than nothing? But that's just my opinion. I'm wondering if we can get any consensus on which of these we actually *want* (or maybe we want some fourth option!), and *then* we can try to figure out the best way to get there? Pretty much any implementation strategy we've talked about could work for any of these, but hard to decide between them if we don't even know what we're trying to do... -- Nathaniel

8 18

[ANN] Euroscipy 2011 - registration now open
by Emmanuelle Gouillart 25 Jun '11

25 Jun '11

Dear all, After some delay due to technical problems, registration for Euroscipy 2011 is now open! Please go to http://www.euroscipy.org/conference/euroscipy2011, login to your account if you have one, or create a new account (right side of the upper banner of the Euroscipy webpage), then click on the registration panel on the sidebar on the left. Choose the items (tutorials and/or conference) that you wish to attend to, and proceed to pay for your items by credit card. Early-bird registration fees are as follows: * for academia, and self-employed persons: 50 euros for the tutorials (two days), and 50 euros for the conference (two days). * for corporate participants: 100 euros for tutorials (two days), and 100 euros for the conference (two days). For all days, lunch will be catered for and is included in the fee. Early-bird fees apply until July 24th; prices will be doubled for late registration. Book early to take advantage of the early-bird prices! As there is a limited number of seats in the lecture halls, registrations shall be accepted on a first-come first-serve basis. All questions regarding registration should be addressed exclusively to org-team(a)lists.euroscipy.org Looking forward to seeing you at Euroscipy! The organizers

1 0

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray
by Laurent Gautier 25 Jun '11

25 Jun '11

On 2011-06-24 17:30, Robert Kern <robert.kern(a)gmail.com> wrote: > On Fri, Jun 24, 2011 at 10:07, Laurent Gautier<lgautier(a)gmail.com> wrote: >> > On 2011-06-24 16:43, Robert Kern<robert.kern(a)gmail.com> wrote: >>> >> >>> >> On Fri, Jun 24, 2011 at 09:33, Charles R Harris >>> >> <charlesr.harris(a)gmail.com> wrote: >>>> >>> >>>>> >>> > >>>>> >>> > ?On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern<robert.kern(a)gmail.com> >>>>> >>> > ?wrote: >>>>> >>>> >>>>>>> >>>> >> ?The alternative proposal would be to add a few new dtypes that are >>>>>>> >>>> >> ?NA-aware. E.g. an nafloat64 would reserve a particular NaN value >>>>>>> >>>> >> ?(there are lots of different NaN bit patterns, we'd just reserve >>>>>>> >>>> >> one) >>>>>>> >>>> >> ?that would represent NA. An naint32 would probably reserve the most >>>>>>> >>>> >> ?negative int32 value (like R does). Using the NA-aware dtypes >>>>>>> >>>> >> signals >>>>>>> >>>> >> ?that you are using NA values; there is no need for an additional >>>>>>> >>>> >> flag. >>>> >>> >>>>> >>> > >>>>> >>> > ?Definitely better names than r-int32. Going this way has the advantage >>>>> >>> > of >>>>> >>> > ?reducing the friction between R and numpy, and since R has pretty much >>>>> >>> > ?become the standard software for statistics that is an important >>>>> >>> > ?consideration. >>> >> >>> >> I would definitely steal their choices of NA value for naint32 and >>> >> nafloat64. I have reservations about their string NA value (i.e. 'NA') >>> >> as anyone doing business in North America and other continents may >>> >> have issues with that.... >> > >> > May be there is not so much need for reservation over the string NA, when >> > making the distinction between: >> > a- the internal representation of a "missing string" (what is stored in >> > memory, and that C-level code would need to be aware of) >> > b- the 'external' representation of a missing string (in Python, what would >> > be returned by repr() ) >> > c- what is assumed to be a missing string value when reading from a file. >> > >> > a/ is not 'NA', c/ should be a parameter in the relevant functions, b/ can >> > be configured as a module-level, class-level, or instance-level variable. > In R, a/ happens to be 'NA', unfortunately. :-/ In a sense yes, in a sense no. There is NA_STRING (that happens to store 'NA', but it could equally be 'foobar' or whatever) and there is "NA". NA_STRING is set once for all, and each time a string element in a vector is set to NA this points to that one. A string "NA" is not the NA_STRING. > I'm not really sure how they handle datasets that use valid 'NA' > values. Presumably, their input routines allow one to convert such > values to something else such that it can use 'NA'==NA internally. That's c/. Example in R's read.table(..., na.strings = "NA", ...). A number of R design choices (that are S design choices here) are empirical and based on experience. Datasets can originally be in a number of different flavours and conversion can be made when reading data into memory / R format. L. > -- Robert Kern

1 0

How to avoid extra copying when forming an array from an iterator
by srean 24 Jun '11

24 Jun '11

Hi, I have a iterator that yields a complex object. I want to make an array out of a numerical attribute that the yielded object possesses and that too very efficiently. My initial plan was to keep writing the numbers to a StringIO object and when done generate the numpy array using StringIO's buffer. But fromstring() method fails on the StringIO object, so does fromfile() or using the StringIO object as the initializer of an array object. After some digging I ran in to this ticket htt projects.scipy.org/numpy/ticket/1634 that has been assigned a low priority. Is there some other way to achieve what I am trying ? Efficiency is important because potentially millions of objects would be yielded. -- srean

2 5

[ANN] Numerical integration with guaranteed precision by interalg
by Dmitrey 24 Jun '11

24 Jun '11

Hi all, some ideas implemented in the solver interalg (INTERval ALGorithm) that already turn out to be more effective than its competitors in numerical optimization (benchmark) appears to be extremely effective in numerical integration with guaranteed precision. Here are some examples where interalg works perfectly while scipy.integrate solvers fail to solve the problems and lie about obtained residual: * 1-D (vs scipy.integrate quad) > * 2-D (vs scipy.integrate dblquad) > * 3-D (vs scipy.integrate tplquad) > see http://openopt.org/IP for more details. Regards, D.

1 0

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray
by Laurent Gautier 24 Jun '11

24 Jun '11

On 2011-06-24 13:59, Nathaniel Smith <njs(a)pobox.com> wrote: > On Thu, Jun 23, 2011 at 5:56 PM, Benjamin Root<ben.root(a)ou.edu> wrote: >> Lastly, I am not entirely familiar with R, so I am also very curious about >> what this magical "NA" value is, and how it compares to how NaNs work. >> Although, Pierre brought up the very good point that NaNs woulldn't work >> anyway with integer arrays (and object arrays, etc.). > Since R is designed for statistics, they made the interesting decision > that *all* of their core types have a special designated "missing" > value. At the R level this is just called "NA". Internally, there are > a bunch of different NA values -- for floats it's a particular NaN, > for integers it's INT_MIN, for booleans it's 2 (IIRC), etc. (You never > notice this, because R will silently cast a NA of one type into NA of > another type whenever needed, and they all print the same.) > > Because any array can contain NA's, all R functions then have to have > some way of handling this -- all their integer arithmetic knows that > INT_MIN is special, for instance. The rules are basically the same as > for NaN's, but NA and NaN are different from each other (because one > means "I don't know, could be anything" and the other means "you tried > to divide by 0, I *know* that's meaningless"). > > That's basically it. > > -- Nathaniel Would the use of R's system for expressing "missing values" be possible in numpy through a special flag ? Any given numpy array could have a boolean flag (say "na_aware") indicating that some of the values are representing a missing cell. If the exact same system is used, interaction with R (through something like rpy2) would be simplified and more robust. L. PS: In R, dividing one by zero returns +/-Inf, not NaN. 0/0 returns NaN.

10 22

correlation in fourier domain
by Alex Flint 24 Jun '11

24 Jun '11

I would like to perform 2d correlations between a large matrix A and a bunch of small matrices B1...Bn. Is there anything built into numpy to help me do this efficiently in the fourier domain (i.e. DFT of A -> multiplication with DFT of B1..Bn)? Cheers, Alex

1 0

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray
by Laurent Gautier 24 Jun '11

24 Jun '11

On 2011-06-24 16:43, Robert Kern <robert.kern(a)gmail.com> wrote: > On Fri, Jun 24, 2011 at 09:33, Charles R Harris > <charlesr.harris(a)gmail.com> wrote: >> > >> > On Fri, Jun 24, 2011 at 8:06 AM, Robert Kern<robert.kern(a)gmail.com> wrote: >>> >> The alternative proposal would be to add a few new dtypes that are >>> >> NA-aware. E.g. an nafloat64 would reserve a particular NaN value >>> >> (there are lots of different NaN bit patterns, we'd just reserve one) >>> >> that would represent NA. An naint32 would probably reserve the most >>> >> negative int32 value (like R does). Using the NA-aware dtypes signals >>> >> that you are using NA values; there is no need for an additional flag. >> > >> > Definitely better names than r-int32. Going this way has the advantage of >> > reducing the friction between R and numpy, and since R has pretty much >> > become the standard software for statistics that is an important >> > consideration. > I would definitely steal their choices of NA value for naint32 and > nafloat64. I have reservations about their string NA value (i.e. 'NA') > as anyone doing business in North America and other continents may > have issues with that.... May be there is not so much need for reservation over the string NA, when making the distinction between: a- the internal representation of a "missing string" (what is stored in memory, and that C-level code would need to be aware of) b- the 'external' representation of a missing string (in Python, what would be returned by repr() ) c- what is assumed to be a missing string value when reading from a file. a/ is not 'NA', c/ should be a parameter in the relevant functions, b/ can be configured as a module-level, class-level, or instance-level variable.

3 3