Mailman 3 March 2012 - NumPy-Discussion

Possible roadmap addendum: building better text file readers
by Wes McKinney March 2, 2012

March 2, 2012

dear all, I haven't read all 180 e-mails, but I didn't see this on Travis's initial list. All of the existing flat file reading solutions I have seen are not suitable for many applications, and they compare very unfavorably to tools present in other languages, like R. Here are some of the main issues I see: - Memory usage: creating millions of Python objects when reading a large file results in horrendously bad memory utilization, which the Python interpreter is loathe to return to the operating system. Any solution using the CSV module (like pandas's parsers-- which are a lot faster than anything else I know of in Python) suffers from this problem because the data come out boxed in tuples of PyObjects. Try loading a 1,000,000 x 20 CSV file into a structured array using np.genfromtxt or into a DataFrame using pandas.read_csv and you will immediately see the problem. R, by contrast, uses very little memory. - Performance: post-processing of Python objects results in poor performance. Also, for the actual parsing, anything regular expression based (like the loadtable effort over the summer, all apologies to those who worked on it), is doomed to failure. I think having a tool with a high degree of compatibility and intelligence for parsing unruly small files does make sense though, but it's not appropriate for large, well-behaved files. - Need to "factorize": as soon as there is an enum dtype in NumPy, we will want to enable the file parsers for structured arrays and DataFrame to be able to "factorize" / convert to enum certain columns (for example, all string columns) during the parsing process, and not afterward. This is very important for enabling fast groupby on large datasets and reducing unnecessary memory usage up front (imagine a column with a million values, with only 10 unique values occurring). This would be trivial to implement using a C hash table implementation like khash.h To be clear: I'm going to do this eventually whether or not it happens in NumPy because it's an existing problem for heavy pandas users. I see no reason why the code can't emit structured arrays, too, so we might as well have a common library component that I can use in pandas and specialize to the DataFrame internal structure. It seems clear to me that this work needs to be done at the lowest level possible, probably all in C (or C++?) or maybe Cython plus C utilities. If anyone wants to get involved in this particular problem right now, let me know! best, Wes

21 57

[Numpy] quadruple precision
by Paweł Biernat March 2, 2012

March 2, 2012

I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding.

8 10

Jaccard & Hamming Problem
by Zayd YAKOUBI March 2, 2012

March 2, 2012

Hello, I use the similarity measure "Jaccard" and "Hamming" of pckage Scipy.spacial.cdist (Python) in a clustering context, I applied to given typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. But I just know that they normally only applies to binary data. The function of these two similarity measures are not specified in the documentation: http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.… . Does anyone of you can help me find these functions? Thank you in advance Regards, Zayd

3 3

Numpy fitting
by Pierre Barthelemy March 1, 2012

March 1, 2012

Dear all, i am writing a program for data analysis. One of the functions of this program gives the possibility to fit the functions. I therefore use the recipe described in : http://www.scipy.org/Cookbook/FittingData<http://www.scipy.org/Cookbook/FittingData> under the section "Simplifying the syntax". The code looks like this: class Parameter: def __init__(self, value): self.value = value self.fixed=False def set(self, value): if not self.fixed: self.value = value def __call__(self): return self.value def fit(function, parameters, y, x = None): def f(params): i = 0 for p in parameters: p.set(params[i]) i += 1 return y - function(x) if x is None: x = arange(y.shape[0]) p = [param() for param in parameters] out=optimize.leastsq(f, p, full_output=1) One thing that i would like to know is how can i get the error on the parameters ? From what i understood from the "Cookbook" page, and from the scipy manual ( http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.…), the second argument returned by the leastsq function gives access to these errors. std_error=std(y-function(x)) param_error=sqrt(diagonal(out[1])*std_error) The param_errors that i get in this case are extremely small. Much smaller than what i expected, and much smaller than what i can get fitting the function with matlab. So i guess i made an error here. Can someone tell me how i should do to retrieve the parameter errors ? Bests, Pierre

3 3

Numpy interpolate: cut through 2D matrix
by Pierre Barthelemy March 1, 2012

March 1, 2012

Hello, for a data analysis tool i am programming, i need to plot a cut through a 2D graph. I then have a 2D array, and the indices start=(start_x,start_y) and stop=(stop_x,stop_y) that are the position of the starting point and stop point of the cut. The code i programmed is placed on the bottom. This code returns only value existing in the original array: if the cut should pass between the column index i and column index i+1, it returns anyway the value at column index i. Is it possible to use the numpy.interpolate library to make such that when passing between i and i+1, the function returns an interpolation of the graph between the points [row,column] and [row,column+1] ? def cut_matrix(array,start,stop,shift=0): ''' Draws a cut through a 2D array, between the positions start=(row,column) and stop=(row,column) ''' n_row=array.shape[0] n_col=array.shape[1] if abs(start[1]-stop[1])>abs(start[0]-stop[0]): if stop[1]<start[1]: start,stop= stop,start col_index=arange(start[1],stop[1]+1).astype(int) row_index=round_(linspace(start[0],stop[0],len(col_index))).astype(int)+int(shift) row=(linspace(start[0],stop[0],len(col_index))) else: if stop[0]<start[0]: start,stop= stop,start row_index=arange(start[0],stop[0]+1).astype(int) col_index=round_(linspace(start[1],stop[1],len(row_index))).astype(int)+int(shift) if max(col_index)>n_col or min(col_index)<0: print 'Error: column index not in range' raise IndexError return if max(row_index)>n_row or min(row_index)<0: print 'Error: row index not in range' raise IndexError return return array[row_index,col_index]

2 1

Re: [Numpy-discussion] Proposed Roadmap Overview
by Richard Hattersley March 1, 2012

March 1, 2012

+1 on the NEP guideline As part of a team building a scientific analysis library, I'm attempting to understand the current state of NumPy development and its likely future (with a view to contributing if appropriate). The proposed NEP process would make that a whole lot easier. And if nothing else, it would reduce the chance of me posting questions about topics that had already been discussed/decided! Without the process the NEPs become another potential source of confusion and mixed messages. On 1 March 2012 03:02, Travis Oliphant wrote: > I Would like to hear the opinions of others on that point, > but yes, I think that is an appropriate procedure. > > Travis > > -- > Travis Oliphant > (on a mobile) > 512-826-7480 > > > On Feb 29, 2012, at 10:54 AM, Matthew Brett > <matthew.brett(a)gmail.com> wrote: > > > Hi, > > > > On Wed, Feb 29, 2012 at 1:46 AM, Travis Oliphant > <travis(a)continuum.io> wrote: > >> We already use the NEP process for such decisions. This > discussion came from simply from the *idea* of writing such a NEP. > >> > >> Nothing has been decided. Only opinions have been shared > that might influence the NEP. This is all pretty premature, > though --- migration to C++ features on a trial branch is > some months away were it to happen. > > > > Fernando can correct me if I'm wrong, but I think he was asking a > > governance question. That is: would you (as BDF$N) consider the > > following guideline: > > > > "As a condition for accepting significant changes to Numpy, for each > > significant change, there will be a NEP. The NEP shall follow the > > same model as the Python PEPs - that is - there will be a summary of > > the changes, the issues arising, the for / against opinions and > > alternatives offered. There will usually be a draft implementation. > > The NEP will contain the resolution of the discussion as it > relates to > > the code" > > > > For example, the masked array NEP, although very > substantial, contains > > little discussion of the controversy arising, or the intended > > resolution of the controversy: > > > > > https://github.com/numpy/numpy/blob/3f685a1a990f7b6e5149c80b52 > 436fb4207e49f5/doc/neps/missing-data.rst > > > > I mean, although it is useful, it is not in the form of a PEP, as > > Fernando has described it. > > > > Would you accept extending the guidelines to the NEP format? > > > > Best, > > > > Matthew > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > >

1 0

IPython 0.12 just entered Debian Testing
by Pierre Haessig March 1, 2012

March 1, 2012

Hi, Just to start the new month on a light & happy topic : IPython 0.12 has entered Debian Testing ! -- Pierre (I'm not at all involved in the process that enabled IPython make its way to Testing. I've been watching this quite closely however. I "suspect" there was a decent amount of work on packaging all the dependencies to make this possible and I'm thankful to the guys who did it)

2 1

Special matrices with structure?
by Jaakko Luttinen March 1, 2012

March 1, 2012

Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For instance: - symmetric matrix could be stored in almost half the memory required by a non-symmetric matrix - diagonal matrix only needs to store the diagonal vector - Toeplitz matrix only needs to store one or two vectors - sparse matrix only needs to store non-zero elements (some implementations in scipy.sparse) - and so on If such classes were implemented, it would be nice if they worked with numpy functions (dot, diag, ...) and operations (+, *, +=, ...) easily. I believe this has been discussed before but google didn't help a lot.. Regards, Jaakko

2 3

Proposed Roadmap Overview
by Travis Oliphant March 1, 2012

March 1, 2012

Mark Wiebe and I have been discussing off and on (as well as talking with Charles) a good way forward to balance two competing desires: * addition of new features that are needed in NumPy * improving the code-base generally and moving towards a more maintainable NumPy I know there are load voices for just focusing on the second of these and avoiding the first until we have finished that. I recognize the need to improve the code base, but I will also be pushing for improvements to the feature-set and user experience in the process. As a result, I am proposing a rough outline for releases over the next year: * NumPy 1.7 to come out as soon as the serious bugs can be eliminated. Bryan, Francesc, Mark, and I are able to help triage some of those. * NumPy 1.8 to come out in July which will have as many ABI-compatible feature enhancements as we can add while improving test coverage and code cleanup. I will post to this list more details of what we plan to address with it later. Included for possible inclusion are: * resolving the NA/missing-data issues * finishing group-by * incorporating the start of label arrays * incorporating a meta-object * a few new dtypes (variable-length string, varialbe-length unicode and an enum type) * adding ufunc support for flexible dtypes and possibly structured arrays * allowing generalized ufuncs to work on more kinds of arrays besides just contiguous * improving the ability for NumPy to receive JIT-generated function pointers for ufuncs and other calculation opportunities * adding "filters" to Input and Output * simple computed fields for dtypes * accepting a Data-Type specification as a class or JSON file * work towards improving the dtype-addition mechanism * re-factoring of code so that it can compile with a C++ compiler and be minimally dependent on Python data-structures. * NumPy 2.0 to come out in January of 2013. Mark Wiebe and I will post to this list a document that explains some of it's proposed features and enhancements. I won't steal his thunder for some of the things he is working on. If there are code issues people would like to see addressed, it would be a great time to speak up and/or propose something that you would like to see. In general NumPy 1.8 will have new features that need to be explored in order that NumPy 2.0 has enough code "experience" in order to be as useful as possible. I recognize that NumPy 1.8 has quite a few proposed features. These have been building up and are the big reason I've committed so many resources to NumPy. The feature-list did not just come out of my head. They are the result of talking and interacting with many NumPy users and watching the code get used (and not used) in the real world. This will be a faster pace of development. But, all of this will be in the open. If the NumPy 2.0 schedule is too aggressive, then we will have a NumPy 1.9 release in order to allow features to come out. Thanks, -Travis

37 196