From pierre.haessig at crans.org Thu Mar 1 03:37:25 2012 From: pierre.haessig at crans.org (Pierre Haessig) Date: Thu, 01 Mar 2012 09:37:25 +0100 Subject: [Numpy-discussion] IPython 0.12 just entered Debian Testing Message-ID: <4F4F3545.6020209@crans.org> Hi, Just to start the new month on a light & happy topic : IPython 0.12 has entered Debian Testing ! -- Pierre (I'm not at all involved in the process that enabled IPython make its way to Testing. I've been watching this quite closely however. I "suspect" there was a decent amount of work on packaging all the dependencies to make this possible and I'm thankful to the guys who did it) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From jaakko.luttinen at aalto.fi Thu Mar 1 05:23:13 2012 From: jaakko.luttinen at aalto.fi (Jaakko Luttinen) Date: Thu, 1 Mar 2012 12:23:13 +0200 Subject: [Numpy-discussion] Special matrices with structure? In-Reply-To: <4F46442C.7090100@aalto.fi> References: <4F46442C.7090100@aalto.fi> Message-ID: <4F4F4E11.3020102@aalto.fi> On 02/23/2012 03:50 PM, Jaakko Luttinen wrote: > Hi! > > I was wondering whether it would be easy/possible/reasonable to have > classes for arrays that have special structure in order to use less > memory and speed up some computations? > > For instance: > - symmetric matrix could be stored in almost half the memory required by > a non-symmetric matrix > - diagonal matrix only needs to store the diagonal vector > - Toeplitz matrix only needs to store one or two vectors > - sparse matrix only needs to store non-zero elements (some > implementations in scipy.sparse) > - and so on Note to self: BLAS has lots of functions for matrices having special structure (symmetric, triangular, banded, ...), so I suppose it would "only" require some Python class wrappers which are compatible with ndarray/matrix. But I don't know how to make these classes compatible with generic numpy functions such as numpy.multiply/numpy.dot/etc.. -Jaakko > > If such classes were implemented, it would be nice if they worked with > numpy functions (dot, diag, ...) and operations (+, *, +=, ...) easily. > > I believe this has been discussed before but google didn't help a lot.. > > Regards, > Jaakko > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From takowl at gmail.com Thu Mar 1 05:26:35 2012 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 1 Mar 2012 10:26:35 +0000 Subject: [Numpy-discussion] IPython 0.12 just entered Debian Testing In-Reply-To: <4F4F3545.6020209@crans.org> References: <4F4F3545.6020209@crans.org> Message-ID: On 1 March 2012 08:37, Pierre Haessig wrote: > Just to start the new month on a light & happy topic : > IPython 0.12 has entered Debian Testing ! > Thanks to Julian Taylor for handling Debian packaging. IPython 0.12 is also in the upcoming Ubuntu 12.04. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhattersley at gmail.com Thu Mar 1 05:30:18 2012 From: rhattersley at gmail.com (Richard Hattersley) Date: Thu, 1 Mar 2012 10:30:18 +0000 Subject: [Numpy-discussion] Proposed Roadmap Overview Message-ID: +1 on the NEP guideline As part of a team building a scientific analysis library, I'm attempting to understand the current state of NumPy development and its likely future (with a view to contributing if appropriate). The proposed NEP process would make that a whole lot easier. And if nothing else, it would reduce the chance of me posting questions about topics that had already been discussed/decided! Without the process the NEPs become another potential source of confusion and mixed messages. On 1 March 2012 03:02, Travis Oliphant wrote: > I Would like to hear the opinions of others on that point, > but yes, I think that is an appropriate procedure. > > Travis > > -- > Travis Oliphant > (on a mobile) > 512-826-7480 > > > On Feb 29, 2012, at 10:54 AM, Matthew Brett > wrote: > > > Hi, > > > > On Wed, Feb 29, 2012 at 1:46 AM, Travis Oliphant > wrote: > >> We already use the NEP process for such decisions. This > discussion came from simply from the *idea* of writing such a NEP. > >> > >> Nothing has been decided. Only opinions have been shared > that might influence the NEP. This is all pretty premature, > though --- migration to C++ features on a trial branch is > some months away were it to happen. > > > > Fernando can correct me if I'm wrong, but I think he was asking a > > governance question. That is: would you (as BDF$N) consider the > > following guideline: > > > > "As a condition for accepting significant changes to Numpy, for each > > significant change, there will be a NEP. The NEP shall follow the > > same model as the Python PEPs - that is - there will be a summary of > > the changes, the issues arising, the for / against opinions and > > alternatives offered. There will usually be a draft implementation. > > The NEP will contain the resolution of the discussion as it > relates to > > the code" > > > > For example, the masked array NEP, although very > substantial, contains > > little discussion of the controversy arising, or the intended > > resolution of the controversy: > > > > > https://github.com/numpy/numpy/blob/3f685a1a990f7b6e5149c80b52 > 436fb4207e49f5/doc/neps/missing-data.rst > > > > I mean, although it is useful, it is not in the form of a PEP, as > > Fernando has described it. > > > > Would you accept extending the guidelines to the NEP format? > > > > Best, > > > > Matthew > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > From barthpi at gmail.com Thu Mar 1 06:35:43 2012 From: barthpi at gmail.com (Pierre Barthelemy) Date: Thu, 1 Mar 2012 12:35:43 +0100 Subject: [Numpy-discussion] Numpy interpolate: cut through 2D matrix Message-ID: Hello, for a data analysis tool i am programming, i need to plot a cut through a 2D graph. I then have a 2D array, and the indices start=(start_x,start_y) and stop=(stop_x,stop_y) that are the position of the starting point and stop point of the cut. The code i programmed is placed on the bottom. This code returns only value existing in the original array: if the cut should pass between the column index i and column index i+1, it returns anyway the value at column index i. Is it possible to use the numpy.interpolate library to make such that when passing between i and i+1, the function returns an interpolation of the graph between the points [row,column] and [row,column+1] ? def cut_matrix(array,start,stop,shift=0): ''' Draws a cut through a 2D array, between the positions start=(row,column) and stop=(row,column) ''' n_row=array.shape[0] n_col=array.shape[1] if abs(start[1]-stop[1])>abs(start[0]-stop[0]): if stop[1]n_col or min(col_index)<0: print 'Error: column index not in range' raise IndexError return if max(row_index)>n_row or min(row_index)<0: print 'Error: row index not in range' raise IndexError return return array[row_index,col_index] -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Thu Mar 1 06:43:05 2012 From: schut at sarvision.nl (Vincent Schut) Date: Thu, 01 Mar 2012 12:43:05 +0100 Subject: [Numpy-discussion] Numpy interpolate: cut through 2D matrix In-Reply-To: References: Message-ID: On 03/01/2012 12:35 PM, Pierre Barthelemy wrote: > Hello, > > for a data analysis tool i am programming, i need to plot a cut through > a 2D graph. I then have a 2D array, and the indices > start=(start_x,start_y) and stop=(stop_x,stop_y) that are the position > of the starting point and stop point of the cut. The code i programmed > is placed on the bottom. > > This code returns only value existing in the original array: if the cut > should pass between the column index i and column index i+1, it returns > anyway the value at column index i. > Is it possible to use the numpy.interpolate library to make such that > when passing between i and i+1, the function returns an interpolation of > the graph between the points [row,column] and [row,column+1] ? Pierre, if you have scipy next to numpy, have a look at scipy.ndimage.map_coordinates. In short, you define the coordinates you want your array to be interpolated at, and it will give you the interpolated values, using spline interpolation. Best, Vincent. From zayd.yakoubi at gmail.com Thu Mar 1 09:43:57 2012 From: zayd.yakoubi at gmail.com (Zayd YAKOUBI) Date: Thu, 1 Mar 2012 15:43:57 +0100 Subject: [Numpy-discussion] Jaccard & Hamming Problem Message-ID: Hello, I use the similarity measure "Jaccard" and "Hamming" of pckage Scipy.spacial.cdist (Python) in a clustering context, I applied to given typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. But I just know that they normally only applies to binary data. The function of these two similarity measures are not specified in the documentation: http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html . Does anyone of you can help me find these functions? Thank you in advance Regards, Zayd -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Thu Mar 1 09:59:00 2012 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 1 Mar 2012 08:59:00 -0600 Subject: [Numpy-discussion] Jaccard & Hamming Problem In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 8:43 AM, Zayd YAKOUBI wrote: > Hello, > > I use the similarity measure "Jaccard" and "Hamming" of pckage > Scipy.spacial.cdist (Python) in a clustering context, I applied to given > typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. But I > just know that they normally only applies to binary data. The function of > these two similarity measures are not specified in the documentation: > http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html > . Does anyone of you can help me find these functions? > Thank you in advance > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.hamming.html#scipy.spatial.distance.hamming http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jaccard.html#scipy.spatial.distance.jaccard Those are the nicely formatted versions of the docstrings of functions. You can also access these in an interactive shell, e.g. >>> from scipy.spatial.distance import hamming >>> help(hamming) or in ipython In [1] from scipy.spatial.distance import hamming In [2] hamming? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From zayd.yakoubi at gmail.com Thu Mar 1 10:10:15 2012 From: zayd.yakoubi at gmail.com (Zayd YAKOUBI) Date: Thu, 1 Mar 2012 16:10:15 +0100 Subject: [Numpy-discussion] Jaccard & Hamming Problem In-Reply-To: References: Message-ID: thank you very much, In fact, the functions of these two measures are for binary vectors, and I have not found their extension to real data such as: 0.7, 0.9, 1.7 .... Knowing that I applied to this data and it worked well.. Have an idea about the version of these functions for this type of data ? thank you for your help Saisissez du texte, l'adresse d'un site Web ou importez un document ? traduire. Annuler Alpha Regards, Zayd 2012/3/1 Warren Weckesser > > > On Thu, Mar 1, 2012 at 8:43 AM, Zayd YAKOUBI wrote: > >> Hello, >> >> I use the similarity measure "Jaccard" and "Hamming" of pckage >> Scipy.spacial.cdist (Python) in a clustering context, I applied to given >> typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. >> But I just know that they normally only applies to binary data. The >> function of these two similarity measures are not specified in the >> documentation: >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html >> . Does anyone of you can help me find these functions? >> Thank you in advance >> >> > > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.hamming.html#scipy.spatial.distance.hamming > > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jaccard.html#scipy.spatial.distance.jaccard > > > Those are the nicely formatted versions of the docstrings of functions. > You can also access these in an interactive shell, e.g. > > >>> from scipy.spatial.distance import hamming > >>> help(hamming) > > or in ipython > > In [1] from scipy.spatial.distance import hamming > In [2] hamming? > > > Warren > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barthpi at gmail.com Thu Mar 1 10:20:14 2012 From: barthpi at gmail.com (Pierre Barthelemy) Date: Thu, 1 Mar 2012 16:20:14 +0100 Subject: [Numpy-discussion] Numpy fitting Message-ID: Dear all, i am writing a program for data analysis. One of the functions of this program gives the possibility to fit the functions. I therefore use the recipe described in : http://www.scipy.org/Cookbook/FittingData under the section "Simplifying the syntax". The code looks like this: class Parameter: def __init__(self, value): self.value = value self.fixed=False def set(self, value): if not self.fixed: self.value = value def __call__(self): return self.value def fit(function, parameters, y, x = None): def f(params): i = 0 for p in parameters: p.set(params[i]) i += 1 return y - function(x) if x is None: x = arange(y.shape[0]) p = [param() for param in parameters] out=optimize.leastsq(f, p, full_output=1) One thing that i would like to know is how can i get the error on the parameters ? From what i understood from the "Cookbook" page, and from the scipy manual ( http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html#scipy.optimize.leastsq), the second argument returned by the leastsq function gives access to these errors. std_error=std(y-function(x)) param_error=sqrt(diagonal(out[1])*std_error) The param_errors that i get in this case are extremely small. Much smaller than what i expected, and much smaller than what i can get fitting the function with matlab. So i guess i made an error here. Can someone tell me how i should do to retrieve the parameter errors ? Bests, Pierre -------------- next part -------------- An HTML attachment was scrubbed... URL: From barthpi at gmail.com Thu Mar 1 10:24:47 2012 From: barthpi at gmail.com (Pierre Barthelemy) Date: Thu, 1 Mar 2012 16:24:47 +0100 Subject: [Numpy-discussion] Numpy fitting In-Reply-To: References: Message-ID: Dear all, i am writing a program for data analysis. One of the functions of this program gives the possibility to fit the functions. I followed the recipe described in : http://www.scipy.org/Cookbook/FittingData under the section "Simplifying the syntax". To fit, i use the function: out=optimize.leastsq(f, p, full_output=1) where f is my function and p a list of parameters. One thing that i would like to know is how can i get the error on the parameters ? From what i understood from the "Cookbook" page, and from the scipy manual ( http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html#scipy.optimize.leastsq), the second argument returned by the leastsq function gives access to these errors. std_error=std(y-function(x)) param_error=sqrt(diagonal(out[1])*std_error) The param_errors that i get in this case are extremely small. Much smaller than what i expected, and much smaller than what i can get fitting the function with matlab. So i guess i made an error here. Can someone tell me how i should do to retrieve the parameter errors ? Bests, Pierre PS: i got the impression something went wrong with my previous message, sorry for that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Thu Mar 1 10:39:13 2012 From: shish at keba.be (Olivier Delalleau) Date: Thu, 1 Mar 2012 10:39:13 -0500 Subject: [Numpy-discussion] Numpy fitting In-Reply-To: References: Message-ID: Sorry I can't help, but I'd just suggest to post this on the scipy mailing list as you may get more replies there. -=- Olivier Le 1 mars 2012 10:24, Pierre Barthelemy a ?crit : > Dear all, > > i am writing a program for data analysis. One of the functions of this > program gives the possibility to fit the functions. I followed the recipe > described in : h*MailScanner soup?onne le lien suivant d'?tre une > tentative de fraude de la part de "www.scipy.org" *ttp://www.scipy.org/Cookbook/FittingData under > the section "Simplifying the syntax". > > To fit, i use the function: > > out=optimize.leastsq(f, p, full_output=1) > > where f is my function and p a list of parameters. > > One thing that i would like to know is how can i get the error on the > parameters ? From what i understood from the "Cookbook" page, and from the > scipy manual ( > http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html#scipy.optimize.leastsq), > the second argument returned by the leastsq function gives access to these > errors. > std_error=std(y-function(x)) > param_error=sqrt(diagonal(out[1])*std_error) > > The param_errors that i get in this case are extremely small. Much smaller > than what i expected, and much smaller than what i can get fitting the > function with matlab. So i guess i made an error here. > > Can someone tell me how i should do to retrieve the parameter errors ? > > Bests, > > Pierre > > PS: i got the impression something went wrong with my previous message, > sorry for that. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Mar 1 12:30:18 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Mar 2012 12:30:18 -0500 Subject: [Numpy-discussion] Numpy fitting In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 10:39 AM, Olivier Delalleau wrote: > Sorry I can't help, but I'd just suggest to post this on the scipy mailing > list as you may get more replies there. > > -=- Olivier > > Le 1 mars 2012 10:24, Pierre Barthelemy a ?crit : >> >> Dear all, >> >> i am writing a program for data analysis. One of the functions of this >> program gives the possibility to fit the functions. I followed the recipe >> described in : hMailScanner soup?onne le lien suivant d'?tre une tentative >> de fraude de la part de "www.scipy.org" >> ttp://www.scipy.org/Cookbook/FittingData?under the section "Simplifying the >> syntax". http://translate.google.com/#auto|en|MailScanner%20soup%C3%A7onne%20le%20lien%20suivant%20d'%C3%AAtre%20une%20tentative%20de%20fraude%20de%20la%20part%20de%20%22www.scipy.org%22 :) Josef >> >> To fit, i use the function: >> >> out=optimize.leastsq(f, p, full_output=1) >> >> where f is my function and p a list of parameters. >> >> One thing that i would like to know is how can i get the error on the >> parameters ? From what i understood from the "Cookbook" page, and from the >> scipy manual >> (http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html#scipy.optimize.leastsq), >> the second argument returned by the leastsq function gives access to these >> errors. >> std_error=std(y-function(x)) >> param_error=sqrt(diagonal(out[1])*std_error) >> >> The param_errors that i get in this case are extremely small. Much smaller >> than what i expected, and much smaller than what i can get fitting the >> function with matlab. So i guess i made an error here. >> >> Can someone tell me how i should do to retrieve the parameter errors ? >> >> Bests, >> >> Pierre >> >> PS: i got the impression something went wrong with my previous message, >> sorry for that. >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jkington at wisc.edu Thu Mar 1 17:44:23 2012 From: jkington at wisc.edu (Joe Kington) Date: Thu, 01 Mar 2012 16:44:23 -0600 Subject: [Numpy-discussion] Floating point "close" function? Message-ID: Is there a numpy function for testing floating point equality that returns a boolean array? I'm aware of np.allclose, but I need a boolean array. Properly handling NaN's and Inf's (as allclose does) would be a nice bonus. I wrote the function below to do this, but I suspect there's a method in numpy that I missed. import numpy as np def close(a, b, rtol=1.e-5, atol=1.e-8, check_invalid=True): """Similar to numpy.allclose, but returns a boolean array. See numpy.allclose for an explanation of *rtol* and *atol*.""" def within_tol(x, y, atol, rtol): return np.less_equal(np.abs(x-y), atol + rtol * np.abs(y)) x = np.array(a, copy=False) y = np.array(b, copy=False) if not check_invalid: return within_tol(x, y, atol, rtol) xfin = np.isfinite(x) yfin = np.isfinite(y) if np.all(xfin) and np.all(yfin): return within_tol(x, y, atol, rtol) else: # Avoid subtraction with infinite/nan values... cond = np.zeros(np.broadcast(x, y).shape, dtype=np.bool) mask = xfin & yfin cond[mask] = within_tol(x[mask], y[mask], atol, rtol) # Inf and -Inf equality... cond[~mask] = (x[~mask] == y[~mask]) # NaN equality... cond[np.isnan(x) & np.isnan(y)] = True return cond # A few quick tests... assert np.any(close(0.300001, np.array([0.1, 0.2, 0.3, 0.4]))) x = np.array([0.1, np.nan, np.inf, -np.inf]) y = np.array([0.1000001, np.nan, np.inf, -np.inf]) assert np.all(close(x, y)) x = np.array([0.1, 0.2, np.inf]) y = np.array([0.101, np.nan, 0.2]) assert not np.all(close(x, y)) Thanks, -Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Mar 1 20:13:03 2012 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Mar 2012 20:13:03 -0500 Subject: [Numpy-discussion] Jaccard & Hamming Problem In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 10:10 AM, Zayd YAKOUBI wrote: > thank you very much, > In fact, the functions of these two measures are? for binary vectors, and I > have not found their extension to real data such as: 0.7, 0.9, 1.7 .... > Knowing that I applied to this data and it worked well.. > > Have an idea about the version of these functions for this type of data ? for hamming just guessing : 1 - np.mean(x==y) which might depend on the implementation >>> spatial.distance.hamming([0,0.5,1,2], np.ones(4)) 0.75 >>> 1 - np.mean([0,0.5,1,2] == np.ones(4)) 0.75 >>> spatial.distance.hamming([0,0.5,1,1], np.ones(4)) 0.5 >>> 1 - np.mean([0,0.5,1,1] == np.ones(4)) 0.5 However I wouldn't trust it for floating point numbers, unless you are sure about the floating point representation >>> [0,0.5,3,2], [0,0.5,np.sqrt(3)**2,2] ([0, 0.5, 3, 2], [0, 0.5, 2.9999999999999996, 2]) >>> spatial.distance.hamming([0,0.5,3,2], [0,0.5,np.sqrt(3)**2,2]) 0.25 >>> spatial.distance.hamming([0,0.5,3,2], [0,0.5,3,2]) 0.0 Josef > > thank you for your help > Saisissez du texte, l'adresse d'un site Web ou importez un document ? > traduire. > Annuler > Alpha > > Regards, > Zayd > > > > > 2012/3/1 Warren Weckesser >> >> >> >> On Thu, Mar 1, 2012 at 8:43 AM, Zayd YAKOUBI >> wrote: >>> >>> Hello, >>> >>> I use the similarity measure "Jaccard" and "Hamming" of pckage >>> Scipy.spacial.cdist (Python) in a clustering context, I applied to given >>> typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. But I >>> just know that they normally only applies to binary data. The function of >>> these two similarity measures are not specified in the documentation: >>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html. >>> Does anyone of you can help me find these functions? >>> Thank you in advance >>> >> >> >> >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.hamming.html#scipy.spatial.distance.hamming >> >> >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jaccard.html#scipy.spatial.distance.jaccard >> >> >> Those are the nicely formatted versions of the docstrings of functions. >> You can also access these in an interactive shell, e.g. >> >> >>> from scipy.spatial.distance import hamming >> >>> help(hamming) >> >> or in ipython >> >> In [1] from scipy.spatial.distance import hamming >> In [2] hamming? >> >> >> Warren >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From jayvius at gmail.com Fri Mar 2 01:58:26 2012 From: jayvius at gmail.com (Jay Bourque) Date: Fri, 2 Mar 2012 00:58:26 -0600 Subject: [Numpy-discussion] Possible roadmap addendum: building better text file readers In-Reply-To: References: <1330092347-sup-3918@rohan> <1330207186-sup-1957@rohan> <1330351883-sup-9943@rohan> <1330365437-sup-7898@rohan> <1330387831-sup-839@rohan> Message-ID: *In an effort to build a consensus of what numpy's New and Improved text file readers should look like, I've put together a short list of the main points discussed in this thread so far:* * * 1. Loading text files using loadtxt/genfromtxt need a significant performance boost (I think at least an order of magnitude increase in performance is very doable based on what I've seen with Erin's recfile code) 2. Improved memory usage. Memory used for reading in a text file shouldn?t be more than the file itself, and less if only reading a subset of file. 3. Keep existing interfaces for reading text files (loadtxt, genfromtxt, etc). No new ones. 4. Underlying code should keep IO iteration and transformation of data separate (awaiting more thoughts from Travis on this). 5. Be able to plug in different transformations of data at low level (also awaiting more thoughts from Travis). 6. memory mapping of text files? 7. Eventually reduce memory usage even more by using same object for duplicate values in array (depends on implementing enum dtype?) Anything else? -Jay Bourque continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwl_b at wp.pl Fri Mar 2 05:47:22 2012 From: pwl_b at wp.pl (=?utf-8?b?UGF3ZcWC?= Biernat) Date: Fri, 2 Mar 2012 10:47:22 +0000 (UTC) Subject: [Numpy-discussion] [Numpy] quadruple precision References: <4F4E81E7.3040107@crans.org> <2078A981-04C4-4A4A-9BCA-B363AEC1D23F@continuum.io> Message-ID: Charles R Harris gmail.com> writes: > > > The quad precision library has been there for a while, and quad precision is also supported by the Intel compiler. I don't know about MSVC. Intel has been working on adding quad precision to their hardware for several years and there is an IEEE spec for it, so some day it will be here, but it isn't here yet. It's a bit sad, I could use quad precision in FORTRAN on a VAX 25 years ago. Mind, I only needed it once ;) I suppose lack of pressing need accounts for the delay.Chuck > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Waiting for hardware support can last forever, and __float128 is already here. Despite being software supported, it is still reasonably fast for people who need it. The slow-down depends on a case and optimization and can be roughly from x2 (using sse) to x10 (without optimization), but you gain x2 significant digits when compared to double, see for example http://locklessinc.com/articles/classifying_floats/. This is still faster than mpfr for example. And gcc-4.6 already supports __float128 on a number of machines: i386, x86_64, ia64 and HP-UX. Also fftw now supports binary128: http://www.fftw.org/release-notes.html (although this might not be the most representative numerical software, it confirms that it is unlikely that __float128 will be ignored by the others unless hardware supported). The portability is broken for numpy.float128 anyway (as I understand, it behaves in different ways on different architectures), so adding a new type (call it, say, quad128) that properly supports binary128 shouldn't be a drawback. Later on, when the hardware support for binary128 shows up, the quad128 will be already there. Pawe?. From njs at pobox.com Fri Mar 2 08:39:47 2012 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Mar 2012 13:39:47 +0000 Subject: [Numpy-discussion] [Numpy] quadruple precision In-Reply-To: References: <4F4E81E7.3040107@crans.org> <2078A981-04C4-4A4A-9BCA-B363AEC1D23F@continuum.io> Message-ID: On Mar 2, 2012 10:48 AM, "Pawe? Biernat" wrote: > The portability is broken for numpy.float128 anyway (as I understand, > it behaves in different ways on different architectures), so adding a > new type (call it, say, quad128) that properly supports binary128 > shouldn't be a drawback. Later on, when the hardware support for > binary128 shows up, the quad128 will be already there. There's already been movement to deprecate using float128 as the name for machine-specific long doubles. This just gives even more reason. If/when someone adds __float128 support to numpy we should really just call it float128, not quad128. (This would even be backwards compatible, since float128 currently gives no guarantees on precision or representation.) - n -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.haessig at crans.org Fri Mar 2 09:39:31 2012 From: pierre.haessig at crans.org (Pierre Haessig) Date: Fri, 02 Mar 2012 15:39:31 +0100 Subject: [Numpy-discussion] [Numpy] quadruple precision In-Reply-To: References: <4F4E81E7.3040107@crans.org> <2078A981-04C4-4A4A-9BCA-B363AEC1D23F@continuum.io> Message-ID: <4F50DBA3.6080201@crans.org> Le 02/03/2012 14:39, Nathaniel Smith a ?crit : > If/when someone adds __float128 support to numpy we should really just > call it float128 I agree! Other types could become "float80_128" and "float80_96", as mentioned about a week ago by Matthew. -- Pierre -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From nouiz at nouiz.org Fri Mar 2 10:26:48 2012 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Fri, 2 Mar 2012 10:26:48 -0500 Subject: [Numpy-discussion] Possible roadmap addendum: building better text file readers In-Reply-To: References: Message-ID: Hi, mmap can give a speed up in some case, but slow down in other. So care must be taken when using it. For example, the speed difference between read and mmap are not the same when the file is local and when it is on NFS. On NFS, you need to read bigger chunk to make it worthwhile. Another example is on an SMP computer. If for example you have a 8 cores computer but have only enought ram for 1 or 2 copy of your dataset, using mmap is a bad idea. If you read the file by chunk normally the OS will keep the file in its cache in ram. So if you launch 8 jobs, they will all use the system cache to shared the data. If you use mmap, I think this bypass the OS cache. So you will always read the file. On NFS with a cluster of computer, this can bring a high load on the file server. So having a way to specify to use or not to use mmap would be great as you can't always guess the right thing to do. (Except if I'm wrong and this don't by pass the OS cache) Anyway, it is great to see people work in this problem, this was just a few comments I had in mind when I read this thread. Fr?d?ric On Sun, Feb 26, 2012 at 4:22 PM, Warren Weckesser wrote: > > > On Sun, Feb 26, 2012 at 3:00 PM, Nathaniel Smith wrote: >> >> On Sun, Feb 26, 2012 at 7:58 PM, Warren Weckesser >> wrote: >> > Right, I got that.? Sorry if the placement of the notes about how to >> > clear >> > the cache seemed to imply otherwise. >> >> OK, cool, np. >> >> >> Clearing the disk cache is very important for getting meaningful, >> >> repeatable benchmarks in code where you know that the cache will >> >> usually be cold and where hitting the disk will have unpredictable >> >> effects (i.e., pretty much anything doing random access, like >> >> databases, which have complicated locality patterns, you may or may >> >> not trigger readahead, etc.). But here we're talking about pure >> >> sequential reads, where the disk just goes however fast it goes, and >> >> your code can either keep up or not. >> >> >> >> One minor point where the OS interface could matter: it's good to set >> >> up your code so it can use mmap() instead of read(), since this can >> >> reduce overhead. read() has to copy the data from the disk into OS >> >> memory, and then from OS memory into your process's memory; mmap() >> >> skips the second step. >> > >> > Thanks for the tip.? Do you happen to have any sample code that >> > demonstrates >> > this?? I'd like to explore this more. >> >> No, I've never actually run into a situation where I needed it myself, >> but I learned the trick from Tridge so I tend to believe it :-). >> mmap() is actually a pretty simple interface -- the only thing I'd >> watch out for is that you want to mmap() the file in pieces (so as to >> avoid VM exhaustion on 32-bit systems), but you want to use pretty big >> pieces (because each call to mmap()/munmap() has overhead). So you >> might want to use chunks in the 32-128 MiB range. Or since I guess >> you're probably developing on a 64-bit system you can just be lazy and >> mmap the whole file for initial testing. git uses mmap, but I'm not >> sure it's very useful example code. >> >> Also it's not going to do magic. Your code has to be fairly quick >> before avoiding a single memcpy() will be noticeable. >> >> HTH, > > > > Yes, thanks! ? I'm working on a mmap version now.? I'm very curious to see > just how much of an improvement it can give. > > Warren > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From xscript at gmx.net Fri Mar 2 11:37:11 2012 From: xscript at gmx.net (=?utf-8?Q?Llu=C3=ADs?=) Date: Fri, 02 Mar 2012 17:37:11 +0100 Subject: [Numpy-discussion] Possible roadmap addendum: building better text file readers In-Reply-To: (=?utf-8?B?IkZyw6lkw6lyaWM=?= Bastien"'s message of "Fri, 2 Mar 2012 10:26:48 -0500") References: Message-ID: <87r4xbvu14.fsf@ginnungagap.bsc.es> Fr?d?ric Bastien writes: > Hi, > mmap can give a speed up in some case, but slow down in other. So care > must be taken when using it. For example, the speed difference between > read and mmap are not the same when the file is local and when it is > on NFS. On NFS, you need to read bigger chunk to make it worthwhile. > Another example is on an SMP computer. If for example you have a 8 > cores computer but have only enought ram for 1 or 2 copy of your > dataset, using mmap is a bad idea. If you read the file by chunk > normally the OS will keep the file in its cache in ram. So if you > launch 8 jobs, they will all use the system cache to shared the data. > If you use mmap, I think this bypass the OS cache. So you will always > read the file. Not according to mmap(2): MAP_SHARED Share this mapping. Updates to the mapping are visible to other processes that map this file, and are carried through to the underlying file. The file may not actually be updated until msync(2) or munmap() is called. My understanding is that all processes will use exactly the same physical memory, and swapping that memory will use the file itself. > On NFS with a cluster of computer, this can bring a > high load on the file server. So having a way to specify to use or not > to use mmap would be great as you can't always guess the right thing > to do. (Except if I'm wrong and this don't by pass the OS cache) > Anyway, it is great to see people work in this problem, this was just > a few comments I had in mind when I read this thread. Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth From matthew.brett at gmail.com Fri Mar 2 18:36:26 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 2 Mar 2012 18:36:26 -0500 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk Message-ID: Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Cheers, Matthew From charlesr.harris at gmail.com Fri Mar 2 20:59:24 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 2 Mar 2012 18:59:24 -0700 Subject: [Numpy-discussion] Where did Numpy-svn come from? Message-ID: Github commits now send out mail: reply-toGitHub tonumpy-svn at scipy.org This is a recent change, did we change something? I liked getting mail directly from github better. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Mar 2 21:05:57 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 2 Mar 2012 19:05:57 -0700 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett wrote: > Hi, > > Sorry that this report is not complete, I don't have full access to > this box but, on a Debian squeeze machine running linux > 2.6.32-5-sparc64-smp: > > nosetests > ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values > > test_user_missing_values (test_io.TestFromTxt) ... Bus error > > This on current master : 1.7.0.dev-b9872b4 > > Hmm, some tests might have been recently enabled. Any chance of doing a bisection? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Mar 3 00:07:55 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 3 Mar 2012 00:07:55 -0500 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: Hi, On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris wrote: > > > On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett > wrote: >> >> Hi, >> >> Sorry that this report is not complete, I don't have full access to >> this box but, on a Debian squeeze machine running linux >> 2.6.32-5-sparc64-smp: >> >> nosetests >> ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values >> >> test_user_missing_values (test_io.TestFromTxt) ... Bus error >> >> This on current master : 1.7.0.dev-b9872b4 >> > > Hmm, some tests might have been recently enabled. Any chance of doing a > bisection? I'm on it - will get back to you tomorrow. See you, Matthew From ralf.gommers at googlemail.com Sat Mar 3 08:59:33 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 3 Mar 2012 14:59:33 +0100 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: > Is there a numpy function for testing floating point equality that returns > a boolean array? > > I'm aware of np.allclose, but I need a boolean array. Properly handling > NaN's and Inf's (as allclose does) would be a nice bonus. > > I wrote the function below to do this, but I suspect there's a method in > numpy that I missed. > I don't think such a function exists, would be nice to have. How about just adding a keyword "return_array" to allclose to do so? Ralf > import numpy as np > > def close(a, b, rtol=1.e-5, atol=1.e-8, check_invalid=True): > """Similar to numpy.allclose, but returns a boolean array. > See numpy.allclose for an explanation of *rtol* and *atol*.""" > def within_tol(x, y, atol, rtol): > return np.less_equal(np.abs(x-y), atol + rtol * np.abs(y)) > x = np.array(a, copy=False) > y = np.array(b, copy=False) > if not check_invalid: > return within_tol(x, y, atol, rtol) > xfin = np.isfinite(x) > yfin = np.isfinite(y) > if np.all(xfin) and np.all(yfin): > return within_tol(x, y, atol, rtol) > else: > # Avoid subtraction with infinite/nan values... > cond = np.zeros(np.broadcast(x, y).shape, dtype=np.bool) > mask = xfin & yfin > cond[mask] = within_tol(x[mask], y[mask], atol, rtol) > # Inf and -Inf equality... > cond[~mask] = (x[~mask] == y[~mask]) > # NaN equality... > cond[np.isnan(x) & np.isnan(y)] = True > return cond > > # A few quick tests... > assert np.any(close(0.300001, np.array([0.1, 0.2, 0.3, 0.4]))) > > x = np.array([0.1, np.nan, np.inf, -np.inf]) > y = np.array([0.1000001, np.nan, np.inf, -np.inf]) > assert np.all(close(x, y)) > > x = np.array([0.1, 0.2, np.inf]) > y = np.array([0.101, np.nan, 0.2]) > assert not np.all(close(x, y)) > > > Thanks, > -Joe > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 3 09:05:30 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 14:05:30 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 13:59, Ralf Gommers wrote: > > > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: >> >> Is there a numpy function for testing floating point equality that returns >> a boolean array? >> >> I'm aware of np.allclose, but I need a boolean array.? Properly handling >> NaN's and Inf's (as allclose does) would be a nice bonus. >> >> I wrote the function below to do this, but I suspect there's a method in >> numpy that I missed. > > > I don't think such a function exists, would be nice to have. How about just > adding a keyword "return_array" to allclose to do so? As a general design principle, adding a boolean flag that changes the return type is worse than making a new function. -- Robert Kern From ralf.gommers at googlemail.com Sat Mar 3 09:31:01 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 3 Mar 2012 15:31:01 +0100 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern wrote: > On Sat, Mar 3, 2012 at 13:59, Ralf Gommers > wrote: > > > > > > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: > >> > >> Is there a numpy function for testing floating point equality that > returns > >> a boolean array? > >> > >> I'm aware of np.allclose, but I need a boolean array. Properly handling > >> NaN's and Inf's (as allclose does) would be a nice bonus. > >> > >> I wrote the function below to do this, but I suspect there's a method in > >> numpy that I missed. > > > > > > I don't think such a function exists, would be nice to have. How about > just > > adding a keyword "return_array" to allclose to do so? > > As a general design principle, adding a boolean flag that changes the > return type is worse than making a new function. > That's certainly true as a general principle. Do you have a concrete suggestion in this case though? Because this is also bad: >>> np. Display all 561 possibilities? (y or n) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 3 09:34:31 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 14:34:31 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 14:31, Ralf Gommers wrote: > > > On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern wrote: >> >> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: >> >> >> >> Is there a numpy function for testing floating point equality that >> >> returns >> >> a boolean array? >> >> >> >> I'm aware of np.allclose, but I need a boolean array.? Properly >> >> handling >> >> NaN's and Inf's (as allclose does) would be a nice bonus. >> >> >> >> I wrote the function below to do this, but I suspect there's a method >> >> in >> >> numpy that I missed. >> > >> > >> > I don't think such a function exists, would be nice to have. How about >> > just >> > adding a keyword "return_array" to allclose to do so? >> >> As a general design principle, adding a boolean flag that changes the >> return type is worse than making a new function. > > > That's certainly true as a general principle. Do you have a concrete > suggestion in this case though? np.close() > Because this is also bad: >>>> np. > Display all 561 possibilities? (y or n) Not as bad as overloading np.allclose(x,y,return_array=True). Or deprecating np.allclose() in favor of np.close().all(). -- Robert Kern From ben.root at ou.edu Sat Mar 3 10:22:07 2012 From: ben.root at ou.edu (Benjamin Root) Date: Sat, 3 Mar 2012 09:22:07 -0600 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Saturday, March 3, 2012, Robert Kern wrote: > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers wrote: >> >> >> On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern wrote: >>> >>> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers >>> wrote: >>> > >>> > >>> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: >>> >> >>> >> Is there a numpy function for testing floating point equality that >>> >> returns >>> >> a boolean array? >>> >> >>> >> I'm aware of np.allclose, but I need a boolean array. Properly >>> >> handling >>> >> NaN's and Inf's (as allclose does) would be a nice bonus. >>> >> >>> >> I wrote the function below to do this, but I suspect there's a method >>> >> in >>> >> numpy that I missed. >>> > >>> > >>> > I don't think such a function exists, would be nice to have. How about >>> > just >>> > adding a keyword "return_array" to allclose to do so? >>> >>> As a general design principle, adding a boolean flag that changes the >>> return type is worse than making a new function. >> >> >> That's certainly true as a general principle. Do you have a concrete >> suggestion in this case though? > > np.close() > When I read that, I mentally think of "close" as in closing a file. I think we need a synonym. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 3 10:26:27 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 15:26:27 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 15:22, Benjamin Root wrote: > > > On Saturday, March 3, 2012, Robert Kern wrote: >> On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> wrote: >>> >>> >>> On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern >>> wrote: >>>> >>>> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers >>>> wrote: >>>> > >>>> > >>>> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington >>>> > wrote: >>>> >> >>>> >> Is there a numpy function for testing floating point equality that >>>> >> returns >>>> >> a boolean array? >>>> >> >>>> >> I'm aware of np.allclose, but I need a boolean array.? Properly >>>> >> handling >>>> >> NaN's and Inf's (as allclose does) would be a nice bonus. >>>> >> >>>> >> I wrote the function below to do this, but I suspect there's a method >>>> >> in >>>> >> numpy that I missed. >>>> > >>>> > >>>> > I don't think such a function exists, would be nice to have. How about >>>> > just >>>> > adding a keyword "return_array" to allclose to do so? >>>> >>>> As a general design principle, adding a boolean flag that changes the >>>> return type is worse than making a new function. >>> >>> >>> That's certainly true as a general principle. Do you have a concrete >>> suggestion in this case though? >> >> np.close() >> > > When I read that, I mentally think of "close" as in closing a file. ?I think > we need a synonym. np.isclose() -- Robert Kern From robert.kern at gmail.com Sat Mar 3 10:27:14 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 15:27:14 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 14:34, Robert Kern wrote: > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers wrote: >> Because this is also bad: >>>>> np. >> Display all 561 possibilities? (y or n) > > Not as bad as overloading np.allclose(x,y,return_array=True). Or > deprecating np.allclose() in favor of np.close().all(). I screwed up this paragraph. I meant that as "Another alternative would be to deprecate ...". -- Robert Kern From shish at keba.be Sat Mar 3 10:51:13 2012 From: shish at keba.be (Olivier Delalleau) Date: Sat, 3 Mar 2012 10:51:13 -0500 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: Le 3 mars 2012 10:27, Robert Kern a ?crit : > On Sat, Mar 3, 2012 at 14:34, Robert Kern wrote: > > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers > wrote: > > >> Because this is also bad: > >>>>> np. > >> Display all 561 possibilities? (y or n) > > > > Not as bad as overloading np.allclose(x,y,return_array=True). Or > > deprecating np.allclose() in favor of np.close().all(). > > I screwed up this paragraph. I meant that as "Another alternative > would be to deprecate ...". > np.close().all() would probably be a lot less efficient in terms of CPU / memory though, wouldn't it? -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 3 11:03:23 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 16:03:23 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: > Le 3 mars 2012 10:27, Robert Kern a ?crit : >> >> On Sat, Mar 3, 2012 at 14:34, Robert Kern wrote: >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> > wrote: >> >> >> Because this is also bad: >> >>>>> np. >> >> Display all 561 possibilities? (y or n) >> > >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or >> > deprecating np.allclose() in favor of np.close().all(). >> >> I screwed up this paragraph. I meant that as "Another alternative >> would be to deprecate ...". > > > np.close().all() would probably be a lot less efficient in terms of CPU / > memory though, wouldn't it? No. np.allclose() is essentially doing exactly this already. -- Robert Kern From shish at keba.be Sat Mar 3 11:06:51 2012 From: shish at keba.be (Olivier Delalleau) Date: Sat, 3 Mar 2012 11:06:51 -0500 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: Le 3 mars 2012 11:03, Robert Kern a ?crit : > On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: > > Le 3 mars 2012 10:27, Robert Kern a ?crit : > >> > >> On Sat, Mar 3, 2012 at 14:34, Robert Kern > wrote: > >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers < > ralf.gommers at googlemail.com> > >> > wrote: > >> > >> >> Because this is also bad: > >> >>>>> np. > >> >> Display all 561 possibilities? (y or n) > >> > > >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or > >> > deprecating np.allclose() in favor of np.close().all(). > >> > >> I screwed up this paragraph. I meant that as "Another alternative > >> would be to deprecate ...". > > > > > > np.close().all() would probably be a lot less efficient in terms of CPU / > > memory though, wouldn't it? > > No. np.allclose() is essentially doing exactly this already. > Ok. What about then, np.allclose() could theoretically be a lot more efficient in terms of CPU / memory? ;) -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From lpc at cmu.edu Sat Mar 3 11:07:16 2012 From: lpc at cmu.edu (Luis Pedro Coelho) Date: Sat, 03 Mar 2012 16:07:16 +0000 Subject: [Numpy-discussion] C++ Example Message-ID: <30965644.bxiDOaWOx7@rabbit> Hi, I sort of missed the big C++ discussion, but I'd like to give some examples of how writing code can become much simpler if you are based on C++. This is from my mahotas package, which has a thin C++ wrapper around numpy's C API https://github.com/luispedro/mahotas/blob/master/mahotas/_morph.cpp and it implements multi-type greyscale erosion. // numpy::aligned_array wraps PyArrayObject* template void erode(numpy::aligned_array res, numpy::aligned_array array, numpy::aligned_array Bc) { // Release the GIL using RAII gil_release nogil; const int N = res.size(); typename numpy::aligned_array::iterator iter = array.begin(); // this is adapted from scipy.ndimage. // it implements the convolution-like filtering. filter_iterator filter(res.raw_array(), Bc.raw_array(), EXTEND_NEAREST, is_bool(T())); const int N2 = filter.size(); T* rpos = res.data(); for (int i = 0; i != N; ++i, ++rpos, filter.iterate_both(iter)) { T value = std::numeric_limits::max(); for (int j = 0; j != N2; ++j) { T arr_val = T(); filter.retrieve(iter, j, arr_val); value = std::min(value, erode_sub(arr_val, filter[j])); } *rpos = value; } } If you compare this with the equivalent scipy.ndimage function, which is very good C code (but mostly write-only?in fact, ndimage has not been maintainable because it is so hard [at least for me, I've tried]): int NI_BinaryErosion(PyArrayObject* input, PyArrayObject* strct, PyArrayObject* mask, PyArrayObject* output, int bdr_value, npy_intp *origins, int invert, int center_is_true, int* changed, NI_CoordinateList **coordinate_list) { npy_intp struct_size = 0, *offsets = NULL, size, *oo, jj; npy_intp ssize, block_size = 0, *current = NULL, border_flag_value; int kk, true, false, msk_value; NI_Iterator ii, io, mi; NI_FilterIterator fi; Bool *ps, out = 0; char *pi, *po, *pm = NULL; NI_CoordinateBlock *block = NULL; ps = (Bool*)PyArray_DATA(strct); ssize = 1; for(kk = 0; kk < strct->nd; kk++) ssize *= strct->dimensions[kk]; for(jj = 0; jj < ssize; jj++) if (ps[jj]) ++struct_size; if (mask) { if (!NI_InitPointIterator(mask, &mi)) return 0; pm = (void *)PyArray_DATA(mask); } /* calculate the filter offsets: */ if (!NI_InitFilterOffsets(input, ps, strct->dimensions, origins, NI_EXTEND_CONSTANT, &offsets, &border_flag_value, NULL)) goto exit; /* initialize input element iterator: */ if (!NI_InitPointIterator(input, &ii)) goto exit; /* initialize output element iterator: */ if (!NI_InitPointIterator(output, &io)) goto exit; /* initialize filter iterator: */ if (!NI_InitFilterIterator(input->nd, strct->dimensions, struct_size, input->dimensions, origins, &fi)) goto exit; /* get data pointers an size: */ pi = (void *)PyArray_DATA(input); po = (void *)PyArray_DATA(output); size = 1; for(kk = 0; kk < input->nd; kk++) size *= input->dimensions[kk]; if (invert) { bdr_value = bdr_value ? 0 : 1; true = 0; false = 1; } else { bdr_value = bdr_value ? 1 : 0; true = 1; false = 0; } if (coordinate_list) { block_size = LIST_SIZE / input->nd / sizeof(int); if (block_size < 1) block_size = 1; if (block_size > size) block_size = size; *coordinate_list = NI_InitCoordinateList(block_size, input->nd); if (!*coordinate_list) goto exit; } /* iterator over the elements: */ oo = offsets; *changed = 0; msk_value = 1; for(jj = 0; jj < size; jj++) { int pchange = 0; if (mask) { switch(mask->descr->type_num) { CASE_GET_MASK(msk_value, pm, Bool); CASE_GET_MASK(msk_value, pm, UInt8); CASE_GET_MASK(msk_value, pm, UInt16); CASE_GET_MASK(msk_value, pm, UInt32); #if HAS_UINT64 CASE_GET_MASK(msk_value, pm, UInt64); #endif CASE_GET_MASK(msk_value, pm, Int8); CASE_GET_MASK(msk_value, pm, Int16); CASE_GET_MASK(msk_value, pm, Int32); CASE_GET_MASK(msk_value, pm, Int64); CASE_GET_MASK(msk_value, pm, Float32); CASE_GET_MASK(msk_value, pm, Float64); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); return 0; } } switch (input->descr->type_num) { CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Bool, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt8, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt16, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); #if HAS_UINT64 CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); #endif CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int8, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int16, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Float32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Float64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); goto exit; } switch (output->descr->type_num) { CASE_OUTPUT(po, out, Bool); CASE_OUTPUT(po, out, UInt8); CASE_OUTPUT(po, out, UInt16); CASE_OUTPUT(po, out, UInt32); #if HAS_UINT64 CASE_OUTPUT(po, out, UInt64); #endif CASE_OUTPUT(po, out, Int8); CASE_OUTPUT(po, out, Int16); CASE_OUTPUT(po, out, Int32); CASE_OUTPUT(po, out, Int64); CASE_OUTPUT(po, out, Float32); CASE_OUTPUT(po, out, Float64); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); goto exit; } if (pchange) { *changed = 1; if (coordinate_list) { if (block == NULL || block->size == block_size) { block = NI_CoordinateListAddBlock(*coordinate_list); current = block->coordinates; } for(kk = 0; kk < input->nd; kk++) *current++ = ii.coordinates[kk]; block->size++; } } if (mask) { NI_FILTER_NEXT3(fi, ii, io, mi, oo, pi, po, pm); } else { NI_FILTER_NEXT2(fi, ii, io, oo, pi, po); } } exit: if (offsets) free(offsets); if (PyErr_Occurred()) { if (coordinate_list) { NI_FreeCoordinateList(*coordinate_list); *coordinate_list = NULL; } return 0; } else { return 1; } return PyErr_Occurred() ? 0 : 1; } HTH -- Luis Pedro Coelho | Institute for Molecular Medicine | http://luispedro.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From robert.kern at gmail.com Sat Mar 3 11:10:12 2012 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Mar 2012 16:10:12 +0000 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 16:06, Olivier Delalleau wrote: > Le 3 mars 2012 11:03, Robert Kern a ?crit : >> >> On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: >> > Le 3 mars 2012 10:27, Robert Kern a ?crit : >> >> >> >> On Sat, Mar 3, 2012 at 14:34, Robert Kern >> >> wrote: >> >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> >> > >> >> > wrote: >> >> >> >> >> Because this is also bad: >> >> >>>>> np. >> >> >> Display all 561 possibilities? (y or n) >> >> > >> >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or >> >> > deprecating np.allclose() in favor of np.close().all(). >> >> >> >> I screwed up this paragraph. I meant that as "Another alternative >> >> would be to deprecate ...". >> > >> > >> > np.close().all() would probably be a lot less efficient in terms of CPU >> > / >> > memory though, wouldn't it? >> >> No. np.allclose() is essentially doing exactly this already. > > > Ok. What about then, np.allclose() could theoretically be a lot more > efficient in terms of CPU / memory? ;) True, but so could a hypothetical np.allequal(), np.allless_equal(), etc. -- Robert Kern From jkington at wisc.edu Sat Mar 3 13:06:54 2012 From: jkington at wisc.edu (Joe Kington) Date: Sat, 03 Mar 2012 12:06:54 -0600 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 10:06 AM, Olivier Delalleau wrote: > Le 3 mars 2012 11:03, Robert Kern a ?crit : > >> On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: >> > Le 3 mars 2012 10:27, Robert Kern a ?crit : >> >> >> >> On Sat, Mar 3, 2012 at 14:34, Robert Kern >> wrote: >> >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers < >> ralf.gommers at googlemail.com> >> >> > wrote: >> >> >> >> >> Because this is also bad: >> >> >>>>> np. >> >> >> Display all 561 possibilities? (y or n) >> >> > >> >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or >> >> > deprecating np.allclose() in favor of np.close().all(). >> >> >> >> I screwed up this paragraph. I meant that as "Another alternative >> >> would be to deprecate ...". >> > >> > >> > np.close().all() would probably be a lot less efficient in terms of CPU >> / >> > memory though, wouldn't it? >> >> No. np.allclose() is essentially doing exactly this already. >> > > Ok. What about then, np.allclose() could theoretically be a lot more > efficient in terms of CPU / memory? ;) > allclose() does short-circuit in a few cases where the pattern of Inf's doesn't match. E.g. if not all(xinf == isinf(y)): return False if not all(x[xinf] == y[xinf]): return False At least for the function I wrote, allclose() would be a bit faster than isclose().all() in those specific cases. It's not likely to be terribly significant, though. -Joe > > -=- Olivier > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkington at wisc.edu Sat Mar 3 13:07:29 2012 From: jkington at wisc.edu (Joe Kington) Date: Sat, 03 Mar 2012 12:07:29 -0600 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 9:26 AM, Robert Kern wrote: > On Sat, Mar 3, 2012 at 15:22, Benjamin Root wrote: > > > > > > On Saturday, March 3, 2012, Robert Kern wrote: > >> On Sat, Mar 3, 2012 at 14:31, Ralf Gommers > > >> wrote: > >>> > >>> > >>> On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern > >>> wrote: > >>>> > >>>> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers < > ralf.gommers at googlemail.com> > >>>> wrote: > >>>> > > >>>> > > >>>> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington > >>>> > wrote: > >>>> >> > >>>> >> Is there a numpy function for testing floating point equality that > >>>> >> returns > >>>> >> a boolean array? > >>>> >> > >>>> >> I'm aware of np.allclose, but I need a boolean array. Properly > >>>> >> handling > >>>> >> NaN's and Inf's (as allclose does) would be a nice bonus. > >>>> >> > >>>> >> I wrote the function below to do this, but I suspect there's a > method > >>>> >> in > >>>> >> numpy that I missed. > >>>> > > >>>> > > >>>> > I don't think such a function exists, would be nice to have. How > about > >>>> > just > >>>> > adding a keyword "return_array" to allclose to do so? > >>>> > >>>> As a general design principle, adding a boolean flag that changes the > >>>> return type is worse than making a new function. > >>> > >>> > >>> That's certainly true as a general principle. Do you have a concrete > >>> suggestion in this case though? > >> > >> np.close() > >> > > > > When I read that, I mentally think of "close" as in closing a file. I > think > > we need a synonym. > > np.isclose() > Would it be helpful if I went ahead and submitted a pull request with the function in my original question called "isclose" (along with a complete docstring and a few tests)? One note: At the moment, it deliberately compares NaN's as equal. E.g. isclose([np.nan, np.nan], [np.nan, np.nan]) will return: [True, True] This obviously runs counter to the standard way NaN's are handled (and indeed the definition of NaN). However, in the context of a floating point "close to" function, I think it makes the most sense. I've had this sitting around in a small project for awhile now, and it's been more useful to have it compare NaN's as "approximately equal" than not for my purposes at least. Nonetheless, it's something that needs additional consideration. Thanks, -Joe > > -- > Robert Kern > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Sat Mar 3 13:50:50 2012 From: shish at keba.be (Olivier Delalleau) Date: Sat, 3 Mar 2012 13:50:50 -0500 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: Le 3 mars 2012 13:07, Joe Kington a ?crit : > > > On Sat, Mar 3, 2012 at 9:26 AM, Robert Kern wrote: > >> On Sat, Mar 3, 2012 at 15:22, Benjamin Root wrote: >> > >> > >> > On Saturday, March 3, 2012, Robert Kern wrote: >> >> On Sat, Mar 3, 2012 at 14:31, Ralf Gommers < >> ralf.gommers at googlemail.com> >> >> wrote: >> >>> >> >>> >> >>> On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern >> >>> wrote: >> >>>> >> >>>> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers < >> ralf.gommers at googlemail.com> >> >>>> wrote: >> >>>> > >> >>>> > >> >>>> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington >> >>>> > wrote: >> >>>> >> >> >>>> >> Is there a numpy function for testing floating point equality that >> >>>> >> returns >> >>>> >> a boolean array? >> >>>> >> >> >>>> >> I'm aware of np.allclose, but I need a boolean array. Properly >> >>>> >> handling >> >>>> >> NaN's and Inf's (as allclose does) would be a nice bonus. >> >>>> >> >> >>>> >> I wrote the function below to do this, but I suspect there's a >> method >> >>>> >> in >> >>>> >> numpy that I missed. >> >>>> > >> >>>> > >> >>>> > I don't think such a function exists, would be nice to have. How >> about >> >>>> > just >> >>>> > adding a keyword "return_array" to allclose to do so? >> >>>> >> >>>> As a general design principle, adding a boolean flag that changes the >> >>>> return type is worse than making a new function. >> >>> >> >>> >> >>> That's certainly true as a general principle. Do you have a concrete >> >>> suggestion in this case though? >> >> >> >> np.close() >> >> >> > >> > When I read that, I mentally think of "close" as in closing a file. I >> think >> > we need a synonym. >> >> np.isclose() >> > > Would it be helpful if I went ahead and submitted a pull request with the > function in my original question called "isclose" (along with a complete > docstring and a few tests)? > > One note: > At the moment, it deliberately compares NaN's as equal. E.g. > > isclose([np.nan, np.nan], [np.nan, np.nan]) > > will return: > > [True, True] > > This obviously runs counter to the standard way NaN's are handled (and > indeed the definition of NaN). > > However, in the context of a floating point "close to" function, I think > it makes the most sense. > > I've had this sitting around in a small project for awhile now, and it's > been more useful to have it compare NaN's as "approximately equal" than not > for my purposes at least. > > Nonetheless, it's something that needs additional consideration. > > Thanks, > -Joe > It would be confusing if numpy.isclose().all() was different from numpy.allclose(). That being said, I agree it's useful to have NaNs compare equal in some cases, maybe it could be a new argument to the function? -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sat Mar 3 15:30:47 2012 From: travis at continuum.io (Travis Oliphant) Date: Sat, 3 Mar 2012 14:30:47 -0600 Subject: [Numpy-discussion] Missing data again Message-ID: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Hi all, I've been thinking a lot about the masked array implementation lately. I finally had the time to look hard at what has been done and now am of the opinion that I do not think that 1.7 can be released with the current state of the masked array implementation *unless* it is clearly marked as experimental and may be changed in 1.8 I wish I had been able to be a bigger part of this conversation last year. But, that is why I took the steps I took to try and figure out another way to feed my family *and* stay involved in the NumPy community. I would love to stay involved in what is happening in the SciPy community, but I am more satisfied with what Ralf, Warren, Robert, Pauli, Josef, Charles, Stefan, and others are doing there right now, and don't have time to keep up with everything. Even though SciPy was the heart and soul of why I even got involved with Python for open source in the first place and took many years of my volunteer labor, I won't be able to spend significant time on SciPy code over the coming months. At some point, I really hope to be able to make contributions again to that code-base. Time will tell whether or not my aspirations will be realized. It depends quite a bit on whether or not my kids have what they need from me (which right now is money and time). NumPy, on the other hand, is not in a position where I can feel comfortable leaving my "baby" to others. I recognize and value the contributions from many people to make NumPy what it is today (e.g. code contributions, code rearrangement and standardization, build and install improvement, and most recently some architectural changes). But, I feel a personal responsibility for the code base as I spent a great many months writing NumPy in the first place, and I've spent a great deal of time interacting with NumPy users and feel like I have at least some sense of their stories. Of course, I built on the shoulders of giants, and much of what is there is *because of* where the code was adapted from (it was not created de-novo). Currently, there remains much that needs to be communicated, improved, and worked on, and I have specific opinions about what some changes and improvements should be, how they should be written, and how the resulting users need to be benefited. It will take time to discuss all of this, and that's where I will spend my open-source time in the coming months. In that vein: Because it is slated to go into release 1.7, we need to re-visit the masked array discussion again. The NEP process is the appropriate one and I'm glad we are taking that route for these discussions. My goal is to get consensus in order for code to get into NumPy (regardless of who writes the code). It may be that we don't come to a consensus (reasonable and intelligent people can disagree on things --- look at the coming election...). We can represent different parts of what is fortunately a very large user-base of NumPy users. First of all, I want to be clear that I think there is much great work that has been done in the current missing data code. There are some nice features in the where clause of the ufunc and the machinery for the iterator that allows re-using ufunc loops that are not re-written to check for missing data. I'm sure there are other things as well that I'm not quite aware of yet. However, I don't think the API presented to the numpy user presently is the correct one for NumPy 1.X. A few particulars: * the reduction operations need to default to "skipna" --- this is the most common use case which has been re-inforced again to me today by a new user to Python who is using masked arrays presently * the mask needs to be visible to the user if they use that approach to missing data (people should be able to get a hold of the mask and work with it in Python) * bit-pattern approaches to missing data (at least for float64 and int32) need to be implemented. * there should be some way when using "masks" (even if it's hidden from most users) for missing data to separate the low-level ufunc operation from the operation on the masks... I have heard from several users that they will *not use the missing data* in NumPy as currently implemented, and I can now see why. For better or for worse, my approach to software is generally very user-driven and very pragmatic. On the other hand, I'm also a mathematician and appreciate the cognitive compression that can come out of well-formed structure. None-the-less, I'm an *applied* mathematician and am ultimately motivated by applications. I will get a hold of the NEP and spend some time with it to discuss some of this in that document. This will take several weeks (as PyCon is next week and I have a tutorial I'm giving there). For now, I do not think 1.7 can be released unless the masked array is labeled *experimental*. Thanks, -Travis From mwwiebe at gmail.com Sat Mar 3 16:46:29 2012 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 3 Mar 2012 13:46:29 -0800 Subject: [Numpy-discussion] Missing data again In-Reply-To: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> References: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Message-ID: On Sat, Mar 3, 2012 at 12:30 PM, Travis Oliphant wrote: > > > First of all, I want to be clear that I think there is much great work > that has been done in the current missing data code. There are some nice > features in the where clause of the ufunc and the machinery for the > iterator that allows re-using ufunc loops that are not re-written to check > for missing data. I'm sure there are other things as well that I'm not > quite aware of yet. However, I don't think the API presented to the > numpy user presently is the correct one for NumPy 1.X. > I thought I might chime in with some implementation-detail notes, as while Travis has dug into the code, I'm still the person who knows it best. A few particulars: > > * the reduction operations need to default to "skipna" --- this is > the most common use case which has been re-inforced again to me today by a > new user to Python who is using masked arrays presently > This is a completely trivial change. I went with the default as I did because it's what R, the primary inspiration for the NA design, does. We'll have to be sure this is well-marked in the documentation about "NumPy NA for R users". > * the mask needs to be visible to the user if they use that > approach to missing data (people should be able to get a hold of the mask > and work with it in Python) > This is relatively easy. Probably the way to do it is with an ndarray.maskna property. It could be in 1.7 if we really push. For the multi-NA future, I think the NPY_MASK dtype, currently an alias for NPY_UBYTE, would need to become its own dtype with separate .exposed and .payload attributes. > * bit-pattern approaches to missing data (at least for float64 and > int32) need to be implemented. > I strongly wanted to do masks first, because of the greater generality and because the bit-patterns would best be implemented sharing mask implementation details. I still believe this was the correct choice, and it set the stage for bit-patterns. It will be possible to make inner loops that specialize for the default hard-coded bit-pattern dtypes. I paid very careful attention in the design making sure high performance is possible without significant rework. The immense scale of the required code changes meant I couldn't actually implement high performance in the time frame. The place I think this affects 1.7 the most is in the default choice for what np.array([1.0, np.NA, 3.0]) and np.array([1, np.NA, 3]) mean. In 1.7, both mean an NA-masked array. In 1.8, I can see a strong case that the first should mean an NA-dtype, and the second an NA-masked array. Also, here's a thought for the usability of NA-float64. As much as global state is a bad idea, something which determines whether implicit float dtypes are NA-float64 or float64 could help. In IPython, "pylab" mode would default to float64, and "statlab" or "pystat" would default to NA-float64. One way to write this might be: >>> np.set_default_float(np.nafloat64) >>> np.array([1.0, 2.0, 3.0]) array([ 1., 2., 3.], dtype=nafloat64) >>> np.set_default_float(np.float64) >>> np.array([1.0, 2.0, 3.0]) array([ 1., 2., 3.], dtype=float64) > * there should be some way when using "masks" (even if it's hidden > from most users) for missing data to separate the low-level ufunc operation > from the operation > on the masks... > This is completely trivial to implement. Maybe ndarray.view(maskna='ignore') is a reasonable way to spell direct access without a mask. Cheers, Mark > I have heard from several users that they will *not use the missing data* > in NumPy as currently implemented, and I can now see why. For better or > for worse, my approach to software is generally very user-driven and very > pragmatic. On the other hand, I'm also a mathematician and appreciate the > cognitive compression that can come out of well-formed structure. > None-the-less, I'm an *applied* mathematician and am ultimately motivated > by applications. > > I will get a hold of the NEP and spend some time with it to discuss some > of this in that document. This will take several weeks (as PyCon is next > week and I have a tutorial I'm giving there). For now, I do not think > 1.7 can be released unless the masked array is labeled *experimental*. > > Thanks, > > -Travis > > > > > > > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Mar 3 16:53:09 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 3 Mar 2012 22:53:09 +0100 Subject: [Numpy-discussion] Missing data again In-Reply-To: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> References: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Message-ID: On Sat, Mar 3, 2012 at 9:30 PM, Travis Oliphant wrote: > Hi all, > > I've been thinking a lot about the masked array implementation lately. > I finally had the time to look hard at what has been done and now am of the > opinion that I do not think that 1.7 can be released with the current state > of the masked array implementation *unless* it is clearly marked as > experimental and may be changed in 1.8 > We had already decided to put an "experimental" label on the implementation. Also on datetime. I will open a ticket for this now to make sure we won't forget. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 3 16:55:04 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 3 Mar 2012 14:55:04 -0700 Subject: [Numpy-discussion] Missing data again In-Reply-To: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> References: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Message-ID: On Sat, Mar 3, 2012 at 1:30 PM, Travis Oliphant wrote: > Hi all, > > I've been thinking a lot about the masked array implementation lately. > I finally had the time to look hard at what has been done and now am of the > opinion that I do not think that 1.7 can be released with the current state > of the masked array implementation *unless* it is clearly marked as > experimental and may be changed in 1.8 > > That was the intention. > I wish I had been able to be a bigger part of this conversation last year. > But, that is why I took the steps I took to try and figure out another > way to feed my family *and* stay involved in the NumPy community. I would > love to stay involved in what is happening in the SciPy community, but I am > more satisfied with what Ralf, Warren, Robert, Pauli, Josef, Charles, > Stefan, and others are doing there right now, and don't have time to keep > up with everything. Even though SciPy was the heart and soul of why I > even got involved with Python for open source in the first place and took > many years of my volunteer labor, I won't be able to spend significant time > on SciPy code over the coming months. At some point, I really hope to be > able to make contributions again to that code-base. Time will tell > whether or not my aspirations will be realized. It depends quite a bit on > whether or not my kids have what they need from me (which right now is > money and time). > > NumPy, on the other hand, is not in a position where I can feel > comfortable leaving my "baby" to others. I recognize and value the > contributions from many people to make NumPy what it is today (e.g. code > contributions, code rearrangement and standardization, build and install > improvement, and most recently some architectural changes). But, I feel > a personal responsibility for the code base as I spent a great many months > writing NumPy in the first place, and I've spent a great deal of time > interacting with NumPy users and feel like I have at least some sense of > their stories. Of course, I built on the shoulders of giants, and much > of what is there is *because of* where the code was adapted from (it was > not created de-novo). Currently, there remains much that needs to be > communicated, improved, and worked on, and I have specific opinions about > what some changes and improvements should be, how they should be written, > and how the resulting users need to be benefited. > It will take time to discuss all of this, and that's where I will spend > my open-source time in the coming months. > > In that vein: > > Because it is slated to go into release 1.7, we need to re-visit the > masked array discussion again. The NEP process is the appropriate one > and I'm glad we are taking that route for these discussions. My goal is > to get consensus in order for code to get into NumPy (regardless of who > writes the code). It may be that we don't come to a consensus > (reasonable and intelligent people can disagree on things --- look at the > coming election...). We can represent different parts of what is > fortunately a very large user-base of NumPy users. > > First of all, I want to be clear that I think there is much great work > that has been done in the current missing data code. There are some nice > features in the where clause of the ufunc and the machinery for the > iterator that allows re-using ufunc loops that are not re-written to check > for missing data. I'm sure there are other things as well that I'm not > quite aware of yet. However, I don't think the API presented to the > numpy user presently is the correct one for NumPy 1.X. > A few particulars: > > * the reduction operations need to default to "skipna" --- this is > the most common use case which has been re-inforced again to me today by a > new user to Python who is using masked arrays presently > > * the mask needs to be visible to the user if they use that > approach to missing data (people should be able to get a hold of the mask > and work with it in Python) > > * bit-pattern approaches to missing data (at least for float64 and > int32) need to be implemented. > > * there should be some way when using "masks" (even if it's hidden > from most users) for missing data to separate the low-level ufunc operation > from the operation > on the masks... > > Mind, Mark only had a few weeks to write code. I think the unfinished state is a direct function of that. > I have heard from several users that they will *not use the missing data* > in NumPy as currently implemented, and I can now see why. For better or > for worse, my approach to software is generally very user-driven and very > pragmatic. On the other hand, I'm also a mathematician and appreciate the > cognitive compression that can come out of well-formed structure. > None-the-less, I'm an *applied* mathematician and am ultimately motivated > by applications. > > I think that would be Wes. I thought the current state wasn't that far away from what he wanted in the only post where he was somewhat explicit. I think it would be useful for him to sit down with Mark at some time and thrash things out since I think there is some misunderstanding involved. > I will get a hold of the NEP and spend some time with it to discuss some > of this in that document. This will take several weeks (as PyCon is next > week and I have a tutorial I'm giving there). For now, I do not think > 1.7 can be released unless the masked array is labeled *experimental*. > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sat Mar 3 17:01:40 2012 From: travis at continuum.io (Travis Oliphant) Date: Sat, 3 Mar 2012 16:01:40 -0600 Subject: [Numpy-discussion] Missing data again In-Reply-To: References: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Message-ID: <36EC0418-E7D2-4C64-A30E-B6FBCDEE3A80@continuum.io> > > Mind, Mark only had a few weeks to write code. I think the unfinished state is a direct function of that. > > I have heard from several users that they will *not use the missing data* in NumPy as currently implemented, and I can now see why. For better or for worse, my approach to software is generally very user-driven and very pragmatic. On the other hand, I'm also a mathematician and appreciate the cognitive compression that can come out of well-formed structure. None-the-less, I'm an *applied* mathematician and am ultimately motivated by applications. > > > I think that would be Wes. I thought the current state wasn't that far away from what he wanted in the only post where he was somewhat explicit. I think it would be useful for him to sit down with Mark at some time and thrash things out since I think there is some misunderstanding involved. > Actually it wasn't Wes. It was 3 other people. I'm already well aware of Wes's perspective and actually think his concerns have been handled already. Also, the person who showed me their use-case was a new user. But, your point about getting people together is well-taken. I also recognize the fact that there have been (and likely continue to be) misunderstandings on multiple fronts. Fortunately, many of us will be at PyCon later this week. We tried really hard to get Mark Wiebe here this weekend as well --- but he could only sacrifice a week away from his degree work to join us for PyCon. It would be great if you could come to PyCon as well. Perhaps we can apply to NumFOCUS for a travel grant to bring NumPy developers together with other interested people to finish the masked array design and implementation. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Sat Mar 3 17:10:44 2012 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 3 Mar 2012 17:10:44 -0500 Subject: [Numpy-discussion] Missing data again In-Reply-To: References: <291B3A7C-A5CB-4559-A143-F4B4FB002B71@continuum.io> Message-ID: On Sat, Mar 3, 2012 at 4:46 PM, Mark Wiebe wrote: > On Sat, Mar 3, 2012 at 12:30 PM, Travis Oliphant >> >> ? ? ? ?* the reduction operations need to default to "skipna" --- this is >> the most common use case which has been re-inforced again to me today by a >> new user to Python who is using masked arrays presently > > > This is a completely trivial change. I went with the default as I did > because it's what R, the primary inspiration for the NA design, does. We'll > have to be sure this is well-marked in the documentation about "NumPy NA for > R users". > It may be trivial to change the code, but this isn't a trivial change. "Most common use case" is hard for me to swallow, since there are so many. Of the different statistical softwares I've used, none that I recall ignores missing data (silently) by default. This sounds dangerous to me. It's one thing to be convenient to work with missing data, but it's another to try to sweep the problem under the rug. I imagine the choice of the R developers was a thoughtful one. Perhaps something like np.seterr should also be implemented for missing data since there's probably no resolution to what's most sensible here. Skipper From jkington at wisc.edu Sat Mar 3 18:33:49 2012 From: jkington at wisc.edu (Joe Kington) Date: Sat, 03 Mar 2012 17:33:49 -0600 Subject: [Numpy-discussion] Floating point "close" function? In-Reply-To: References: Message-ID: On Sat, Mar 3, 2012 at 12:50 PM, Olivier Delalleau wrote: > >> Would it be helpful if I went ahead and submitted a pull request with the >> function in my original question called "isclose" (along with a complete >> docstring and a few tests)? >> >> One note: >> At the moment, it deliberately compares NaN's as equal. E.g. >> >> isclose([np.nan, np.nan], [np.nan, np.nan]) >> >> will return: >> >> [True, True] >> >> This obviously runs counter to the standard way NaN's are handled (and >> indeed the definition of NaN). >> >> However, in the context of a floating point "close to" function, I think >> it makes the most sense. >> >> I've had this sitting around in a small project for awhile now, and it's >> been more useful to have it compare NaN's as "approximately equal" than not >> for my purposes at least. >> >> Nonetheless, it's something that needs additional consideration. >> >> Thanks, >> -Joe >> > > It would be confusing if numpy.isclose().all() was different from > numpy.allclose(). That being said, I agree it's useful to have NaNs compare > equal in some cases, maybe it could be a new argument to the function? > Good point. I went ahead and added an "equal_nan" kwarg and removed the "check_invalid" kwarg I had in before. I also made it mimic what np.equal() does in the case of two scalars (return a scalar instead of an array). I went ahead an make a pull request: https://github.com/numpy/numpy/pull/224 Hope that's alright. Cheers, -Joe > > -=- Olivier > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Mar 3 19:38:53 2012 From: cournape at gmail.com (David Cournapeau) Date: Sat, 3 Mar 2012 16:38:53 -0800 Subject: [Numpy-discussion] C++ Example In-Reply-To: <30965644.bxiDOaWOx7@rabbit> References: <30965644.bxiDOaWOx7@rabbit> Message-ID: On Sat, Mar 3, 2012 at 8:07 AM, Luis Pedro Coelho wrote: > Hi, > > I sort of missed the big C++ discussion, but I'd like to give some examples of > how writing code can become much simpler if you are based on C++. This is from > my mahotas package, which has a thin C++ wrapper around numpy's C API > > https://github.com/luispedro/mahotas/blob/master/mahotas/_morph.cpp > > and it implements multi-type greyscale erosion. > > > // numpy::aligned_array wraps PyArrayObject* > template > void erode(numpy::aligned_array res, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?numpy::aligned_array array, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?numpy::aligned_array Bc) { > > > ? ? ? ?// Release the GIL using RAII > ? ?gil_release nogil; > ? ?const int N = res.size(); > ? ?typename numpy::aligned_array::iterator iter = array.begin(); > ? ?// this is adapted from scipy.ndimage. > ? ?// it implements the convolution-like filtering. > ? ?filter_iterator filter(res.raw_array(), > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Bc.raw_array(), > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EXTEND_NEAREST, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?is_bool(T())); > ? ?const int N2 = filter.size(); > ? ?T* rpos = res.data(); > > ? ?for (int i = 0; i != N; ++i, ++rpos, filter.iterate_both(iter)) { > ? ? ? ?T value = std::numeric_limits::max(); > ? ? ? ?for (int j = 0; j != N2; ++j) { > ? ? ? ? ? ?T arr_val = T(); > ? ? ? ? ? ?filter.retrieve(iter, j, arr_val); > ? ? ? ? ? ?value = std::min(value, erode_sub(arr_val, filter[j])); > ? ? ? ?} > ? ? ? ?*rpos = value; > ? ?} > } > > If you compare this with the equivalent scipy.ndimage function, which is very > good C code (but mostly write-only?in fact, ndimage has not been maintainable > because it is so hard [at least for me, I've tried]): The fact that this is good C is matter of opinon :) I don't think the code is comparable either - some of the stuff done in the C code is done in the C++ code your are calling. The C code could be significantly improved. Even more important here: almost none of this code should be written anymore anyway, C++ or not. This is really the kind of code that should be done in cython, as it is mostly about wrapping C code into the python C API. cheers, David From gael.varoquaux at normalesup.org Sun Mar 4 09:07:53 2012 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 4 Mar 2012 15:07:53 +0100 Subject: [Numpy-discussion] C++ Example In-Reply-To: References: <30965644.bxiDOaWOx7@rabbit> Message-ID: <20120304140753.GC705@phare.normalesup.org> On Sat, Mar 03, 2012 at 04:38:53PM -0800, David Cournapeau wrote: > This is really the kind of code that should be done in cython, as it is > mostly about wrapping C code into the python C API. +1 Gael From chaoyuejoy at gmail.com Sun Mar 4 13:01:55 2012 From: chaoyuejoy at gmail.com (Chao YUE) Date: Sun, 4 Mar 2012 19:01:55 +0100 Subject: [Numpy-discussion] copy mask from existing masked array? Message-ID: Dear all, I have a matrix with dimension of (360,720) but with all global data. I have another land-sea mask matrix with only 2 unique values in it (land=1, sea=-1). So I can easily create transform the second array to a masked array. the problem is, how can I quickly transform the first one to a masked array using the same mask as the land-sea mask array? I hope my question is clear. If not, here is an example: In [93]: a=np.arange(10).reshape(2,5) In [95]: a=np.ma.masked_equal(a,2 In [96]: a=np.ma.masked_equal(a,8) In [97]: a Out[97]: masked_array(data = [[0 1 -- 3 4] [5 6 7 -- 9]], mask = [[False False True False False] [False False False True False]], fill_value = 8) In [100]: b=np.random.normal(0,2,size=(2,5)) I want to convert b to a masked array using exactly the same mask as a. thanks to all, cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Sun Mar 4 13:42:08 2012 From: shish at keba.be (Olivier Delalleau) Date: Sun, 4 Mar 2012 13:42:08 -0500 Subject: [Numpy-discussion] copy mask from existing masked array? In-Reply-To: References: Message-ID: Should work with: b = numpy.ma.masked_array(b, mask=a.mask) -=- Olivier Le 4 mars 2012 13:01, Chao YUE a ?crit : > Dear all, > > I have a matrix with dimension of (360,720) but with all global data. > I have another land-sea mask matrix with only 2 unique values in it > (land=1, sea=-1). > So I can easily create transform the second array to a masked array. > the problem is, how can I quickly transform the first one to a masked > array using the same mask as the land-sea mask array? > > I hope my question is clear. If not, here is an example: > > In [93]: a=np.arange(10).reshape(2,5) > In [95]: a=np.ma.masked_equal(a,2 > In [96]: a=np.ma.masked_equal(a,8) > > In [97]: a > Out[97]: > masked_array(data = > [[0 1 -- 3 4] > [5 6 7 -- 9]], > mask = > [[False False True False False] > [False False False True False]], > fill_value = 8) > > In [100]: b=np.random.normal(0,2,size=(2,5)) > > I want to convert b to a masked array using exactly the same mask as a. > > thanks to all, > cheers, > > Chao > -- > > *********************************************************************************** > Chao YUE > Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) > UMR 1572 CEA-CNRS-UVSQ > Batiment 712 - Pe 119 > 91191 GIF Sur YVETTE Cedex > Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 > > ************************************************************************************ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Mar 4 14:27:47 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 4 Mar 2012 14:27:47 -0500 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: Hi, On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett wrote: > Hi, > > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris > wrote: >> >> >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> Sorry that this report is not complete, I don't have full access to >>> this box but, on a Debian squeeze machine running linux >>> 2.6.32-5-sparc64-smp: >>> >>> nosetests >>> ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values >>> >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error >>> >>> This on current master : 1.7.0.dev-b9872b4 >>> >> >> Hmm, some tests might have been recently enabled. Any chance of doing a >> bisection? Struggling because compilation is very slow and there are lots of untestable commits. df907e6 is the first known bad. Here's the output from a log: * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build mode because of a static function in two separate files (6 months ago) [Mark Wiebe] * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) BUG: nditer: The nditer was reusing the reduce loop inappropriately (#1938) (6 months ago) [Mark Wiebe] * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 months ago) [Mark Wiebe] * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) TST: missingdata: Finish up NA mask tests for np.std and np.var (6 months ago) [Mark Wiebe] * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue (6 months ago) [Mark Wiebe] * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) ENH: missingdata: Implement tests for np.std, add skipna= and keepdims= parameters to more functions (6 months ago) [Mark Wiebe] * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) ENH: missingdata: Implement skipna= support for np.std and np.var (6 months ago) [Mark Wiebe] * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months ago) [Mark Wiebe] * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) ENH: missingdata: Create count_reduce_items function (6 months ago) [Mark Wiebe] * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) ENH: missingdata: Move the Reduce boilerplate into a function PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) ENH: missingdata: Finish count_nonzero as a full-fledged reduction operation (6 months ago) [Mark Wiebe] * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) ENH: missingdata: Towards making count_nonzero a full-featured reduction operation (6 months ago) [Mark Wiebe] * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) ENH: missingdata: Move some of the refactored reduction code into the API (6 months ago) [Mark Wiebe] * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) ENH: missingdata: Change default to create NA-mask when NAs are in lists (6 months ago) [Mark Wiebe] * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) ENH: missingdata: trying some more functions to see how they treat NAs (6 months ago) [Mark Wiebe] * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 months ago) [Mark Wiebe] * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it (6 months ago) [Mark Wiebe] * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA masks (6 months ago) [Mark Wiebe] * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 months ago) [Mark Wiebe] * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to always return a view (6 months ago) [Mark Wiebe] So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad) There seems to be a compilation error for the skipped commits - here's the one I tested, 9194b3a: gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 See you, Matthew From mwwiebe at gmail.com Sun Mar 4 14:41:53 2012 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 4 Mar 2012 11:41:53 -0800 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett wrote: > Hi, > > On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett > wrote: > > Hi, > > > > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett > >> wrote: > >>> > >>> Hi, > >>> > >>> Sorry that this report is not complete, I don't have full access to > >>> this box but, on a Debian squeeze machine running linux > >>> 2.6.32-5-sparc64-smp: > >>> > >>> nosetests > >>> > ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values > >>> > >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error > >>> > >>> This on current master : 1.7.0.dev-b9872b4 > >>> > >> > >> Hmm, some tests might have been recently enabled. Any chance of doing a > >> bisection? > > Struggling because compilation is very slow and there are lots of > untestable commits. df907e6 is the first known bad. Here's the > output from a log: > > * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build > mode because of a static function in two separate files (6 months ago) > [Mark Wiebe] > * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) > BUG: nditer: The nditer was reusing the reduce loop inappropriately > (#1938) (6 months ago) [Mark Wiebe] > * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) > DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] > * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) > BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 > months ago) [Mark Wiebe] > * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) > TST: missingdata: Finish up NA mask tests for np.std and np.var (6 > months ago) [Mark Wiebe] > * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) > BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue > (6 months ago) [Mark Wiebe] > * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) > ENH: missingdata: Implement tests for np.std, add skipna= and > keepdims= parameters to more functions (6 months ago) [Mark Wiebe] > * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) > ENH: missingdata: Implement skipna= support for np.std and np.var (6 > months ago) [Mark Wiebe] > * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) > ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months > ago) [Mark Wiebe] > * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) > ENH: missingdata: Create count_reduce_items function (6 months ago) > [Mark Wiebe] > * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) > ENH: missingdata: Move the Reduce boilerplate into a function > PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] > * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) > ENH: missingdata: Finish count_nonzero as a full-fledged reduction > operation (6 months ago) [Mark Wiebe] > * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) > ENH: missingdata: Towards making count_nonzero a full-featured > reduction operation (6 months ago) [Mark Wiebe] > * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) > ENH: missingdata: Move some of the refactored reduction code into the > API (6 months ago) [Mark Wiebe] > * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) > ENH: missingdata: Change default to create NA-mask when NAs are in > lists (6 months ago) [Mark Wiebe] > * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) > ENH: missingdata: trying some more functions to see how they treat NAs > (6 months ago) [Mark Wiebe] > * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) > ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 > months ago) [Mark Wiebe] > * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) > BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it > (6 months ago) [Mark Wiebe] > * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) > BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA > masks (6 months ago) [Mark Wiebe] > * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) > ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 > months ago) [Mark Wiebe] > * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) > ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to > always return a view (6 months ago) [Mark Wiebe] > > So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad) > > There seems to be a compilation error for the skipped commits - here's > the one I tested, 9194b3a: > If you enable separate compilation by setting an environment variable, these commits should build as well. $ export ENABLE_SEPARATE_COMPILATION=1 Cheers, Mark > gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv > -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include > -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include > -I/usr/include/python2.6 > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c > numpy/core/src/multiarray/multiarraymodule_onefile.c -o > > build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" > failed with exit status 1 > > See you, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 4 14:50:33 2012 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 4 Mar 2012 12:50:33 -0700 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: On Sun, Mar 4, 2012 at 12:27 PM, Matthew Brett wrote: > Hi, > > On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett > wrote: > > Hi, > > > > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris > > wrote: > >> > >> > >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett > >> wrote: > >>> > >>> Hi, > >>> > >>> Sorry that this report is not complete, I don't have full access to > >>> this box but, on a Debian squeeze machine running linux > >>> 2.6.32-5-sparc64-smp: > >>> > >>> nosetests > >>> > ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values > >>> > >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error > >>> > >>> This on current master : 1.7.0.dev-b9872b4 > >>> > >> > >> Hmm, some tests might have been recently enabled. Any chance of doing a > >> bisection? > > Struggling because compilation is very slow and there are lots of > untestable commits. df907e6 is the first known bad. Here's the > output from a log: > The effort is much appreciated. At least we are down to a 3 day period. > > * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build > mode because of a static function in two separate files (6 months ago) > [Mark Wiebe] > * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) > BUG: nditer: The nditer was reusing the reduce loop inappropriately > (#1938) (6 months ago) [Mark Wiebe] > * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) > DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] > * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) > BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 > months ago) [Mark Wiebe] > * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) > TST: missingdata: Finish up NA mask tests for np.std and np.var (6 > months ago) [Mark Wiebe] > * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) > BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue > (6 months ago) [Mark Wiebe] > * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) > ENH: missingdata: Implement tests for np.std, add skipna= and > keepdims= parameters to more functions (6 months ago) [Mark Wiebe] > * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) > ENH: missingdata: Implement skipna= support for np.std and np.var (6 > months ago) [Mark Wiebe] > * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) > ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months > ago) [Mark Wiebe] > * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) > ENH: missingdata: Create count_reduce_items function (6 months ago) > [Mark Wiebe] > * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) > ENH: missingdata: Move the Reduce boilerplate into a function > PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] > * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) > ENH: missingdata: Finish count_nonzero as a full-fledged reduction > operation (6 months ago) [Mark Wiebe] > * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) > ENH: missingdata: Towards making count_nonzero a full-featured > reduction operation (6 months ago) [Mark Wiebe] > * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) > ENH: missingdata: Move some of the refactored reduction code into the > API (6 months ago) [Mark Wiebe] > * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) > ENH: missingdata: Change default to create NA-mask when NAs are in > lists (6 months ago) [Mark Wiebe] > * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) > ENH: missingdata: trying some more functions to see how they treat NAs > (6 months ago) [Mark Wiebe] > * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) > ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 > months ago) [Mark Wiebe] > * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) > BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it > (6 months ago) [Mark Wiebe] > * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) > BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA > masks (6 months ago) [Mark Wiebe] > * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) > ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 > months ago) [Mark Wiebe] > * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) > ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to > always return a view (6 months ago) [Mark Wiebe] > > So - the problem arises somewhere between 99a21ef (good) and df907e6 (bad) > > There seems to be a compilation error for the skipped commits - here's > the one I tested, 9194b3a: > > gcc: numpy/core/src/multiarray/multiarraymodule_onefile.c > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv > -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include > -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include > -I/usr/include/python2.6 > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c > numpy/core/src/multiarray/multiarraymodule_onefile.c -o > > build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" > failed with exit status 1 > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Mar 4 15:56:03 2012 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 4 Mar 2012 21:56:03 +0100 Subject: [Numpy-discussion] test errors on deprecation/runtime warnings In-Reply-To: References: Message-ID: On Sat, Feb 18, 2012 at 10:13 AM, Ralf Gommers wrote: > > > 2012/2/17 St?fan van der Walt > >> Hi Ralf >> >> On Thu, Feb 16, 2012 at 11:05 AM, Ralf Gommers >> wrote: >> > Last week we merged https://github.com/numpy/numpy/pull/201, which >> causes >> > DeprecationWarning's and RuntimeWarning's to be converted to errors if >> they >> > occur when running the test suite. >> >> It looks like this change affects other packages, too, > > > It does, which is why I wanted to bring it up here. > > >> which may legitimately raise RuntimeWarnings while running their test >> suites >> (unless I read the patch wrong). Would it be an option to rather add >> a flag (False by default) to enable this behaviour, and enable it >> inside of numpy.test() ? >> > > Well, the idea is that this behavior is the correct one for all packages. > It calls attention to those RuntimeWarnings, which may only occur on > certain platforms. If they're legitimate, you silence them in the test > suite of that package. If not, you fix them. Would you agree with that? Or > would you prefer to just ignore DeprecationWarnings and/or RuntimeWarnings > in skimage for example? > > Note that the changed behavior would only be visible for people running > numpy master. > This behavior has been made configurable, and I added instructions in HOWTO_RELEASE to turn it off in maintenance branches, in https://github.com/rgommers/numpy/compare/pull-219-warnings. There is some discussion at https://github.com/numpy/numpy/pull/219. The intention is to merge this soon, so now is the time to comment. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lpc at cmu.edu Sun Mar 4 17:18:40 2012 From: lpc at cmu.edu (Luis Pedro Coelho) Date: Sun, 04 Mar 2012 22:18:40 +0000 Subject: [Numpy-discussion] C++ Example In-Reply-To: References: <30965644.bxiDOaWOx7@rabbit> Message-ID: <1375080.6SoljH8rGC@rabbit> On Saturday, March 03, 2012 04:38:53 PM David Cournapeau wrote: > I don't think the code is comparable either - some of the stuff done > in the C code is done in the C++ code your are calling. The C code > could be significantly improved. Actually, that's not 100% accurate. The C code calls the same functions. Most of the extra cruft is that it needs to do all of this error checking and type- dispatch, while in C++ you can have RAII and templates. > Even more important here: almost none > of this code should be written anymore anyway, C++ or not. This is > really the kind of code that should be done in cython, as it is mostly > about wrapping C code into the python C API. At least last time I read up on it, cython was not able to do multi-type code, i.e., have code that works on arrays of multiple types. Does it support it now? Best, -- Luis Pedro Coelho | University of Lisbon | http://luispedro.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From jswhit at fastmail.fm Sun Mar 4 20:57:24 2012 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Sun, 04 Mar 2012 18:57:24 -0700 Subject: [Numpy-discussion] C++ Example In-Reply-To: <1375080.6SoljH8rGC@rabbit> References: <30965644.bxiDOaWOx7@rabbit> <1375080.6SoljH8rGC@rabbit> Message-ID: <4F541D84.1050706@fastmail.fm> On 3/4/12 3:18 PM, Luis Pedro Coelho wrote: > On Saturday, March 03, 2012 04:38:53 PM David Cournapeau wrote: >> I don't think the code is comparable either - some of the stuff done >> in the C code is done in the C++ code your are calling. The C code >> could be significantly improved. > Actually, that's not 100% accurate. The C code calls the same functions. Most > of the extra cruft is that it needs to do all of this error checking and type- > dispatch, while in C++ you can have RAII and templates. > >> Even more important here: almost none >> of this code should be written anymore anyway, C++ or not. This is >> really the kind of code that should be done in cython, as it is mostly >> about wrapping C code into the python C API. > At least last time I read up on it, cython was not able to do multi-type code, > i.e., have code that works on arrays of multiple types. Does it support it > now? > > Best, Coming soon in version 0.16: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/fusedtypes.html -Jeff From matthew.brett at gmail.com Sun Mar 4 23:08:51 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 4 Mar 2012 20:08:51 -0800 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: Hi, On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe wrote: > On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett >> wrote: >> > Hi, >> > >> > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris >> > wrote: >> >> >> >> >> >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett >> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> Sorry that this report is not complete, I don't have full access to >> >>> this box but, on a Debian squeeze machine running linux >> >>> 2.6.32-5-sparc64-smp: >> >>> >> >>> nosetests >> >>> >> >>> ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values >> >>> >> >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error >> >>> >> >>> This on current master : 1.7.0.dev-b9872b4 >> >>> >> >> >> >> Hmm, some tests might have been recently enabled. Any chance of doing a >> >> bisection? >> >> Struggling because compilation is very slow and there are lots of >> untestable commits. ?df907e6 is the first known bad. ?Here's the >> output from a log: >> >> * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build >> mode because of a static function in two separate files (6 months ago) >> [Mark Wiebe] >> * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) >> BUG: nditer: The nditer was reusing the reduce loop inappropriately >> (#1938) (6 months ago) [Mark Wiebe] >> * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) >> DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] >> * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) >> BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 >> months ago) [Mark Wiebe] >> * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) >> TST: missingdata: Finish up NA mask tests for np.std and np.var (6 >> months ago) [Mark Wiebe] >> * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) >> BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue >> (6 months ago) [Mark Wiebe] >> * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) >> ENH: missingdata: Implement tests for np.std, add skipna= and >> keepdims= parameters to more functions (6 months ago) [Mark Wiebe] >> * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) >> ENH: missingdata: Implement skipna= support for np.std and np.var (6 >> months ago) [Mark Wiebe] >> * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) >> ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months >> ago) [Mark Wiebe] >> * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) >> ENH: missingdata: Create count_reduce_items function (6 months ago) >> [Mark Wiebe] >> * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) >> ENH: missingdata: Move the Reduce boilerplate into a function >> PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] >> * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) >> ENH: missingdata: Finish count_nonzero as a full-fledged reduction >> operation (6 months ago) [Mark Wiebe] >> * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) >> ENH: missingdata: Towards making count_nonzero a full-featured >> reduction operation (6 months ago) [Mark Wiebe] >> * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) >> ENH: missingdata: Move some of the refactored reduction code into the >> API (6 months ago) [Mark Wiebe] >> * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) >> ENH: missingdata: Change default to create NA-mask when NAs are in >> lists (6 months ago) [Mark Wiebe] >> * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) >> ENH: missingdata: trying some more functions to see how they treat NAs >> (6 months ago) [Mark Wiebe] >> * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) >> ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 >> months ago) [Mark Wiebe] >> * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) >> BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it >> (6 months ago) [Mark Wiebe] >> * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) >> BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA >> masks (6 months ago) [Mark Wiebe] >> * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) >> ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 >> months ago) [Mark Wiebe] >> * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) >> ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to >> always return a view (6 months ago) [Mark Wiebe] >> >> So - the problem arises somewhere between 99a21ef (good) and ?df907e6 >> (bad) >> >> There seems to be a compilation error for the skipped commits - here's >> the one I tested, 9194b3a: > > > If you enable separate compilation by setting an environment variable, these > commits should build as well. > > $ export?ENABLE_SEPARATE_COMPILATION=1 > I might be doing something wrong but: In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here In file included from numpy/core/src/multiarray/scalartypes.c.src:25, from numpy/core/src/multiarray/multiarraymodule_onefile.c:10: numpy/core/src/multiarray/_datetime.h:9: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:13: numpy/core/src/multiarray/datetime.c:34: warning: function declaration isn't a prototype In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:41: numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of 'intp_abs' numpy/core/src/multiarray/shape.c:927: note: previous definition of 'intp_abs' was here error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" failed with exit status 1 (bare-env)[matthew at vagus ~/dev_trees/numpy ((9194b3a...)|BISECTING)]$ echo $ENABLE_SEPARATE_COMPILATION 1 Best, Matthew From mwwiebe at gmail.com Sun Mar 4 23:32:35 2012 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 4 Mar 2012 22:32:35 -0600 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: On Sun, Mar 4, 2012 at 10:08 PM, Matthew Brett wrote: > Hi, > > On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe wrote: > > On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett > > >> wrote: > >> > Hi, > >> > > >> > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris > >> > wrote: > >> >> > >> >> > >> >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett < > matthew.brett at gmail.com> > >> >> wrote: > >> >>> > >> >>> Hi, > >> >>> > >> >>> Sorry that this report is not complete, I don't have full access to > >> >>> this box but, on a Debian squeeze machine running linux > >> >>> 2.6.32-5-sparc64-smp: > >> >>> > >> >>> nosetests > >> >>> > >> >>> > ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values > >> >>> > >> >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error > >> >>> > >> >>> This on current master : 1.7.0.dev-b9872b4 > >> >>> > >> >> > >> >> Hmm, some tests might have been recently enabled. Any chance of > doing a > >> >> bisection? > >> > >> Struggling because compilation is very slow and there are lots of > >> untestable commits. df907e6 is the first known bad. Here's the > >> output from a log: > >> > >> * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build > >> mode because of a static function in two separate files (6 months ago) > >> [Mark Wiebe] > >> * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) > >> BUG: nditer: The nditer was reusing the reduce loop inappropriately > >> (#1938) (6 months ago) [Mark Wiebe] > >> * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) > >> DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] > >> * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) > >> BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 > >> months ago) [Mark Wiebe] > >> * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) > >> TST: missingdata: Finish up NA mask tests for np.std and np.var (6 > >> months ago) [Mark Wiebe] > >> * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) > >> BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue > >> (6 months ago) [Mark Wiebe] > >> * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) > >> ENH: missingdata: Implement tests for np.std, add skipna= and > >> keepdims= parameters to more functions (6 months ago) [Mark Wiebe] > >> * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) > >> ENH: missingdata: Implement skipna= support for np.std and np.var (6 > >> months ago) [Mark Wiebe] > >> * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) > >> ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months > >> ago) [Mark Wiebe] > >> * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) > >> ENH: missingdata: Create count_reduce_items function (6 months ago) > >> [Mark Wiebe] > >> * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) > >> ENH: missingdata: Move the Reduce boilerplate into a function > >> PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] > >> * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) > >> ENH: missingdata: Finish count_nonzero as a full-fledged reduction > >> operation (6 months ago) [Mark Wiebe] > >> * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) > >> ENH: missingdata: Towards making count_nonzero a full-featured > >> reduction operation (6 months ago) [Mark Wiebe] > >> * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) > >> ENH: missingdata: Move some of the refactored reduction code into the > >> API (6 months ago) [Mark Wiebe] > >> * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) > >> ENH: missingdata: Change default to create NA-mask when NAs are in > >> lists (6 months ago) [Mark Wiebe] > >> * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) > >> ENH: missingdata: trying some more functions to see how they treat NAs > >> (6 months ago) [Mark Wiebe] > >> * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) > >> ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 > >> months ago) [Mark Wiebe] > >> * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) > >> BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it > >> (6 months ago) [Mark Wiebe] > >> * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) > >> BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA > >> masks (6 months ago) [Mark Wiebe] > >> * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) > >> ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 > >> months ago) [Mark Wiebe] > >> * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) > >> ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to > >> always return a view (6 months ago) [Mark Wiebe] > >> > >> So - the problem arises somewhere between 99a21ef (good) and df907e6 > >> (bad) > >> > >> There seems to be a compilation error for the skipped commits - here's > >> the one I tested, 9194b3a: > > > > > > If you enable separate compilation by setting an environment variable, > these > > commits should build as well. > > > > $ export ENABLE_SEPARATE_COMPILATION=1 > > > > I might be doing something wrong but: > I made a mistake, sorry! I even copy/pasted the variable name to make sure I didn't misspell it, but I didn't notice that setup.py uses a different but nearly identical name internally as the environment variable. That should have been $ export NPY_SEPARATE_COMPILATION=1 The place this is used inside of setup.py is here: https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L14 I really dislike this build feature, it repeatedly trips me up. In my opinion, the build should be changed to always do separate compilation, and the single file mode should be eradicated. Thanks , Mark > > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > In file included from numpy/core/src/multiarray/scalartypes.c.src:25, > from > numpy/core/src/multiarray/multiarraymodule_onefile.c:10: > numpy/core/src/multiarray/_datetime.h:9: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:13: > numpy/core/src/multiarray/datetime.c:34: warning: function declaration > isn't a prototype > In file included from > numpy/core/src/multiarray/multiarraymodule_onefile.c:41: > numpy/core/src/multiarray/nditer_constr.c:2373: error: redefinition of > 'intp_abs' > numpy/core/src/multiarray/shape.c:927: note: previous definition of > 'intp_abs' was here > error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv > -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include > -Ibuild/src.linux-sparc64-2.6/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include > -I/usr/include/python2.6 > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/multiarray > -Ibuild/src.linux-sparc64-2.6/numpy/core/src/umath -c > numpy/core/src/multiarray/multiarraymodule_onefile.c -o > > build/temp.linux-sparc64-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o" > failed with exit status 1 > > (bare-env)[matthew at vagus ~/dev_trees/numpy ((9194b3a...)|BISECTING)]$ > echo $ENABLE_SEPARATE_COMPILATION > 1 > > Best, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Mar 5 01:34:56 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 4 Mar 2012 22:34:56 -0800 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: Hi, On Sun, Mar 4, 2012 at 8:32 PM, Mark Wiebe wrote: > On Sun, Mar 4, 2012 at 10:08 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sun, Mar 4, 2012 at 11:41 AM, Mark Wiebe wrote: >> > On Sun, Mar 4, 2012 at 11:27 AM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Sat, Mar 3, 2012 at 12:07 AM, Matthew Brett >> >> >> >> wrote: >> >> > Hi, >> >> > >> >> > On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris >> >> > wrote: >> >> >> >> >> >> >> >> >> On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett >> >> >> >> >> >> wrote: >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> Sorry that this report is not complete, I don't have full access to >> >> >>> this box but, on a Debian squeeze machine running linux >> >> >>> 2.6.32-5-sparc64-smp: >> >> >>> >> >> >>> nosetests >> >> >>> >> >> >>> >> >> >>> ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values >> >> >>> >> >> >>> test_user_missing_values (test_io.TestFromTxt) ... Bus error >> >> >>> >> >> >>> This on current master : 1.7.0.dev-b9872b4 >> >> >>> >> >> >> >> >> >> Hmm, some tests might have been recently enabled. Any chance of >> >> >> doing a >> >> >> bisection? >> >> >> >> Struggling because compilation is very slow and there are lots of >> >> untestable commits. ?df907e6 is the first known bad. ?Here's the >> >> output from a log: >> >> >> >> * df907e6 - (HEAD, refs/bisect/bad) BLD: Failure in single file build >> >> mode because of a static function in two separate files (6 months ago) >> >> [Mark Wiebe] >> >> * 01b200b - (refs/bisect/skip-01b200b10149312f51234448e44b230b1b548046) >> >> BUG: nditer: The nditer was reusing the reduce loop inappropriately >> >> (#1938) (6 months ago) [Mark Wiebe] >> >> * f45fd67 - (refs/bisect/skip-f45fd67fe8eefc8fd2e4b914ab4e376ab5226887) >> >> DOC: Small tweak to release notes (6 months ago) [Mark Wiebe] >> >> * 73be11d - (refs/bisect/skip-73be11db794d115a7d9bd2e822c0d8008bc14a28) >> >> BUG: Some bugs in squeeze and concatenate found by testing SciPy (6 >> >> months ago) [Mark Wiebe] >> >> * c873295 - (refs/bisect/skip-c8732958c8e07f2306029dfde2178faf9c01d049) >> >> TST: missingdata: Finish up NA mask tests for np.std and np.var (6 >> >> months ago) [Mark Wiebe] >> >> * e15712c - (refs/bisect/skip-e15712cf5df41806980f040606744040a433b331) >> >> BUG: nditer: NA masks in arrays with leading 1 dimensions had an issue >> >> (6 months ago) [Mark Wiebe] >> >> * ded81ae - (refs/bisect/skip-ded81ae7d529ac0fba641b7e5e3ecf52e120700f) >> >> ENH: missingdata: Implement tests for np.std, add skipna= and >> >> keepdims= parameters to more functions (6 months ago) [Mark Wiebe] >> >> * a112fc4 - (refs/bisect/skip-a112fc4a6b28fbb85e1b0c6d423095d13cf7b226) >> >> ENH: missingdata: Implement skipna= support for np.std and np.var (6 >> >> months ago) [Mark Wiebe] >> >> * 0fa4f22 - (refs/bisect/skip-0fa4f22fec4b19e2a8c1d93e5a1f955167c9addd) >> >> ENH: missingdata: Support 'skipna=' parameter in np.mean (6 months >> >> ago) [Mark Wiebe] >> >> * bfda229 - (refs/bisect/skip-bfda229ec93d37b1ee2cdd8b9443ec4e34536bbf) >> >> ENH: missingdata: Create count_reduce_items function (6 months ago) >> >> [Mark Wiebe] >> >> * d9b3f90 - (refs/bisect/skip-d9b3f90de3213ece9a78b77088fdec17910e81d9) >> >> ENH: missingdata: Move the Reduce boilerplate into a function >> >> PyArray_ReduceWrapper (6 months ago) [Mark Wiebe] >> >> * 67ece6b - (refs/bisect/skip-67ece6bdd2b35d011893e78154dbff6ab51c7d35) >> >> ENH: missingdata: Finish count_nonzero as a full-fledged reduction >> >> operation (6 months ago) [Mark Wiebe] >> >> * 6bfd819 - (refs/bisect/skip-6bfd819a0897caf6e6db244930c40ed0d17b9e62) >> >> ENH: missingdata: Towards making count_nonzero a full-featured >> >> reduction operation (6 months ago) [Mark Wiebe] >> >> * a1faa1b - (refs/bisect/skip-a1faa1b6883c47333508a0476c1304b0a8a3f64e) >> >> ENH: missingdata: Move some of the refactored reduction code into the >> >> API (6 months ago) [Mark Wiebe] >> >> * f597374 - (refs/bisect/skip-f597374edc298810083799e8539c99fc0a93b319) >> >> ENH: missingdata: Change default to create NA-mask when NAs are in >> >> lists (6 months ago) [Mark Wiebe] >> >> * 965e4cf - (refs/bisect/skip-965e4cff5c4c50e8ff051a3363adc6cf6aa640cd) >> >> ENH: missingdata: trying some more functions to see how they treat NAs >> >> (6 months ago) [Mark Wiebe] >> >> * b1cb211 - (refs/bisect/skip-b1cb211d159c617ee4ebd16266d6f1042417ef75) >> >> ENH: missingdata: Add nastr= parameter to np.set_printoptions() (6 >> >> months ago) [Mark Wiebe] >> >> * ba4d116 - (refs/bisect/skip-ba4d1161fe4943cb720f35c0abfd0581628255d6) >> >> BUG: missingdata: Fix mask usage in PyArray_TakeFrom, add tests for it >> >> (6 months ago) [Mark Wiebe] >> >> * a3a0ee8 - (refs/bisect/skip-a3a0ee8c72fdd55ffacb96bbb1fa9c3569cfb3e9) >> >> BUG: missingdata: The ndmin parameter to np.array wasn't respecting NA >> >> masks (6 months ago) [Mark Wiebe] >> >> * 9194b3a - (refs/bisect/skip-9194b3af704df71aa9b1ff2f53f169848d0f9dc7) >> >> ENH: missingdata: Rewrite PyArray_Concatenate to work with NA masks (6 >> >> months ago) [Mark Wiebe] >> >> * 99a21ef - (refs/bisect/good-99a21efff4b1f2292dc370c7c9c7c58f10385f2a) >> >> ENH: missingdata: Add NA support to np.diagonal, change np.diagonal to >> >> always return a view (6 months ago) [Mark Wiebe] >> >> >> >> So - the problem arises somewhere between 99a21ef (good) and ?df907e6 >> >> (bad) >> >> >> >> There seems to be a compilation error for the skipped commits - here's >> >> the one I tested, 9194b3a: >> > >> > >> > If you enable separate compilation by setting an environment variable, >> > these >> > commits should build as well. >> > >> > $ export?ENABLE_SEPARATE_COMPILATION=1 >> > >> >> I might be doing something wrong but: > > > I made a mistake, sorry! I even copy/pasted the variable name to make sure I > didn't misspell it, but I didn't notice that setup.py uses a different but > nearly identical name internally as the environment variable. That should > have been > > $ export?NPY_SEPARATE_COMPILATION=1 Thanks, that did it: 9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit Let me know if I can debug further, See you, Matthew From mwwiebe at gmail.com Mon Mar 5 02:53:09 2012 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 4 Mar 2012 23:53:09 -0800 Subject: [Numpy-discussion] Bus error for Debian / SPARC on current trunk In-Reply-To: References: Message-ID: On Sun, Mar 4, 2012 at 10:34 PM, Matthew Brett wrote: > > > $ export NPY_SEPARATE_COMPILATION=1 > > Thanks, that did it: > > 9194b3af704df71aa9b1ff2f53f169848d0f9dc7 is the first bad commit > > Let me know if I can debug further, > That commit was a rewrite of np.concatenate, and I've traced the test function you got the crash in. The only call to concatenate is as follows: >>> a = np.array([True], dtype=object) >>> np.concatenate((a,)*3) array([True, True, True], dtype=object) >>> Can you try this and see if it crashes? Another thing you can do is compile with debug information enabled, then run the crashing case in gdb. This will look something like this: $ *export CFLAGS=-g* $ *rm -rf build* # make sure it's a fresh build from scratch $ *python setup.py install --prefix=* # or however you do it [... build printout] $ *gdb python