From ambrose.li at gmail.com Sun Sep 1 11:32:14 2013 From: ambrose.li at gmail.com (Ambrose LI) Date: Sun, 1 Sep 2013 11:32:14 -0400 Subject: [Numpy-discussion] numpy.org down - any ETA on when it's back up? Message-ID: Hi, numpy.org has been down for at least a few days and is currently accessible only through archive.org. So I?m just wondering if anyone with admin access to the site is working on bringing numpy.org back up. Apparently, github has recently changed their project subdomains from github.com to github.io, and (presumably as a result) numpy.github.com is no longer responding. I checked and saw that numpy.org?s cname has a ttl of just 60 seconds, so fixing it should fix the problem right away provided the fix isn?t going to cause a redirection loop. Thanks much -- cheers, -ambrose From charlesr.harris at gmail.com Sun Sep 1 12:54:53 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 1 Sep 2013 10:54:53 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release Message-ID: Hi all, I'm happy to announce the first beta release of Numpy 1.8.0. Please try this beta and report any issues on the numpy-dev mailing list. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows and OS X installers will follow when the infrastructure issues are dealt with. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Sun Sep 1 13:15:54 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Sun, 1 Sep 2013 19:15:54 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: Message-ID: On 1 September 2013 18:54, Charles R Harris wrote: > I'm happy to announce the first beta release of Numpy 1.8.0. Please try > this beta and report any issues on the numpy-dev mailing list. > In an old thread there was discussion about adding minmax and sincos; is there a plan for implementing these? -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 1 13:22:40 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 1 Sep 2013 11:22:40 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: Message-ID: On Sun, Sep 1, 2013 at 11:15 AM, Da?id wrote: > On 1 September 2013 18:54, Charles R Harris wrote: > >> I'm happy to announce the first beta release of Numpy 1.8.0. Please try >> this beta and report any issues on the numpy-dev mailing list. >> > > In an old thread there was discussion about adding minmax and sincos; is > there a plan for implementing these? > > That would be for 1.9, you should open a new thread for that question. I think a good argument can be made for both. Another pair that might be useful would be stdmean (or a better name). Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From malorian at me.com Sun Sep 1 18:38:02 2013 From: malorian at me.com (Daniel Smith) Date: Sun, 01 Sep 2013 17:38:02 -0500 Subject: [Numpy-discussion] Faster np.triu_indices Message-ID: Hello all, I was noticing that `np.triu_indices` took quite awhile and discovered it creates an upper triu array and then uses `np.where`. This seems quite inefficient and I was curious if something like the following would be better: """ def fast_triu_indices(dim,k=0): tmp_range = np.arange(dim-k) rows = np.repeat(tmp_range,(tmp_range+1)[::-1]) cols = np.ones(rows.shape[0],dtype=np.int) inds = np.cumsum(tmp_range[1:][::-1]+1) np.put(cols,inds,np.arange(dim*-1+2+k,1)) cols[0] = k np.cumsum(cols,out=cols) return (rows,cols) """ This is just a first run at the function, and unfortunately does not work for k<0. However, it does return the correct results for k>=0 and is between 2-8 faster on my machine then `np.triu_indices`. Any thoughts on this? -Daniel From joseluismietta at yahoo.com.ar Sun Sep 1 18:55:04 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Sun, 1 Sep 2013 15:55:04 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm Message-ID: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> Hi experts! I wanna study the intersection between line segments (sticks). I wrote a algorithm that generate a matrix, M, with N rows and N columns. The M-element Mij is 1 if stick number 'i' intersect stick number 'j' (obviously M is symmetric). Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. Any idea for that? Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sun Sep 1 20:18:04 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 1 Sep 2013 17:18:04 -0700 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> Message-ID: On Sun, Sep 1, 2013 at 3:55 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > Given two arbitrary sticks, i need a simple and effective algorithm that > determinate if that two sticks are conected by a 'intersected-sticks' path. > > do you mean a test to see if two line segments intersect? This looks reasonable: http://wiki.processing.org/w/Line-Line_intersection It probably makes sense to translate to Cython (or use the C and call with cython). I"ve also got similar code in a little package of mine: https://github.com/ChrisBarker-NOAA/py_geometry Already Cython, and includes code to check a whole bunch at once, stored in numpy arrays: https://github.com/ChrisBarker-NOAA/py_geometry/blob/master/py_geometry/line_crossings.pyx I hope it's useful to you. -Chris -Chris -Chris > Any idea for that? > > Thanks a lot! > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseluismietta at yahoo.com.ar Sun Sep 1 20:50:04 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Sun, 1 Sep 2013 17:50:04 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> Message-ID: <1378083004.8117.YahooMailNeo@web142304.mail.bf1.yahoo.com> Hi Chris. I wanna see if two line segments are connected by a path formed by line segments intersected. Waiting for your answers. Thanks a lot! De: Chris Barker - NOAA Federal Para: Discussion of Numerical Python Enviado: domingo, 1 de septiembre de 2013 21:18 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On Sun, Sep 1, 2013 at 3:55 PM, Jos? Luis Mietta wrote: Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. > > > do you mean a test to see if two line segments intersect? This looks reasonable: http://wiki.processing.org/w/Line-Line_intersection It probably makes sense to translate to Cython (or use the C and call with cython). I"ve also got similar code in a little package of mine: https://github.com/ChrisBarker-NOAA/py_geometry Already Cython, and includes code to check a whole bunch at once, stored in numpy arrays: https://github.com/ChrisBarker-NOAA/py_geometry/blob/master/py_geometry/line_crossings.pyx I hope it's useful to you. -Chris -Chris -Chris ? Any idea for that? > > >Thanks a lot! > > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959?? voice 7600 Sand Point Way NE ??(206) 526-6329?? fax Seattle, WA ?98115 ? ? ??(206) 526-6317?? main reception Chris.Barker at noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Sun Sep 1 21:06:27 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Mon, 2 Sep 2013 03:06:27 +0200 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378083004.8117.YahooMailNeo@web142304.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378083004.8117.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: On 2 September 2013 02:50, Jos? Luis Mietta wrote: > I wanna see if two line segments are connected by a path formed by line > segments intersected. > You could build a network where each segment is a node and an intersection is a link. Then, you could use NetworkX connected_components to get groups of segments together. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Sep 2 03:38:41 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 2 Sep 2013 09:38:41 +0200 Subject: [Numpy-discussion] Faster np.triu_indices In-Reply-To: References: Message-ID: On Mon, Sep 2, 2013 at 12:38 AM, Daniel Smith wrote: > This is just a first run at the function, and unfortunately does not work for k<0. However, it does return the correct results for k>=0 and is between 2-8 faster on my machine then `np.triu_indices`. Any thoughts on this? That looks like a good start to me. Let's get a pull request up that fixes it for k < 0 as well, and then continue the discussion there. St?fan From juanlu001 at gmail.com Mon Sep 2 03:53:52 2013 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Mon, 02 Sep 2013 09:53:52 +0200 Subject: [Numpy-discussion] Are NEPs still a thing? Message-ID: <52244410.6070402@gmail.com> The subject speaks for itself, I was diving into NumPy source code and saw some NEPs, and I wonder if there still function as a mechanism to propose changes to NumPy, and if it is documented somewhere. From juanlu001 at gmail.com Mon Sep 2 04:43:16 2013 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Mon, 02 Sep 2013 10:43:16 +0200 Subject: [Numpy-discussion] Deprecation of financial routines In-Reply-To: References: Message-ID: <52244FA4.7070704@gmail.com> [snip] On 08/19/2013 05:06 PM, Joe Harrington wrote: > A reorg that would bring us to a very heirarchical structure would be > disruptive to existing code. Yet, there are maintenance and startup > speed arguments in favor of a heirarchy. However we resolve it, I don't > know that singling out the financial routines is the right short-term > approach, and it wouldn't make much of a dent. So, I'd suggest we take > it as two issues to solve separately 1) keeping or tossing the fin > routines and 2) moving toward a heirarchy of routines or not. Talking about existing code, honestly I don't think we should worry about that much. I made a not-so-shallow search on GitHub and Google and I saw zero mention of numpy.financial in the NumPy User Guide, zero in the SciPy Cookbook, no mention on GitHub (people are pushing their virtualenvs to their repos, so NumPy source code is replicated a thousand times on the internets ;) including financial_test.py), and, well, I didn't go through all the ooooo's of Google but you get the picture. Making people used to IDL and MATLAB use namespaces is an issue completely unrelated to this in my opinion, or better put, much more general, because it affects the whole scientific Python ecosystem. Josef mentioned matplotlib.finance: https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/finance.py#L5 "This module is deprecated in 1.4 and will be moved to `mpl_toolkits`or it's own project in the future.". They might be in a similar situation than NumPy. I don't have an opinion on the usefulness of financial functions on NumPy and I have zero knowledge on finance myself, but if this discussion makes someone write a nice tutorial on using them (being on a subpackage or not) and attract people to actually use them (I would bet very few are doing so as of now), then great! My proposal is moving them to a separate subpackage as a first step to their disappearance. Maybe when someone sees the move and/or the deprecation note on the release notes that tutorial will be written :) Juan Luis Cano From robert.kern at gmail.com Mon Sep 2 05:39:02 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Sep 2013 10:39:02 +0100 Subject: [Numpy-discussion] Are NEPs still a thing? In-Reply-To: <52244410.6070402@gmail.com> References: <52244410.6070402@gmail.com> Message-ID: On Mon, Sep 2, 2013 at 8:53 AM, Juan Luis Cano wrote: > > The subject speaks for itself, I was diving into NumPy source code and > saw some NEPs, and I wonder if there still function as a mechanism to > propose changes to NumPy, and if it is documented somewhere. Yes, the NEP "process", such as it is, is still a mechanism for proposing some kinds of changes to numpy, but it's not the only one. Generally, it's only needed for sweeping changes or ones that require significant amounts of considered design and choosing between close alternatives. Most changes do not fall into this category. The process itself is not really documented, per se. Post what you are thinking about to the mailing list. The discussion may reach a point where someone says, "Hey, it would be a good idea to write all this up into a NEP". Then you write a NEP. Alternative designs are documented and discussed, one is chosen, implementations are made, PRs are merged. Or not. If you want information about how to format a NEP and what to include, consult PEP 1 and PEP 12, as we stole this process from the core Python development team: http://www.python.org/dev/peps/pep-0001/ http://www.python.org/dev/peps/pep-0012/ -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 2 09:40:34 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Sep 2013 14:40:34 +0100 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> Message-ID: On Sun, Sep 1, 2013 at 11:55 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > > Hi experts! > > I wanna study the intersection between line segments (sticks). > I wrote a algorithm that generate a matrix, M, with N rows and N columns. The M-element Mij is 1 if stick number 'i' intersect stick number 'j' (obviously M is symmetric). > > Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. You may find it helpful to reword your problem using standard terminology. If I hadn't read your previous question, I would not have been able to understand what you meant by intersected sticks (or, as Chris thought at first, that you needed help determining the intersections). This will also help in searching Google for background and pre-existing software to solve your problem. You have an "undirected graph" (the sticks are "nodes", and the intersections are "edges") and you want to find if two given nodes are "reachable" from each other. You are currently representing your graph as an "adjacency matrix" `M` where `M[i,j]` is 1 iff nodes `i` and `j` have an edge between them and 0 otherwise. The "transitive closure" of your graph `M` is the graph that connects two nodes with an edge iff the two nodes are reachable from each other in the original graph `M`. There are several graph theory packages out there, like NetworkX, that will do this for you. Depending on the kinds of queries you would like to do, as David points out, want to compute the "connected components" of your graph; a connected component is a subgraph of your original graph such that all of the nodes are reachable from each other. You can also look up Python code for computing the transitive closure of a graph; it's not a complicated algorithm. However, this algorithm is usually implemented using a different representation of a graph than an adjacency matrix, so you may need to do a conversion. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 2 10:11:38 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Sep 2013 15:11:38 +0100 Subject: [Numpy-discussion] numpy.org down - any ETA on when it's back up? In-Reply-To: References: Message-ID: On Sun, Sep 1, 2013 at 4:32 PM, Ambrose LI wrote: > > Hi, > > numpy.org has been down for at least a few days and is currently > accessible only through archive.org. So I?m just wondering if anyone > with admin access to the site is working on bringing numpy.org back > up. > > Apparently, github has recently changed their project subdomains from > github.com to github.io, and (presumably as a result) numpy.github.com > is no longer responding. I checked and saw that numpy.org?s cname has > a ttl of just 60 seconds, so fixing it should fix the problem right > away provided the fix isn?t going to cause a redirection loop. The problem appears to have corrected itself without any changes. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregorio.bastardo at gmail.com Mon Sep 2 10:21:11 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Mon, 2 Sep 2013 16:21:11 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values Message-ID: Hi, I'd like to find the smallest possible representation of an array given a set of possible values. I've checked the function 'np.min_scalar_type', it works well for scalar input, but contrary to my assumption when array-like param is given: array's dtype is simply returned, instead of finding a more compact dtype for the array values. >>> np.version.version '1.7.0' >>> np.min_scalar_type(0) # ok dtype('uint8') >>> np.min_scalar_type(-1) # ok dtype('int8') >>> np.min_scalar_type( [0,-1] ) # int8 expected, returns platform-default int dtype('int32') >>> np.min_scalar_type( [0,256] ) # uint16 expected dtype('int32') >>> np.min_scalar_type([-1,256]) # int16 expected dtype('int32') Am I missing something? Anyone knows how to achieve the desired operation? Thanks a lot, Gregorio From stefan at sun.ac.za Mon Sep 2 10:55:55 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 2 Sep 2013 16:55:55 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: On Mon, Sep 2, 2013 at 4:21 PM, Gregorio Bastardo wrote: >>>> np.min_scalar_type([-1,256]) # int16 expected > dtype('int32') > > Am I missing something? Anyone knows how to achieve the desired operation? The docstring states explicitly that this use case is not supported. Here's one way of doing it: https://gist.github.com/stefanv/6413742 St?fan From cournape at gmail.com Mon Sep 2 11:30:23 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 2 Sep 2013 16:30:23 +0100 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release In-Reply-To: References: Message-ID: On Fri, Aug 30, 2013 at 7:16 PM, David Cournapeau wrote: > It looks like it broke the build with MKL as well (in, surprised, ARPACK). > I will investigate this further this WE > Ok, I think the commit 5935030f8cced33e433804a21bdb15572d1d38e8 is quite wrong. It conflates the issue of dealing with Accelerate brokenness and using g77 ABI. I would suggest reverting it for 0.13.0 (and re-disable single precision), as fixing this correctly may require quite some time/testing. David > > > On Thu, Aug 22, 2013 at 2:12 PM, Ralf Gommers wrote: > >> Hi all, >> >> I'm happy to announce the availability of the first beta release of Scipy >> 0.13.0. Please try this beta and report any issues on the scipy-dev mailing >> list. >> >> Source tarballs and release notes can be found at >> https://sourceforge.net/projects/scipy/files/scipy/0.13.0b1/. Windows >> and OS X installers will follow later (we have a minor infrastructure issue >> to solve, and I'm at EuroScipy now). >> >> Cheers, >> Ralf >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 2 12:46:48 2013 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 02 Sep 2013 19:46:48 +0300 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, 02.09.2013 18:30, David Cournapeau kirjoitti: [clip] > Ok, I think the commit 5935030f8cced33e433804a21bdb15572d1d38e8 is > quite wrong. > > It conflates the issue of dealing with Accelerate brokenness and > using g77 ABI. I would suggest reverting it for 0.13.0 (and > re-disable single precision), as fixing this correctly may require > quite some time/testing. I'm -1 on returning to the previous situation where many routines on OSX are simply broken (which was the situation previously --- "disabling single precision" left several things still broken). I'd rather just postpone the 0.13.0 release until this issue is solved properly. Can you say what exactly is wrong -- as far as I know, Accelerate on OSX simply uses g77 ABI. There were some bugs previously where it in places did things differently, but on recent OSX releases this is no longer the case? I guess you are trying to link with MKL? In that case, I would rather propose restoring the previous MKL wrappers (with trivial extensions for the missing w* symbols). - -- Pauli Virtanen -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlIkwPQACgkQ6BQxb7O0pWBnkwCfeKj32CYYCPdEWVcYYMtq/OZM 32wAoIn9yG/HNtUeh+XwqAm2uAS9sVQ5 =CKBN -----END PGP SIGNATURE----- From cournape at gmail.com Mon Sep 2 14:12:00 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 2 Sep 2013 19:12:00 +0100 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release In-Reply-To: References: Message-ID: On Mon, Sep 2, 2013 at 5:46 PM, Pauli Virtanen wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > 02.09.2013 18:30, David Cournapeau kirjoitti: > [clip] > > Ok, I think the commit 5935030f8cced33e433804a21bdb15572d1d38e8 is > > quite wrong. > > > > It conflates the issue of dealing with Accelerate brokenness and > > using g77 ABI. I would suggest reverting it for 0.13.0 (and > > re-disable single precision), as fixing this correctly may require > > quite some time/testing. > > I'm -1 on returning to the previous situation where many routines on > OSX are simply broken (which was the situation previously --- > "disabling single precision" left several things still broken). I'd > rather just postpone the 0.13.0 release until this issue is solved > properly. > I see, I missed that it was more than just reverting to slower versions. > > Can you say what exactly is wrong -- as far as I know, Accelerate on > OSX simply uses g77 ABI. There were some bugs previously where it in > places did things differently, but on recent OSX releases this is no > longer the case? > > I guess you are trying to link with MKL? In that case, I would rather > propose restoring the previous MKL wrappers (with trivial extensions > for the missing w* symbols). > The pb is specific to MKL, yes, but I've found a workaround: in the case of MKL, you just need to workaround the g77 ABI of the MKL vs gfortran ABI, but the LAPACK interface is the actual LAPACK, not CLAPACK. So for MKL, you need non dummy wrappers iff the function returns a complex. A dirty patch seems to confirm that this fixes the issue, I will prepare an actual patch tomorrow, David > > - -- > Pauli Virtanen > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > > iEYEARECAAYFAlIkwPQACgkQ6BQxb7O0pWBnkwCfeKj32CYYCPdEWVcYYMtq/OZM > 32wAoIn9yG/HNtUeh+XwqAm2uAS9sVQ5 > =CKBN > -----END PGP SIGNATURE----- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregorio.bastardo at gmail.com Tue Sep 3 03:22:04 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Tue, 3 Sep 2013 09:22:04 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: Thanks St?fan, your script works well. There's a small typo on line 12. I also discovered the functions 'np.iinfo' and 'np.finfo' for machine limits on integer/float types (a note for myself, you might be already familiar with them). After having read the docstring, I was only curious why this feature is not provided by the function itself, as returning the input array's dtype seems not so useful (can't imagine such a use case). Gregorio 2013/9/2 St?fan van der Walt : > On Mon, Sep 2, 2013 at 4:21 PM, Gregorio Bastardo > wrote: >>>>> np.min_scalar_type([-1,256]) # int16 expected >> dtype('int32') >> >> Am I missing something? Anyone knows how to achieve the desired operation? > > The docstring states explicitly that this use case is not supported. > > Here's one way of doing it: https://gist.github.com/stefanv/6413742 > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Tue Sep 3 08:47:26 2013 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 Sep 2013 13:47:26 +0100 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: On Mon, Sep 2, 2013 at 3:55 PM, St?fan van der Walt wrote: > > On Mon, Sep 2, 2013 at 4:21 PM, Gregorio Bastardo > wrote: > >>>> np.min_scalar_type([-1,256]) # int16 expected > > dtype('int32') > > > > Am I missing something? Anyone knows how to achieve the desired operation? > > The docstring states explicitly that this use case is not supported. > > Here's one way of doing it: https://gist.github.com/stefanv/6413742 You can probably reduce the amount of work by only comparing a.min() and a.max() instead of the whole array. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Sep 3 12:31:46 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 3 Sep 2013 18:31:46 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: On Tue, Sep 3, 2013 at 2:47 PM, Robert Kern wrote: >> Here's one way of doing it: https://gist.github.com/stefanv/6413742 > > You can probably reduce the amount of work by only comparing a.min() and > a.max() instead of the whole array. Thanks, fixed. St?fan From cgohlke at uci.edu Tue Sep 3 17:23:59 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 Sep 2013 14:23:59 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: Message-ID: <5226536F.9030509@uci.edu> On 9/1/2013 9:54 AM, Charles R Harris wrote: > Hi all, > > I'm happy to announce the first beta release of Numpy 1.8.0. Please try > this beta and report any issues on the numpy-dev mailing list. > > Source tarballs and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows > and OS X installers will follow when the infrastructure issues are dealt > with. > > Chuck > Hello, I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on win-amd64-py2.7. It builds OK but there are 23 test errors and 6 failures (attached). Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, pandas and matplotlib) that were built against numpy-MKL 1.7 fail tests when used with numpy-MKL 1.8. Other packages test OK (e.g. skimage, sklearn, statsmodels, mahotas, pygame). See compared to . I have not looked in more detail or at other Python versions yet. Christoph -------------- next part -------------- ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.....................................................................................................................................................................................................................................................................................................K...................................................................................................................................................................................................................................................K........................................................................K..K...............................K...SK.S.......S.................................................................................SSSSSSSSSSSSSSSSSSSSS.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............E..K..............................................................EEEE.EEE................................................................................................................................................................................S.........SS..SS.....................................................................................................SSSSSSSS......................................................................................................................................................................................................................................................................................S......................E.............E..................................E............F...........................................................................................F...........................EEEEEEE..E..E..................EF...E........F............................F...E.............................F....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................... ====================================================================== ERROR: test_big_arrays (test_io.TestSavezLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "Test\numpy-build\numpy\lib\tests\test_io.py", line 148, in test_big_arrays a = np.empty(L, dtype=np.uint8) ValueError: negative dimensions are not allowed ====================================================================== ERROR: Basic test of join_by ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 557, in test_inner_join test = join_by('a', a, b, jointype='inner') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: test_join (test_recfunctions.TestJoinBy) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 568, in test_join test = join_by(('a', 'b'), a, b) File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: test_leftouter_join (test_recfunctions.TestJoinBy) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 606, in test_leftouter_join test = join_by(('a', 'b'), a, b, 'leftouter') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: test_outer_join (test_recfunctions.TestJoinBy) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 578, in test_outer_join test = join_by(('a', 'b'), a, b, 'outer') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Basic test of join_by no_r1postfix ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 634, in test_no_r1postfix test = join_by('a', a, b, r1postfix='', r2postfix='2', jointype='inner') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Basic test of join_by no_r2postfix ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 652, in test_no_r2postfix test = join_by('a', a, b, r1postfix='1', r2postfix='', jointype='inner') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: test_two_keys_two_vars (test_recfunctions.TestJoinBy2) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\lib\tests\test_recfunctions.py", line 678, in test_two_keys_two_vars test = join_by(['a', 'k'], a, b, r1postfix='1', r2postfix='2', jointype='inner') File "Test\numpy-build\numpy\lib\recfunctions.py", line 919, in join_by flag_in = ma.concatenate(([False], aux[1:] == aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3629, in __eq__ check = ndarray.__eq__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Tests the concatenation on flexible arrays. ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_core.py", line 146, in test_concatenate_flexible assert_equal_records(test, data) File "Test\numpy-build\numpy\ma\testutils.py", line 78, in assert_equal_records assert_equal(operator.getitem(a, f), operator.getitem(b, f)) File "Test\numpy-build\numpy\ma\testutils.py", line 123, in assert_equal return assert_array_equal(actual, desired, err_msg) File "Test\numpy-build\numpy\ma\testutils.py", line 196, in assert_array_equal header='Arrays are not equal') File "Test\numpy-build\numpy\ma\testutils.py", line 176, in assert_array_compare m = mask_or(getmask(x), getmask(y)) File "Test\numpy-build\numpy\ma\core.py", line 1621, in mask_or return make_mask(umath.logical_or(m1, m2), copy=copy, shrink=shrink) ValueError: operands could not be broadcast together with shapes (9) (10) ====================================================================== ERROR: Tests conversions and indexing ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_core.py", line 247, in test_indexing assert_equal(x1[:], x2[:]) File "Test\numpy-build\numpy\ma\testutils.py", line 123, in assert_equal return assert_array_equal(actual, desired, err_msg) File "Test\numpy-build\numpy\ma\testutils.py", line 196, in assert_array_equal header='Arrays are not equal') File "Test\numpy-build\numpy\ma\testutils.py", line 177, in assert_array_compare x = masked_array(x, copy=False, mask=m, keep_mask=False, subok=False) File "Test\numpy-build\numpy\ma\core.py", line 2701, in __new__ raise MaskError(msg % (nd, nm)) MaskError: Mask and data not compatible: data size is 4, mask size is 3. ====================================================================== ERROR: test_methods_with_output (test_core.TestMaskedArrayArithmetic) ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_core.py", line 1115, in test_methods_with_output result = xmmeth(axis=0, out=output) File "Test\numpy-build\numpy\ma\core.py", line 4813, in var out.flat = dvar File "Test\numpy-build\numpy\ma\core.py", line 3314, in _set_flat y[:] = value File "Test\numpy-build\numpy\ma\core.py", line 3109, in __setslice__ self.__setitem__(slice(i, j), value) File "Test\numpy-build\numpy\ma\core.py", line 3063, in __setitem__ ndarray.__setitem__(_data, indx, dval) ValueError: could not broadcast input array from shape (4) into shape (3) ====================================================================== ERROR: Test in1d ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 858, in test_in1d test = in1d(a, b) File "Test\numpy-build\numpy\ma\extras.py", line 1141, in in1d ar1, rev_idx = unique(ar1, return_inverse=True) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 185, in unique flag = np.concatenate(([True], aux[1:] != aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test in1d's invert parameter ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 872, in test_in1d_invert assert_equal(np.invert(in1d(a, b)), in1d(a, b, invert=True)) File "Test\numpy-build\numpy\ma\extras.py", line 1141, in in1d ar1, rev_idx = unique(ar1, return_inverse=True) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 185, in unique flag = np.concatenate(([True], aux[1:] != aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test intersect1d ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 824, in test_intersect1d test = intersect1d(x, y) File "Test\numpy-build\numpy\ma\extras.py", line 1093, in intersect1d aux = ma.concatenate((unique(ar1), unique(ar2))) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 198, in unique flag = np.concatenate(([True], ar[1:] != ar[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test setdiff1d ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 894, in test_setdiff1d test = setdiff1d(a, b) File "Test\numpy-build\numpy\ma\extras.py", line 1198, in setdiff1d ar1 = unique(ar1) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 198, in unique flag = np.concatenate(([True], ar[1:] != ar[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test setdiff1d_charray ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 905, in test_setdiff1d_char_array assert_array_equal(setdiff1d(a, b), np.array(['c'])) File "Test\numpy-build\numpy\ma\extras.py", line 1200, in setdiff1d aux = in1d(ar1, ar2, assume_unique=True) File "Test\numpy-build\numpy\ma\extras.py", line 1153, in in1d bool_ar = (sar[1:] == sar[:-1]) File "Test\numpy-build\numpy\ma\core.py", line 3620, in __eq__ check = ndarray.__eq__(self.filled(0), other) ValueError: shape mismatch: objects cannot be broadcast to a single shape ====================================================================== ERROR: Test setxor1d ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 832, in test_setxor1d test = setxor1d(a, b) File "Test\numpy-build\numpy\ma\extras.py", line 1110, in setxor1d ar1 = unique(ar1) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 198, in unique flag = np.concatenate(([True], ar[1:] != ar[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test union1d ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 884, in test_union1d test = union1d(a, b) File "Test\numpy-build\numpy\ma\extras.py", line 1174, in union1d return unique(ma.concatenate((ar1, ar2))) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 198, in unique flag = np.concatenate(([True], ar[1:] != ar[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test unique on masked data w/use_mask=True ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 721, in test_unique_onmaskedarray test = unique(data, return_index=True, return_inverse=True) File "Test\numpy-build\numpy\ma\extras.py", line 1056, in unique return_inverse=return_inverse) File "Test\numpy-build\numpy\lib\arraysetops.py", line 185, in unique flag = np.concatenate(([True], aux[1:] != aux[:-1])) File "Test\numpy-build\numpy\ma\core.py", line 3662, in __ne__ check = ndarray.__ne__(self.filled(0), odata).view(type(self)) AttributeError: 'NotImplementedType' object has no attribute 'view' ====================================================================== ERROR: Test of average. ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 129, in test_testAverage1 assert_equal(average(ott, axis=0).mask, [True]) AttributeError: 'numpy.float64' object has no attribute 'mask' ====================================================================== ERROR: Tests median w/ 2D ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 503, in test_2d z[:, 0] = x[:] File "Test\numpy-build\numpy\ma\core.py", line 3063, in __setitem__ ndarray.__setitem__(_data, indx, dval) ValueError: could not broadcast input array from shape (100) into shape (101) ====================================================================== ERROR: Tests polyfit ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 681, in test_polyfit full=True) File "Test\numpy-build\numpy\lib\polynomial.py", line 555, in polyfit raise TypeError("expected x and y to have same length") TypeError: expected x and y to have same length ====================================================================== ERROR: Test of conversions and indexing ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_old_ma.py", line 219, in test_testCI assert_(eq(x1[:], x2[:])) File "Test\numpy-build\numpy\ma\tests\test_old_ma.py", line 15, in eq result = allclose(v, w) File "Test\numpy-build\numpy\ma\core.py", line 6916, in allclose xinf = np.isinf(masked_array(x, copy=False, mask=m)).filled(False) File "Test\numpy-build\numpy\ma\core.py", line 2701, in __new__ raise MaskError(msg % (nd, nm)) MaskError: Mask and data not compatible: data size is 4, mask size is 3. ====================================================================== FAIL: Test hard_mask ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_core.py", line 1243, in test_hardmask assert_equal(xs._data, [1, 1, 1, 1, 1]) File "Test\numpy-build\numpy\ma\testutils.py", line 123, in assert_equal return assert_array_equal(actual, desired, err_msg) File "Test\numpy-build\numpy\ma\testutils.py", line 196, in assert_array_equal header='Arrays are not equal') File "Test\numpy-build\numpy\ma\testutils.py", line 189, in assert_array_compare verbose=verbose, header=header) File "Test\numpy-build\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 20.0%) x: array([ 1, 1, 1, 1, 40]) y: array([1, 1, 1, 1, 1]) ====================================================================== FAIL: Test getmaskarray on flexible dtype ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_core.py", line 3465, in test_getmaskarray dtype=[('a', '|b1'), ('b', '|b1')])) File "Test\numpy-build\numpy\ma\testutils.py", line 123, in assert_equal return assert_array_equal(actual, desired, err_msg) File "Test\numpy-build\numpy\ma\testutils.py", line 196, in assert_array_equal header='Arrays are not equal') File "Test\numpy-build\numpy\ma\testutils.py", line 189, in assert_array_compare verbose=verbose, header=header) File "Test\numpy-build\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 33.3333333333%) x: array([(True, True), (True, True), (False, False)], dtype=[('a', '?'), ('b', '?')]) y: array([(True, True), (True, True), (True, True)], dtype=[('a', '?'), ('b', '?')]) ====================================================================== FAIL: Tests median w/ 2D arrays and different axis. ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_extras.py", line 517, in test_2d_waxis assert_equal(median(x), 14.5) File "Test\numpy-build\numpy\ma\testutils.py", line 100, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 16.0 DESIRED: 14.5 ====================================================================== FAIL: Tests setting fields. ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_mrecords.py", line 134, in test_set_fields assert_equal(mbase.a._mask, [0, 1, 0, 0, 0]) File "Test\numpy-build\numpy\ma\testutils.py", line 123, in assert_equal return assert_array_equal(actual, desired, err_msg) File "Test\numpy-build\numpy\ma\testutils.py", line 196, in assert_array_equal header='Arrays are not equal') File "Test\numpy-build\numpy\ma\testutils.py", line 189, in assert_array_compare verbose=verbose, header=header) File "Test\numpy-build\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 20.0%) x: array([False, True, False, False, True], dtype=bool) y: array([0, 1, 0, 0, 0]) ====================================================================== FAIL: Test of average. ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_old_ma.py", line 529, in test_testAverage1 self.assertTrue(average(ott, axis=0) is masked) AssertionError: False is not true ====================================================================== FAIL: Issue gh-2757 ---------------------------------------------------------------------- Traceback (most recent call last): File "Test\numpy-build\numpy\ma\tests\test_regression.py", line 62, in test_var_sets_maskedarray_scalar assert_(mout._data == 0) File "Test\numpy-build\numpy\testing\utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError ---------------------------------------------------------------------- Ran 4721 tests in 36.084s FAILED (KNOWNFAIL=10, SKIP=39, errors=23, failures=6) Running unit tests for numpy NumPy version 1.8.0.dev-86a6e6c NumPy is installed in Test\numpy-build\numpy Python version 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] nose version 1.3.0 From charlesr.harris at gmail.com Tue Sep 3 17:51:09 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 15:51:09 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <5226536F.9030509@uci.edu> References: <5226536F.9030509@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke wrote: > On 9/1/2013 9:54 AM, Charles R Harris wrote: > >> Hi all, >> >> I'm happy to announce the first beta release of Numpy 1.8.0. Please try >> this beta and report any issues on the numpy-dev mailing list. >> >> Source tarballs and release notes can be found at >> https://sourceforge.net/**projects/numpy/files/NumPy/1.**8.0b1/. >> The Windows >> and OS X installers will follow when the infrastructure issues are dealt >> with. >> >> Chuck >> >> > Hello, > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 failures > (attached). > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, pandas > and matplotlib) that were built against numpy-MKL 1.7 fail tests when used > with numpy-MKL 1.8. Other packages test OK (e.g. skimage, sklearn, > statsmodels, mahotas, pygame). See gohlke/pythonlibs/tests/**20130902-win-amd64-py2.7-** > numpy-1.8.0.dev-86a6e6c/> > compared to 20130902-win-amd64-py2.7/>. > > > I have not looked in more detail or at other Python versions yet. > > Thanks Christoph, Looks like some work to do. I wonder what is different between windows and linux here? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 3 19:32:13 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 17:32:13 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <5226536F.9030509@uci.edu> References: <5226536F.9030509@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke wrote: > On 9/1/2013 9:54 AM, Charles R Harris wrote: > >> Hi all, >> >> I'm happy to announce the first beta release of Numpy 1.8.0. Please try >> this beta and report any issues on the numpy-dev mailing list. >> >> Source tarballs and release notes can be found at >> https://sourceforge.net/**projects/numpy/files/NumPy/1.**8.0b1/. >> The Windows >> and OS X installers will follow when the infrastructure issues are dealt >> with. >> >> Chuck >> >> > Hello, > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 failures > (attached). > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, pandas > and matplotlib) that were built against numpy-MKL 1.7 fail tests when used > with numpy-MKL 1.8. Other packages test OK (e.g. skimage, sklearn, > statsmodels, mahotas, pygame). See gohlke/pythonlibs/tests/**20130902-win-amd64-py2.7-** > numpy-1.8.0.dev-86a6e6c/> > compared to 20130902-win-amd64-py2.7/>. > > > I have not looked in more detail or at other Python versions yet. > > It's pretty clear that I will need a windows environment to debug this. I have windows 7 running in a virtual machine, and have downloaded the vsc9 express compiler and isos. Do I need to burn those guys to a disk in order to install or is there some windows magic that will let me install them directly from the isos? That done, I assume I can just download python 2.7 for windows, clone the repository, and do the usual python setup.py install thing. Anything I need to be wary about, any pointers? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Sep 3 19:40:36 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 Sep 2013 16:40:36 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> Message-ID: <52267374.7060602@uci.edu> On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > wrote: > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > Hi all, > > I'm happy to announce the first beta release of Numpy 1.8.0. > Please try > this beta and report any issues on the numpy-dev mailing list. > > Source tarballs and release notes can be found at > https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > . > The Windows > and OS X installers will follow when the infrastructure issues > are dealt > with. > > Chuck > > > Hello, > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 > failures (attached). > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, > pandas and matplotlib) that were built against numpy-MKL 1.7 fail > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. > skimage, sklearn, statsmodels, mahotas, pygame). See > > > compared to > >. > > > I have not looked in more detail or at other Python versions yet. > > > Thanks Christoph, > > Looks like some work to do. I wonder what is different between windows > and linux here? > > Chuck > Looks like the fundamental PyArray_PyIntAsIntp function is broken on 64 bit Windows. 64 bit PyLong values are intermediately stored in a 32 bit C long variable. But maybe I am missing something... Christoph From charlesr.harris at gmail.com Tue Sep 3 19:45:38 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 17:45:38 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <52267374.7060602@uci.edu> References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke wrote: > On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > > wrote: > > > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > > Hi all, > > > > I'm happy to announce the first beta release of Numpy 1.8.0. > > Please try > > this beta and report any issues on the numpy-dev mailing list. > > > > Source tarballs and release notes can be found at > > https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > > . > > The Windows > > and OS X installers will follow when the infrastructure issues > > are dealt > > with. > > > > Chuck > > > > > > Hello, > > > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 > > failures (attached). > > > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, > > pandas and matplotlib) that were built against numpy-MKL 1.7 fail > > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. > > skimage, sklearn, statsmodels, mahotas, pygame). See > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ > >> > > compared to > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ > >>. > > > > > > I have not looked in more detail or at other Python versions yet. > > > > > > Thanks Christoph, > > > > Looks like some work to do. I wonder what is different between windows > > and linux here? > > > > Chuck > > > > Looks like the fundamental PyArray_PyIntAsIntp function is broken on 64 > bit Windows. 64 bit PyLong values are intermediately stored in a 32 bit > C long variable. But maybe I am missing something... > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 > > > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 > > > > My, that does look suspicious. That function is new in 1.8 I believe. Looks like it needs fixing whatever else it fixes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Sep 3 19:55:46 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 Sep 2013 16:55:46 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> Message-ID: <52267702.3060105@uci.edu> On 9/3/2013 4:32 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > wrote: > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > Hi all, > > I'm happy to announce the first beta release of Numpy 1.8.0. > Please try > this beta and report any issues on the numpy-dev mailing list. > > Source tarballs and release notes can be found at > https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > . > The Windows > and OS X installers will follow when the infrastructure issues > are dealt > with. > > Chuck > > > Hello, > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 > failures (attached). > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, > pandas and matplotlib) that were built against numpy-MKL 1.7 fail > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. > skimage, sklearn, statsmodels, mahotas, pygame). See > > > compared to > >. > > > I have not looked in more detail or at other Python versions yet. > > > It's pretty clear that I will need a windows environment to debug this. > I have windows 7 running in a virtual machine, and have downloaded the > vsc9 express compiler and isos. Do I need to burn those guys to a disk > in order to install or is there some windows magic that will let me > install them directly from the isos? That done, I assume I can just > download python 2.7 for windows, clone the repository, and do the usual > python setup.py install thing. Anything I need to be wary about, any > pointers? > > Chuck > I would not recommend the VS Express version. Instead use the "Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1" , which contains compatible 32 and 64 bit compilers for Python 2.6 to 3.2. Use the web installer or mount the ISO with VirtualCloneDrive . Then, on a command prompt in the numpy source directory type (not tested, but should work for 64 bit Python 2.7): setlocal EnableDelayedExpansion call "%ProgramFiles%\Microsoft SDKs\Windows\v7.0\Bin\SetEnv.Cmd" /Release /x64 /vista set DISTUTILS_USE_SDK=1 C:\Python27\python.exe setup.py build Christoph From charlesr.harris at gmail.com Tue Sep 3 19:52:42 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 17:52:42 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 5:45 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke wrote: > >> On 9/3/2013 2:51 PM, Charles R Harris wrote: >> > >> > >> > >> > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > > > wrote: >> > >> > On 9/1/2013 9:54 AM, Charles R Harris wrote: >> > >> > Hi all, >> > >> > I'm happy to announce the first beta release of Numpy 1.8.0. >> > Please try >> > this beta and report any issues on the numpy-dev mailing list. >> > >> > Source tarballs and release notes can be found at >> > https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ >> > . >> > The Windows >> > and OS X installers will follow when the infrastructure issues >> > are dealt >> > with. >> > >> > Chuck >> > >> > >> > Hello, >> > >> > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on >> > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 >> > failures (attached). >> > >> > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, >> > pandas and matplotlib) that were built against numpy-MKL 1.7 fail >> > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. >> > skimage, sklearn, statsmodels, mahotas, pygame). See >> > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ >> > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ >> >> >> > compared to >> > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ >> > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ >> >>. >> > >> > >> > I have not looked in more detail or at other Python versions yet. >> > >> > >> > Thanks Christoph, >> > >> > Looks like some work to do. I wonder what is different between windows >> > and linux here? >> > >> > Chuck >> > >> >> Looks like the fundamental PyArray_PyIntAsIntp function is broken on 64 >> bit Windows. 64 bit PyLong values are intermediately stored in a 32 bit >> C long variable. But maybe I am missing something... >> < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 >> > >> < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 >> > >> >> My, that does look suspicious. That function is new in 1.8 I believe. > Looks like it needs fixing whatever else it fixes. > > BTW, do the tests pass with a 32 build? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Sep 3 20:09:59 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 Sep 2013 17:09:59 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> Message-ID: <52267A57.4010506@uci.edu> On 9/3/2013 4:45 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > wrote: > > On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > > >> wrote: > > > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > > Hi all, > > > > I'm happy to announce the first beta release of Numpy 1.8.0. > > Please try > > this beta and report any issues on the numpy-dev mailing list. > > > > Source tarballs and release notes can be found at > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > > . > > The Windows > > and OS X installers will follow when the infrastructure issues > > are dealt > > with. > > > > Chuck > > > > > > Hello, > > > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 > > failures (attached). > > > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, > > pandas and matplotlib) that were built against numpy-MKL 1.7 fail > > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. > > skimage, sklearn, statsmodels, mahotas, pygame). See > > > > > > > compared to > > > > >. > > > > > > I have not looked in more detail or at other Python versions yet. > > > > > > Thanks Christoph, > > > > Looks like some work to do. I wonder what is different between windows > > and linux here? > > > > Chuck > > > > Looks like the fundamental PyArray_PyIntAsIntp function is broken on 64 > bit Windows. 64 bit PyLong values are intermediately stored in a 32 bit > C long variable. But maybe I am missing something... > > > > My, that does look suspicious. That function is new in 1.8 I believe. > Looks like it needs fixing whatever else it fixes. > > Chuck > In fact, using a npy_longlong instead of npy_long fixes all numpy test errors and failures. But it probably foils the recent optimizations. Christoph From carl.canuck.official at gmail.com Tue Sep 3 20:12:31 2013 From: carl.canuck.official at gmail.com (Carl Canuck) Date: Tue, 3 Sep 2013 20:12:31 -0400 Subject: [Numpy-discussion] (no subject) Message-ID: Hello, I'm new to numpy, and I'm a stuck on my first real project with it. I am trying to take the rfft of a numpy array, like this: >>> my_rfft = numpy.fft.rfft(my_numpy_array) and replace the amplitudes that can be obtained with: >>> my_amplitudes = numpy.abs(my_rfft) with amplitudes from an arbitrary numpy array's rFFT, which is to then be converted back using numpy.fft.irfft . Alternately, some future plans will involve having to modify individual array element amplitudes directly based on other parameters. I would think that modifying and re-synthesizing signals using FFT is a fairly common use-case, but my attempts at Googling example code have been fruitless. I'm not sure if my rudimentary knowledge of FFT is failing me, or if I'm just not understanding how numpy represents and exposes the data, but I would really appreciate any help I can get :) /carl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 3 20:18:33 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 18:18:33 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <52267A57.4010506@uci.edu> References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> <52267A57.4010506@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 6:09 PM, Christoph Gohlke wrote: > On 9/3/2013 4:45 PM, Charles R Harris wrote: > > > > > > > > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > > wrote: > > > > On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > > > > > > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke cgohlke at uci.edu> > > > >> wrote: > > > > > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > > > > Hi all, > > > > > > I'm happy to announce the first beta release of Numpy > 1.8.0. > > > Please try > > > this beta and report any issues on the numpy-dev mailing > list. > > > > > > Source tarballs and release notes can be found at > > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > > > < > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/>. > > > The Windows > > > and OS X installers will follow when the infrastructure > issues > > > are dealt > > > with. > > > > > > Chuck > > > > > > > > > Hello, > > > > > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > > > win-amd64-py2.7. It builds OK but there are 23 test errors and > 6 > > > failures (attached). > > > > > > Some 3rd party packages (e.g. scipy, numexpr, pytables, > bottleneck, > > > pandas and matplotlib) that were built against numpy-MKL 1.7 > fail > > > tests when used with numpy-MKL 1.8. Other packages test OK > (e.g. > > > skimage, sklearn, statsmodels, mahotas, pygame). See > > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ > > < > http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ > > > > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ > >> > > > compared to > > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ > > < > http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ > > > > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ > >>. > > > > > > > > > I have not looked in more detail or at other Python versions > yet. > > > > > > > > > Thanks Christoph, > > > > > > Looks like some work to do. I wonder what is different between > windows > > > and linux here? > > > > > > Chuck > > > > > > > Looks like the fundamental PyArray_PyIntAsIntp function is broken on > 64 > > bit Windows. 64 bit PyLong values are intermediately stored in a 32 > bit > > C long variable. But maybe I am missing something... > > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 > > > > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 > > > > > > My, that does look suspicious. That function is new in 1.8 I believe. > > Looks like it needs fixing whatever else it fixes. > > > > Chuck > > > > In fact, using a npy_longlong instead of npy_long fixes all numpy test > errors and failures. But it probably foils the recent optimizations. > Great! I think the function is not used for numeric things so I'm not sure what optimizations could be affected. I'll put up a PR and backport it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Tue Sep 3 20:24:00 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 Sep 2013 17:24:00 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> Message-ID: <52267DA0.3050305@uci.edu> On 9/3/2013 4:52 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 5:45 PM, Charles R Harris > > wrote: > > > > > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > wrote: > > On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > > >> wrote: > > > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > > Hi all, > > > > I'm happy to announce the first beta release of Numpy 1.8.0. > > Please try > > this beta and report any issues on the numpy-dev mailing list. > > > > Source tarballs and release notes can be found at > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > > . > > The Windows > > and OS X installers will follow when the infrastructure issues > > are dealt > > with. > > > > Chuck > > > > > > Hello, > > > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > > win-amd64-py2.7. It builds OK but there are 23 test errors and 6 > > failures (attached). > > > > Some 3rd party packages (e.g. scipy, numexpr, pytables, bottleneck, > > pandas and matplotlib) that were built against numpy-MKL 1.7 fail > > tests when used with numpy-MKL 1.8. Other packages test OK (e.g. > > skimage, sklearn, statsmodels, mahotas, pygame). See > > > > > > > compared to > > > > >. > > > > > > I have not looked in more detail or at other Python versions yet. > > > > > > Thanks Christoph, > > > > Looks like some work to do. I wonder what is different between windows > > and linux here? > > > > Chuck > > > > Looks like the fundamental PyArray_PyIntAsIntp function is > broken on 64 > bit Windows. 64 bit PyLong values are intermediately stored in a > 32 bit > C long variable. But maybe I am missing something... > > > > My, that does look suspicious. That function is new in 1.8 I > believe. Looks like it needs fixing whatever else it fixes. > > > BTW, do the tests pass with a 32 build? > > Chuck > The 32 bit build fails two tests (unrelated to the above 64 bit issue): ====================================================================== FAIL: test_invalid (test_errstate.TestErrstate) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\numpy\testing\decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "X:\Python27\lib\site-packages\numpy\core\tests\test_errstate.py", line 23, in test_invalid self.fail("Did not raise an invalid error") AssertionError: Did not raise an invalid error ====================================================================== FAIL: simd tests on max/min ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", line 678, in test_minmax_blocked msg=repr(inp) + '\n' + msg) AssertionError: array([ 0., 1., nan, 3., 4., 5., 6., 7., 8., 9., 10.], dtype=float32) unary offset=(0, 0), size=11, dtype=, out of place ---------------------------------------------------------------------- Christoph From chris.barker at noaa.gov Tue Sep 3 20:51:24 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 3 Sep 2013 17:51:24 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> <52267A57.4010506@uci.edu> Message-ID: <5861044329138124247@unknownmsgid> FWIW, You all may know this already, but a "long" is 64 bit on most 64 bit platforms, but 32 bit on Windows. Can we start using stdint.h and int32_t and friends? -CHB On Sep 3, 2013, at 5:18 PM, Charles R Harris wrote: On Tue, Sep 3, 2013 at 6:09 PM, Christoph Gohlke wrote: > On 9/3/2013 4:45 PM, Charles R Harris wrote: > > > > > > > > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > > wrote: > > > > On 9/3/2013 2:51 PM, Charles R Harris wrote: > > > > > > > > > > > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke cgohlke at uci.edu> > > > >> wrote: > > > > > > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > > > > Hi all, > > > > > > I'm happy to announce the first beta release of Numpy > 1.8.0. > > > Please try > > > this beta and report any issues on the numpy-dev mailing > list. > > > > > > Source tarballs and release notes can be found at > > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ > > > < > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/>. > > > The Windows > > > and OS X installers will follow when the infrastructure > issues > > > are dealt > > > with. > > > > > > Chuck > > > > > > > > > Hello, > > > > > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on > > > win-amd64-py2.7. It builds OK but there are 23 test errors and > 6 > > > failures (attached). > > > > > > Some 3rd party packages (e.g. scipy, numexpr, pytables, > bottleneck, > > > pandas and matplotlib) that were built against numpy-MKL 1.7 > fail > > > tests when used with numpy-MKL 1.8. Other packages test OK > (e.g. > > > skimage, sklearn, statsmodels, mahotas, pygame). See > > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ > > < > http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ > > > > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ > >> > > > compared to > > > < > http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ > > < > http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ > > > > > < > http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ > >>. > > > > > > > > > I have not looked in more detail or at other Python versions > yet. > > > > > > > > > Thanks Christoph, > > > > > > Looks like some work to do. I wonder what is different between > windows > > > and linux here? > > > > > > Chuck > > > > > > > Looks like the fundamental PyArray_PyIntAsIntp function is broken on > 64 > > bit Windows. 64 bit PyLong values are intermediately stored in a 32 > bit > > C long variable. But maybe I am missing something... > > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 > > > > < > https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 > > > > > > My, that does look suspicious. That function is new in 1.8 I believe. > > Looks like it needs fixing whatever else it fixes. > > > > Chuck > > > > In fact, using a npy_longlong instead of npy_long fixes all numpy test > errors and failures. But it probably foils the recent optimizations. > Great! I think the function is not used for numeric things so I'm not sure what optimizations could be affected. I'll put up a PR and backport it. Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 3 21:01:56 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Sep 2013 19:01:56 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> <52267A57.4010506@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 6:18 PM, Charles R Harris wrote: > > > > On Tue, Sep 3, 2013 at 6:09 PM, Christoph Gohlke wrote: > >> On 9/3/2013 4:45 PM, Charles R Harris wrote: >> > >> > >> > >> > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > > > wrote: >> > >> > On 9/3/2013 2:51 PM, Charles R Harris wrote: >> > > >> > > >> > > >> > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > cgohlke at uci.edu> >> > > >> wrote: >> > > >> > > On 9/1/2013 9:54 AM, Charles R Harris wrote: >> > > >> > > Hi all, >> > > >> > > I'm happy to announce the first beta release of Numpy >> 1.8.0. >> > > Please try >> > > this beta and report any issues on the numpy-dev mailing >> list. >> > > >> > > Source tarballs and release notes can be found at >> > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ >> > > < >> https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/>. >> > > The Windows >> > > and OS X installers will follow when the infrastructure >> issues >> > > are dealt >> > > with. >> > > >> > > Chuck >> > > >> > > >> > > Hello, >> > > >> > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on >> > > win-amd64-py2.7. It builds OK but there are 23 test errors >> and 6 >> > > failures (attached). >> > > >> > > Some 3rd party packages (e.g. scipy, numexpr, pytables, >> bottleneck, >> > > pandas and matplotlib) that were built against numpy-MKL 1.7 >> fail >> > > tests when used with numpy-MKL 1.8. Other packages test OK >> (e.g. >> > > skimage, sklearn, statsmodels, mahotas, pygame). See >> > > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ >> > < >> http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ >> > >> > > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ >> >> >> > > compared to >> > > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ >> > < >> http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ >> > >> > > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ >> >>. >> > > >> > > >> > > I have not looked in more detail or at other Python versions >> yet. >> > > >> > > >> > > Thanks Christoph, >> > > >> > > Looks like some work to do. I wonder what is different between >> windows >> > > and linux here? >> > > >> > > Chuck >> > > >> > >> > Looks like the fundamental PyArray_PyIntAsIntp function is broken >> on 64 >> > bit Windows. 64 bit PyLong values are intermediately stored in a 32 >> bit >> > C long variable. But maybe I am missing something... >> > < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 >> > >> > < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 >> > >> > >> > My, that does look suspicious. That function is new in 1.8 I believe. >> > Looks like it needs fixing whatever else it fixes. >> > >> > Chuck >> > >> >> In fact, using a npy_longlong instead of npy_long fixes all numpy test >> errors and failures. But it probably foils the recent optimizations. >> > > Great! I think the function is not used for numeric things so I'm not sure > what optimizations could be affected. I'll put up a PR and backport it. > > Looks like there are several errors in that function. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Tue Sep 3 21:05:43 2013 From: tim at cerazone.net (Cera, Tim) Date: Tue, 3 Sep 2013 21:05:43 -0400 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: > I am trying to take the rfft of a numpy array, like this: >>>> my_rfft = numpy.fft.rfft(my_numpy_array) > > and replace the amplitudes that can be obtained with: > >>>> my_amplitudes = numpy.abs(my_rfft) > > with amplitudes from an arbitrary numpy array's rFFT, which is to then be > converted back using numpy.fft.irfft . Alternately, some future plans will > involve having to modify individual array element amplitudes directly based > on other parameters. I would think that modifying and re-synthesizing > signals using FFT is a fairly common use-case, but my attempts at Googling > example code have been fruitless. I have FFT transform filter in my tidal analysis package. See http://sourceforge.net/apps/mediawiki/tappy/index.php?title=CompareTidalFilters for a comparison and short description. See my function below. My earlier self made some poor variable name choices. The 'low_bound' variable is actually where frequencies greater are set to zero ('factor[freq > low_bound] = 0.0'), then factor is ramped from 0 at 'low_bound' to 1 at 'high_bound'. To filter out tidal signals if your water elevations are hourly then 'low_bound' = 1/30.0 and 'high_bound' = 1/40.0. Having this gradual change in the frequency domain rather than an abrupt change makes a better filter. def fft_lowpass(nelevation, low_bound, high_bound): """ Performs a low pass filter on the nelevation series. low_bound and high_bound specifies the boundary of the filter. """ import numpy.fft as F if len(nelevation) % 2: result = F.rfft(nelevation, len(nelevation)) else: result = F.rfft(nelevation) freq = F.fftfreq(len(nelevation))[:len(nelevation)/2] factor = np.ones_like(result) factor[freq > low_bound] = 0.0 sl = np.logical_and(high_bound < freq, freq < low_bound) a = factor[sl] # Create float array of required length and reverse a = np.arange(len(a) + 2).astype(float)[::-1] # Ramp from 1 to 0 exclusive a = (a/a[0])[1:-1] # Insert ramp into factor factor[sl] = a result = result * factor print 'result=', len(result) relevation = F.irfft(result, len(nelevation)) print 'result=', len(relevation) return relevation Kindest regards, Tim From carl.canuck.official at gmail.com Tue Sep 3 22:04:13 2013 From: carl.canuck.official at gmail.com (Carl Canuck) Date: Tue, 3 Sep 2013 22:04:13 -0400 Subject: [Numpy-discussion] (no subject) In-Reply-To: References: Message-ID: Hi Tim, Brilliant! Many thanks... I think this is exactly what I need, I owe you a beer (or other beverage of your choice). I'm now going to lock myself in the basement until I can work out an implementation of this for my use-case :) /Carl On Tue, Sep 3, 2013 at 9:05 PM, Cera, Tim wrote: > > I am trying to take the rfft of a numpy array, like this: > >>>> my_rfft = numpy.fft.rfft(my_numpy_array) > > > > and replace the amplitudes that can be obtained with: > > > >>>> my_amplitudes = numpy.abs(my_rfft) > > > > with amplitudes from an arbitrary numpy array's rFFT, which is to then be > > converted back using numpy.fft.irfft . Alternately, some future plans > will > > involve having to modify individual array element amplitudes directly > based > > on other parameters. I would think that modifying and re-synthesizing > > signals using FFT is a fairly common use-case, but my attempts at > Googling > > example code have been fruitless. > > I have FFT transform filter in my tidal analysis package. See > > http://sourceforge.net/apps/mediawiki/tappy/index.php?title=CompareTidalFilters > for a comparison and short description. > > See my function below. My earlier self made some poor variable name > choices. The 'low_bound' variable is actually where frequencies > greater are set to zero ('factor[freq > low_bound] = 0.0'), then > factor is ramped from 0 at 'low_bound' to 1 at 'high_bound'. To > filter out tidal signals if your water elevations are hourly then > 'low_bound' = 1/30.0 and 'high_bound' = 1/40.0. Having this gradual > change in the frequency domain rather than an abrupt change makes a > better filter. > > def fft_lowpass(nelevation, low_bound, high_bound): > """ Performs a low pass filter on the nelevation series. > low_bound and high_bound specifies the boundary of the filter. > """ > import numpy.fft as F > if len(nelevation) % 2: > result = F.rfft(nelevation, len(nelevation)) > else: > result = F.rfft(nelevation) > freq = F.fftfreq(len(nelevation))[:len(nelevation)/2] > factor = np.ones_like(result) > factor[freq > low_bound] = 0.0 > > sl = np.logical_and(high_bound < freq, freq < low_bound) > > a = factor[sl] > # Create float array of required length and reverse > a = np.arange(len(a) + 2).astype(float)[::-1] > > # Ramp from 1 to 0 exclusive > a = (a/a[0])[1:-1] > > # Insert ramp into factor > factor[sl] = a > > result = result * factor > print 'result=', len(result) > relevation = F.irfft(result, len(nelevation)) > print 'result=', len(relevation) > return relevation > > > Kindest regards, > Tim > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Sep 4 02:20:43 2013 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Sep 2013 07:20:43 +0100 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <5861044329138124247@unknownmsgid> References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> <52267A57.4010506@uci.edu> <5861044329138124247@unknownmsgid> Message-ID: On Wed, Sep 4, 2013 at 1:51 AM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > FWIW, > > You all may know this already, but a "long" is 64 bit on most 64 bit > platforms, but 32 bit on Windows. > The correct solution in this context is to use (s)size_t, intptr_t and ptrdiff_t depending on the use case, not hardcoding the bitwidth. David > > Can we start using stdint.h and int32_t and friends? > > -CHB > > > On Sep 3, 2013, at 5:18 PM, Charles R Harris > wrote: > > > > > On Tue, Sep 3, 2013 at 6:09 PM, Christoph Gohlke wrote: > >> On 9/3/2013 4:45 PM, Charles R Harris wrote: >> > >> > >> > >> > On Tue, Sep 3, 2013 at 5:40 PM, Christoph Gohlke > > > wrote: >> > >> > On 9/3/2013 2:51 PM, Charles R Harris wrote: >> > > >> > > >> > > >> > > On Tue, Sep 3, 2013 at 3:23 PM, Christoph Gohlke > cgohlke at uci.edu> >> > > >> wrote: >> > > >> > > On 9/1/2013 9:54 AM, Charles R Harris wrote: >> > > >> > > Hi all, >> > > >> > > I'm happy to announce the first beta release of Numpy >> 1.8.0. >> > > Please try >> > > this beta and report any issues on the numpy-dev mailing >> list. >> > > >> > > Source tarballs and release notes can be found at >> > >https://sourceforge.net/__projects/numpy/files/NumPy/1.__8.0b1/ >> > > < >> https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/>. >> > > The Windows >> > > and OS X installers will follow when the infrastructure >> issues >> > > are dealt >> > > with. >> > > >> > > Chuck >> > > >> > > >> > > Hello, >> > > >> > > I tried numpy-1.8.0.dev-86a6e6c with msvc9 and MKL 11.1 on >> > > win-amd64-py2.7. It builds OK but there are 23 test errors >> and 6 >> > > failures (attached). >> > > >> > > Some 3rd party packages (e.g. scipy, numexpr, pytables, >> bottleneck, >> > > pandas and matplotlib) that were built against numpy-MKL 1.7 >> fail >> > > tests when used with numpy-MKL 1.8. Other packages test OK >> (e.g. >> > > skimage, sklearn, statsmodels, mahotas, pygame). See >> > > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ >> > < >> http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7-__numpy-1.8.0.dev-86a6e6c/ >> > >> > > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7-numpy-1.8.0.dev-86a6e6c/ >> >> >> > > compared to >> > > < >> http://www.lfd.uci.edu/~__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ >> > < >> http://www.lfd.uci.edu/%7E__gohlke/pythonlibs/tests/__20130902-win-amd64-py2.7/ >> > >> > > < >> http://www.lfd.uci.edu/%7Egohlke/pythonlibs/tests/20130902-win-amd64-py2.7/ >> >>. >> > > >> > > >> > > I have not looked in more detail or at other Python versions >> yet. >> > > >> > > >> > > Thanks Christoph, >> > > >> > > Looks like some work to do. I wonder what is different between >> windows >> > > and linux here? >> > > >> > > Chuck >> > > >> > >> > Looks like the fundamental PyArray_PyIntAsIntp function is broken >> on 64 >> > bit Windows. 64 bit PyLong values are intermediately stored in a 32 >> bit >> > C long variable. But maybe I am missing something... >> > < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L729 >> > >> > < >> https://github.com/numpy/numpy/blob/maintenance/1.8.x/numpy/core/src/multiarray/conversion_utils.c#L767 >> > >> > >> > My, that does look suspicious. That function is new in 1.8 I believe. >> > Looks like it needs fixing whatever else it fixes. >> > >> > Chuck >> > >> >> In fact, using a npy_longlong instead of npy_long fixes all numpy test >> errors and failures. But it probably foils the recent optimizations. >> > > Great! I think the function is not used for numeric things so I'm not sure > what optimizations could be affected. I'll put up a PR and backport it. > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregorio.bastardo at gmail.com Wed Sep 4 03:48:44 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Wed, 4 Sep 2013 09:48:44 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: @St?fan: the 'np.all' calls are now unnecessary on line 26 @St?fan, Robert: Is it worth to bring this solution into numpy? I mean it's probably not a rare problem, and now users have to bring this snippet into their codebase. Gregorio 2013/9/3 St?fan van der Walt : > On Tue, Sep 3, 2013 at 2:47 PM, Robert Kern wrote: >>> Here's one way of doing it: https://gist.github.com/stefanv/6413742 >> >> You can probably reduce the amount of work by only comparing a.min() and >> a.max() instead of the whole array. > > Thanks, fixed. > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From grb at skogoglandskap.no Wed Sep 4 06:05:32 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Wed, 4 Sep 2013 10:05:32 +0000 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy Message-ID: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> In my current GIS raster work I often have a situation where I generate code something like this: np.any([A>4, A==2, B==5, ...]) However, np.any() is quite slow. It's possible to use np.logical_or to solve the problem, but then you get nested logical_or's, since logical_or combines only two parameters. It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. The question is: which is best (syntactically, in terms of performance, etc)? I've written a little helper function to provide a faster version of any() and all(). It's embarrassingly simple - just a for loop. However, I think there's a syntactic advantage to using a helper function for this situation rather than writing it idiomatically each time; and it reduces the chance of a bug in idiomatic implementation. However, the code does not cover all the use cases currently addressed by np.any() and np.all(). I benchmarked to pick the fastest underlying implementation (logical_or rather than integer maths). The result is 14 to 17x faster than np.any() for this use case.* Code & benchmark here: https://github.com/gbb/numpy-fast-any-all Please feel welcome to use it or improve it :-) Graeme. * (Should this become an execution path in np.any()... ?) From robert.kern at gmail.com Wed Sep 4 06:21:09 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Sep 2013 11:21:09 +0100 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> Message-ID: On Wed, Sep 4, 2013 at 11:05 AM, Graeme B. Bell wrote: > > In my current GIS raster work I often have a situation where I generate code something like this: > > np.any([A>4, A==2, B==5, ...]) > > However, np.any() is quite slow. > > It's possible to use np.logical_or to solve the problem, but then you get nested logical_or's, since logical_or combines only two parameters. > It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. > > The question is: which is best (syntactically, in terms of performance, etc)? > > I've written a little helper function to provide a faster version of any() and all(). It's embarrassingly simple - just a for loop. However, I think there's a syntactic advantage to using a helper function for this situation rather than writing it idiomatically each time; and it reduces the chance of a bug in idiomatic implementation. However, the code does not cover all the use cases currently addressed by np.any() and np.all(). > > I benchmarked to pick the fastest underlying implementation (logical_or rather than integer maths). > > The result is 14 to 17x faster than np.any() for this use case.* > > Code & benchmark here: > > https://github.com/gbb/numpy-fast-any-all > > Please feel welcome to use it or improve it :-) Try the following: any(map(np.any, inputs)) all(map(np.all, inputs)) -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Sep 4 08:00:10 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 04 Sep 2013 08:00:10 -0400 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release References: Message-ID: Failed building on fedora 19 x86_64 using atlas: creating build/temp.linux-x86_64-2.7/numpy/linalg creating build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite compile options: '-DATLAS_INFO="\"3.8.4\"" -I/usr/include -Inumpy/core/include - Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private - Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath - Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort - Inumpy/core/include -I/usr/include/python2.7 -c' gcc: numpy/linalg/lapack_litemodule.c gcc: numpy/linalg/lapack_lite/python_xerbla.c /usr/bin/gfortran -Wall -Wl,-rpath=/opt/intel/mkl/lib/intel64 build/temp.linux- x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux- x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/usr/lib64/atlas - L/usr/lib64 -Lbuild/temp.linux-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas -lpython2.7 -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: error: ld returned 1 exit status /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: error: ld returned 1 exit status Build command was: env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- rpath=/opt/intel/mkl/lib/intel64" python setup.py build attached site.cfg -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: site.cfg URL: From ndbecker2 at gmail.com Wed Sep 4 08:10:26 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 04 Sep 2013 08:10:26 -0400 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release References: Message-ID: Built on fedora linux 19 x86_64 using mkl: build OK using: env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- rpath=/opt/intel/mkl/lib/intel64" python setup.py build and attached site.cfg: ====================================================================== FAIL: test_linalg.test_xerbla ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla assert_(False) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError ---------------------------------------------------------------------- Ran 5271 tests in 57.567s FAILED (KNOWNFAIL=5, SKIP=13, failures=1) -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: site.cfg URL: From grb at skogoglandskap.no Wed Sep 4 08:14:51 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Wed, 4 Sep 2013 12:14:51 +0000 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> Message-ID: <3204E4D5-1920-4DC6-9D5F-1FB3D7CA1415@skogoglandskap.no> Sorry, I should have been more clear. As shown in the benchmark/example, the method is replacing the behaviour of np.any(inputs, 0) not the behaviour of np.any(inputs) Here, where I'm making decisions based on overlaying layers of raster data in the same shape, I don't want to map the entire dataset to a single boolean, rather I want to preserve the layers' shape but identify if a condition was matched in any of the overlaid layers, generating a mask. For example, this type of reasoning: def mask(): for all pixel locations in the images, A, B and C: if A[location] is 3, 19, or between 21 and 30 AND B[location] is any value AND C[location] is 1-4, 9-13... pixel=True This naturally fits the any/all metaphor. Will update the description on github. Graeme. On Sep 4, 2013, at 12:05 PM, Graeme Bell wrote: > In my current GIS raster work I often have a situation where I generate code something like this: > > np.any([A>4, A==2, B==5, ...]) > > However, np.any() is quite slow. > > It's possible to use np.logical_or to solve the problem, but then you get nested logical_or's, since logical_or combines only two parameters. > It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. > > The question is: which is best (syntactically, in terms of performance, etc)? > > I've written a little helper function to provide a faster version of any() and all(). It's embarrassingly simple - just a for loop. However, I think there's a syntactic advantage to using a helper function for this situation rather than writing it idiomatically each time; and it reduces the chance of a bug in idiomatic implementation. However, the code does not cover all the use cases currently addressed by np.any() and np.all(). > > I benchmarked to pick the fastest underlying implementation (logical_or rather than integer maths). > > The result is 14 to 17x faster than np.any() for this use case.* > > Code & benchmark here: > > https://github.com/gbb/numpy-fast-any-all > > Please feel welcome to use it or improve it :-) > > Graeme. > > > * (Should this become an execution path in np.any()... ?) From cournape at gmail.com Wed Sep 4 08:33:54 2013 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Sep 2013 13:33:54 +0100 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release In-Reply-To: References: Message-ID: On Wed, Sep 4, 2013 at 1:00 PM, Neal Becker wrote: > Failed building on fedora 19 x86_64 using atlas: > > creating build/temp.linux-x86_64-2.7/numpy/linalg > creating build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite > compile options: '-DATLAS_INFO="\"3.8.4\"" -I/usr/include > -Inumpy/core/include - > Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private - > Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath - > Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort > - > Inumpy/core/include -I/usr/include/python2.7 -c' > gcc: numpy/linalg/lapack_litemodule.c > gcc: numpy/linalg/lapack_lite/python_xerbla.c > /usr/bin/gfortran -Wall -Wl,-rpath=/opt/intel/mkl/lib/intel64 > build/temp.linux- > x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux- > x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/usr/lib64/atlas - > L/usr/lib64 -Lbuild/temp.linux-x86_64-2.7 -llapack -lptf77blas -lptcblas > -latlas > -lpython2.7 -lgfortran -o > build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so > /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In > function > `_start': > (.text+0x20): undefined reference to `main' > collect2: error: ld returned 1 exit status > /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In > function > `_start': > (.text+0x20): undefined reference to `main' > collect2: error: ld returned 1 exit status > > Build command was: > env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 > CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- > rpath=/opt/intel/mkl/lib/intel64" python setup.py build > This command never worked: you need to add the -shared flag to LDFLAGS (and you may want to remove rpath to MKL if you use ATLAS). David > > attached site.cfg > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Sep 4 08:59:51 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 04 Sep 2013 08:59:51 -0400 Subject: [Numpy-discussion] ANN: Scipy 0.13.0 beta 1 release References: Message-ID: David Cournapeau wrote: > On Wed, Sep 4, 2013 at 1:00 PM, Neal Becker wrote: > >> Failed building on fedora 19 x86_64 using atlas: >> >> creating build/temp.linux-x86_64-2.7/numpy/linalg >> creating build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite >> compile options: '-DATLAS_INFO="\"3.8.4\"" -I/usr/include >> -Inumpy/core/include - >> Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >> -Inumpy/core/src/private - >> Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath - >> Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort >> - >> Inumpy/core/include -I/usr/include/python2.7 -c' >> gcc: numpy/linalg/lapack_litemodule.c >> gcc: numpy/linalg/lapack_lite/python_xerbla.c >> /usr/bin/gfortran -Wall -Wl,-rpath=/opt/intel/mkl/lib/intel64 >> build/temp.linux- >> x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux- >> x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/usr/lib64/atlas - >> L/usr/lib64 -Lbuild/temp.linux-x86_64-2.7 -llapack -lptf77blas -lptcblas >> -latlas >> -lpython2.7 -lgfortran -o >> build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so >> /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In >> function >> `_start': >> (.text+0x20): undefined reference to `main' >> collect2: error: ld returned 1 exit status >> /usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../lib64/crt1.o: In >> function >> `_start': >> (.text+0x20): undefined reference to `main' >> collect2: error: ld returned 1 exit status >> >> Build command was: >> env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 >> CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- >> rpath=/opt/intel/mkl/lib/intel64" python setup.py build >> > > This command never worked: you need to add the -shared flag to LDFLAGS (and > you may want to remove rpath to MKL if you use ATLAS). > > David > OK, building with -shared (and removing rpath) works. numpy.test('full') reports no unexpected failures. From joseluismietta at yahoo.com.ar Wed Sep 4 10:17:30 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Wed, 4 Sep 2013 07:17:30 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> Message-ID: <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> Hi experts! If I do: G =Graph(M) That is: to use the associated intersection graph, where the vertices are the sticks and there is an edge between the two vertices if they intersect. Two sticks are "connected by a 'intersected-stick' path" if they are in the same connected component of this graph. It turns out that the matrix I consider (M) is the adjacency matrix of this graph. Then, I can do: first_stick inG.connected_component_containing_vertex(second_stick) This is 'True' if 'first_stick' is in the sub-graph formed by 'second_stick' and all sticks 'connected' with 'second_stick'. The problem is that G.connected_component_containing_vertex() explore all the sub-graph. How can I do (what is the code) for stop the iteration when the algorithm found 'first-stick'? Waiting for your answers. Thanks a lot!! ________________________________ De: Robert Kern Para: Discussion of Numerical Python Enviado: lunes, 2 de septiembre de 2013 10:40 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On Sun, Sep 1, 2013 at 11:55 PM, Jos? Luis Mietta wrote: > > Hi experts! > > I wanna study the intersection between line segments (sticks). > I wrote a algorithm that generate a matrix, M, with N rows and N columns. The M-element Mij is 1 if stick number 'i' intersect stick number 'j' (obviously M is symmetric). > > Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. You may find it helpful to reword your problem using standard terminology. If I hadn't read your previous question, I would not have been able to understand what you meant by intersected sticks (or, as Chris thought at first, that you needed help determining the intersections). This will also help in searching Google for background and pre-existing software to solve your problem. You have an "undirected graph" (the sticks are "nodes", and the intersections are "edges") and you want to find if two given nodes are "reachable" from each other. You are currently representing your graph as an "adjacency matrix" `M` where `M[i,j]` is 1 iff nodes `i` and `j` have an edge between them and 0 otherwise. The "transitive closure" of your graph `M` is the graph that connects two nodes with an edge iff the two nodes are reachable from each other in the original graph `M`. There are several graph theory packages out there, like NetworkX, that will do this for you. Depending on the kinds of queries you would like to do, as David points out, want to compute the "connected components" of your graph; a connected component is a subgraph of your original graph such that all of the nodes are reachable from each other. You can also look up Python code for computing the transitive closure of a graph; it's not a complicated algorithm. However, this algorithm is usually implemented using a different representation of a graph than an adjacency matrix, so you may need to do a conversion. -- Robert Kern _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From pelson.pub at gmail.com Wed Sep 4 11:00:40 2013 From: pelson.pub at gmail.com (Phil Elson) Date: Wed, 4 Sep 2013 16:00:40 +0100 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <3204E4D5-1920-4DC6-9D5F-1FB3D7CA1415@skogoglandskap.no> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> <3204E4D5-1920-4DC6-9D5F-1FB3D7CA1415@skogoglandskap.no> Message-ID: For the record, I started a discussion about 6 months ago about a "find_first" type function which avoided running the logic over the whole array (using lambdas instead). This spilled into a discussion about implementing a short-cutted "any" or "all" function: http://numpy-discussion.10968.n7.nabble.com/Implementing-a-find-first-style-function-tp33085.htmlwith some interesting results. Nothing more has been done with those discussions, but you may find it of interest. (And I'd still be interested in taking it forwards if you have any comments) Cheers, On 4 September 2013 13:14, Graeme B. Bell wrote: > Sorry, I should have been more clear. > > As shown in the benchmark/example, the method is replacing the behaviour of > > np.any(inputs, 0) > > not the behaviour of > > np.any(inputs) > > Here, where I'm making decisions based on overlaying layers of raster data > in the same shape, I don't want to map the entire dataset to a single > boolean, rather I want to preserve the layers' shape but identify if a > condition was matched in any of the overlaid layers, generating a mask. > > For example, this type of reasoning: > > def mask(): > for all pixel locations in the images, A, B and C: > if A[location] is 3, 19, or between 21 and 30 AND B[location] is any > value AND C[location] is 1-4, 9-13... > pixel=True > > This naturally fits the any/all metaphor. > > Will update the description on github. > > Graeme. > > On Sep 4, 2013, at 12:05 PM, Graeme Bell wrote: > > > In my current GIS raster work I often have a situation where I generate > code something like this: > > > > np.any([A>4, A==2, B==5, ...]) > > > > However, np.any() is quite slow. > > > > It's possible to use np.logical_or to solve the problem, but then you > get nested logical_or's, since logical_or combines only two parameters. > > It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. > > > > The question is: which is best (syntactically, in terms of performance, > etc)? > > > > I've written a little helper function to provide a faster version of > any() and all(). It's embarrassingly simple - just a for loop. However, I > think there's a syntactic advantage to using a helper function for this > situation rather than writing it idiomatically each time; and it reduces > the chance of a bug in idiomatic implementation. However, the code does not > cover all the use cases currently addressed by np.any() and np.all(). > > > > I benchmarked to pick the fastest underlying implementation (logical_or > rather than integer maths). > > > > The result is 14 to 17x faster than np.any() for this use case.* > > > > Code & benchmark here: > > > > https://github.com/gbb/numpy-fast-any-all > > > > Please feel welcome to use it or improve it :-) > > > > Graeme. > > > > > > * (Should this become an execution path in np.any()... ?) > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 4 11:18:05 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Sep 2013 09:18:05 -0600 Subject: [Numpy-discussion] Warning: 1.8.0b2 this weekend. Message-ID: Hi All, I'd like to make a 1.8.0b2 this weekend to take care of the error cascade on 64 bit windows. If you have any fixes/bugs you would like to see in that beta now is the time to get the PRs in. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 4 12:28:15 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Sep 2013 17:28:15 +0100 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <3204E4D5-1920-4DC6-9D5F-1FB3D7CA1415@skogoglandskap.no> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> <3204E4D5-1920-4DC6-9D5F-1FB3D7CA1415@skogoglandskap.no> Message-ID: On Wed, Sep 4, 2013 at 1:14 PM, Graeme B. Bell wrote: > > Sorry, I should have been more clear. > > As shown in the benchmark/example, the method is replacing the behaviour of > > np.any(inputs, 0) > > not the behaviour of > > np.any(inputs) reduce(np.logical_or, inputs, False) reduce(np.logical_and, inputs, True) Your fast_logic() is basically reduce(). -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.collette at gmail.com Wed Sep 4 12:48:22 2013 From: andrew.collette at gmail.com (Andrew Collette) Date: Wed, 4 Sep 2013 10:48:22 -0600 Subject: [Numpy-discussion] ANN: HDF5 for Python (h5py) 2.2.0 Message-ID: Announcing HDF5 for Python (h5py) 2.2.0 ======================================= We are proud to announce that HDF5 for Python 2.2.0 is now available. Thanks to everyone who helped put together this release! The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. H5py uses straightforward NumPy and Python metaphors, like dictionary and NumPy array syntax. For example, you can iterate over datasets in a file, or check out the .shape or .dtype attributes of datasets. You don't need to know anything special about HDF5 to get started. Documentation and download links are available at: http://www.h5py.org Parallel HDF5 ============= This version of h5py introduces support for MPI/Parallel HDF5, using the mpi4py package. Parallel HDF5 is the "native" method in HDF5 for sharing files and objects across multiple processes, in contrast to the "threading" package or "multiprocessing". There is a guide to using Parallel HDF5 at the h5py web site: http://www.h5py.org/docs/topics/mpi.html Other new features ================== * Support for Python 3.3 * Support for 16-bit "mini" floats * Access to the HDF5 scale-offset filter * Field names are now allowed when writing to a dataset * Region references now preserve the shape of their selections * File-resident "committed" types can be linked to datasets and attributes * Make object mtime tracking optional * A new "move" method on Group objects * Many new options for Group.copy * Access to HDF5's get_vfd_handle * Many bug fixes Acknowledgments =============== Special thanks to: * Thomas A Caswell * Konrad Hinsen * Darren Dale * Matt Zwier * Toon Verstraelen * Noel Dawe * Barry Wardel * Bradley M. Froehle * Dan Meliza * Johannes Meixner * John Benediktsson * Matthew Turk * syhw And everyone else who posted a bug report, contributed on the mailing list, or otherwise helped with this release. From jtaylor.debian at googlemail.com Wed Sep 4 13:38:58 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 04 Sep 2013 19:38:58 +0200 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> Message-ID: <52277032.40304@googlemail.com> On 04.09.2013 12:05, Graeme B. Bell wrote: > In my current GIS raster work I often have a situation where I generate code something like this: > > np.any([A>4, A==2, B==5, ...]) > > However, np.any() is quite slow. > > It's possible to use np.logical_or to solve the problem, but then you get nested logical_or's, since logical_or combines only two parameters. > It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. > > The question is: which is best (syntactically, in terms of performance, etc)? > > I've written a little helper function to provide a faster version of any() and all(). It's embarrassingly simple - just a for loop. However, I think there's a syntactic advantage to using a helper function for this situation rather than writing it idiomatically each time; and it reduces the chance of a bug in idiomatic implementation. However, the code does not cover all the use cases currently addressed by np.any() and np.all(). > > I benchmarked to pick the fastest underlying implementation (logical_or rather than integer maths). > > The result is 14 to 17x faster than np.any() for this use case.* any/all and boolean operations have been significantly speed up by vectorization in numpy 1.8 [0]. They are now around 10 times faster than before, especially if the boolean array fits into one of the cpu caching layers. If they don't I recommend using a blocking utility function, something like: for i in range(0, n, blocksize): view = d[i:i+blocksize] #dostuff on view with this method and the new vectorizations in numpy you are almost as fast as numexpr for floats and probably a lot faster with bools. [0] http://www.onerussian.com/tmp/numpy-vbench/vb_vb_ufunc.html#numpy-and-bool http://www.onerussian.com/tmp/numpy-vbench/vb_vb_reduce.html#numpy-any-slow (the dip before 1.7 was part of the NA branch and never released, 1.8 adds some of its optimizations back) From nouiz at nouiz.org Wed Sep 4 13:49:49 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Wed, 4 Sep 2013 13:49:49 -0400 Subject: [Numpy-discussion] Minimal NumPy for distribution Message-ID: Hi, I have done some exploratory work with Theano to generate a shared library from a Theano function. This link with numpy c api. If we want to distribute this library and call it from C and/or python, what is the minimal installed part of NumPy needed? I suppose that only the c api is needed. Do someone already checked that? >From what I found on linux, numpy linked with a dynamic BLAS take 13MB. There is 2.7MB from .so, 5.3MB from .py and 3.8MB from .pyc. Can we just keep all .so and remove all .py and .pyc file? I suppose we need to modify the __init__.py file too. On Windows, with the unofficial 64 bit binary of NumPy linked with MKL, the numpy directory take 56MB. So I would also like to know witch shared library(dll) is needed. Here is the .so file found on linux: /lib/python2.7/site-packages/numpy/core/multiarray_tests.so /lib/python2.7/site-packages/numpy/core/multiarray.so /lib/python2.7/site-packages/numpy/core/_dotblas.so /lib/python2.7/site-packages/numpy/core/_sort.so /lib/python2.7/site-packages/numpy/core/scalarmath.so /lib/python2.7/site-packages/numpy/core/umath_tests.so /lib/python2.7/site-packages/numpy/core/umath.so /lib/python2.7/site-packages/numpy/numarray/_capi.so /lib/python2.7/site-packages/numpy/linalg/lapack_lite.so /lib/python2.7/site-packages/numpy/random/mtrand.so /lib/python2.7/site-packages/numpy/fft/fftpack_lite.so /lib/python2.7/site-packages/numpy/lib/_compiled_base.so Can I get rid of all shared lib outside of core? thanks Fr?d?ric -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 4 14:14:36 2013 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Sep 2013 19:14:36 +0100 Subject: [Numpy-discussion] Minimal NumPy for distribution In-Reply-To: References: Message-ID: There do exist numpy c functions that call .py file routines. I don't know how likely you are to find them in practice, but it definitely happens. You don't need .py files if you have .pyc files, and those can be compressed (python can import directly from .zip files). -n On 4 Sep 2013 18:52, "Fr?d?ric Bastien" wrote: > Hi, > > I have done some exploratory work with Theano to generate a shared library > from a Theano function. This link with numpy c api. If we want to > distribute this library and call it from C and/or python, what is the > minimal installed part of NumPy needed? I suppose that only the c api is > needed. Do someone already checked that? > > From what I found on linux, numpy linked with a dynamic BLAS take 13MB. > There is 2.7MB from .so, 5.3MB from .py and 3.8MB from .pyc. Can we just > keep all .so and remove all .py and .pyc file? I suppose we need to modify > the __init__.py file too. > > On Windows, with the unofficial 64 bit binary of NumPy linked with MKL, > the numpy directory take 56MB. So I would also like to know witch shared > library(dll) is needed. Here is the .so file found on linux: > > /lib/python2.7/site-packages/numpy/core/multiarray_tests.so > /lib/python2.7/site-packages/numpy/core/multiarray.so > /lib/python2.7/site-packages/numpy/core/_dotblas.so > /lib/python2.7/site-packages/numpy/core/_sort.so > /lib/python2.7/site-packages/numpy/core/scalarmath.so > /lib/python2.7/site-packages/numpy/core/umath_tests.so > /lib/python2.7/site-packages/numpy/core/umath.so > /lib/python2.7/site-packages/numpy/numarray/_capi.so > /lib/python2.7/site-packages/numpy/linalg/lapack_lite.so > /lib/python2.7/site-packages/numpy/random/mtrand.so > /lib/python2.7/site-packages/numpy/fft/fftpack_lite.so > /lib/python2.7/site-packages/numpy/lib/_compiled_base.so > > Can I get rid of all shared lib outside of core? > > thanks > > Fr?d?ric > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 4 15:17:22 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Sep 2013 13:17:22 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <52267DA0.3050305@uci.edu> References: <5226536F.9030509@uci.edu> <52267374.7060602@uci.edu> <52267DA0.3050305@uci.edu> Message-ID: On Tue, Sep 3, 2013 at 6:24 PM, Christoph Gohlke wrote: > > The 32 bit build fails two tests (unrelated to the above 64 bit issue): > > ====================================================================== > FAIL: test_invalid (test_errstate.TestErrstate) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python27\lib\site-packages\numpy\testing\decorators.py", > line 146, in skipper_func > return f(*args, **kwargs) > File > "X:\Python27\lib\site-packages\numpy\core\tests\test_errstate.py", line > 23, in test_invalid > self.fail("Did not raise an invalid error") > AssertionError: Did not raise an invalid error > > ====================================================================== > FAIL: simd tests on max/min > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python27\lib\site-packages\numpy\core\tests\test_umath.py", > line 678, in test_minmax_blocked > msg=repr(inp) + '\n' + msg) > AssertionError: array([ 0., 1., nan, 3., 4., 5., 6., 7., > 8., 9., 10.], dtype=float32) > unary offset=(0, 0), size=11, dtype=, out of place > > ---------------------------------------------------------------------- > > I opened issues for both of these. Christoph, Julian is looking at the second , could you help him debug the issue? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 4 17:49:34 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 Sep 2013 22:49:34 +0100 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> Message-ID: On Wed, Sep 4, 2013 at 3:17 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > > Hi experts! > > If I do: > > G = Graph(M) > > That is: to use the associated intersection graph, where the vertices are the sticks and there is an edge between the two vertices if they intersect. Two sticks are "connected by a 'intersected-stick' path" if they are in the same connected component of this graph. > It turns out that the matrix I consider (M) is the adjacency matrix of this graph. > > Then, I can do: > > first_stick in G.connected_component_containing_vertex(second_stick) > > This is 'True' if 'first_stick' is in the sub-graph formed by 'second_stick' and all sticks 'connected' with 'second_stick'. > > The problem is that > > G.connected_component_containing_vertex() > > explore all the sub-graph. > > How can I do (what is the code) for stop the iteration when the algorithm found 'first-stick'? networkx.has_path(G, first_stick, second_stick) http://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.shortest_paths.generic.has_path.html#networkx.algorithms.shortest_paths.generic.has_path If you are going to be doing many queries, you should compute all of the path lengths, then you can query the final data structure easily. lengths = networkx.all_pairs_shortest_path_length(G) second_stick in lengths[first_stick] -- Robert Kern On Wed, Sep 4, 2013 at 3:17 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > Hi experts! > > If I do: > > G = Graph(M) > > That is: to use the associated intersection graph, > where the vertices are the sticks and there is an edge between the two > vertices if they intersect. Two sticks are "connected by a > 'intersected-stick' path" if they are in the same connected component of > this graph. > It turns out that the matrix I consider (M) is the adjacency matrixof this graph. > > Then, I can do: > > first_stick in G.connected_component_containing_vertex(second_stick) > > This is 'True' if 'first_stick' is in the sub-graph formed by > 'second_stick' and all sticks 'connected' with 'second_stick'. > > The problem is that > > G.connected_component_containing_vertex() > > explore all the sub-graph. > > How can I do (what is the code) for stop the iteration when the algorithm > found 'first-stick'? > > Waiting for your answers. > > Thanks a lot!! > > > ------------------------------ > *De:* Robert Kern > > *Para:* Discussion of Numerical Python > *Enviado:* lunes, 2 de septiembre de 2013 10:40 > > *Asunto:* Re: [Numpy-discussion] Stick intersection path algorithm > > On Sun, Sep 1, 2013 at 11:55 PM, Jos? Luis Mietta < > joseluismietta at yahoo.com.ar> wrote: > > > > Hi experts! > > > > I wanna study the intersection between line segments (sticks). > > I wrote a algorithm that generate a matrix, M, with N rows and N > columns. The M-element Mij is 1 if stick number 'i' intersect stick number > 'j' (obviously M is symmetric). > > > > Given two arbitrary sticks, i need a simple and effective algorithm that > determinate if that two sticks are conected by a 'intersected-sticks' path. > > You may find it helpful to reword your problem using standard terminology. > If I hadn't read your previous question, I would not have been able to > understand what you meant by intersected sticks (or, as Chris thought at > first, that you needed help determining the intersections). This will also > help in searching Google for background and pre-existing software to solve > your problem. > > You have an "undirected graph" (the sticks are "nodes", and the > intersections are "edges") and you want to find if two given nodes are > "reachable" from each other. You are currently representing your graph as > an "adjacency matrix" `M` where `M[i,j]` is 1 iff nodes `i` and `j` have an > edge between them and 0 otherwise. The "transitive closure" of your graph > `M` is the graph that connects two nodes with an edge iff the two nodes are > reachable from each other in the original graph `M`. There are several > graph theory packages out there, like NetworkX, that will do this for you. > Depending on the kinds of queries you would like to do, as David points > out, want to compute the "connected components" of your graph; a connected > component is a subgraph of your original graph such that all of the nodes > are reachable from each other. > > You can also look up Python code for computing the transitive closure of a > graph; it's not a complicated algorithm. However, this algorithm is usually > implemented using a different representation of a graph than an adjacency > matrix, so you may need to do a conversion. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From orion at cora.nwra.com Wed Sep 4 19:05:45 2013 From: orion at cora.nwra.com (Orion Poplawski) Date: Wed, 04 Sep 2013 17:05:45 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: Message-ID: <5227BCC9.9030202@cora.nwra.com> On 09/01/2013 10:54 AM, Charles R Harris wrote: > Hi all, > > I'm happy to announce the first beta release of Numpy 1.8.0. Please try this > beta and report any issues on the numpy-dev mailing list. > > Source tarballs and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows and > OS X installers will follow when the infrastructure issues are dealt with. > > Chuck > The Fedora packages appear to build fine on F19, F20, and Rawhide. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion at nwra.com Boulder, CO 80301 http://www.nwra.com From charlesr.harris at gmail.com Wed Sep 4 21:37:51 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Sep 2013 19:37:51 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <5227BCC9.9030202@cora.nwra.com> References: <5227BCC9.9030202@cora.nwra.com> Message-ID: On Wed, Sep 4, 2013 at 5:05 PM, Orion Poplawski wrote: > On 09/01/2013 10:54 AM, Charles R Harris wrote: > > Hi all, > > > > I'm happy to announce the first beta release of Numpy 1.8.0. Please try > this > > beta and report any issues on the numpy-dev mailing list. > > > > Source tarballs and release notes can be found at > > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The > Windows and > > OS X installers will follow when the infrastructure issues are dealt > with. > > > > Chuck > > > > The Fedora packages appear to build fine on F19, F20, and Rawhide. > > > Thanks for the report. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Thu Sep 5 01:08:32 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 04 Sep 2013 22:08:32 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: References: Message-ID: <522811D0.8030003@uci.edu> On 9/1/2013 9:54 AM, Charles R Harris wrote: > Hi all, > > I'm happy to announce the first beta release of Numpy 1.8.0. Please try > this beta and report any issues on the numpy-dev mailing list. > > Source tarballs and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows > and OS X installers will follow when the infrastructure issues are dealt > with. > > Chuck > Hello, is this IndexError intentional in numpy 1.8? Matplotlib 1.3 fails some tests because of this. >>> numpy.zeros(1)[[0], :] Traceback (most recent call last): File "", line 1, in IndexError: too many indices With numpy 1.7: >>> numpy.zeros(1)[[0], :] array([ 0.]) Christoph From sebastian at sipsolutions.net Thu Sep 5 04:18:06 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 05 Sep 2013 10:18:06 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <522811D0.8030003@uci.edu> References: <522811D0.8030003@uci.edu> Message-ID: <1378369086.8377.1.camel@sebastian-laptop> On Wed, 2013-09-04 at 22:08 -0700, Christoph Gohlke wrote: > On 9/1/2013 9:54 AM, Charles R Harris wrote: > > Hello, > > is this IndexError intentional in numpy 1.8? Matplotlib 1.3 fails some > tests because of this. > > >>> numpy.zeros(1)[[0], :] > Traceback (most recent call last): > File "", line 1, in > IndexError: too many indices > > With numpy 1.7: > >>> numpy.zeros(1)[[0], :] > array([ 0.]) > Yes certainly is intentional, since you actually do have too many indices (you have one dimension, you can only have one index). That it worked before was a bug. If this is a real problem, maybe we have to temporarily allow it? - Sebastian > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From grb at skogoglandskap.no Thu Sep 5 04:47:28 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Thu, 5 Sep 2013 08:47:28 +0000 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy Message-ID: <258E27F0-2EF4-4F11-9EA8-59DBCF29AA31@skogoglandskap.no> Hi Robert, Thanks for proposing an alternative implementation approach. However, did you test your proposal before you made the assertion about its behaviour? >reduce(np.logical_or, inputs, False) >reduce(np.logical_and, inputs, True) This code consistently benchmarks 20% slower than the method I use (tested on two different machines several times). >Your fast_logic() is basically reduce(). No, it isn't. Updated benchmarks for your proposal and also for another alternative implemenation using boolean indexing at: https://github.com/gbb/numpy-fast-any-all/blob/master/BENCHMARK.md Three general points arising from this: 1 - idioms don't have test coverage Generally, by using idioms rather than functions, you risk mistyping or misusing the form of the idiom and thus introducing a bug. You also lose out on explicit testing and implicit 'real world testing' that tends to build up around library functions. 2 - idioms aren't maintained or updated (and they have a unknown shelf life) An idiom might be fast today (or not), it may be correct today, but tomorrow is unknown. A key problem is that the relative performance of the parts of a library like numpy will keep changing - sometimes substantially - and idiomatic approaches to overcome performance difficulties in the short term tend to become outdated and even harmful very quickly. As in this example, they can even be harmful from the moment they're written. Browsing a site like stackoverflow should show you both new and experienced users often taking inefficient approaches because of outdated idiomatic advice. 3 - idioms are OK, but functions are better, because implementation hiding and abstraction are good things. If you use a benchmarked/tested function which acknowledges a range of alternative implementations, you have a reasonable degree of confidence that you're getting the best performance and correct behaviour, because you can actually see the effects of the alternative implementations in benchmarks/test output. It's a lot more sensible to use a function from a publicly available library - any library - than to manually maintain a set of idioms and have to continually search your software for the idioms, benchmark them to see if they're still beneficial, and modify them when they're not. Graeme From grb at skogoglandskap.no Thu Sep 5 04:54:32 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Thu, 5 Sep 2013 08:54:32 +0000 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <52277032.40304@googlemail.com> References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> <52277032.40304@googlemail.com> Message-ID: Hi Julian, Thanks for the post. It's great to hear that the main numpy function is improving in 1.8, though I think there is still plenty of value here for performance junkies :-) I don't have 1.8beta installed (and I can't conveniently install it on my machines just now). If you have time, and have the beta installed, could you try this and mail me the output from the benchmark? I'm curious to know. # git clone https://github.com/gbb/numpy-fast-any-all.git # cd numpy-fast-any-all # python test-fast-any-all.py Graeme On Sep 4, 2013, at 7:38 PM, Julian Taylor wrote: >> >> The result is 14 to 17x faster than np.any() for this use case.* > > any/all and boolean operations have been significantly speed up by > vectorization in numpy 1.8 [0]. > They are now around 10 times faster than before, especially if the > boolean array fits into one of the cpu caching layers. From gregorio.bastardo at gmail.com Thu Sep 5 05:16:51 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Thu, 5 Sep 2013 11:16:51 +0200 Subject: [Numpy-discussion] find appropriate dtype based on a set of values In-Reply-To: References: Message-ID: Hi St?fan, I ran into a problem: >>> min_typecode( (18446744073709551615L,) ) # ok >>> min_typecode( (0, 18446744073709551615L,) ) # ? Traceback (most recent call last): ... ValueError: Can only handle integer arrays. It seems that np.asarray converts the input sequence into a float64 array in the second case (same behaviour with np.array). Anyone knows the reason behind? python 2.7.4 win32 numpy 1.7.1 Gregorio 2013/9/4 Gregorio Bastardo : > @St?fan: the 'np.all' calls are now unnecessary on line 26 > > @St?fan, Robert: Is it worth to bring this solution into numpy? I mean > it's probably not a rare problem, and now users have to bring this > snippet into their codebase. > > Gregorio > > 2013/9/3 St?fan van der Walt : >> On Tue, Sep 3, 2013 at 2:47 PM, Robert Kern wrote: >>>> Here's one way of doing it: https://gist.github.com/stefanv/6413742 >>> >>> You can probably reduce the amount of work by only comparing a.min() and >>> a.max() instead of the whole array. >> >> Thanks, fixed. >> >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion From joseluismietta at yahoo.com.ar Thu Sep 5 07:14:10 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Thu, 5 Sep 2013 04:14:10 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> Message-ID: <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> Thanks experts! Thanks Robert Kern! Two more questions about it: 1. networkx.has_path(G, first_stick, second_stick) stop when find second_stick or compute all the sub-graph and then evaluate if first_stick and second_stick are connected? 2. Using networkx or other tool: how can I obtain the 'clusters size' distribution (that is: number of clusters of size 'D', for all cluster-sizes)? Thanks a lot!! Jos? Luis ________________________________ De: Robert Kern Para: Discussion of Numerical Python Enviado: mi?rcoles, 4 de septiembre de 2013 18:49 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On Wed, Sep 4, 2013 at 3:17 PM, Jos? Luis Mietta wrote: > > Hi experts! > > If I do: > > G = Graph(M) > > That is: to use the associated intersection graph, where the vertices are the sticks and there is an edge between the two vertices if they intersect. Two sticks are "connected by a 'intersected-stick' path" if they are in the same connected component of this graph. > It turns out that the matrix I consider (M) is the adjacency matrix of this graph. > > Then, I can do: > > first_stick in G.connected_component_containing_vertex(second_stick) > > This is 'True' if 'first_stick' is in the sub-graph formed by 'second_stick' and all sticks 'connected' with 'second_stick'. > > The problem is that > > G.connected_component_containing_vertex() > > explore all the sub-graph. > > How can I do (what is the code) for stop the iteration when the algorithm found 'first-stick'? networkx.has_path(G, first_stick, second_stick) http://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.shortest_paths.generic.has_path.html#networkx.algorithms.shortest_paths.generic.has_path If you are going to be doing many queries, you should compute all of the path lengths, then you can query the final data structure easily. lengths = networkx.all_pairs_shortest_path_length(G) second_stick in lengths[first_stick] -- Robert Kern On Wed, Sep 4, 2013 at 3:17 PM, Jos? Luis Mietta wrote: Hi experts! > > > >If I do: >G =Graph(M) > > >That is: to use the associated intersection graph, where the vertices are the sticks and there is an edge between the two vertices if they intersect. Two sticks are "connected by a 'intersected-stick' path" if they are in the same connected component of this graph. >It turns out that the matrix I consider (M) is the adjacency matrix of this graph. > > >Then, I can do: >first_stick inG.connected_component_containing_vertex(second_stick) >This is 'True' if 'first_stick' is in the sub-graph formed by 'second_stick' and all sticks 'connected' with 'second_stick'. > > >The problem is that > >G.connected_component_containing_vertex() > >explore all the sub-graph. > >How can I do (what is the code) for stop the iteration when the algorithm found 'first-stick'? > >Waiting for your answers. > >Thanks a lot!! > > > > > >________________________________ > De: Robert Kern > >Para: Discussion of Numerical Python >Enviado: lunes, 2 de septiembre de 2013 10:40 > >Asunto: Re: [Numpy-discussion] Stick intersection path algorithm > > > >On Sun, Sep 1, 2013 at 11:55 PM, Jos? Luis Mietta wrote: >> >> Hi experts! >> >> I wanna study the intersection between line segments (sticks). >> I wrote a algorithm that generate a matrix, M, with N rows and N columns. The M-element Mij is 1 if stick number 'i' intersect stick number 'j' (obviously M is symmetric). >> >> Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. > >You may find it helpful to reword your problem using standard terminology. If I hadn't read your previous question, I would not have been able to understand what you meant by intersected sticks (or, as Chris thought at first, that you needed help determining the intersections). This will also help in searching Google for background and pre-existing software to solve your problem. > >You have an "undirected graph" (the sticks are "nodes", and the intersections are "edges") and you want to find if two given nodes are "reachable" from each other. You are currently representing your graph as an "adjacency matrix" `M` where `M[i,j]` is 1 iff nodes `i` and `j` have an edge between them and 0 otherwise. The "transitive closure" of your graph `M` is the graph that connects two nodes with an edge iff the two nodes are reachable from each other in the original graph `M`. There are several graph theory packages out there, like NetworkX, that will do this for you. Depending on the kinds of queries you would like to do, as David points out, want to compute the "connected components" of your graph; a connected component is a subgraph of your original graph such that all of the nodes are reachable from each other. > > >You can also look up Python code for computing the transitive closure of a graph; it's not a complicated algorithm. However, this algorithm is usually implemented using a different representation of a graph than an adjacency matrix, so you may need to do a conversion. > > >-- >Robert Kern > > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Robert Kern _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Sep 5 07:34:14 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 05 Sep 2013 07:34:14 -0400 Subject: [Numpy-discussion] numpy 1.8.0b1 mkl test_xerbla failure References: Message-ID: Just want to make sure this post had been noted: Neal Becker wrote: > Built on fedora linux 19 x86_64 using mkl: > > build OK using: > env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 > CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- > rpath=/opt/intel/mkl/lib/intel64" python setup.py build > > and attached site.cfg: > > ====================================================================== > FAIL: test_linalg.test_xerbla > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File "/home/nbecker/.local/lib/python2.7/site- > packages/numpy/testing/decorators.py", line 146, in skipper_func > return f(*args, **kwargs) > File "/home/nbecker/.local/lib/python2.7/site- > packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla > assert_(False) > File "/home/nbecker/.local/lib/python2.7/site- > packages/numpy/testing/utils.py", line 44, in assert_ > raise AssertionError(msg) > AssertionError > > ---------------------------------------------------------------------- > Ran 5271 tests in 57.567s > > FAILED (KNOWNFAIL=5, SKIP=13, failures=1) > From bahtiyor_zohidov at mail.ru Thu Sep 5 09:35:16 2013 From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=) Date: Thu, 05 Sep 2013 17:35:16 +0400 Subject: [Numpy-discussion] =?utf-8?q?Error_occurance_on_Skimage_0=2E9=2E0?= =?utf-8?q?_version?= Message-ID: <1378388116.548479125@f150.i.mail.ru> Hello, I downloaded skimage 0.9.0dev version and installed like following on Windows; cd C:\skimage\python setup.py install. One error occurred: Unable to find vcvarsall.bat . It is not the first time I have been taken such an error. What is the best way to avoid this problem??? Any answers will be appreciated -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Thu Sep 5 10:02:05 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Thu, 5 Sep 2013 10:02:05 -0400 Subject: [Numpy-discussion] Minimal NumPy for distribution In-Reply-To: References: Message-ID: Hi, thanks for the information. It is very useful to know that the c code could call back to python. Fred On Wed, Sep 4, 2013 at 2:14 PM, Nathaniel Smith wrote: > There do exist numpy c functions that call .py file routines. I don't know > how likely you are to find them in practice, but it definitely happens. > > You don't need .py files if you have .pyc files, and those can be > compressed (python can import directly from .zip files). > > -n > On 4 Sep 2013 18:52, "Fr?d?ric Bastien" wrote: > >> Hi, >> >> I have done some exploratory work with Theano to generate a shared >> library from a Theano function. This link with numpy c api. If we want to >> distribute this library and call it from C and/or python, what is the >> minimal installed part of NumPy needed? I suppose that only the c api is >> needed. Do someone already checked that? >> >> From what I found on linux, numpy linked with a dynamic BLAS take 13MB. >> There is 2.7MB from .so, 5.3MB from .py and 3.8MB from .pyc. Can we just >> keep all .so and remove all .py and .pyc file? I suppose we need to modify >> the __init__.py file too. >> >> On Windows, with the unofficial 64 bit binary of NumPy linked with MKL, >> the numpy directory take 56MB. So I would also like to know witch shared >> library(dll) is needed. Here is the .so file found on linux: >> >> /lib/python2.7/site-packages/numpy/core/multiarray_tests.so >> /lib/python2.7/site-packages/numpy/core/multiarray.so >> /lib/python2.7/site-packages/numpy/core/_dotblas.so >> /lib/python2.7/site-packages/numpy/core/_sort.so >> /lib/python2.7/site-packages/numpy/core/scalarmath.so >> /lib/python2.7/site-packages/numpy/core/umath_tests.so >> /lib/python2.7/site-packages/numpy/core/umath.so >> /lib/python2.7/site-packages/numpy/numarray/_capi.so >> /lib/python2.7/site-packages/numpy/linalg/lapack_lite.so >> /lib/python2.7/site-packages/numpy/random/mtrand.so >> /lib/python2.7/site-packages/numpy/fft/fftpack_lite.so >> /lib/python2.7/site-packages/numpy/lib/_compiled_base.so >> >> Can I get rid of all shared lib outside of core? >> >> thanks >> >> Fr?d?ric >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Sep 5 10:06:36 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 5 Sep 2013 16:06:36 +0200 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: On 5 September 2013 13:14, Jos? Luis Mietta wrote: > 2. Using networkx or other tool: how can I obtain the 'clusters size' > distribution (that is: number of clusters of size 'D', for all > cluster-sizes)? This is best asked in their mailing list. A possible way is: np.bincount([len(i) for i in nx.connected_components(G)]) For example: np.bincount([len(i) for i in nx.connected_components(nx.erdos_renyi_graph(100, 0.01))]) array([ 0, 39, 7, 3, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jerome.Kieffer at esrf.fr Thu Sep 5 10:10:14 2013 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Thu, 5 Sep 2013 16:10:14 +0200 Subject: [Numpy-discussion] Error occurance on Skimage 0.9.0 version In-Reply-To: <1378388116.548479125@f150.i.mail.ru> References: <1378388116.548479125@f150.i.mail.ru> Message-ID: <20130905161014.88030e05.Jerome.Kieffer@esrf.fr> On Thu, 05 Sep 2013 17:35:16 +0400 Happyman wrote: > Hello, > > I downloaded skimage 0.9.0dev version and installed like following on Windows; > cd C:\skimage\python setup.py install. > > One error occurred: Unable to find vcvarsall.bat . It is not the first time I have been taken such an error. > What is the best way to avoid this problem??? > > Any answers will be appreciated it looks like it is looking for the visual studio compiler ... do you have it ? if not try mingw32 (only for windows32 bits) it often works. Cheers, -- J?r?me Kieffer On-Line Data analysis / Software Group ISDD / ESRF tel +33 476 882 445 From joseluismietta at yahoo.com.ar Thu Sep 5 11:03:57 2013 From: joseluismietta at yahoo.com.ar (=?utf-8?B?Sm9zw6ggTHVpcyBNaWV0dGE=?=) Date: Thu, 5 Sep 2013 08:03:57 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: <1378393437.79314.YahooMailNeo@web142304.mail.bf1.yahoo.com> Hi experts! The array ([ 0, 39, ?7, ?3, ?1, ?1, ?0, ?0, ?0, ?0, ?1, ?0, ?0, ?0, ?0, ?0, ?0, 0, ?0, ?1]) means that in the sistem (graph) are : 4 cluster of size 1, one cluster of size 3, one cluster of size 7 and one cluste of size 39? What does means 'zero' (13 times) in the array? Thans a lot! Jos? Luis ________________________________ De: Da?id Para: Discussion of Numerical Python Enviado: jueves, 5 de septiembre de 2013 11:06 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On 5 September 2013 13:14, Jos? Luis Mietta wrote: 2. Using networkx or other tool: how can I obtain the 'clusters size' distribution (that is: number of clusters of size 'D', for all cluster-sizes)? This is best asked in their mailing list. A possible way is: np.bincount([len(i) for i in nx.connected_components(G)]) For example: np.bincount([len(i) for i in nx.connected_components(nx.erdos_renyi_graph(100, 0.01))]) array([ 0, 39, ?7, ?3, ?1, ?1, ?0, ?0, ?0, ?0, ?1, ?0, ?0, ?0, ?0, ?0, ?0, 0, ?0, ?1]) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahtiyor_zohidov at mail.ru Thu Sep 5 11:20:03 2013 From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=) Date: Thu, 05 Sep 2013 19:20:03 +0400 Subject: [Numpy-discussion] =?utf-8?q?Error_occurance_on_Skimage_0=2E9=2E0?= =?utf-8?q?_version?= In-Reply-To: <20130905161014.88030e05.Jerome.Kieffer@esrf.fr> References: <1378388116.548479125@f150.i.mail.ru> <20130905161014.88030e05.Jerome.Kieffer@esrf.fr> Message-ID: <1378394403.578576573@f142.i.mail.ru> Yeap I have installed visual studio, but I am a bit far away from installation and dealing vs such problems.. :)) My computer is Win64. I woulb be grateful if you could give me the steps of installation with some comments for Win64 thanks ???????, 5 ???????? 2013, 16:10 +02:00 ?? Jerome Kieffer : >On Thu, 05 Sep 2013 17:35:16 +0400 >Happyman < bahtiyor_zohidov at mail.ru > wrote: > >> Hello, >> >> I downloaded skimage 0.9.0dev version and installed like following on Windows; >> cd C:\skimage\python setup.py install. >> >> One error occurred: Unable to find vcvarsall.bat . It is not the first time I have been taken such an error. >> What is the best way to avoid this problem??? >> >> Any answers will be appreciated > >it looks like it is looking for the visual studio compiler ... do you have it ? >if not try mingw32 (only for windows32 bits) it often works. > >Cheers, > >-- >J?r?me Kieffer >On-Line Data analysis / Software Group >ISDD / ESRF >tel +33 476 882 445 >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion -- happy Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Sep 5 11:55:25 2013 From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=) Date: Thu, 5 Sep 2013 08:55:25 -0700 Subject: [Numpy-discussion] Error occurance on Skimage 0.9.0 version In-Reply-To: <1378394403.578576573@f142.i.mail.ru> References: <1378388116.548479125@f150.i.mail.ru> <20130905161014.88030e05.Jerome.Kieffer@esrf.fr> <1378394403.578576573@f142.i.mail.ru> Message-ID: I haven't tried to compile skimage, but the easiest way to get set up for compilation with Windows and MSVC is to follow the instructions in the Cython wiki. It is routinely spammed, so here's a link to the last non-corrupt version as of right now: http://wiki.cython.org/64BitCythonExtensionsOnWindows?action=recall&rev=54 Choose the SDK you need (3.5 SP1, unless you want to compile for Python 3.3, then it's 4) and install it. After installation, if you search for 'cmd' it will find both the standard console, 'cmd.exe', and a version tuned for the SDK,, 'CMD Shell' on my system. Open the second one and from the console type: set DISTUTILS_USE_SDK=1 setenv /x64 /release If you now run setup.py install, everything should work fine. Good luck! Jaime On Thu, Sep 5, 2013 at 8:20 AM, Happyman wrote: > Yeap I have installed visual studio, but I am a bit far away from > installation and dealing vs such problems.. :)) > My computer is Win64. I woulb be grateful if you could give me the steps > of installation with some comments for Win64 > > thanks > > > > ???????, 5 ???????? 2013, 16:10 +02:00 ?? Jerome Kieffer < > Jerome.Kieffer at esrf.fr>: > > On Thu, 05 Sep 2013 17:35:16 +0400 > Happyman > > wrote: > > > Hello, > > > > I downloaded skimage 0.9.0dev version and installed like following on > Windows; > > cd C:\skimage\python setup.py install. > > > > One error occurred: Unable to find vcvarsall.bat . It is not the first > time I have been taken such an error. > > What is the best way to avoid this problem??? > > > > Any answers will be appreciated > > it looks like it is looking for the visual studio compiler ... do you have > it ? > if not try mingw32 (only for windows32 bits) it often works. > > Cheers, > > -- > J?r?me Kieffer > On-Line Data analysis / Software Group > ISDD / ESRF > tel +33 476 882 445 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > happy Man > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Sep 5 11:56:19 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 5 Sep 2013 17:56:19 +0200 Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: <1378393437.79314.YahooMailNeo@web142304.mail.bf1.yahoo.com> References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> <1378393437.79314.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: On 5 September 2013 17:03, Jos? Luis Mietta wrote: > The array ([ 0, 39, 7, 3, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, > 0, 0, 0, 0, 1]) means that in the sistem (graph) are : 4 cluster of > size 1, one cluster of size 3, one cluster of size 7 and one cluste of size > 39? > No, it means there are 39 of size 1, 7 of size 2 and so on. David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Thu Sep 5 11:56:43 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 05 Sep 2013 08:56:43 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.8.0 beta 1 release In-Reply-To: <1378369086.8377.1.camel@sebastian-laptop> References: <522811D0.8030003@uci.edu> <1378369086.8377.1.camel@sebastian-laptop> Message-ID: <5228A9BB.3000003@uci.edu> On 9/5/2013 1:18 AM, Sebastian Berg wrote: > On Wed, 2013-09-04 at 22:08 -0700, Christoph Gohlke wrote: >> On 9/1/2013 9:54 AM, Charles R Harris wrote: > > > >> >> Hello, >> >> is this IndexError intentional in numpy 1.8? Matplotlib 1.3 fails some >> tests because of this. >> >>>>> numpy.zeros(1)[[0], :] >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: too many indices >> >> With numpy 1.7: >>>>> numpy.zeros(1)[[0], :] >> array([ 0.]) >> > > Yes certainly is intentional, since you actually do have too many > indices (you have one dimension, you can only have one index). That it > worked before was a bug. If this is a real problem, maybe we have to > temporarily allow it? > > - Sebastian > Thanks. Not a real problem. It is better to fix matplotlib . Christoph From joseluismietta at yahoo.com.ar Thu Sep 5 12:51:09 2013 From: joseluismietta at yahoo.com.ar (=?utf-8?B?Sm9zw6ggTHVpcyBNaWV0dGE=?=) Date: Thu, 5 Sep 2013 09:51:09 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> <1378393437.79314.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: <1378399869.51210.YahooMailNeo@web142306.mail.bf1.yahoo.com> Thanks so much!! ? Best regards, Jos? Luis ________________________________ De: Da?id Para: Discussion of Numerical Python Enviado: jueves, 5 de septiembre de 2013 12:56 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On 5 September 2013 17:03, Jos? Luis Mietta wrote: The array ([ 0, 39, ?7, ?3, ?1, ?1, ?0, ?0, ?0, ?0, ?1, ?0, ?0, ?0, ?0, ?0, ?0, 0, ?0, ?1]) means that in the sistem (graph) are : 4 cluster of size 1, one cluster of size 3, one cluster of size 7 and one cluste of size 39? No, it means there are 39 of size 1, 7 of size 2 and so on. David. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.collette at gmail.com Thu Sep 5 13:13:50 2013 From: andrew.collette at gmail.com (Andrew Collette) Date: Thu, 5 Sep 2013 11:13:50 -0600 Subject: [Numpy-discussion] dtype "metadata" attribute Message-ID: Hi, We have a pull request pending at the h5py project to use the dtype "metadata" attribute to implement special type support, as opposed to the current hacked-together solution based on field names: >>> dt = dtype('i', metadata={"key": 42}) >>> dt.metadata However, I can't find documentation on it anywhere; the "metadata" keyword doesn't even appear in the dtype docstring. Is this an officially supported feature in NumPy? I am mostly concerned about it going away in a future release. Thanks! Andrew Collette From chris.barker at noaa.gov Thu Sep 5 13:16:06 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 5 Sep 2013 10:16:06 -0700 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> <52277032.40304@googlemail.com> Message-ID: This is good stuff, but I can't help thinking that if I needed to do an any/all test on a number of arrays with common and/or combos -- I'd probably write a Cython function to do it. It could be a bit tricky to make it really general, but not bad for a couple specific dtypes / use cases. -just a thought... Also -- how does this work with numexpr? It would be nice if it could handle these kinds of cases. -Chris On Thu, Sep 5, 2013 at 1:54 AM, Graeme B. Bell wrote: > > > Hi Julian, > > Thanks for the post. It's great to hear that the main numpy function is > improving in 1.8, though I think there is still plenty of value here for > performance junkies :-) > > I don't have 1.8beta installed (and I can't conveniently install it on my > machines just now). If you have time, and have the beta installed, could > you try this and mail me the output from the benchmark? I'm curious to > know. > > # git clone https://github.com/gbb/numpy-fast-any-all.git > # cd numpy-fast-any-all > # python test-fast-any-all.py > > Graeme > > > On Sep 4, 2013, at 7:38 PM, Julian Taylor > wrote: > > >> > >> The result is 14 to 17x faster than np.any() for this use case.* > > > > any/all and boolean operations have been significantly speed up by > > vectorization in numpy 1.8 [0]. > > They are now around 10 times faster than before, especially if the > > boolean array fits into one of the cpu caching layers. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Sep 5 14:11:59 2013 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 5 Sep 2013 11:11:59 -0700 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? Message-ID: Hi all, I just ran into this rather weird behavior: http://nbviewer.ipython.org/6453869 In summary, as far as I can tell, shuffle is misbehaving when acting on arrays that have structured dtypes. I've seen the problem on 1.7.1 (official on ubuntu 13.04) as well as master as of a few minutes ago. Is this my misuse? It really looks like a bug to me... Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail From warren.weckesser at gmail.com Thu Sep 5 14:29:09 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Thu, 5 Sep 2013 14:29:09 -0400 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: Message-ID: On Thu, Sep 5, 2013 at 2:11 PM, Fernando Perez wrote: > Hi all, > > I just ran into this rather weird behavior: > > http://nbviewer.ipython.org/6453869 > > In summary, as far as I can tell, shuffle is misbehaving when acting > on arrays that have structured dtypes. I've seen the problem on 1.7.1 > (official on ubuntu 13.04) as well as master as of a few minutes ago. > > Is this my misuse? It really looks like a bug to me... > > Definitely a bug: In [1]: np.__version__ Out[1]: '1.9.0.dev-573b3b0' In [2]: z = np.array([(0,),(1,),(2,),(3,),(4,)], dtype=[('a',int)]) In [3]: z Out[3]: array([(0,), (1,), (2,), (3,), (4,)], dtype=[('a', ' > f > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad.froehle at gmail.com Thu Sep 5 14:30:25 2013 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Thu, 5 Sep 2013 11:30:25 -0700 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: Message-ID: Looks like a bug. FWIW, NumPy 1.6.1 on Scientific Linux 6.4 does not suffer from this malady. -Brad $ python Python 2.6.6 (r266:84292, Feb 21 2013, 19:26:11) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> import numpy as np >>> from numpy.random import shuffle >>> >>> print 'Numpy version:', np.__version__ Numpy version: 1.6.2 >>> >>> a = np.arange(5) >>> for i in range(5): ... print a ... shuffle(a) ... [0 1 2 3 4] [2 1 0 4 3] [0 4 3 1 2] [4 1 0 2 3] [2 0 4 3 1] >>> >>> b = np.zeros(5, dtype=[('n', 'S1'), ('i', int)]) >>> b['i'] = range(5) >>> print b [('', 0) ('', 1) ('', 2) ('', 3) ('', 4)] >>> print b.dtype [('n', '|S1'), ('i', '>> for i in range(5): ... print b ... shuffle(b) ... [('', 0) ('', 1) ('', 2) ('', 3) ('', 4)] [('', 0) ('', 2) ('', 4) ('', 1) ('', 3)] [('', 2) ('', 0) ('', 4) ('', 1) ('', 3)] [('', 2) ('', 1) ('', 3) ('', 4) ('', 0)] [('', 0) ('', 1) ('', 2) ('', 4) ('', 3)] On Thu, Sep 5, 2013 at 11:11 AM, Fernando Perez wrote: > Hi all, > > I just ran into this rather weird behavior: > > http://nbviewer.ipython.org/6453869 > > In summary, as far as I can tell, shuffle is misbehaving when acting > on arrays that have structured dtypes. I've seen the problem on 1.7.1 > (official on ubuntu 13.04) as well as master as of a few minutes ago. > > Is this my misuse? It really looks like a bug to me... > > Cheers, > > f > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jjhelmus at gmail.com Thu Sep 5 14:35:14 2013 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Thu, 05 Sep 2013 13:35:14 -0500 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: Message-ID: <5228CEE2.6020305@gmail.com> On 09/05/2013 01:29 PM, Warren Weckesser wrote: > > > On Thu, Sep 5, 2013 at 2:11 PM, Fernando Perez > wrote: > > Hi all, > > I just ran into this rather weird behavior: > > http://nbviewer.ipython.org/6453869 > > In summary, as far as I can tell, shuffle is misbehaving when acting > on arrays that have structured dtypes. I've seen the problem on 1.7.1 > (official on ubuntu 13.04) as well as master as of a few minutes ago. > > Is this my misuse? It really looks like a bug to me... > > > > Definitely a bug: > > In [1]: np.__version__ > Out[1]: '1.9.0.dev-573b3b0' > > In [2]: z = np.array([(0,),(1,),(2,),(3,),(4,)], dtype=[('a',int)]) > > In [3]: z > Out[3]: > array([(0,), (1,), (2,), (3,), (4,)], > dtype=[('a', ' > In [4]: shuffle(z) > > In [5]: z > Out[5]: > array([(0,), (1,), (2,), (0,), (0,)], > dtype=[('a', ' > > > Nothing in the docstring suggests that it shouldn't work for > structured dtypes. > > Warren Looks to stemming from the fact that elements of records arrays cannot be swapped: In [1]: import numpy as np In [2]: x = np.zeros(5, dtype=[('n', 'S1'), ('i', int)]) In [3]: x['i'] = range(5) In [4]: print x [('', 0) ('', 1) ('', 2) ('', 3) ('', 4)] In [5]: x[0], x[1] = x[1], x[0] In [6]: print x [('', 1) ('', 1) ('', 2) ('', 3) ('', 4)] This is with numpy 1.7.1 Cheers, - Jonathan Helmus > > > > > Cheers, > > f > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 5 14:36:11 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 12:36:11 -0600 Subject: [Numpy-discussion] numpy 1.8.0b1 mkl test_xerbla failure In-Reply-To: References: Message-ID: On Thu, Sep 5, 2013 at 5:34 AM, Neal Becker wrote: > Just want to make sure this post had been noted: > > Neal Becker wrote: > > > Built on fedora linux 19 x86_64 using mkl: > > > > build OK using: > > env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 > LAPACK=/usr/lib64 > > CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- > > rpath=/opt/intel/mkl/lib/intel64" python setup.py build > > > > and attached site.cfg: > > > > ====================================================================== > > FAIL: test_linalg.test_xerbla > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in > runTest > > self.test(*self.arg) > > File "/home/nbecker/.local/lib/python2.7/site- > > packages/numpy/testing/decorators.py", line 146, in skipper_func > > return f(*args, **kwargs) > > File "/home/nbecker/.local/lib/python2.7/site- > > packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla > > assert_(False) > > File "/home/nbecker/.local/lib/python2.7/site- > > packages/numpy/testing/utils.py", line 44, in assert_ > > raise AssertionError(msg) > > AssertionError > > > > ---------------------------------------------------------------------- > > Ran 5271 tests in 57.567s > > > > FAILED (KNOWNFAIL=5, SKIP=13, failures=1) > > > > > What version of MKL is this? The bug doesn't show in Christolph's compiles with MKL on windows, so it might be an MKL bug. Is it repeatable? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 5 14:43:12 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 12:43:12 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: <5228CEE2.6020305@gmail.com> References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 12:35 PM, Jonathan Helmus wrote: > On 09/05/2013 01:29 PM, Warren Weckesser wrote: > > > > On Thu, Sep 5, 2013 at 2:11 PM, Fernando Perez wrote: > >> Hi all, >> >> I just ran into this rather weird behavior: >> >> http://nbviewer.ipython.org/6453869 >> >> In summary, as far as I can tell, shuffle is misbehaving when acting >> on arrays that have structured dtypes. I've seen the problem on 1.7.1 >> (official on ubuntu 13.04) as well as master as of a few minutes ago. >> >> Is this my misuse? It really looks like a bug to me... >> >> > > Definitely a bug: > > In [1]: np.__version__ > Out[1]: '1.9.0.dev-573b3b0' > > In [2]: z = np.array([(0,),(1,),(2,),(3,),(4,)], dtype=[('a',int)]) > > In [3]: z > Out[3]: > array([(0,), (1,), (2,), (3,), (4,)], > dtype=[('a', ' > In [4]: shuffle(z) > > In [5]: z > Out[5]: > array([(0,), (1,), (2,), (0,), (0,)], > dtype=[('a', ' > > > Nothing in the docstring suggests that it shouldn't work for structured > dtypes. > > Warren > > > Looks to stemming from the fact that elements of records arrays cannot be > swapped: > > In [1]: import numpy as np > > In [2]: x = np.zeros(5, dtype=[('n', 'S1'), ('i', int)]) > > In [3]: x['i'] = range(5) > > In [4]: print x > > [('', 0) ('', 1) ('', 2) ('', 3) ('', 4)] > > In [5]: x[0], x[1] = x[1], x[0] > > In [6]: print x > > [('', 1) ('', 1) ('', 2) ('', 3) ('', 4)] > > This is with numpy 1.7.1 > > Oh, nice one ;) Should be fixable if you want to submit a patch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Sep 5 14:50:42 2013 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 5 Sep 2013 11:50:42 -0700 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris wrote: > Oh, nice one ;) Should be fixable if you want to submit a patch. Strategy? One option is to do, for structured arrays, a shuffle of the indices and then an in-place arr = arr[shuffled_indices] But there may be a cleaner/faster way to do it. I'm happy to submit a patch, but I'm not familiar enough with the internals to know what the best approach should be. Cheers, f From charlesr.harris at gmail.com Thu Sep 5 14:52:22 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 12:52:22 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: <5228CEE2.6020305@gmail.com> References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 12:35 PM, Jonathan Helmus wrote: > On 09/05/2013 01:29 PM, Warren Weckesser wrote: > > > > On Thu, Sep 5, 2013 at 2:11 PM, Fernando Perez wrote: > >> Hi all, >> >> I just ran into this rather weird behavior: >> >> http://nbviewer.ipython.org/6453869 >> >> In summary, as far as I can tell, shuffle is misbehaving when acting >> on arrays that have structured dtypes. I've seen the problem on 1.7.1 >> (official on ubuntu 13.04) as well as master as of a few minutes ago. >> >> Is this my misuse? It really looks like a bug to me... >> >> > > Definitely a bug: > > In [1]: np.__version__ > Out[1]: '1.9.0.dev-573b3b0' > > In [2]: z = np.array([(0,),(1,),(2,),(3,),(4,)], dtype=[('a',int)]) > > In [3]: z > Out[3]: > array([(0,), (1,), (2,), (3,), (4,)], > dtype=[('a', ' > In [4]: shuffle(z) > > In [5]: z > Out[5]: > array([(0,), (1,), (2,), (0,), (0,)], > dtype=[('a', ' > > > Nothing in the docstring suggests that it shouldn't work for structured > dtypes. > > Warren > > > Looks to stemming from the fact that elements of records arrays cannot be > swapped: > > In [1]: import numpy as np > > In [2]: x = np.zeros(5, dtype=[('n', 'S1'), ('i', int)]) > > In [3]: x['i'] = range(5) > > In [4]: print x > > [('', 0) ('', 1) ('', 2) ('', 3) ('', 4)] > > In [5]: x[0], x[1] = x[1], x[0] > > In [6]: print x > > [('', 1) ('', 1) ('', 2) ('', 3) ('', 4)] > > This is with numpy 1.7.1 > > The elements seem to be by reference, making copies works. In [16]: x[0], x[1] = x[1].copy(), x[0].copy() In [17]: x Out[17]: array([('', 1), ('', 0), ('', 2), ('', 3), ('', 4)], dtype=[('n', 'S1'), ('i', ' From jtaylor.debian at googlemail.com Thu Sep 5 14:53:38 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 05 Sep 2013 20:53:38 +0200 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: References: <047D6950-46F9-4EF1-B768-20F722E21BF3@skogoglandskap.no> <52277032.40304@googlemail.com> Message-ID: <5228D332.2090406@googlemail.com> hi, its not np.any that is slow in this case its np.array([A, B, C]) np.dstack([A, B, C]) is better but writing it like this has the same performance as your code: a = empty([3] list(A.shape) a[0] = A>5; a[1] = B<2; a[2] = A>10; np.any(a, 0) I'll check if creating an array from a sequence can be improved for this case. On 05.09.2013 10:54, Graeme B. Bell wrote: > > > Hi Julian, > > Thanks for the post. It's great to hear that the main numpy function is improving in 1.8, though I think there is still plenty of value here for performance junkies :-) > > I don't have 1.8beta installed (and I can't conveniently install it on my machines just now). If you have time, and have the beta installed, could you try this and mail me the output from the benchmark? I'm curious to know. > > # git clone https://github.com/gbb/numpy-fast-any-all.git > # cd numpy-fast-any-all > # python test-fast-any-all.py > > Graeme > > > On Sep 4, 2013, at 7:38 PM, Julian Taylor wrote: > >>> >>> The result is 14 to 17x faster than np.any() for this use case.* >> >> any/all and boolean operations have been significantly speed up by >> vectorization in numpy 1.8 [0]. >> They are now around 10 times faster than before, especially if the >> boolean array fits into one of the cpu caching layers. > From jjhelmus at gmail.com Thu Sep 5 14:57:05 2013 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Thu, 05 Sep 2013 13:57:05 -0500 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: <5228D401.8060904@gmail.com> On 09/05/2013 01:50 PM, Fernando Perez wrote: > On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris > wrote: > > >> Oh, nice one ;) Should be fixable if you want to submit a patch. > Strategy? One option is to do, for structured arrays, a shuffle of the > indices and then an in-place > > arr = arr[shuffled_indices] > > But there may be a cleaner/faster way to do it. > > I'm happy to submit a patch, but I'm not familiar enough with the > internals to know what the best approach should be. > > Cheers, > > f Fixing the shuffle function can be done by adding a check to see if x[0] is of type numpy.void on line 4429 of numpy/random/mtrand/mtrand.pyx and using the top if block of code which uses a buffer for element swapping if it is. But it wouldn't fix the problem with swapping of records array elements which is the real problem. - Jonathan Helmus From charlesr.harris at gmail.com Thu Sep 5 14:58:26 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 12:58:26 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 12:50 PM, Fernando Perez wrote: > On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris > wrote: > > > > Oh, nice one ;) Should be fixable if you want to submit a patch. > > Strategy? One option is to do, for structured arrays, a shuffle of the > indices and then an in-place > > arr = arr[shuffled_indices] > > But there may be a cleaner/faster way to do it. > > I'm happy to submit a patch, but I'm not familiar enough with the > internals to know what the best approach should be. > > Better open an issue. It looks like a bug in the indexing code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad.froehle at gmail.com Thu Sep 5 15:34:46 2013 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Thu, 5 Sep 2013 12:34:46 -0700 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: I put this test case through `git bisect run` and here's what came back. I haven't confirmed this manually yet, but the blamed commit does seem reasonable: b26c675e2a91e1042f8f8d634763942c87fbbb6e is the first bad commit commit b26c675e2a91e1042f8f8d634763942c87fbbb6e Author: Nathaniel J. Smith Date: Thu Jul 12 13:20:20 2012 +0100 [FIX] Make np.random.shuffle less brain-dead The logic in np.random.shuffle was... not very sensible. Fixes trac ticket #2074. This patch also exposes a completely unrelated issue in numpy.testing. Filed as Github issue #347 and marked as knownfail for now. :040000 040000 6f3cf0c85a64664db6a71bd59909903f18b51639 0b6c8571dd3c9de8f023389f6bd963e42b12cc26 M numpy bisect run success On Thu, Sep 5, 2013 at 11:58 AM, Charles R Harris wrote: > > > > On Thu, Sep 5, 2013 at 12:50 PM, Fernando Perez > wrote: >> >> On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris >> wrote: >> >> >> > Oh, nice one ;) Should be fixable if you want to submit a patch. >> >> Strategy? One option is to do, for structured arrays, a shuffle of the >> indices and then an in-place >> >> arr = arr[shuffled_indices] >> >> But there may be a cleaner/faster way to do it. >> >> I'm happy to submit a patch, but I'm not familiar enough with the >> internals to know what the best approach should be. >> > > Better open an issue. It looks like a bug in the indexing code. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Sep 5 15:45:49 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 13:45:49 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 1:34 PM, Bradley M. Froehle wrote: > I put this test case through `git bisect run` and here's what came > back. I haven't confirmed this manually yet, but the blamed commit > does seem reasonable: > > b26c675e2a91e1042f8f8d634763942c87fbbb6e is the first bad commit > commit b26c675e2a91e1042f8f8d634763942c87fbbb6e > Author: Nathaniel J. Smith > Date: Thu Jul 12 13:20:20 2012 +0100 > > [FIX] Make np.random.shuffle less brain-dead > > The logic in np.random.shuffle was... not very sensible. Fixes trac > ticket #2074. > > This patch also exposes a completely unrelated issue in > numpy.testing. Filed as Github issue #347 and marked as knownfail for > now. > > :040000 040000 6f3cf0c85a64664db6a71bd59909903f18b51639 > 0b6c8571dd3c9de8f023389f6bd963e42b12cc26 M numpy > bisect run success > > On Thu, Sep 5, 2013 at 11:58 AM, Charles R Harris > wrote: > > > > > > > > On Thu, Sep 5, 2013 at 12:50 PM, Fernando Perez > > wrote: > >> > >> On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris > >> wrote: > >> > >> > >> > Oh, nice one ;) Should be fixable if you want to submit a patch. > >> > >> Strategy? One option is to do, for structured arrays, a shuffle of the > >> indices and then an in-place > >> > >> arr = arr[shuffled_indices] > >> > >> But there may be a cleaner/faster way to do it. > >> > >> I'm happy to submit a patch, but I'm not familiar enough with the > >> internals to know what the best approach should be. > >> > > > > Better open an issue. It looks like a bug in the indexing code. > > > Also fails for string arrays. In [6]: x = np.zeros(5, dtype=[('n', 'S1'), ('s', 'S1')]) In [7]: x['s'] = [c for c in 'abcde'] In [8]: x Out[8]: array([('', 'a'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], dtype=[('n', 'S1'), ('s', 'S1')]) In [9]: x[0], x[1] = x[1], x[0] In [10]: x Out[10]: array([('', 'b'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], dtype=[('n', 'S1'), ('s', 'S1')]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 5 16:06:44 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 14:06:44 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 1:45 PM, Charles R Harris wrote: > > > > On Thu, Sep 5, 2013 at 1:34 PM, Bradley M. Froehle > wrote: > >> I put this test case through `git bisect run` and here's what came >> back. I haven't confirmed this manually yet, but the blamed commit >> does seem reasonable: >> >> b26c675e2a91e1042f8f8d634763942c87fbbb6e is the first bad commit >> commit b26c675e2a91e1042f8f8d634763942c87fbbb6e >> Author: Nathaniel J. Smith >> Date: Thu Jul 12 13:20:20 2012 +0100 >> >> [FIX] Make np.random.shuffle less brain-dead >> >> The logic in np.random.shuffle was... not very sensible. Fixes trac >> ticket #2074. >> >> This patch also exposes a completely unrelated issue in >> numpy.testing. Filed as Github issue #347 and marked as knownfail for >> now. >> >> :040000 040000 6f3cf0c85a64664db6a71bd59909903f18b51639 >> 0b6c8571dd3c9de8f023389f6bd963e42b12cc26 M numpy >> bisect run success >> >> On Thu, Sep 5, 2013 at 11:58 AM, Charles R Harris >> wrote: >> > >> > >> > >> > On Thu, Sep 5, 2013 at 12:50 PM, Fernando Perez >> > wrote: >> >> >> >> On Thu, Sep 5, 2013 at 11:43 AM, Charles R Harris >> >> wrote: >> >> >> >> >> >> > Oh, nice one ;) Should be fixable if you want to submit a patch. >> >> >> >> Strategy? One option is to do, for structured arrays, a shuffle of the >> >> indices and then an in-place >> >> >> >> arr = arr[shuffled_indices] >> >> >> >> But there may be a cleaner/faster way to do it. >> >> >> >> I'm happy to submit a patch, but I'm not familiar enough with the >> >> internals to know what the best approach should be. >> >> >> > >> > Better open an issue. It looks like a bug in the indexing code. >> > >> > > Also fails for string arrays. > > In [6]: x = np.zeros(5, dtype=[('n', 'S1'), ('s', 'S1')]) > > In [7]: x['s'] = [c for c in 'abcde'] > > In [8]: x > Out[8]: > array([('', 'a'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > dtype=[('n', 'S1'), ('s', 'S1')]) > > In [9]: x[0], x[1] = x[1], x[0] > > In [10]: x > Out[10]: > array([('', 'b'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > dtype=[('n', 'S1'), ('s', 'S1')]) > > This behavior is not new, it is also present in 1.6.x In [1]: x = np.zeros(5, dtype=[('n', 'S1'), ('s', 'S1')]) In [2]: x['s'] = [c for c in 'abcde'] In [3]: x Out[3]: array([('', 'a'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], dtype=[('n', '|S1'), ('s', '|S1')]) In [4]: x[0], x[1] = x[1], x[0] In [5]: x Out[5]: array([('', 'b'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], dtype=[('n', '|S1'), ('s', '|S1')]) In [6]: np.__version__ Out[6]: '1.6.3.dev-3f58621' So it looks like it needs to be decided if this is a bug or not. I think the returned scalars should be copies of the data. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 5 16:27:31 2013 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Sep 2013 21:27:31 +0100 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 9:06 PM, Charles R Harris wrote: > This behavior is not new, it is also present in 1.6.x > > In [1]: x = np.zeros(5, dtype=[('n', 'S1'), ('s', 'S1')]) > > In [2]: x['s'] = [c for c in 'abcde'] > > In [3]: x > Out[3]: > array([('', 'a'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > dtype=[('n', '|S1'), ('s', '|S1')]) > > In [4]: x[0], x[1] = x[1], x[0] > > In [5]: x > Out[5]: > array([('', 'b'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > dtype=[('n', '|S1'), ('s', '|S1')]) > > In [6]: np.__version__ > Out[6]: '1.6.3.dev-3f58621' > > So it looks like it needs to be decided if this is a bug or not. I think the > returned scalars should be copies of the data. I'm not a fan of the weird behaviour where for every other dtype, scalar indexing returns an immutable copy, but for voids it returns a mutable view. So +1 if we can do it. However, isn't this going to mess up stuff like x[0]["n"] = "a" ? -n From charlesr.harris at gmail.com Thu Sep 5 17:21:26 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 15:21:26 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 2:27 PM, Nathaniel Smith wrote: > On Thu, Sep 5, 2013 at 9:06 PM, Charles R Harris > wrote: > > This behavior is not new, it is also present in 1.6.x > > > > In [1]: x = np.zeros(5, dtype=[('n', 'S1'), ('s', 'S1')]) > > > > In [2]: x['s'] = [c for c in 'abcde'] > > > > In [3]: x > > Out[3]: > > array([('', 'a'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > > dtype=[('n', '|S1'), ('s', '|S1')]) > > > > In [4]: x[0], x[1] = x[1], x[0] > > > > In [5]: x > > Out[5]: > > array([('', 'b'), ('', 'b'), ('', 'c'), ('', 'd'), ('', 'e')], > > dtype=[('n', '|S1'), ('s', '|S1')]) > > > > In [6]: np.__version__ > > Out[6]: '1.6.3.dev-3f58621' > > > > So it looks like it needs to be decided if this is a bug or not. I think > the > > returned scalars should be copies of the data. > > I'm not a fan of the weird behaviour where for every other dtype, > scalar indexing returns an immutable copy, but for voids it returns a > mutable view. So +1 if we can do it. > > However, isn't this going to mess up stuff like > x[0]["n"] = "a" > ? > > Left and right are different, but I'm not sure how the assignment op handles that. So maybe no and maybe yes ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Sep 5 17:40:00 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Sep 2013 17:40:00 -0400 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: >> > >> > In [4]: x[0], x[1] = x[1], x[0] Is this ever predictable? sounds to me like the inplace question a few days ago result depends on the underlying iterator >>> a = np.arange(5*3).reshape((5,3), order='F') >>> a array([[ 0, 5, 10], [ 1, 6, 11], [ 2, 7, 12], [ 3, 8, 13], [ 4, 9, 14]]) >>> b = a[::-1] >>> b array([[ 4, 9, 14], [ 3, 8, 13], [ 2, 7, 12], [ 1, 6, 11], [ 0, 5, 10]]) >>> b[0], b[1] = b[1], b[0] >>> b array([[ 3, 8, 13], [ 3, 8, 13], [ 2, 7, 12], [ 1, 6, 11], [ 0, 5, 10]]) >>> a array([[ 0, 5, 10], [ 1, 6, 11], [ 2, 7, 12], [ 3, 8, 13], [ 3, 8, 13]]) The example works as "expected" if a is 1d Josef From joseluismietta at yahoo.com.ar Thu Sep 5 18:24:17 2013 From: joseluismietta at yahoo.com.ar (=?utf-8?B?Sm9zw6ggTHVpcyBNaWV0dGE=?=) Date: Thu, 5 Sep 2013 15:24:17 -0700 (PDT) Subject: [Numpy-discussion] Stick intersection path algorithm In-Reply-To: References: <1378076104.58147.YahooMailNeo@web142301.mail.bf1.yahoo.com> <1378304250.32441.YahooMailNeo@web142303.mail.bf1.yahoo.com> <1378379650.92309.YahooMailNeo@web142304.mail.bf1.yahoo.com> <1378393437.79314.YahooMailNeo@web142304.mail.bf1.yahoo.com> Message-ID: <1378419857.67570.YahooMailNeo@web142301.mail.bf1.yahoo.com> Hi experts! How can I create a?networkx graph from the adjacency matrix M? ? Thanks a lot, Jos? Luis ________________________________ De: Da?id Para: Discussion of Numerical Python Enviado: jueves, 5 de septiembre de 2013 12:56 Asunto: Re: [Numpy-discussion] Stick intersection path algorithm On 5 September 2013 17:03, Jos? Luis Mietta wrote: The array ([ 0, 39, ?7, ?3, ?1, ?1, ?0, ?0, ?0, ?0, ?1, ?0, ?0, ?0, ?0, ?0, ?0, 0, ?0, ?1]) means that in the sistem (graph) are : 4 cluster of size 1, one cluster of size 3, one cluster of size 7 and one cluste of size 39? No, it means there are 39 of size 1, 7 of size 2 and so on. David. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Thu Sep 5 19:09:45 2013 From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=) Date: Thu, 5 Sep 2013 16:09:45 -0700 Subject: [Numpy-discussion] Weird behavior of gufuncs Message-ID: Hi all, I am seeing some very weird behavior on a gufunc I coded. It has a pretty complicated signature: '(r,c,p),(i,j,k,n),(u,v),(d),(n,q)->(q,r,c)' And a single registered loop function, for types: uint8, uint16, uint16, uintp, uint8->uint8. In general it performs beautifully well, returning the expected results about x10 faster than a MATLAB version using a 12 core CPU (!) But today I found out that for certain sizes of the inputs, it doesn't even call the loop function and returns a zeroed output array immediately. I am using Python 2.7.5 and Numpy 1.7.1 under Windows 7, and have seen the behavior in both x64 and x86 versions. The thresholds vary, and I don't fully understand the logic behind it, but there is a clear threshold of sizes: keeping all other inputs the same, one byte more in the other argument and it returns all zeros, one byte less and everything works fine. The thresholds are different for 64 and 32 bit versions. I have a strong suspicion that this may be a NumPy 1.7 bug, because a colleague running MacOS and NumPy 1.8 compiled from the development branch isn't seeing any of this issues I have. It could of course be a Windows thing, but somehow that seems less likely, especially since there are other bugs in gufuncs that have been fixed in 1.8 but haven't been backported to 1.7. Tomorrow I am going to try and get access to a Linux box with Numpy 1.7 to try to reproduce it there. Does this sound familiar to anyone? Any known (and hopefully solved) issue that could trigger this behavior? A search for ufunc in github's issue tracker didn't give me any meaningful leads... Thanks! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 5 21:33:55 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Sep 2013 19:33:55 -0600 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 3:40 PM, wrote: > >> > > >> > In [4]: x[0], x[1] = x[1], x[0] > > Is this ever predictable? > sounds to me like the inplace question a few days ago > result depends on the underlying iterator > > >>> a = np.arange(5*3).reshape((5,3), order='F') > >>> a > array([[ 0, 5, 10], > [ 1, 6, 11], > [ 2, 7, 12], > [ 3, 8, 13], > [ 4, 9, 14]]) > >>> b = a[::-1] > >>> b > array([[ 4, 9, 14], > [ 3, 8, 13], > [ 2, 7, 12], > [ 1, 6, 11], > [ 0, 5, 10]]) > >>> b[0], b[1] = b[1], b[0] > >>> b > array([[ 3, 8, 13], > [ 3, 8, 13], > [ 2, 7, 12], > [ 1, 6, 11], > [ 0, 5, 10]]) > >>> a > array([[ 0, 5, 10], > [ 1, 6, 11], > [ 2, 7, 12], > [ 3, 8, 13], > [ 3, 8, 13]]) > > The example works as "expected" if a is 1d > The rows are views, so that is expected. What is unexpected is that the void/string scalars are also views. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Sep 5 23:16:04 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Sep 2013 23:16:04 -0400 Subject: [Numpy-discussion] A bug in numpy.random.shuffle? In-Reply-To: References: <5228CEE2.6020305@gmail.com> Message-ID: On Thu, Sep 5, 2013 at 9:33 PM, Charles R Harris wrote: > > > > On Thu, Sep 5, 2013 at 3:40 PM, wrote: >> >> >> > >> >> > In [4]: x[0], x[1] = x[1], x[0] >> >> Is this ever predictable? >> sounds to me like the inplace question a few days ago >> result depends on the underlying iterator >> >> >>> a = np.arange(5*3).reshape((5,3), order='F') >> >>> a >> array([[ 0, 5, 10], >> [ 1, 6, 11], >> [ 2, 7, 12], >> [ 3, 8, 13], >> [ 4, 9, 14]]) >> >>> b = a[::-1] >> >>> b >> array([[ 4, 9, 14], >> [ 3, 8, 13], >> [ 2, 7, 12], >> [ 1, 6, 11], >> [ 0, 5, 10]]) >> >>> b[0], b[1] = b[1], b[0] >> >>> b >> array([[ 3, 8, 13], >> [ 3, 8, 13], >> [ 2, 7, 12], >> [ 1, 6, 11], >> [ 0, 5, 10]]) >> >>> a >> array([[ 0, 5, 10], >> [ 1, 6, 11], >> [ 2, 7, 12], >> [ 3, 8, 13], >> [ 3, 8, 13]]) >> >> The example works as "expected" if a is 1d > > > The rows are views, so that is expected. What is unexpected is that the > void/string scalars are also views. I don't see it with string, only with structured dtypes IIUC >>> b = np.arange(5).astype('S2') >>> b array(['0', '1', '2', '3', '4'], dtype='|S2') >>> b[1], b[0] = b[0], b[1] >>> b array(['1', '0', '2', '3', '4'], dtype='|S2') >>> b[0], b[1] = b[1], b[0] >>> b array(['0', '1', '2', '3', '4'], dtype='|S2') >>> type(b[0]) >>> np.__version__ '1.5.1' > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ndbecker2 at gmail.com Fri Sep 6 06:58:56 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 06 Sep 2013 06:58:56 -0400 Subject: [Numpy-discussion] numpy 1.8.0b1 mkl test_xerbla failure References: Message-ID: Charles R Harris wrote: > On Thu, Sep 5, 2013 at 5:34 AM, Neal Becker wrote: > >> Just want to make sure this post had been noted: >> >> Neal Becker wrote: >> >> > Built on fedora linux 19 x86_64 using mkl: >> > >> > build OK using: >> > env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 >> LAPACK=/usr/lib64 >> > CFLAGS="-mtune=native -march=native -O3" LDFLAGS="-Wl,- >> > rpath=/opt/intel/mkl/lib/intel64" python setup.py build >> > >> > and attached site.cfg: >> > >> > ====================================================================== >> > FAIL: test_linalg.test_xerbla >> > ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in >> runTest >> > self.test(*self.arg) >> > File "/home/nbecker/.local/lib/python2.7/site- >> > packages/numpy/testing/decorators.py", line 146, in skipper_func >> > return f(*args, **kwargs) >> > File "/home/nbecker/.local/lib/python2.7/site- >> > packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla >> > assert_(False) >> > File "/home/nbecker/.local/lib/python2.7/site- >> > packages/numpy/testing/utils.py", line 44, in assert_ >> > raise AssertionError(msg) >> > AssertionError >> > >> > ---------------------------------------------------------------------- >> > Ran 5271 tests in 57.567s >> > >> > FAILED (KNOWNFAIL=5, SKIP=13, failures=1) >> > >> >> >> > What version of MKL is this? The bug doesn't show in Christolph's compiles > with MKL on windows, so it might be an MKL bug. Is it repeatable? > seems to be 2013.3.163 From njs at pobox.com Fri Sep 6 08:09:39 2013 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 6 Sep 2013 13:09:39 +0100 Subject: [Numpy-discussion] Weird behavior of gufuncs In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 12:09 AM, Jaime Fern?ndez del R?o wrote: > Hi all, > > I am seeing some very weird behavior on a gufunc I coded. > > It has a pretty complicated signature: > '(r,c,p),(i,j,k,n),(u,v),(d),(n,q)->(q,r,c)' > > And a single registered loop function, for types: > uint8, uint16, uint16, uintp, uint8->uint8. > > In general it performs beautifully well, returning the expected results > about x10 faster than a MATLAB version using a 12 core CPU (!) But today I > found out that for certain sizes of the inputs, it doesn't even call the > loop function and returns a zeroed output array immediately. > > I am using Python 2.7.5 and Numpy 1.7.1 under Windows 7, and have seen the > behavior in both x64 and x86 versions. The thresholds vary, and I don't > fully understand the logic behind it, but there is a clear threshold of > sizes: keeping all other inputs the same, one byte more in the other > argument and it returns all zeros, one byte less and everything works fine. > The thresholds are different for 64 and 32 bit versions. > > I have a strong suspicion that this may be a NumPy 1.7 bug, because a > colleague running MacOS and NumPy 1.8 compiled from the development branch > isn't seeing any of this issues I have. It could of course be a Windows > thing, but somehow that seems less likely, especially since there are other > bugs in gufuncs that have been fixed in 1.8 but haven't been backported to > 1.7. Tomorrow I am going to try and get access to a Linux box with Numpy 1.7 > to try to reproduce it there. > > Does this sound familiar to anyone? Any known (and hopefully solved) issue > that could trigger this behavior? A search for ufunc in github's issue > tracker didn't give me any meaningful leads... Doesn't ring any bell directly, but as you note there are definitely some major gufunc bug fixes in 1.8 and master, so your best bet on tracking it down might be to (a) check whether it occurs in master, (b) if not, bisect it. -n From joseluismietta at yahoo.com.ar Fri Sep 6 10:45:04 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Fri, 6 Sep 2013 07:45:04 -0700 (PDT) Subject: [Numpy-discussion] Networkx graph - numpy array - networkx.has_path() function In-Reply-To: References: Message-ID: <1378478704.36423.YahooMailNeo@web142302.mail.bf1.yahoo.com> Hi experts! I wanna use networkx.has_path(). This is applies, necesary, to a networkx graph. I have a adjacency matrix of a undirected graph (M, wich is a numpy matrix (array of N x N elements)). How can I do for use M in? networkx.has_path()? If I must transform M into a networkx graph: how can I do that? Waiting for your answers. Thanks a lot!! Jos? Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: From bergstrj at iro.umontreal.ca Fri Sep 6 12:19:08 2013 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Fri, 6 Sep 2013 12:19:08 -0400 Subject: [Numpy-discussion] Funny business with 'is' operator? Message-ID: Hi, could someone help me understand why this assertion fails? def test_is(self): a = np.empty(1) b = np.empty(1) if a.data is not b.data: assert id(a.data) != id(b.data) # <-- fail I'm trying to write an alternate may_share_memory function. Thanks, - James -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 6 12:28:42 2013 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 6 Sep 2013 17:28:42 +0100 Subject: [Numpy-discussion] Funny business with 'is' operator?: Message-ID: The .data attribute is generated on the fly when accessed. So it returns an anonymous temporary that's deallocated as soon as it's no longer needed. a.data is b.data needs both objects, so both get allocated and then compared. In the second one though, each object gets allocated one at a time and then immediately released. So the a.data and b.data objects are different... but since they have non-overlapping lifetimes, they happen to be out at the same memory location. id() is only guaranteed to be unique while the object is alive. Notice also: a.data is a.data -> False -n On 6 Sep 2013 17:21, "James Bergstra" wrote: > Hi, could someone help me understand why this assertion fails? > > def test_is(self): > a = np.empty(1) > b = np.empty(1) > if a.data is not b.data: > assert id(a.data) != id(b.data) # <-- fail > > I'm trying to write an alternate may_share_memory function. > > Thanks, > - James > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Sep 6 12:30:34 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 6 Sep 2013 09:30:34 -0700 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 9:19 AM, James Bergstra wrote: > def test_is(self): > a = np.empty(1) > b = np.empty(1) > if a.data is not b.data: > assert id(a.data) != id(b.data) # <-- fail > > I'm not familiar with the internals, but: In [27]: a = np.empty(1) In [28]: a.data Out[28]: so .data is a buffer object, wrapped around a particular pointer. In [29]: id(a.data) Out[29]: 55581376 In [30]: id(a.data) Out[30]: 55581728 but it looks like the buffer object itself is created on the fly -- so you are getting a different pyton object, even though it wraps the same data pointer. So you need to compare the value of the pointer compared, not the identity of the buffer object. Not sure how to get that value though, other that parsing __repr__ In [44]: a.data.__repr__().split()[-1][:-1] Out[44]: '0x3512100' ugly hack -- there must be a better way! But: In [38]: a = np.zeros((100,)) In [39]: b = a[50:] In [40]: a.data Out[40]: In [41]: b.data Out[41]: a and b share memory, but don't have the same data pointer as b's is offset. HTH, -Chris > I'm trying to write an alternate may_share_memory function. > > Thanks, > - James > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 6 12:32:12 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Sep 2013 10:32:12 -0600 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra wrote: > Hi, could someone help me understand why this assertion fails? > > def test_is(self): > a = np.empty(1) > b = np.empty(1) > if a.data is not b.data: > assert id(a.data) != id(b.data) # <-- fail > > I'm trying to write an alternate may_share_memory function. > > id seems not as advertised: In [22]: for i in range(10): print id(a.data) 66094640 66095792 66095792 66095792 66095792 66095792 66094640 66094640 66094640 66094640 Not sure why. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bergstrj at iro.umontreal.ca Fri Sep 6 12:48:55 2013 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Fri, 6 Sep 2013 12:48:55 -0400 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: Thanks for the tips! FWIW my guess is that since '.data' is dynamically generated property rather than an attribute, it is being freed and re-allocated in the loop, and once for each of my id() expressions. On Fri, Sep 6, 2013 at 12:32 PM, Charles R Harris wrote: > > > > On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra > wrote: > >> Hi, could someone help me understand why this assertion fails? >> >> def test_is(self): >> a = np.empty(1) >> b = np.empty(1) >> if a.data is not b.data: >> assert id(a.data) != id(b.data) # <-- fail >> >> I'm trying to write an alternate may_share_memory function. >> >> > id seems not as advertised: > > In [22]: for i in range(10): print id(a.data) > 66094640 > 66095792 > 66095792 > 66095792 > 66095792 > 66095792 > 66094640 > 66094640 > 66094640 > 66094640 > > Not sure why. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- http://www-etud.iro.umontreal.ca/~bergstrj -------------- next part -------------- An HTML attachment was scrubbed... URL: From bergstrj at iro.umontreal.ca Fri Sep 6 12:58:31 2013 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Fri, 6 Sep 2013 12:58:31 -0400 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: I'm stumped. I can't figure out how to extract from e.g. view = A[:, 3] that the view starts at element 3 of A. I was planning to make a may_share_memory implementation based on the idea of swapping in a buffer of 0s, and using the shapes, strides, itemsize etc. to increment just the parts of the 0s buffer touched by the two ndarrays. If there are any 2s in the incremented buffer, it's because the two views overlapped. It's not the best algorithm for comparing tiny views of huge arrays, I was wondering if in my case it would have been quicker than the built-in method (I don't know how it works). I actually have another data structure around to pull out that shape and stride info, but it's a shame to use it, because then I can't use the algorithm to compare ndarrays in general (or at least ones that have been created by typical construction methods and slicing). On Fri, Sep 6, 2013 at 12:48 PM, James Bergstra wrote: > Thanks for the tips! FWIW my guess is that since '.data' is dynamically > generated property rather than an attribute, it is being freed and > re-allocated in the loop, and once for each of my id() expressions. > > > On Fri, Sep 6, 2013 at 12:32 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Fri, Sep 6, 2013 at 10:19 AM, James Bergstra < >> bergstrj at iro.umontreal.ca> wrote: >> >>> Hi, could someone help me understand why this assertion fails? >>> >>> def test_is(self): >>> a = np.empty(1) >>> b = np.empty(1) >>> if a.data is not b.data: >>> assert id(a.data) != id(b.data) # <-- fail >>> >>> I'm trying to write an alternate may_share_memory function. >>> >>> >> id seems not as advertised: >> >> In [22]: for i in range(10): print id(a.data) >> 66094640 >> 66095792 >> 66095792 >> 66095792 >> 66095792 >> 66095792 >> 66094640 >> 66094640 >> 66094640 >> 66094640 >> >> Not sure why. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > http://www-etud.iro.umontreal.ca/~bergstrj > -- http://www-etud.iro.umontreal.ca/~bergstrj -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Fri Sep 6 13:06:23 2013 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Fri, 6 Sep 2013 13:06:23 -0400 Subject: [Numpy-discussion] Join and stacking for structured arrays Message-ID: For the astropy Table class (which wraps numpy structured arrays), I wrote functions that perform table joins and concatenate tables along rows or columns. These are reasonably full-featured and handle most of the common needs for these operations. The join function here addresses some limitations in the lib.recfunctions.join_by() function. You can see the documentation for the Table interface to these functions at [1]. The reason I'm writing is that the implementation is split into a pure numpy-based module (no astropy dependence [5]) and the astropy bit. Essentially all of the real work is done in the numpy-based module, so I'm wondering if the numpy project is interested in this code for inclusion into numpy. If so, I'd be happy to put together a pull request. I would just need direction on: - Where to put it - What function names to use - How to setup for included Cython code The Python and Cython code and tests can be see at [2], [3], [4]. The tests are currently based on astropy Tables, so these would be modified to use plain ndarray or masked arrays. Cheers, Tom [1]: http://astropy.readthedocs.org/en/latest/table/operations.html [2]: https://github.com/astropy/astropy/blob/master/astropy/table/np_utils.py [3]: https://github.com/astropy/astropy/blob/master/astropy/table/_np_utils.pyx [4]: https://github.com/astropy/astropy/blob/master/astropy/table/tests/test_operations.py [5]: There is a requirement for an OrderedDict which is currently provided by astropy for python 2.6. From robert.kern at gmail.com Fri Sep 6 13:27:51 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 6 Sep 2013 18:27:51 +0100 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 5:58 PM, James Bergstra wrote: > > I'm stumped. I can't figure out how to extract from e.g. > > view = A[:, 3] > > that the view starts at element 3 of A. I was planning to make a may_share_memory implementation based on the idea of swapping in a buffer of 0s, and using the shapes, strides, itemsize etc. to increment just the parts of the 0s buffer touched by the two ndarrays. If there are any 2s in the incremented buffer, it's because the two views overlapped. It's not the best algorithm for comparing tiny views of huge arrays, I was wondering if in my case it would have been quicker than the built-in method (I don't know how it works). It certainly won't be faster. may_share_memory() is very simplistic. It just checks if the outermost bounds of the each array overlap. Thus, it can give false positives for arrays that are interleaved but do not actually intersect. This is why it is named may_share_memory() instead of does_share_memory(). > I actually have another data structure around to pull out that shape and stride info, but it's a shame to use it, because then I can't use the algorithm to compare ndarrays in general (or at least ones that have been created by typical construction methods and slicing). All of the information you need is stored in the __array_interface__ attribute of an ndarray. The right way to solve this is to solve a bounded, linear Diophantine equation. That's where you should be looking if you want to crack this problem. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From bergstrj at iro.umontreal.ca Fri Sep 6 13:50:48 2013 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Fri, 6 Sep 2013 13:50:48 -0400 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: Thanks, this is exactly what I was looking for. I'll look into what this Diophantine equation is. Also, relatedly, a few months ago Julian Taylor at least wrote what was there in C, which made it faster, if not better. - James On Fri, Sep 6, 2013 at 1:27 PM, Robert Kern wrote: > On Fri, Sep 6, 2013 at 5:58 PM, James Bergstra > wrote: > > > > I'm stumped. I can't figure out how to extract from e.g. > > > > view = A[:, 3] > > > > that the view starts at element 3 of A. I was planning to make a > may_share_memory implementation based on the idea of swapping in a buffer > of 0s, and using the shapes, strides, itemsize etc. to increment just the > parts of the 0s buffer touched by the two ndarrays. If there are any 2s in > the incremented buffer, it's because the two views overlapped. It's not the > best algorithm for comparing tiny views of huge arrays, I was wondering if > in my case it would have been quicker than the built-in method (I don't > know how it works). > > It certainly won't be faster. may_share_memory() is very simplistic. It > just checks if the outermost bounds of the each array overlap. Thus, it can > give false positives for arrays that are interleaved but do not actually > intersect. This is why it is named may_share_memory() instead of > does_share_memory(). > > > I actually have another data structure around to pull out that shape and > stride info, but it's a shame to use it, because then I can't use the > algorithm to compare ndarrays in general (or at least ones that have been > created by typical construction methods and slicing). > > All of the information you need is stored in the __array_interface__ > attribute of an ndarray. > > The right way to solve this is to solve a bounded, linear Diophantine > equation. That's where you should be looking if you want to crack this > problem. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- http://www-etud.iro.umontreal.ca/~bergstrj -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at onerussian.com Fri Sep 6 15:21:24 2013 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 6 Sep 2013 15:21:24 -0400 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: <20130506143241.GV5140@onerussian.com> References: <20130506143241.GV5140@onerussian.com> Message-ID: <20130906192124.GR27621@onerussian.com> FWIW -- updated runs of the benchmarks are available at http://yarikoptic.github.io/numpy-vbench which now include also maintenance/1.8.x branch (no divergences were detected yet). There are only recent improvements as I see and no new (but some old ones are still there, some might be specific to my CPU here) performance regressions. Cheers, -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From davidmenhur at gmail.com Fri Sep 6 15:34:13 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 6 Sep 2013 21:34:13 +0200 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: <20130906192124.GR27621@onerussian.com> References: <20130506143241.GV5140@onerussian.com> <20130906192124.GR27621@onerussian.com> Message-ID: On 6 September 2013 21:21, Yaroslav Halchenko wrote: > some old ones are > still there, some might be specific to my CPU here > How long does one run take? Maybe I can run it in my machine (Intel i5) for comparison. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Sep 6 15:37:29 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Sep 2013 15:37:29 -0400 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: <20130906192124.GR27621@onerussian.com> References: <20130506143241.GV5140@onerussian.com> <20130906192124.GR27621@onerussian.com> Message-ID: On Fri, Sep 6, 2013 at 3:21 PM, Yaroslav Halchenko wrote: > FWIW -- updated runs of the benchmarks are available at > http://yarikoptic.github.io/numpy-vbench which now include also > maintenance/1.8.x branch (no divergences were detected yet). There are > only recent improvements as I see and no new (but some old ones are > still there, some might be specific to my CPU here) performance > regressions. You would have enough data to add some quality control bands to the charts (like cusum chart for example). Then it would be possible to send a congratulation note or ring an alarm bell without looking at all the plots. Josef > > Cheers, > -- > Yaroslav O. Halchenko, Ph.D. > http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org > Senior Research Associate, Psychological and Brain Sciences Dept. > Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 > Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 > WWW: http://www.linkedin.com/in/yarik > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Fri Sep 6 15:53:39 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Sep 2013 13:53:39 -0600 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: <20130906192124.GR27621@onerussian.com> References: <20130506143241.GV5140@onerussian.com> <20130906192124.GR27621@onerussian.com> Message-ID: On Fri, Sep 6, 2013 at 1:21 PM, Yaroslav Halchenko wrote: > FWIW -- updated runs of the benchmarks are available at > http://yarikoptic.github.io/numpy-vbench which now include also > maintenance/1.8.x branch (no divergences were detected yet). There are > only recent improvements as I see and no new (but some old ones are > still there, some might be specific to my CPU here) performance > regressions. > > This work is really nice. Thank you Yaroslav. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at onerussian.com Sat Sep 7 10:33:56 2013 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sat, 7 Sep 2013 10:33:56 -0400 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: References: <20130506143241.GV5140@onerussian.com> <20130906192124.GR27621@onerussian.com> Message-ID: <20130907143356.GS27621@onerussian.com> On Fri, 06 Sep 2013, Da?id wrote: > some old ones are > still there, some might be specific to my CPU here > How long does one run take? Maybe I can run it in my machine (Intel i5) > for comparison. In current configuration where I "target" benchmark run to around 200ms (thus possibly jumping up to 400ms) and thus 1-2 sec for 3 actual runs to figure out min among those -- on that elderly box it takes about a day to run "end of the day" commits (iirc around 400 of them) and then 3-4 days for a full run (all commits). I am not sure if targetting 200ms is of any benefit, as opposed to 100ms which would run twice faster. you are welcome to give it a shout right away http://github.com/yarikoptic/numpy-vbench it is still a bit ad-hoc and I also use additional shell wrapper to set cpu affinity (taskset -cp 1) and renice to -10 the benchmarking process. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From lists at onerussian.com Sat Sep 7 10:37:23 2013 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sat, 7 Sep 2013 10:37:23 -0400 Subject: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011) In-Reply-To: References: <20130506143241.GV5140@onerussian.com> <20130906192124.GR27621@onerussian.com> Message-ID: <20130907143723.GT27621@onerussian.com> On Fri, 06 Sep 2013, josef.pktd at gmail.com wrote: > On Fri, Sep 6, 2013 at 3:21 PM, Yaroslav Halchenko wrote: > > FWIW -- updated runs of the benchmarks are available at > > http://yarikoptic.github.io/numpy-vbench which now include also > > maintenance/1.8.x branch (no divergences were detected yet). There are > > only recent improvements as I see and no new (but some old ones are > > still there, some might be specific to my CPU here) performance > > regressions. > You would have enough data to add some quality control bands to the > charts (like cusum chart for example). > Then it would be possible to send a congratulation note or ring an > alarm bell without looking at all the plots. well -- I did cook up some basic "detector" but I believe I haven't adjusted it for multiple branches yet: http://yarikoptic.github.io/numpy-vbench/#benchmarks-performance-analysis you are welcome to introduce additional (or replacement) detection goodness http://github.com/yarikoptic/vbench/blob/HEAD/vbench/analysis.py and plotting is done here I believe: https://github.com/yarikoptic/vbench/blob/HEAD/vbench/benchmark.py#L155 -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From robert.kern at gmail.com Sat Sep 7 18:02:01 2013 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 7 Sep 2013 23:02:01 +0100 Subject: [Numpy-discussion] Networkx graph - numpy array - networkx.has_path() function In-Reply-To: <1378478704.36423.YahooMailNeo@web142302.mail.bf1.yahoo.com> References: <1378478704.36423.YahooMailNeo@web142302.mail.bf1.yahoo.com> Message-ID: On Fri, Sep 6, 2013 at 3:45 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > > > Hi experts! > > I wanna use networkx.has_path(). This is applies, necesary, to a networkx graph. I have a adjacency matrix of a undirected graph (M, wich is a numpy matrix (array of N x N elements)). > > How can I do for use M in networkx.has_path()? > > If I must transform M into a networkx graph: how can I do that? networkx has some nice, searchable documentation. http://networkx.github.io/documentation/latest/reference/generated/networkx.convert.from_numpy_matrix.html And its own mailing list. https://groups.google.com/forum/#!forum/networkx-discuss -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Sep 7 19:56:52 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 7 Sep 2013 17:56:52 -0600 Subject: [Numpy-discussion] PEP8 Message-ID: Hi All, I've been doing some PEP8 work using autopep8. One problem that has turned up is that the default behavior of autopep8 is version dependent. I'd like to put a script in numpy tools that runs autopep8 with some features disabled, namely 1. E226 -- puts spaces around arithmetic operators (+, -, *, /, **). 2. E241 -- allows only single spaces after ',' The first leaves expression formatting in the hands of the coder and avoids things like "2 ** 3". The second allows array entries to be vertically aligned, which can be useful in clarifying the values used in tests. A few other things that might need decisions: 1. [:,:, 2] or [:, :, 2] 2. Blank line before first function after class Foo(): The advantage of having a fixed script for PEP8 compliance is that once the code is brought into compliance it will stay there and we can recommend running the script before pushing. That will not only keep the code looking neat and regular, but will also strip trailing whitespace. Of course, the autopep8 folks might make further changes in the defaults that will cause trouble in the future, but I hope that will be a minor nuisance that we can deal with. There are options for autopep8and pep8 in case folks have other suggestions for features to add or disable. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 8 15:14:28 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Sep 2013 13:14:28 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. Message-ID: Hi all, I'm happy to announce the second beta release of Numpy 1.8.0. This release should solve the Windows problems encountered in the first beta. Many thanks to Christolph Gohlke and Julian Taylor for their hard work in getting those issues settled. It would be good if folks running OS X could try out this release and report any issues on the numpy-dev mailing list. Unfortunately the files still need to be installed from source as dmg files are not avalable at this time. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The Windows and OS X installers will follow when the infrastructure issues are dealt with. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 8 15:26:52 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Sep 2013 13:26:52 -0600 Subject: [Numpy-discussion] Removal of beta and rc files from Sourceforge Message-ID: Hi All, Currently the beta and rc files for numpy versions >= 1.6.1 are still up on sourceforge. I think at this point they are at best clutter and propose that they be removed. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Sep 8 15:29:23 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 8 Sep 2013 19:29:23 +0000 Subject: [Numpy-discussion] Removal of beta and rc files from Sourceforge In-Reply-To: References: Message-ID: On Sun, Sep 8, 2013 at 7:26 PM, Charles R Harris wrote: > > Hi All, > > Currently the beta and rc files for numpy versions >= 1.6.1 are still up > on sourceforge. I think at this point they are at best clutter and propose > that they be removed. Thoughts? > +1 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 8 16:07:44 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Sep 2013 14:07:44 -0600 Subject: [Numpy-discussion] Removal of beta and rc files from Sourceforge In-Reply-To: References: Message-ID: On Sun, Sep 8, 2013 at 1:29 PM, Ralf Gommers wrote: > > > > On Sun, Sep 8, 2013 at 7:26 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> Hi All, >> >> Currently the beta and rc files for numpy versions >= 1.6.1 are still up >> on sourceforge. I think at this point they are at best clutter and propose >> that they be removed. Thoughts? >> > > +1 > > Done. I suppose at some point we can remove some of the releases also, but that doesn't seem pressing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.j.a.cock at googlemail.com Sun Sep 8 16:24:52 2013 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sun, 8 Sep 2013 21:24:52 +0100 Subject: [Numpy-discussion] Removal of beta and rc files from Sourceforge In-Reply-To: References: Message-ID: On Sun, Sep 8, 2013 at 9:07 PM, Charles R Harris wrote: > > On Sun, Sep 8, 2013 at 1:29 PM, Ralf Gommers wrote: >> >> On Sun, Sep 8, 2013 at 7:26 PM, Charles R Harris wrote: >>> >>> Hi All, >>> >>> Currently the beta and rc files for numpy versions >= 1.6.1 are still up >>> on sourceforge. I think at this point they are at best clutter and propose >>> that they be removed. Thoughts? >> >> >> +1 >> > > Done. I suppose at some point we can remove some of the releases > also, but that doesn't seem pressing. -1 on removing old releases, sometimes they can be useful for tracing an old bug or repeating an old analysis. They still have some small value - and don't cost much to leave online. Peter From cgohlke at uci.edu Sun Sep 8 17:45:10 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 08 Sep 2013 14:45:10 -0700 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: Message-ID: <522CEFE6.2050306@uci.edu> On 9/8/2013 12:14 PM, Charles R Harris wrote: > Hi all, > > I'm happy to announce the second beta release of Numpy 1.8.0. This > release should solve the Windows problems encountered in the first beta. > Many thanks to Christolph Gohlke and Julian Taylor for their hard work > in getting those issues settled. > > It would be good if folks running OS X could try out this release and > report any issues on the numpy-dev mailing list. Unfortunately the files > still need to be installed from source as dmg files are not avalable at > this time. > > Source tarballs and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The Windows > and OS X installers will follow when the infrastructure issues are dealt > with. > > Chuck > Hello, I tested numpy 1.8.0b2 with Visual Studio and Intel MKL on Python 2.7 and 3.3 for Windows, 32 and 64 bit. There's only a single test failure on win-amd64-py3.3, which looks strange since the test expects a TypeError to be raised. ====================================================================== ERROR: test_record_no_hash (test_multiarray.TestRecord) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\numpy\core\tests\test_multiarray.py", line 2464, in test_record_no_hash self.assertRaises(TypeError, hash, a[0]) File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises return context.handle('assertRaises', callableObj, args, kwargs) File "X:\Python33\lib\unittest\case.py", line 135, in handle callable_obj(*args, **kwargs) File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ self.obj_name)) TypeError: unhashable type: 'writeable void-scalar' In case it is useful: there are a few failures in the scipy.io.matlab and scipy.fftpack modules when testing scipy 0.12.0 (built against numpy 1.7.1) with numpy 1.8.0b2. I have not looked in detail. On Python 3.3 only, 32 and 64 bit: ====================================================================== ERROR: test_mio.test_str_round ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33-x32\lib\site-packages\nose\case.py", line 198, in runTest self.test(*self.arg) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\tests\test_mio.py", line 923, in test_str_round savemat_future(stream, dict(a=in_from_str)) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\mio.py", line 270, in savemat MW.put_variables(mdict) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\mio5.py", line 866, in put_variables self._matrix_writer.write_top(var, asbytes(name), is_global) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\mio5.py", line 617, in write_top self.write(arr) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\mio5.py", line 652, in write self.write_char(narr, codec) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\mio5.py", line 698, in write_char arr = arr_to_chars(arr) File "X:\Python33-x32\lib\site-packages\scipy\io\matlab\miobase.py", line 392, in arr_to_chars buffer=arr) ValueError: ndarray is not C-contiguous On Python 2.7 only, 32 and 64 bit (this kind of error also occurs in the tests of pytables 3.3): ====================================================================== ERROR: test_mio.test_load('unicode', ['X:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testunicode_7.1 _GLNX86.mat', 'X:\\Python27\\lib\\site-packages\\scipy\\io\\matlab\\tests\\data\\testunicode_7.4_GLNX86.mat'], {'testuni code': array([ u'Japanese: \n\u3059\u3079\u3066\u306e\u4eba\u9593\u306f\u3001\u751f\u307e\u308c\u306a\u304c\u3089\u306b\ u3057\u3066\u81ea\u7531\u3067\u3042\u308a\u3001\n\u304b\u3064\u3001\u5c0a\u53b3\u3068\u6a29\u5229\u3068 \u306b\u3064\u30 44\u3066\u5e73\u7b49\u3067\u3042\u308b\u3002\n\u4eba\u9593\u306f\u3001\u7406\u6027\u3068\u826f\u5fc3\u3068\u3092\u6388\u 3051\u3089\u308c\u3066\u304a\u308a\u3001\n\u4e92\u3044\u306b\u540c\u80de\u306e\u7cbe\u795e\u3092\u3082\u3063\u3066\u884c \u52d5\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u3002'], ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\io\matlab\tests\test_mio.py", line 314, in _load_check_case matdict = loadmat(file_name, struct_as_record=True) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio.py", line 176, in loadmat matfile_dict = MR.get_variables(variable_names) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 294, in get_variables res = self.read_var_array(hdr, process) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 257, in read_var_array return self._matrix_reader.array_from_header(header, process) File "mio5_utils.pyx", line 624, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5922) File "mio5_utils.pyx", line 665, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5479) File "mio5_utils.pyx", line 822, in scipy.io.matlab.mio5_utils.VarReader5.read_char (scipy\io\matlab\mio5_utils.c:7325 ) TypeError: buffer is too small for requested array ====================================================================== ERROR: test_mio.test_round_trip('unicode_round_trip', {'testunicode': array([ u'Japanese: \n\u3059\u3079\u3066\u306e\u4e ba\u9593\u306f\u3001\u751f\u307e\u308c\u306a\u304c\u3089\u306b\u3057\u3066\u81ea\u7531\u3067\u3042\u308a\u3001\n\u304b\u 3064\u3001\u5c0a\u53b3\u3068\u6a29\u5229\u3068 \u306b\u3064\u3044\u3066\u5e73\u7b49\u3067\u3042\u308b\u3002\n\u4eba\u959 3\u306f\u3001\u7406\u6027\u3068\u826f\u5fc3\u3068\u3092\u6388\u3051\u3089\u308c\u3066\u304a\u308a\u3001\n\u4e92\u3044\u3 06b\u540c\u80de\u306e\u7cbe\u795e\u3092\u3082\u3063\u3066\u884c\u52d5\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3 044\u3002'], ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\io\matlab\tests\test_mio.py", line 344, in _rt_check_case _load_check_case(name, [mat_stream], expected) File "X:\Python27\lib\site-packages\scipy\io\matlab\tests\test_mio.py", line 314, in _load_check_case matdict = loadmat(file_name, struct_as_record=True) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio.py", line 176, in loadmat matfile_dict = MR.get_variables(variable_names) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 294, in get_variables res = self.read_var_array(hdr, process) File "X:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 257, in read_var_array return self._matrix_reader.array_from_header(header, process) File "mio5_utils.pyx", line 624, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5922) File "mio5_utils.pyx", line 665, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5479) File "mio5_utils.pyx", line 822, in scipy.io.matlab.mio5_utils.VarReader5.read_char (scipy\io\matlab\mio5_utils.c:7325 ) TypeError: buffer is too small for requested array On Python 2.7 and 3.3, 32 bit only: ====================================================================== FAIL: test_basic.TestOverwrite.test_fft ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 714, in test_fft self._check_1d(fft, dtype, (16,), -1, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 709, in _check_1d should_overwrite=should_overwrite) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 689, in _check raise AssertionError("no overwrite in %s" % sig) AssertionError: no overwrite in fft(complex128(16,), 8, axis=-1, overwrite_x=True) ====================================================================== FAIL: test_basic.TestOverwrite.test_fftn ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 793, in test_fftn self._check_nd(fftn, dtype, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 776, in _check_nd self._check_nd_one(routine, dtype, (16,), None, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 768, in _check_nd_one should_overwrite=should_overwrite) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 689, in _check raise AssertionError("no overwrite in %s" % sig) AssertionError: no overwrite in fftn(complex128(16,), (8,), axis=None, overwrite_x=True) ====================================================================== FAIL: test_basic.TestOverwrite.test_ifft ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 721, in test_ifft self._check_1d(ifft, dtype, (16,), -1, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 709, in _check_1d should_overwrite=should_overwrite) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 689, in _check raise AssertionError("no overwrite in %s" % sig) AssertionError: no overwrite in ifft(complex128(16,), 8, axis=-1, overwrite_x=True) ====================================================================== FAIL: test_basic.TestOverwrite.test_ifftn ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 798, in test_ifftn self._check_nd(ifftn, dtype, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 776, in _check_nd self._check_nd_one(routine, dtype, (16,), None, overwritable) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 768, in _check_nd_one should_overwrite=should_overwrite) File "X:\Python27\lib\site-packages\scipy\fftpack\tests\test_basic.py", line 689, in _check raise AssertionError("no overwrite in %s" % sig) AssertionError: no overwrite in ifftn(complex128(16,), (8,), axis=None, overwrite_x=True) Christoph From charlesr.harris at gmail.com Sun Sep 8 20:36:06 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Sep 2013 18:36:06 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: <522CEFE6.2050306@uci.edu> References: <522CEFE6.2050306@uci.edu> Message-ID: On Sun, Sep 8, 2013 at 3:45 PM, Christoph Gohlke wrote: > On 9/8/2013 12:14 PM, Charles R Harris wrote: > > Hi all, > > > > I'm happy to announce the second beta release of Numpy 1.8.0. This > > release should solve the Windows problems encountered in the first beta. > > Many thanks to Christolph Gohlke and Julian Taylor for their hard work > > in getting those issues settled. > > > > It would be good if folks running OS X could try out this release and > > report any issues on the numpy-dev mailing list. Unfortunately the files > > still need to be installed from source as dmg files are not avalable at > > this time. > > > > Source tarballs and release notes can be found at > > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The Windows > > and OS X installers will follow when the infrastructure issues are dealt > > with. > > > > Chuck > > > > > Hello, > > I tested numpy 1.8.0b2 with Visual Studio and Intel MKL on Python 2.7 > and 3.3 for Windows, 32 and 64 bit. There's only a single test failure > on win-amd64-py3.3, which looks strange since the test expects a > TypeError to be raised. > > ====================================================================== > ERROR: test_record_no_hash (test_multiarray.TestRecord) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "X:\Python33\lib\site-packages\numpy\core\tests\test_multiarray.py", > line 2464, in test_record_no_hash > self.assertRaises(TypeError, hash, a[0]) > File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises > return context.handle('assertRaises', callableObj, args, kwargs) > File "X:\Python33\lib\unittest\case.py", line 135, in handle > callable_obj(*args, **kwargs) > File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ > self.obj_name)) > TypeError: unhashable type: 'writeable void-scalar' > > > Hmm, that *is* strange. I don't know what to make of the scipy errors at first glance. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 8 20:55:25 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Sep 2013 18:55:25 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Sun, Sep 8, 2013 at 6:36 PM, Charles R Harris wrote: > > > > On Sun, Sep 8, 2013 at 3:45 PM, Christoph Gohlke wrote: > >> On 9/8/2013 12:14 PM, Charles R Harris wrote: >> > Hi all, >> > >> > I'm happy to announce the second beta release of Numpy 1.8.0. This >> > release should solve the Windows problems encountered in the first beta. >> > Many thanks to Christolph Gohlke and Julian Taylor for their hard work >> > in getting those issues settled. >> > >> > It would be good if folks running OS X could try out this release and >> > report any issues on the numpy-dev mailing list. Unfortunately the files >> > still need to be installed from source as dmg files are not avalable at >> > this time. >> > >> > Source tarballs and release notes can be found at >> > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The >> Windows >> > and OS X installers will follow when the infrastructure issues are dealt >> > with. >> > >> > Chuck >> > >> >> >> Hello, >> >> I tested numpy 1.8.0b2 with Visual Studio and Intel MKL on Python 2.7 >> and 3.3 for Windows, 32 and 64 bit. There's only a single test failure >> on win-amd64-py3.3, which looks strange since the test expects a >> TypeError to be raised. >> >> ====================================================================== >> ERROR: test_record_no_hash (test_multiarray.TestRecord) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "X:\Python33\lib\site-packages\numpy\core\tests\test_multiarray.py", >> line 2464, in test_record_no_hash >> self.assertRaises(TypeError, hash, a[0]) >> File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises >> return context.handle('assertRaises', callableObj, args, kwargs) >> File "X:\Python33\lib\unittest\case.py", line 135, in handle >> callable_obj(*args, **kwargs) >> File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ >> self.obj_name)) >> TypeError: unhashable type: 'writeable void-scalar' >> >> >> > Hmm, that *is* strange. > > I don't know what to make of the scipy errors at first glance. > > > > I'm going to guess self.assertRaises tried to hash it again, raising the error again, and we see the second one. The assertRaises from numpy.testing might not have that problem. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From grb at skogoglandskap.no Mon Sep 9 03:40:56 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Mon, 9 Sep 2013 07:40:56 +0000 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 84, Issue 19 In-Reply-To: References: Message-ID: <8B623060-BF0D-4B24-B8B8-ECE66147A6EA@skogoglandskap.no> Hi again Julian, > its not np.any that is slow in this case its np.array([A, B, C]) Not really. The 4 unique alternative implementations provided all work quickly and correctly as-is, without a need to jump through a strange idiomatic hoop, so the problem does indeed lie with the implementation approach used in the current numpy.any(). It is unreasonable to expect users to know and always prepare their data using the idiom you propose if they might want to use the .any() function. If quirky idioms are needed to fix the performance of .any(), they should be placed in the implementation of .any(). > this has the same performance as your code: > a = empty([3] list(A.shape) How do you know? This isn't valid Python, it won't run. Please can you send me a working version I can benchmark and compare? Graeme. On Sep 5, 2013, at 8:52 PM, wrote: > > hi, > its not np.any that is slow in this case its np.array([A, B, C]) > > np.dstack([A, B, C]) is better but writing it like this has the same > performance as your code: > a = empty([3] list(A.shape) > a[0] = A>5; a[1] = B<2; a[2] = A>10; > np.any(a, 0) > > I'll check if creating an array from a sequence can be improved for this > case. From grb at skogoglandskap.no Mon Sep 9 04:56:27 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Mon, 9 Sep 2013 08:56:27 +0000 Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 84, Issue 18 In-Reply-To: References: Message-ID: <714AB97D-4E08-4A04-9AE0-7FD69ABE508F@skogoglandskap.no> > > This is good stuff, but I can't help thinking that if I needed to do an > any/all test on a number of arrays with common and/or combos -- > I'd probably write a Cython function to do it. Hi Chris, I agree, if you want the best performance and have no constraints on implementation, then using something like cython might be the way to do it. However, not everyone has the option of using cython. I'm using a GIS program which accepts a single numpy expression and performs it across every cell. What I'm doing has to a) be compatible with the standard release b) should 'just work' for all users, not just me and preferably c) shouldn't be complicated to implement/test. I don't know if GDAL and SWIG will work out of the box with a hybrid numpy/Cython approach. From what I've read, I believe it probably won't be practical for most users, particularly on Windows. Whereas python based or core numpy improvements are drop-in upgrades to performance. I suppose this leads to a more general point that using cython instead of improving numpy performance during projects, is locally good, but globally bad, rather like the situation with idiomatic workarounds for performance as opposed to fixing the implementation. Graeme. From robert.kern at gmail.com Mon Sep 9 08:10:42 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 9 Sep 2013 13:10:42 +0100 Subject: [Numpy-discussion] Funny business with 'is' operator? In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 6:50 PM, James Bergstra wrote: > > Thanks, this is exactly what I was looking for. I'll look into what this Diophantine equation is. Let's say we have two arrays with shape tuples `shape0` and `shape1`, stride tuples `stride0` and `stride1` and memory offsets `offset0` and `offset1`. Without loss of generality, let's assume that itemsize=1 since it is trivial to convert the general case to itemsize=1. Now, if you will permit Einstein summation notation, you can generate the memory address of every item in the first array like so: index0[i]*stride0[i] + offset0 0 <= i < len(shape0) 0 <= index0[i] < shape0[i] There exists an overlap between the two arrays iff there exists two tuples `index0` and `index1` such that index0[i]*stride0[i] + offset0 = index1[j]*stride0[j] + offset1 0 <= i < len(shape0) 0 <= index0[i] < shape0[i] 0 <= j < len(shape1) 0 <= index1[j] < shape1[j] This is a bounded linear Diophantine equation. Diophantine because the variables `index0[i]` and `index1[j]` are integer-valued, linear because the variables only appear multiplied by a constant, and bounded because each variable must stay within the size of its corresponding axis. Unbounded linear Diophantine equations can be solved using an extended version of the Euclidean GCD algorithm. Bounded linear Diophantine equations are NP- to solve. With the right heuristics, a branch-and-bound approach could probably work well. > Also, relatedly, a few months ago Julian Taylor at least wrote what was there in C, which made it faster, if not better. The main benefit of that was to make it available at the C level for other C-implemented routines, IIRC, not speed. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Sep 9 09:43:24 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Sep 2013 09:43:24 -0400 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. References: Message-ID: Charles R Harris wrote: > Hi all, > > I'm happy to announce the second beta release of Numpy 1.8.0. This release > should solve the Windows problems encountered in the first beta. Many > thanks to Christolph Gohlke and Julian Taylor for their hard work in > getting those issues settled. > > It would be good if folks running OS X could try out this release and > report any issues on the numpy-dev mailing list. Unfortunately the files > still need to be installed from source as dmg files are not avalable at > this time. > > Source tarballs and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The Windows > and OS X installers will follow when the infrastructure issues are dealt > with. > > Chuck Fedora 19 linux x86_64 mkl Package ID: l_mkl_11.0.3.163 MKL ERROR: Parameter 4 was incorrect on entry to DGETRF. ... FAIL: test_linalg.test_xerbla ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla assert_(False) File "/home/nbecker/.local/lib/python2.7/site- packages/numpy/testing/utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError ---------------------------------------------------------------------- Ran 5271 tests in 56.843s FAILED (KNOWNFAIL=5, SKIP=13, failures=1) >>> From cckidder at gmail.com Mon Sep 9 09:46:13 2013 From: cckidder at gmail.com (Chad Kidder) Date: Mon, 9 Sep 2013 07:46:13 -0600 Subject: [Numpy-discussion] Bug (?) converting list to array Message-ID: I'm trying to enter a 2-D array and np.array() is returning a 1-D array of lists. I'm using Python (x,y) on Windows 7 with numpy 1.7.1. Here's the code that is giving me issues. >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, -62, -70, -72, -73.5, -77]] >>> f1a = np.array(f1) >>> f1a array([[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, -62, -70, -72, -73.5, -77]], dtype=object) What am I missing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Mon Sep 9 09:46:18 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 9 Sep 2013 09:46:18 -0400 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: Hi, I checkout the dev version of numpy and it fail to compile with this error: creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' gcc: numpy/core/src/multiarray/sequence.c gcc: numpy/core/src/multiarray/descriptor.c gcc: numpy/core/src/multiarray/getset.c gcc: numpy/core/src/multiarray/arrayobject.c gcc: numpy/core/src/multiarray/methods.c numpy/core/src/multiarray/methods.c: In function ?array_partition?: numpy/core/src/multiarray/methods.c:1199:38: error: ?PyArray_SelectkindConverter? undeclared (first use in this function) numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared identifier is reported only once for each function it appears in numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: numpy/core/src/multiarray/methods.c:1316:38: error: ?PyArray_SelectkindConverter? undeclared (first use in this function) numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes pointer from integer without a cast numpy/core/src/multiarray/methods.c: In function ?array_partition?: numpy/core/src/multiarray/methods.c:1199:38: error: ?PyArray_SelectkindConverter? undeclared (first use in this function) numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared identifier is reported only once for each function it appears in numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: numpy/core/src/multiarray/methods.c:1316:38: error: ?PyArray_SelectkindConverter? undeclared (first use in this function) numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes pointer from integer without a cast error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c numpy/core/src/multiarray/methods.c -o build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed with exit status 1 PyArray_SelectkindConverter is defined in numpy/core/src/multiarray/conversion_utils.c. methods.c include conversion_utils.h, but there is no fct declaration of this fct in this file. Is that normal? Fred On Sun, Sep 8, 2013 at 8:55 PM, Charles R Harris wrote: > > > > On Sun, Sep 8, 2013 at 6:36 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Sun, Sep 8, 2013 at 3:45 PM, Christoph Gohlke wrote: >> >>> On 9/8/2013 12:14 PM, Charles R Harris wrote: >>> > Hi all, >>> > >>> > I'm happy to announce the second beta release of Numpy 1.8.0. This >>> > release should solve the Windows problems encountered in the first >>> beta. >>> > Many thanks to Christolph Gohlke and Julian Taylor for their hard work >>> > in getting those issues settled. >>> > >>> > It would be good if folks running OS X could try out this release and >>> > report any issues on the numpy-dev mailing list. Unfortunately the >>> files >>> > still need to be installed from source as dmg files are not avalable at >>> > this time. >>> > >>> > Source tarballs and release notes can be found at >>> > https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b2/. The >>> Windows >>> > and OS X installers will follow when the infrastructure issues are >>> dealt >>> > with. >>> > >>> > Chuck >>> > >>> >>> >>> Hello, >>> >>> I tested numpy 1.8.0b2 with Visual Studio and Intel MKL on Python 2.7 >>> and 3.3 for Windows, 32 and 64 bit. There's only a single test failure >>> on win-amd64-py3.3, which looks strange since the test expects a >>> TypeError to be raised. >>> >>> ====================================================================== >>> ERROR: test_record_no_hash (test_multiarray.TestRecord) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File >>> "X:\Python33\lib\site-packages\numpy\core\tests\test_multiarray.py", >>> line 2464, in test_record_no_hash >>> self.assertRaises(TypeError, hash, a[0]) >>> File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises >>> return context.handle('assertRaises', callableObj, args, kwargs) >>> File "X:\Python33\lib\unittest\case.py", line 135, in handle >>> callable_obj(*args, **kwargs) >>> File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ >>> self.obj_name)) >>> TypeError: unhashable type: 'writeable void-scalar' >>> >>> >>> >> Hmm, that *is* strange. >> >> I don't know what to make of the scipy errors at first glance. >> >> >> >> > I'm going to guess self.assertRaises tried to hash it again, raising the > error again, and we see the second one. The assertRaises from numpy.testing > might not have that problem. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Sep 9 09:50:39 2013 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 9 Sep 2013 09:50:39 -0400 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: The two lists are of different sizes. Had to count twice to catch that. Ben Root On Mon, Sep 9, 2013 at 9:46 AM, Chad Kidder wrote: > I'm trying to enter a 2-D array and np.array() is returning a 1-D array of > lists. I'm using Python (x,y) on Windows 7 with numpy 1.7.1. Here's the > code that is giving me issues. > > >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, > -62, -70, -72, -73.5, -77]] > >>> f1a = np.array(f1) > >>> f1a > array([[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], > [-45, -57, -62, -70, -72, -73.5, -77]], dtype=object) > > What am I missing? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 9 09:52:39 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 9 Sep 2013 14:52:39 +0100 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: One list has 6 entries and one has 7, so they can't be aligned into a single array. Possibly it would be better to raise an error here instead of returning an object array, but that's what's going on. -n On 9 Sep 2013 14:49, "Chad Kidder" wrote: > I'm trying to enter a 2-D array and np.array() is returning a 1-D array of > lists. I'm using Python (x,y) on Windows 7 with numpy 1.7.1. Here's the > code that is giving me issues. > > >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, > -62, -70, -72, -73.5, -77]] > >>> f1a = np.array(f1) > >>> f1a > array([[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], > [-45, -57, -62, -70, -72, -73.5, -77]], dtype=object) > > What am I missing? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cckidder at gmail.com Mon Sep 9 09:54:40 2013 From: cckidder at gmail.com (Chad Kidder) Date: Mon, 9 Sep 2013 07:54:40 -0600 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: Oh, so there was a bug in the user... On Mon, Sep 9, 2013 at 7:52 AM, Nathaniel Smith wrote: > One list has 6 entries and one has 7, so they can't be aligned into a > single array. Possibly it would be better to raise an error here instead of > returning an object array, but that's what's going on. > > -n > On 9 Sep 2013 14:49, "Chad Kidder" wrote: > >> I'm trying to enter a 2-D array and np.array() is returning a 1-D array >> of lists. I'm using Python (x,y) on Windows 7 with numpy 1.7.1. Here's >> the code that is giving me issues. >> >> >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, >> -62, -70, -72, -73.5, -77]] >> >>> f1a = np.array(f1) >> >>> f1a >> array([[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], >> [-45, -57, -62, -70, -72, -73.5, -77]], dtype=object) >> >> What am I missing? >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Mon Sep 9 09:56:47 2013 From: ben.root at ou.edu (Benjamin Root) Date: Mon, 9 Sep 2013 09:56:47 -0400 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: On Sat, Sep 7, 2013 at 7:56 PM, Charles R Harris wrote: > Hi All, > > I've been doing some PEP8 work using autopep8. One problem that has turned > up is that the default behavior of autopep8 is version dependent. I'd like > to put a script in numpy tools that runs autopep8 with some features > disabled, namely > > 1. E226 -- puts spaces around arithmetic operators (+, -, *, /, **). > 2. E241 -- allows only single spaces after ',' > > Something we have done in matplotlib is that we have made PEP8 a part of the tests. We are transitioning, but the idea is that eventually, with Travis, all pull requests will get PEP8 checked. I am very leary of automatic PEP8-ing. I would rather have the tests fail and let me manually fix it rather than have code automatically changed. The first leaves expression formatting in the hands of the coder and avoids > things like "2 ** 3". The second allows array entries to be vertically > aligned, which can be useful in clarifying the values used in tests. A few > other things that might need decisions: > > 1. [:,:, 2] or [:, :, 2] > 2. Blank line before first function after class Foo(): > > For the first one, I prefer spaces. For the second one, I prefer no blank lines. Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhattersley at gmail.com Mon Sep 9 10:12:15 2013 From: rhattersley at gmail.com (Richard Hattersley) Date: Mon, 9 Sep 2013 15:12:15 +0100 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: > Something we have done in matplotlib is that we have made PEP8 a part of the tests. In Iris and Cartopy we've also done this and it works well. While we transition we have an exclusion list (which is gradually getting shorter). We've had mixed experiences with automatic reformatting, so prefer to keep the human in the loop. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 9 10:29:33 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 08:29:33 -0600 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: On Mon, Sep 9, 2013 at 8:12 AM, Richard Hattersley wrote: > > Something we have done in matplotlib is that we have made PEP8 a part of > the tests. > > In Iris and Cartopy we've also done this and it works well. While we > transition we have an exclusion list (which is gradually getting shorter). > We've had mixed experiences with automatic reformatting, so prefer to keep > the human in the loop. > > I agree with keeping a human in the loop, the script would be intended to get things into the right neighborhood, the submitter would have to review the changes after. If the script isn't too strict there will be than one way to do some things and those bits would rely on the good taste of the coder. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Sep 9 10:45:37 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Sep 2013 10:45:37 -0400 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: On Mon, Sep 9, 2013 at 9:52 AM, Nathaniel Smith wrote: > One list has 6 entries and one has 7, so they can't be aligned into a single > array. Possibly it would be better to raise an error here instead of > returning an object array, but that's what's going on. It did at some point (and I relied on the exception to catch bugs, since I'm still using mainly numpy 1.5) >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, -62, -70, -72, -73.5, -77]] >>> np.array(f1) Traceback (most recent call last): File "", line 1, in ValueError: setting an array element with a sequence. >>> np.__version__ '1.5.1' now we get object arrays (in scipy.stats, and I didn't know what to do with them) I don't remember any discussion on this. Josef > > -n > > On 9 Sep 2013 14:49, "Chad Kidder" wrote: >> >> I'm trying to enter a 2-D array and np.array() is returning a 1-D array of >> lists. I'm using Python (x,y) on Windows 7 with numpy 1.7.1. Here's the >> code that is giving me issues. >> >> >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, >> >>> -62, -70, -72, -73.5, -77]] >> >>> f1a = np.array(f1) >> >>> f1a >> array([[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], >> [-45, -57, -62, -70, -72, -73.5, -77]], dtype=object) >> >> What am I missing? >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Sep 9 10:51:02 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 08:51:02 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien wrote: > Hi, > > I checkout the dev version of numpy and it fail to compile with this error: > > creating > build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray > > compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/src/npysort -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' > > gcc: numpy/core/src/multiarray/sequence.c > > gcc: numpy/core/src/multiarray/descriptor.c > > gcc: numpy/core/src/multiarray/getset.c > > gcc: numpy/core/src/multiarray/arrayobject.c > > gcc: numpy/core/src/multiarray/methods.c > > numpy/core/src/multiarray/methods.c: In function ?array_partition?: > > numpy/core/src/multiarray/methods.c:1199:38: error: > ?PyArray_SelectkindConverter? undeclared (first use in this function) > > numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared > identifier is reported only once for each function it appears in > > numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: > > numpy/core/src/multiarray/methods.c:1316:38: error: > ?PyArray_SelectkindConverter? undeclared (first use in this function) > > numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes > pointer from integer without a cast > > numpy/core/src/multiarray/methods.c: In function ?array_partition?: > > numpy/core/src/multiarray/methods.c:1199:38: error: > ?PyArray_SelectkindConverter? undeclared (first use in this function) > > numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared > identifier is reported only once for each function it appears in > > numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: > > numpy/core/src/multiarray/methods.c:1316:38: error: > ?PyArray_SelectkindConverter? undeclared (first use in this function) > > numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes > pointer from integer without a cast > > error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 > -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/src/npysort -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c > numpy/core/src/multiarray/methods.c -o > build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed > with exit status 1 > > > PyArray_SelectkindConverter is defined in > numpy/core/src/multiarray/conversion_utils.c. methods.c include > conversion_utils.h, but there is no fct declaration of this fct in this > file. Is that normal? > > No, it looks like a bug. What is strange is that it doesn't show up on my machine. What compiler flags are you using? Could you make a PR for this? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 9 11:02:39 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 09:02:39 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 8:51 AM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien wrote: > >> Hi, >> >> I checkout the dev version of numpy and it fail to compile with this >> error: >> >> creating >> build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray >> >> compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >> -Inumpy/core/src/npysort -Inumpy/core/include >> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' >> >> gcc: numpy/core/src/multiarray/sequence.c >> >> gcc: numpy/core/src/multiarray/descriptor.c >> >> gcc: numpy/core/src/multiarray/getset.c >> >> gcc: numpy/core/src/multiarray/arrayobject.c >> >> gcc: numpy/core/src/multiarray/methods.c >> >> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >> >> numpy/core/src/multiarray/methods.c:1199:38: error: >> ?PyArray_SelectkindConverter? undeclared (first use in this function) >> >> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >> identifier is reported only once for each function it appears in >> >> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >> >> numpy/core/src/multiarray/methods.c:1316:38: error: >> ?PyArray_SelectkindConverter? undeclared (first use in this function) >> >> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >> pointer from integer without a cast >> >> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >> >> numpy/core/src/multiarray/methods.c:1199:38: error: >> ?PyArray_SelectkindConverter? undeclared (first use in this function) >> >> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >> identifier is reported only once for each function it appears in >> >> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >> >> numpy/core/src/multiarray/methods.c:1316:38: error: >> ?PyArray_SelectkindConverter? undeclared (first use in this function) >> >> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >> pointer from integer without a cast >> >> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >> -Inumpy/core/src/npysort -Inumpy/core/include >> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >> numpy/core/src/multiarray/methods.c -o >> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed >> with exit status 1 >> >> >> PyArray_SelectkindConverter is defined in >> numpy/core/src/multiarray/conversion_utils.c. methods.c include >> conversion_utils.h, but there is no fct declaration of this fct in this >> file. Is that normal? >> >> > No, it looks like a bug. What is strange is that it doesn't show up on my > machine. What compiler flags are you using? > > Could you make a PR for this? > > > > Wait a minute, it is in the API. Try a clean build and see what happens. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Mon Sep 9 11:33:35 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 9 Sep 2013 11:33:35 -0400 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: I tried it and retried and it still fail. This is in an virtualenv $git show commit c9b06111227f7a4ec213571f97e1b8d19b9c23f5 Merge: 73fbfb2 8edccea Author: Charles Harris Date: Sun Sep 8 19:47:21 2013 -0700 Merge pull request #3701 from cgohlke/patch-2 ENH: add support for Python 3.4 ast.NameConstant $rm -rf build ## Fail as there is no such directory $pip install . # fail with the same error $pip uninstall numpy $python setup.py install --user # fail with the same error $pip install . ## fail with the same error: $git grep PyArray_SelectkindConverter doc/release/1.8.0-notes.rst:* PyArray_SelectkindConverter numpy/core/code_generators/numpy_api.py: 'PyArray_SelectkindConverter': 298, numpy/core/src/multiarray/conversion_utils.c:PyArray_SelectkindConverter(PyObject *obj, NPY_SELECTKIND *selectkind) numpy/core/src/multiarray/methods.c: PyArray_SelectkindConverter, &sortkind, numpy/core/src/multiarray/methods.c: PyArray_SelectkindConverter, &sortkind, Here I don't see PyArray_SelectkindConverter in conversion_utils.h as you said it is present. Witch commit do you use? Fred On Mon, Sep 9, 2013 at 11:02 AM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 8:51 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien wrote: >> >>> Hi, >>> >>> I checkout the dev version of numpy and it fail to compile with this >>> error: >>> >>> creating >>> build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray >>> >>> compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>> -Inumpy/core/src/npysort -Inumpy/core/include >>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' >>> >>> gcc: numpy/core/src/multiarray/sequence.c >>> >>> gcc: numpy/core/src/multiarray/descriptor.c >>> >>> gcc: numpy/core/src/multiarray/getset.c >>> >>> gcc: numpy/core/src/multiarray/arrayobject.c >>> >>> gcc: numpy/core/src/multiarray/methods.c >>> >>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>> >>> numpy/core/src/multiarray/methods.c:1199:38: error: >>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>> >>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>> identifier is reported only once for each function it appears in >>> >>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>> >>> numpy/core/src/multiarray/methods.c:1316:38: error: >>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>> >>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>> pointer from integer without a cast >>> >>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>> >>> numpy/core/src/multiarray/methods.c:1199:38: error: >>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>> >>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>> identifier is reported only once for each function it appears in >>> >>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>> >>> numpy/core/src/multiarray/methods.c:1316:38: error: >>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>> >>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>> pointer from integer without a cast >>> >>> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >>> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>> -Inumpy/core/src/npysort -Inumpy/core/include >>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >>> numpy/core/src/multiarray/methods.c -o >>> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed >>> with exit status 1 >>> >>> >>> PyArray_SelectkindConverter is defined in >>> numpy/core/src/multiarray/conversion_utils.c. methods.c include >>> conversion_utils.h, but there is no fct declaration of this fct in this >>> file. Is that normal? >>> >>> >> No, it looks like a bug. What is strange is that it doesn't show up on my >> machine. What compiler flags are you using? >> >> Could you make a PR for this? >> >> >> >> > Wait a minute, it is in the API. Try a clean build and see what happens. > > Chuck > >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 9 11:35:17 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 9 Sep 2013 16:35:17 +0100 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: On 9 Sep 2013 15:50, wrote: > > On Mon, Sep 9, 2013 at 9:52 AM, Nathaniel Smith wrote: > > One list has 6 entries and one has 7, so they can't be aligned into a single > > array. Possibly it would be better to raise an error here instead of > > returning an object array, but that's what's going on. > > It did at some point (and I relied on the exception to catch bugs, > since I'm still using mainly numpy 1.5) > > >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, -62, -70, -72, -73.5, -77]] > >>> np.array(f1) > Traceback (most recent call last): > File "", line 1, in > ValueError: setting an array element with a sequence. > >>> np.__version__ > '1.5.1' > > now we get object arrays (in scipy.stats, and I didn't know what to do > with them) > > I don't remember any discussion on this. There may not have been any. Feel free to submit a PR and we can argue about which way is better... (I also prefer the 1.5 approach personally.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 9 11:44:43 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 09:44:43 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 9:33 AM, Fr?d?ric Bastien wrote: > I tried it and retried and it still fail. This is in an virtualenv > > $git show > commit c9b06111227f7a4ec213571f97e1b8d19b9c23f5 > Merge: 73fbfb2 8edccea > Author: Charles Harris > Date: Sun Sep 8 19:47:21 2013 -0700 > > Merge pull request #3701 from cgohlke/patch-2 > > ENH: add support for Python 3.4 ast.NameConstant > > > > $rm -rf build ## Fail as there is no such directory > $pip install . # fail with the same error > > $pip uninstall numpy > $python setup.py install --user # fail with the same error > $pip install . ## fail with the same error: > > $git grep PyArray_SelectkindConverter > doc/release/1.8.0-notes.rst:* PyArray_SelectkindConverter > numpy/core/code_generators/numpy_api.py: > 'PyArray_SelectkindConverter': 298, > numpy/core/src/multiarray/conversion_utils.c:PyArray_SelectkindConverter(PyObject > *obj, NPY_SELECTKIND *selectkind) > numpy/core/src/multiarray/methods.c: > PyArray_SelectkindConverter, &sortkind, > numpy/core/src/multiarray/methods.c: > PyArray_SelectkindConverter, &sortkind, > > > Here I don't see PyArray_SelectkindConverter in conversion_utils.h as you > said it is present. Witch commit do you use? > > It's not there, it is part of the API. I've got a PR to add it to the *.h file. The question is why you are the only person (so far) to have a problem compiling. What are your CFLAGS? Chuck > Fred > > > > On Mon, Sep 9, 2013 at 11:02 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Sep 9, 2013 at 8:51 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> >>> On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien wrote: >>> >>>> Hi, >>>> >>>> I checkout the dev version of numpy and it fail to compile with this >>>> error: >>>> >>>> creating >>>> build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray >>>> >>>> compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>>> -Inumpy/core/src/npysort -Inumpy/core/include >>>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' >>>> >>>> gcc: numpy/core/src/multiarray/sequence.c >>>> >>>> gcc: numpy/core/src/multiarray/descriptor.c >>>> >>>> gcc: numpy/core/src/multiarray/getset.c >>>> >>>> gcc: numpy/core/src/multiarray/arrayobject.c >>>> >>>> gcc: numpy/core/src/multiarray/methods.c >>>> >>>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>>> >>>> numpy/core/src/multiarray/methods.c:1199:38: error: >>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>> >>>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>>> identifier is reported only once for each function it appears in >>>> >>>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>>> >>>> numpy/core/src/multiarray/methods.c:1316:38: error: >>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>> >>>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>>> pointer from integer without a cast >>>> >>>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>>> >>>> numpy/core/src/multiarray/methods.c:1199:38: error: >>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>> >>>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>>> identifier is reported only once for each function it appears in >>>> >>>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>>> >>>> numpy/core/src/multiarray/methods.c:1316:38: error: >>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>> >>>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>>> pointer from integer without a cast >>>> >>>> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >>>> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>>> -Inumpy/core/src/npysort -Inumpy/core/include >>>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >>>> numpy/core/src/multiarray/methods.c -o >>>> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed >>>> with exit status 1 >>>> >>>> >>>> PyArray_SelectkindConverter is defined in >>>> numpy/core/src/multiarray/conversion_utils.c. methods.c include >>>> conversion_utils.h, but there is no fct declaration of this fct in this >>>> file. Is that normal? >>>> >>>> >>> No, it looks like a bug. What is strange is that it doesn't show up on >>> my machine. What compiler flags are you using? >>> >>> Could you make a PR for this? >>> >>> >>> >>> >> Wait a minute, it is in the API. Try a clean build and see what happens. >> >> Chuck >> >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Sep 9 11:52:49 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Sep 2013 11:52:49 -0400 Subject: [Numpy-discussion] Bug (?) converting list to array In-Reply-To: References: Message-ID: On Mon, Sep 9, 2013 at 11:35 AM, Nathaniel Smith wrote: > On 9 Sep 2013 15:50, wrote: >> >> On Mon, Sep 9, 2013 at 9:52 AM, Nathaniel Smith wrote: >> > One list has 6 entries and one has 7, so they can't be aligned into a >> > single >> > array. Possibly it would be better to raise an error here instead of >> > returning an object array, but that's what's going on. >> >> It did at some point (and I relied on the exception to catch bugs, >> since I'm still using mainly numpy 1.5) >> >> >>> f1 = [[15.207, 15.266, 15.181, 15.189, 15.215, 15.198], [-45, -57, >> >>> -62, -70, -72, -73.5, -77]] >> >>> np.array(f1) >> Traceback (most recent call last): >> File "", line 1, in >> ValueError: setting an array element with a sequence. >> >>> np.__version__ >> '1.5.1' >> >> now we get object arrays (in scipy.stats, and I didn't know what to do >> with them) >> >> I don't remember any discussion on this. > > There may not have been any. Isn't it too late now? > > Feel free to submit a PR and we can argue about which way is better... (I > also prefer the 1.5 approach personally.) I'm just a balcony muppet (and user) (and I lost the argument against object arrays in scipy.stats) Josef > > -n > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From njs at pobox.com Mon Sep 9 11:54:40 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 9 Sep 2013 16:54:40 +0100 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: On Mon, Sep 9, 2013 at 3:29 PM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 8:12 AM, Richard Hattersley > wrote: >> >> > Something we have done in matplotlib is that we have made PEP8 a part of >> > the tests. >> >> In Iris and Cartopy we've also done this and it works well. While we >> transition we have an exclusion list (which is gradually getting shorter). >> We've had mixed experiences with automatic reformatting, so prefer to keep >> the human in the loop. >> > > I agree with keeping a human in the loop, the script would be intended to > get things into the right neighborhood, the submitter would have to review > the changes after. If the script isn't too strict there will be than one way > to do some things and those bits would rely on the good taste of the coder. So if I understand right, the goal is to have some script that developers can run before (or after) submitting a PR, like tools/autopep8-my-changes numpy/ that will fix up their changes, but leave the rest of numpy alone? And the proposed mechanism is to come up with a combination of changes to the numpy source and an autopep8 configuration such that autopep8 --our-config numpy/ becomes a no-op, and then we can use this as an implementation of tools/autopep8-my-changes? If that's right then my feeling is that the goal seems worthwhile but the approach seems difficult and unlikely to survive for long. As soon as someone overrides autopep8 once, we either have to disable the rule for the whole project or keep overriding it manually forever. You're already suggesting taking out the spaces-around-arithmetic rule, which strikes me as one of the most useful -- sure, it gets things wrongs sometimes, but I feel like we're constantly reviewing PRs where all*the*(arithmetic+is)-written**like*this. Maybe a better approach would be to spend that time hacking up some script that uses git and autopep8 together to run autopep8 over all and only those lines which the current branch has actually touched? It's pretty easy to parse 'git diff' output to get a list of all line numbers which have been modified, and then we could run autopep8 over the modified files and pull out only those changes which touch those lines. -n P.S.: definitely [:, :, 2] From blake.a.griffith at gmail.com Mon Sep 9 12:08:46 2013 From: blake.a.griffith at gmail.com (Blake Griffith) Date: Mon, 9 Sep 2013 11:08:46 -0500 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: I think a good solution would to use add a git_hooks directory with a pre-commit git hook along with an git hook installation script. And a note should be added to DEV_README.txt suggesting installing the git hooks for pep8 compatibility. I personally use this as a pre-commit #!/bin/sh FILES=$(git diff --cached --name-status | grep -v ^D | awk '$1 $2 { print $2}' | grep -e .py$) if [ -n "$FILES" ]; then pep8 -r $FILES fi which is from here: https://gist.github.com/lentil/810399#comment-303703 On Mon, Sep 9, 2013 at 10:54 AM, Nathaniel Smith wrote: > On Mon, Sep 9, 2013 at 3:29 PM, Charles R Harris > wrote: > > > > > > > > On Mon, Sep 9, 2013 at 8:12 AM, Richard Hattersley < > rhattersley at gmail.com> > > wrote: > >> > >> > Something we have done in matplotlib is that we have made PEP8 a part > of > >> > the tests. > >> > >> In Iris and Cartopy we've also done this and it works well. While we > >> transition we have an exclusion list (which is gradually getting > shorter). > >> We've had mixed experiences with automatic reformatting, so prefer to > keep > >> the human in the loop. > >> > > > > I agree with keeping a human in the loop, the script would be intended to > > get things into the right neighborhood, the submitter would have to > review > > the changes after. If the script isn't too strict there will be than one > way > > to do some things and those bits would rely on the good taste of the > coder. > > So if I understand right, the goal is to have some script that > developers can run before (or after) submitting a PR, like > tools/autopep8-my-changes numpy/ > that will fix up their changes, but leave the rest of numpy alone? > > And the proposed mechanism is to come up with a combination of changes > to the numpy source and an autopep8 configuration such that > autopep8 --our-config numpy/ > becomes a no-op, and then we can use this as an implementation of > tools/autopep8-my-changes? > > If that's right then my feeling is that the goal seems worthwhile but > the approach seems difficult and unlikely to survive for long. As soon > as someone overrides autopep8 once, we either have to disable the rule > for the whole project or keep overriding it manually forever. You're > already suggesting taking out the spaces-around-arithmetic rule, which > strikes me as one of the most useful -- sure, it gets things wrongs > sometimes, but I feel like we're constantly reviewing PRs where > all*the*(arithmetic+is)-written**like*this. > > Maybe a better approach would be to spend that time hacking up some > script that uses git and autopep8 together to run autopep8 over all > and only those lines which the current branch has actually touched? > It's pretty easy to parse 'git diff' output to get a list of all line > numbers which have been modified, and then we could run autopep8 over > the modified files and pull out only those changes which touch those > lines. > > -n > > P.S.: definitely [:, :, 2] > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Mon Sep 9 13:09:17 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Mon, 9 Sep 2013 13:09:17 -0400 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: I don't have CFLAGS defined. But I have iothers env variable that point to other python stuff like CPATH. But even in that case, I don't understand how other people could have compiled methods.c. The include aren't part of the env variable, but in the file. Anyway, I think your PR is the good fix. I checked our PR and now I have this new error: gcc: numpy/core/src/multiarray/multiarraymodule.c In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: error: ?PyArray_Partition? undeclared here (not in a function) build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: error: ?PyArray_ArgPartition? undeclared here (not in a function) In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: error: ?PyArray_Partition? undeclared here (not in a function) build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: error: ?PyArray_ArgPartition? undeclared here (not in a function) error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c numpy/core/src/multiarray/multiarraymodule.c -o build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" failed with exit status 1 So it seam we have the same problem with those 2 functions. They are defined in numpy/core/src/multiarray/item_selection.c, but not in the .h file. thanks Fred On Mon, Sep 9, 2013 at 11:44 AM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 9:33 AM, Fr?d?ric Bastien wrote: > >> I tried it and retried and it still fail. This is in an virtualenv >> >> $git show >> commit c9b06111227f7a4ec213571f97e1b8d19b9c23f5 >> Merge: 73fbfb2 8edccea >> Author: Charles Harris >> Date: Sun Sep 8 19:47:21 2013 -0700 >> >> Merge pull request #3701 from cgohlke/patch-2 >> >> ENH: add support for Python 3.4 ast.NameConstant >> >> >> >> $rm -rf build ## Fail as there is no such directory >> $pip install . # fail with the same error >> >> $pip uninstall numpy >> $python setup.py install --user # fail with the same error >> $pip install . ## fail with the same error: >> >> $git grep PyArray_SelectkindConverter >> doc/release/1.8.0-notes.rst:* PyArray_SelectkindConverter >> numpy/core/code_generators/numpy_api.py: >> 'PyArray_SelectkindConverter': 298, >> numpy/core/src/multiarray/conversion_utils.c:PyArray_SelectkindConverter(PyObject >> *obj, NPY_SELECTKIND *selectkind) >> numpy/core/src/multiarray/methods.c: >> PyArray_SelectkindConverter, &sortkind, >> numpy/core/src/multiarray/methods.c: >> PyArray_SelectkindConverter, &sortkind, >> >> >> Here I don't see PyArray_SelectkindConverter in conversion_utils.h as you >> said it is present. Witch commit do you use? >> >> > It's not there, it is part of the API. I've got a PR to add it to the *.h > file. The question is why you are the only person (so far) to have a > problem compiling. What are your CFLAGS? > > Chuck > >> Fred >> >> >> >> On Mon, Sep 9, 2013 at 11:02 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> >>> On Mon, Sep 9, 2013 at 8:51 AM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> >>>> >>>> >>>> On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien wrote: >>>> >>>>> Hi, >>>>> >>>>> I checkout the dev version of numpy and it fail to compile with this >>>>> error: >>>>> >>>>> creating >>>>> build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray >>>>> >>>>> compile options: '-DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>>>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>>>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>>>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>>>> -Inumpy/core/src/npysort -Inumpy/core/include >>>>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' >>>>> >>>>> gcc: numpy/core/src/multiarray/sequence.c >>>>> >>>>> gcc: numpy/core/src/multiarray/descriptor.c >>>>> >>>>> gcc: numpy/core/src/multiarray/getset.c >>>>> >>>>> gcc: numpy/core/src/multiarray/arrayobject.c >>>>> >>>>> gcc: numpy/core/src/multiarray/methods.c >>>>> >>>>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>>>> >>>>> numpy/core/src/multiarray/methods.c:1199:38: error: >>>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>>> >>>>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>>>> identifier is reported only once for each function it appears in >>>>> >>>>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>>>> >>>>> numpy/core/src/multiarray/methods.c:1316:38: error: >>>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>>> >>>>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>>>> pointer from integer without a cast >>>>> >>>>> numpy/core/src/multiarray/methods.c: In function ?array_partition?: >>>>> >>>>> numpy/core/src/multiarray/methods.c:1199:38: error: >>>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>>> >>>>> numpy/core/src/multiarray/methods.c:1199:38: note: each undeclared >>>>> identifier is reported only once for each function it appears in >>>>> >>>>> numpy/core/src/multiarray/methods.c: In function ?array_argpartition?: >>>>> >>>>> numpy/core/src/multiarray/methods.c:1316:38: error: >>>>> ?PyArray_SelectkindConverter? undeclared (first use in this function) >>>>> >>>>> numpy/core/src/multiarray/methods.c:1352:9: warning: assignment makes >>>>> pointer from integer without a cast >>>>> >>>>> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >>>>> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>>>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>>>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>>>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>>>> -Inumpy/core/src/npysort -Inumpy/core/include >>>>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >>>>> numpy/core/src/multiarray/methods.c -o >>>>> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" failed >>>>> with exit status 1 >>>>> >>>>> >>>>> PyArray_SelectkindConverter is defined in >>>>> numpy/core/src/multiarray/conversion_utils.c. methods.c include >>>>> conversion_utils.h, but there is no fct declaration of this fct in this >>>>> file. Is that normal? >>>>> >>>>> >>>> No, it looks like a bug. What is strange is that it doesn't show up on >>>> my machine. What compiler flags are you using? >>>> >>>> Could you make a PR for this? >>>> >>>> >>>> >>>> >>> Wait a minute, it is in the API. Try a clean build and see what happens. >>> >>> Chuck >>> >>>> >>>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Sep 9 13:12:05 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 09 Sep 2013 19:12:05 +0200 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: <522E0165.9010309@googlemail.com> try rebuilding everything from scratch. setup.py dependency handling is a bit dodgy with the generated files. On 09.09.2013 19:09, Fr?d?ric Bastien wrote: > I don't have CFLAGS defined. But I have iothers env variable that point > to other python stuff like CPATH. > > But even in that case, I don't understand how other people could have > compiled methods.c. The include aren't part of the env variable, but in > the file. > > Anyway, I think your PR is the good fix. I checked our PR and now I have > this new error: > > gcc: numpy/core/src/multiarray/multiarraymodule.c > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 > -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c > numpy/core/src/multiarray/multiarraymodule.c -o > build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" failed > with exit status 1 > > > So it seam we have the same problem with those 2 functions. They are > defined in numpy/core/src/multiarray/item_selection.c, but not in the .h > file. > > thanks > > Fred > > > On Mon, Sep 9, 2013 at 11:44 AM, Charles R Harris > > wrote: > > > > > On Mon, Sep 9, 2013 at 9:33 AM, Fr?d?ric Bastien > wrote: > > I tried it and retried and it still fail. This is in an virtualenv > > $git show > commit c9b06111227f7a4ec213571f97e1b8d19b9c23f5 > Merge: 73fbfb2 8edccea > Author: Charles Harris > > Date: Sun Sep 8 19:47:21 2013 -0700 > > Merge pull request #3701 from cgohlke/patch-2 > > ENH: add support for Python 3.4 ast.NameConstant > > > > $rm -rf build ## Fail as there is no such directory > $pip install . # fail with the same error > > $pip uninstall numpy > $python setup.py install --user # fail with the same error > $pip install . ## fail with the same error: > > $git grep PyArray_SelectkindConverter > doc/release/1.8.0-notes.rst:* PyArray_SelectkindConverter > numpy/core/code_generators/numpy_api.py: > 'PyArray_SelectkindConverter': 298, > numpy/core/src/multiarray/conversion_utils.c:PyArray_SelectkindConverter(PyObject > *obj, NPY_SELECTKIND *selectkind) > numpy/core/src/multiarray/methods.c: > PyArray_SelectkindConverter, &sortkind, > numpy/core/src/multiarray/methods.c: > PyArray_SelectkindConverter, &sortkind, > > > Here I don't see PyArray_SelectkindConverter in > conversion_utils.h as you said it is present. Witch commit do > you use? > > > It's not there, it is part of the API. I've got a PR to add it to > the *.h file. The question is why you are the only person (so far) > to have a problem compiling. What are your CFLAGS? > > Chuck > > Fred > > > > On Mon, Sep 9, 2013 at 11:02 AM, Charles R Harris > > > wrote: > > > > > On Mon, Sep 9, 2013 at 8:51 AM, Charles R Harris > > wrote: > > > > > On Mon, Sep 9, 2013 at 7:46 AM, Fr?d?ric Bastien > > wrote: > > Hi, > > I checkout the dev version of numpy and it fail to > compile with this error: > > creating > build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/multiarray > > compile options: '-DHAVE_NPY_CONFIG_H=1 > -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private > -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort > -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c' > > gcc: numpy/core/src/multiarray/sequence.c > > gcc: numpy/core/src/multiarray/descriptor.c > > gcc: numpy/core/src/multiarray/getset.c > > gcc: numpy/core/src/multiarray/arrayobject.c > > gcc: numpy/core/src/multiarray/methods.c > > numpy/core/src/multiarray/methods.c: In function > ?array_partition?: > > numpy/core/src/multiarray/methods.c:1199:38: error: > ?PyArray_SelectkindConverter? undeclared (first use > in this function) > > numpy/core/src/multiarray/methods.c:1199:38: note: > each undeclared identifier is reported only once for > each function it appears in > > numpy/core/src/multiarray/methods.c: In function > ?array_argpartition?: > > numpy/core/src/multiarray/methods.c:1316:38: error: > ?PyArray_SelectkindConverter? undeclared (first use > in this function) > > numpy/core/src/multiarray/methods.c:1352:9: warning: > assignment makes pointer from integer without a cast > > numpy/core/src/multiarray/methods.c: In function > ?array_partition?: > > numpy/core/src/multiarray/methods.c:1199:38: error: > ?PyArray_SelectkindConverter? undeclared (first use > in this function) > > numpy/core/src/multiarray/methods.c:1199:38: note: > each undeclared identifier is reported only once for > each function it appears in > > numpy/core/src/multiarray/methods.c: In function > ?array_argpartition?: > > numpy/core/src/multiarray/methods.c:1316:38: error: > ?PyArray_SelectkindConverter? undeclared (first use > in this function) > > numpy/core/src/multiarray/methods.c:1352:9: warning: > assignment makes pointer from integer without a cast > > error: Command "gcc -pthread -fno-strict-aliasing -g > -O2 -DNDEBUG -O2 -fPIC -DHAVE_NPY_CONFIG_H=1 > -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private > -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/src/npysort > -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c > numpy/core/src/multiarray/methods.c -o > build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/methods.o" > failed with exit status 1 > > > PyArray_SelectkindConverter is defined in > numpy/core/src/multiarray/conversion_utils.c. > methods.c include conversion_utils.h, but there is > no fct declaration of this fct in this file. Is that > normal? > > > No, it looks like a bug. What is strange is that it > doesn't show up on my machine. What compiler flags are > you using? > > Could you make a PR for this? > > > > > Wait a minute, it is in the API. Try a clean build and see > what happens. > > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From njs at pobox.com Mon Sep 9 13:46:01 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 9 Sep 2013 18:46:01 +0100 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 6:09 PM, Fr?d?ric Bastien wrote: > I don't have CFLAGS defined. But I have iothers env variable that point to > other python stuff like CPATH. > > But even in that case, I don't understand how other people could have > compiled methods.c. The include aren't part of the env variable, but in the > file. > > Anyway, I think your PR is the good fix. I checked our PR and now I have > this new error: > > gcc: numpy/core/src/multiarray/multiarraymodule.c > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC > -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/src/npysort -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c > numpy/core/src/multiarray/multiarraymodule.c -o > build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" > failed with exit status 1 > > > So it seam we have the same problem with those 2 functions. They are defined > in numpy/core/src/multiarray/item_selection.c, but not in the .h file. Something is going horribly wrong with your build process. The API generation code should automatically be putting prototypes for these functions into __multiarray_api.h. I don't understand how you could have a reference to these functions in your __multiarray_api.c without having them in your __multiarray_api.h, though... -n From charlesr.harris at gmail.com Mon Sep 9 14:04:51 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 12:04:51 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 11:09 AM, Fr?d?ric Bastien wrote: > I don't have CFLAGS defined. But I have iothers env variable that point to > other python stuff like CPATH. > > But even in that case, I don't understand how other people could have > compiled methods.c. The include aren't part of the env variable, but in the > file. > > Anyway, I think your PR is the good fix. I checked our PR and now I have > this new error: > > gcc: numpy/core/src/multiarray/multiarraymodule.c > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: > error: ?PyArray_Partition? undeclared here (not in a function) > > build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: > error: ?PyArray_ArgPartition? undeclared here (not in a function) > > error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 > -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/src/npysort -Inumpy/core/include > -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c > numpy/core/src/multiarray/multiarraymodule.c -o > build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" > failed with exit status 1 > > > So it seam we have the same problem with those 2 functions. They are > defined in numpy/core/src/multiarray/item_selection.c, but not in the .h > file. > > I'm going to guess that there is something special about your virtualenv. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 9 14:14:49 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Sep 2013 12:14:49 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: On Mon, Sep 9, 2013 at 12:04 PM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 11:09 AM, Fr?d?ric Bastien wrote: > >> I don't have CFLAGS defined. But I have iothers env variable that point >> to other python stuff like CPATH. >> >> But even in that case, I don't understand how other people could have >> compiled methods.c. The include aren't part of the env variable, but in the >> file. >> >> Anyway, I think your PR is the good fix. I checked our PR and now I have >> this new error: >> >> gcc: numpy/core/src/multiarray/multiarraymodule.c >> >> In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: >> >> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: >> error: ?PyArray_Partition? undeclared here (not in a function) >> >> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: >> error: ?PyArray_ArgPartition? undeclared here (not in a function) >> >> In file included from numpy/core/src/multiarray/multiarraymodule.c:3753:0: >> >> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: >> error: ?PyArray_Partition? undeclared here (not in a function) >> >> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: >> error: ?PyArray_ArgPartition? undeclared here (not in a function) >> >> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >> -Inumpy/core/src/npysort -Inumpy/core/include >> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >> numpy/core/src/multiarray/multiarraymodule.c -o >> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" >> failed with exit status 1 >> >> >> So it seam we have the same problem with those 2 functions. They are >> defined in numpy/core/src/multiarray/item_selection.c, but not in the .h >> file. >> >> > I'm going to guess that there is something special about your virtualenv. > > > > The prototypes should be in `arrayobject.h` and `ndarrayobject.h`. Perhaps old versions are being used and cached somewhere. Are they precompiled somewhere? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim at cerazone.net Mon Sep 9 15:40:10 2013 From: tim at cerazone.net (Cera, Tim) Date: Mon, 9 Sep 2013 15:40:10 -0400 Subject: [Numpy-discussion] PEP8 In-Reply-To: References: Message-ID: I made a PR request about supplying a git hooks framework at https://github.com/numpy/numpy/pull/200. I asked for it to be closed because I couldn't easily figure our how to handle x-platform issues. If you have an answer, what I was working on might be a start. But your script is an example of the x-platform challenges since it will only run on Windows that has a Linux environment installed (Mingw or the like). What I have started to use in my projects is 'tox', 'coverage', and 'flake8'. Really nice stuff. There is an issue though with the 'flake8' report. There are only very localized places in a few files that I want to ignore some portion or other of PEP8, so I don' t insist that 'flake8' pass, I just ignore the output where I need to ignore it. Usually what I want to ignore is the spaces around brackets and commas in order to line up array values as mentioned in an earlier post within this thread. Here is an example of my project setup based in part on 'cookiecutter' - https://bitbucket.org/timcera/astronomia/src. Yes, I use bitbucket - so sue me. Magic happens inside the 'tox.ini' file. I suspect that for numpy, making something like 'flake8' part of the tests would work better than 'autopep8'. 'autopep8' can be configured to just report with the --list-fixes option, which would give people a little more confidence to use it rather that it's primary mission as an editor. Another plug for 'flake8'; it can be configured on a file by file basis to ignore particular errors or warnings, and exits with an error status if the file doesn't pass. I do think the idea has merit, but this entire process would be a lot of work and I am not stepping forward to do it. At this point I simply have to say that I am a 'balcony muppet'. Much thanks to Josef for the reminder about where I learned my curmudgeony ways. Those guys made the show! Kindest regards, Tim On Mon, Sep 9, 2013 at 12:08 PM, Blake Griffith wrote: > I think a good solution would to use add a git_hooks directory with a > pre-commit git hook along with an git hook installation script. And a note > should be added to DEV_README.txt suggesting installing the git hooks for > pep8 compatibility. I personally use this as a pre-commit > > #!/bin/sh > > FILES=$(git diff --cached --name-status | grep -v ^D | awk '$1 $2 { print > $2}' | grep -e .py$) > if [ -n "$FILES" ]; then > pep8 -r $FILES > fi > > which is from here: https://gist.github.com/lentil/810399#comment-303703 > > > On Mon, Sep 9, 2013 at 10:54 AM, Nathaniel Smith wrote: >> >> On Mon, Sep 9, 2013 at 3:29 PM, Charles R Harris >> wrote: >> > >> > >> > >> > On Mon, Sep 9, 2013 at 8:12 AM, Richard Hattersley >> > >> > wrote: >> >> >> >> > Something we have done in matplotlib is that we have made PEP8 a part >> >> > of >> >> > the tests. >> >> >> >> In Iris and Cartopy we've also done this and it works well. While we >> >> transition we have an exclusion list (which is gradually getting >> >> shorter). >> >> We've had mixed experiences with automatic reformatting, so prefer to >> >> keep >> >> the human in the loop. >> >> >> > >> > I agree with keeping a human in the loop, the script would be intended >> > to >> > get things into the right neighborhood, the submitter would have to >> > review >> > the changes after. If the script isn't too strict there will be than one >> > way >> > to do some things and those bits would rely on the good taste of the >> > coder. >> >> So if I understand right, the goal is to have some script that >> developers can run before (or after) submitting a PR, like >> tools/autopep8-my-changes numpy/ >> that will fix up their changes, but leave the rest of numpy alone? >> >> And the proposed mechanism is to come up with a combination of changes >> to the numpy source and an autopep8 configuration such that >> autopep8 --our-config numpy/ >> becomes a no-op, and then we can use this as an implementation of >> tools/autopep8-my-changes? >> >> If that's right then my feeling is that the goal seems worthwhile but >> the approach seems difficult and unlikely to survive for long. As soon >> as someone overrides autopep8 once, we either have to disable the rule >> for the whole project or keep overriding it manually forever. You're >> already suggesting taking out the spaces-around-arithmetic rule, which >> strikes me as one of the most useful -- sure, it gets things wrongs >> sometimes, but I feel like we're constantly reviewing PRs where >> all*the*(arithmetic+is)-written**like*this. >> >> Maybe a better approach would be to spend that time hacking up some >> script that uses git and autopep8 together to run autopep8 over all >> and only those lines which the current branch has actually touched? >> It's pretty easy to parse 'git diff' output to get a list of all line >> numbers which have been modified, and then we could run autopep8 over >> the modified files and pull out only those changes which touch those >> lines. >> >> -n >> >> P.S.: definitely [:, :, 2] >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david.reed.c at gmail.com Tue Sep 10 09:52:04 2013 From: david.reed.c at gmail.com (David Reed) Date: Tue, 10 Sep 2013 09:52:04 -0400 Subject: [Numpy-discussion] 2-D Histogram Message-ID: Hi there, Is there a faster way to perform a 2D Histogram from a 2D matrix than what I have below: def spatial_histogram(frame, n_bins): shape = frame.shape h_len = shape[0]/n_bins w_len = shape[1]/n_bins h_ind = range(0, shape[0], h_len) w_ind = range(0, shape[1], w_len) max_val = 255*h_len*w_len out = np.empty((n_bins, n_bins), np.uint8) for ii in range(n_bins): for jj in range(n_bins): out[ii, jj] = np.sum(frame[h_ind[ii]:h_ind[ii]+h_len, w_ind[jj]:w_ind[jj]+w_len])/max_val*255 return out Should I try implementing this in Cython, or is there something I can do in Numpy? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Tue Sep 10 09:59:32 2013 From: lists at hilboll.de (Andreas Hilboll) Date: Tue, 10 Sep 2013 15:59:32 +0200 Subject: [Numpy-discussion] 2-D Histogram In-Reply-To: References: Message-ID: <522F25C4.7020907@hilboll.de> On 10.09.2013 15:52, David Reed wrote: > Hi there, > > Is there a faster way to perform a 2D Histogram from a 2D matrix than > what I have below: > > def spatial_histogram(frame, n_bins): > shape = frame.shape > > h_len = shape[0]/n_bins > w_len = shape[1]/n_bins > > h_ind = range(0, shape[0], h_len) > w_ind = range(0, shape[1], w_len) > > max_val = 255*h_len*w_len > > out = np.empty((n_bins, n_bins), np.uint8) > > for ii in range(n_bins): > for jj in range(n_bins): > out[ii, jj] = np.sum(frame[h_ind[ii]:h_ind[ii]+h_len, > w_ind[jj]:w_ind[jj]+w_len])/max_val*255 > > return out > > Should I try implementing this in Cython, or is there something I can do > in Numpy? > > Thanks! David, are you aware of Scipy's binne_statistic_2d method? http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binned_statistic_2d.html At first glance it can do what you're trying to do. Andreas. From josef.pktd at gmail.com Tue Sep 10 10:47:55 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 10 Sep 2013 10:47:55 -0400 Subject: [Numpy-discussion] 2-D Histogram In-Reply-To: <522F25C4.7020907@hilboll.de> References: <522F25C4.7020907@hilboll.de> Message-ID: On Tue, Sep 10, 2013 at 9:59 AM, Andreas Hilboll wrote: > On 10.09.2013 15:52, David Reed wrote: >> Hi there, >> >> Is there a faster way to perform a 2D Histogram from a 2D matrix than >> what I have below: >> >> def spatial_histogram(frame, n_bins): >> shape = frame.shape >> >> h_len = shape[0]/n_bins >> w_len = shape[1]/n_bins >> >> h_ind = range(0, shape[0], h_len) >> w_ind = range(0, shape[1], w_len) >> >> max_val = 255*h_len*w_len >> >> out = np.empty((n_bins, n_bins), np.uint8) >> >> for ii in range(n_bins): >> for jj in range(n_bins): >> out[ii, jj] = np.sum(frame[h_ind[ii]:h_ind[ii]+h_len, >> w_ind[jj]:w_ind[jj]+w_len])/max_val*255 >> >> return out >> >> Should I try implementing this in Cython, or is there something I can do >> in Numpy? >> >> Thanks! > > David, > > are you aware of Scipy's binne_statistic_2d method? > > > http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binned_statistic_2d.html That's modeled after numpy.histogram2d, and just calculates other statistics besides histogram Josef > > At first glance it can do what you're trying to do. > > Andreas. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From ndbecker2 at gmail.com Tue Sep 10 11:19:20 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Sep 2013 11:19:20 -0400 Subject: [Numpy-discussion] my code (python version) Message-ID: Here's code I use for basic 2d histogram -------------- next part -------------- A non-text attachment was scrubbed... Name: histogram2d.py Type: text/x-python Size: 1155 bytes Desc: not available URL: From antlarac at gmail.com Wed Sep 11 06:33:52 2013 From: antlarac at gmail.com (antlarac) Date: Wed, 11 Sep 2013 03:33:52 -0700 (PDT) Subject: [Numpy-discussion] strange behaviour with numpy arrays Message-ID: <1378895632501-35123.post@n7.nabble.com> Hi, I have a numpy array, and I want to create another variable equal to it, to back it up for later calculations, because the first array will change. But after the first array changes, the second automatically changes to the same value. An example of what happens: import numpy as np a=np.zeros((4,4)) b=a print(b) a[3,3]=3 print(' ') print(b) gives the result: [[ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 0.]] [[ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 3.]] As you can see, when the value of a changes, the value of b automatically changes, even when this is not asked. Is there a way of avoiding this? This problem does not happen with normal python variables. Thank you for your time. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/strange-behaviour-with-numpy-arrays-tp35123.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From jtaylor.debian at googlemail.com Wed Sep 11 06:40:32 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Wed, 11 Sep 2013 12:40:32 +0200 Subject: [Numpy-discussion] strange behaviour with numpy arrays In-Reply-To: <1378895632501-35123.post@n7.nabble.com> References: <1378895632501-35123.post@n7.nabble.com> Message-ID: <523048A0.20602@googlemail.com> On 11.09.2013 12:33, antlarac wrote: > Hi, I have a numpy array, and I want to create another variable equal to it, > to back it up for later calculations, because the first array will change. > But after the first array changes, the second automatically changes to the > same value. An example of what happens: > > import numpy as np > a=np.zeros((4,4)) > b=a > print(b) > a[3,3]=3 > print(' ') > print(b) > > gives the result: > > [[ 0. 0. 0. 0.] > [ 0. 0. 0. 0.] > [ 0. 0. 0. 0.] > [ 0. 0. 0. 0.]] > > [[ 0. 0. 0. 0.] > [ 0. 0. 0. 0.] > [ 0. 0. 0. 0.] > [ 0. 0. 0. 3.]] > > As you can see, when the value of a changes, the value of b automatically > changes, even when this is not asked. Is there a way of avoiding this? > > This problem does not happen with normal python variables. > Thank you for your time. > this is normal, python tracks its mutable variables as references. b=a makes b a reference to a so changing a changes b too. python lists work the same way: In [1]: a = [1,2,3] In [2]: b = a In [3]: b[2] = 9 In [4]: b Out[4]: [1, 2, 9] In [5]: a Out[5]: [1, 2, 9] note that python integers and strings a not mutable, so it does not behave the same way. to avoid it make explicit copies. b = a.copy() Also note that slices of array (a[:5]) in numpy are *not* copies but views on the original array. This is different than python list slices which are shallow copies. From grb at skogoglandskap.no Wed Sep 11 09:20:58 2013 From: grb at skogoglandskap.no (Graeme B. Bell) Date: Wed, 11 Sep 2013 13:20:58 +0000 Subject: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy In-Reply-To: <8B623060-BF0D-4B24-B8B8-ECE66147A6EA@skogoglandskap.no> References: <8B623060-BF0D-4B24-B8B8-ECE66147A6EA@skogoglandskap.no> Message-ID: >From my previous mail: >> this has the same performance as your code: >> a = empty([3] list(A.shape) For anyone that is interested. I ran a benchmark on the code after Julian kindly provided me with a correction to the listing he posted. >> a = empty([3] + list(A.shape)) >> a[0] = A>5; a[1] = B<2; a[2] = A>10; >> np.any(a, 0) Julian also suggested trying the idiom "np.vstack([A,B,C])" instead of [A,B,C]. Revised benchmarks here. I've moved the [A>5, B<2, A>10] creation outside the timing loop in all cases since it was distorting results due to array creation, which shouldn't be part of the any() timing measurement. I'm also now using separate test arrays to avoid the possibility of side effects between tests of different functions. The following results are produced consistently: np.any() -> 2.68 s np.any() with Julian's first idiom above: -> 0.24s faa.any() (original version) -> 0.2s np.any() with vstack(): 0.14s faa.any() with vstack: 0.1s faa.any() without vstack: 0.08s (alternative faa implementations: 0.11-0.12s) Conclusion: fast_any_all is 30x faster than numpy.any() 1.7 fast_any_all is 43% faster than numpy.any() 1.7 with the vstack() idiom, which I understand to be the basis for the new approach in numpy.any() 1.8 development branch. I'd be really interested to see the benchmarks under the current 1.8 master branch of numpy. Please can someone try this and send me the file? # git clone https://github.com/gbb/numpy-fast-any-all.git (read the source code to make sure I'm not evil) # cd numpy-fast-any-all # python test_fast_any_all.py > BENCHMARK.txt Incidentally, this is an appropriate example of a case where a 'performance idiom' becomes a 'penalty idiom' unexpectedly when the underlying implementation changes (vstack). Thanks for your suggestions, Julian. Graeme. From nouiz at nouiz.org Wed Sep 11 14:12:24 2013 From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=) Date: Wed, 11 Sep 2013 14:12:24 -0400 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: Hi, I create an Ubuntu VM, cloned numpy and it compiled correctly in it. I tried in my normal development environment (instead of a virtualenv) with my old clone that is updated of numpy and it failed. So this isn't virtualenv. I created a new clone of numpy and it compiled correctly with my normal development environment. So this seam to indicate that the __multiarray_api.* are cached. In the new clone, I have this after compilation: $find . -name '*multiarray_api*' ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/multiarray_api.txt ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h In my old clone, I have this: $cd ../numpy [bastienf at oolong numpy]$find . -name '*multiarray_api*' ./numpy/core/include/numpy/__multiarray_api.h ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/multiarray_api.txt ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h So for some reason, the __multiarray_api.h file ended in the source of numpy. When this happen all following compilation reuse it instead of regenerating it. If in the new numpy clone, I run: python setup.py build_ext --inplace I get: $find . -name '*multiarray_api*' ./numpy/core/include/numpy/__multiarray_api.c ./numpy/core/include/numpy/multiarray_api.txt ./numpy/core/include/numpy/__multiarray_api.h ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/multiarray_api.txt ./build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h So this could explain how I ended up with a numpy clone in such state. I think we should make numpy always regenerate the auto-generated files even if they are present, or it would need to check if they are older the the source of thoses files. Do you think this is the good solution too? Do someone have guidance on where those file are generated? Do you and how do you support the partial rebuild vs the full rebuild? Or is it just not supported to build inplace? In the .gitignore file, there is 41 files that seam to be generated. thanks Fr?d?ric On Mon, Sep 9, 2013 at 2:14 PM, Charles R Harris wrote: > > > > On Mon, Sep 9, 2013 at 12:04 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Sep 9, 2013 at 11:09 AM, Fr?d?ric Bastien wrote: >> >>> I don't have CFLAGS defined. But I have iothers env variable that point >>> to other python stuff like CPATH. >>> >>> But even in that case, I don't understand how other people could have >>> compiled methods.c. The include aren't part of the env variable, but in the >>> file. >>> >>> Anyway, I think your PR is the good fix. I checked our PR and now I have >>> this new error: >>> >>> gcc: numpy/core/src/multiarray/multiarraymodule.c >>> >>> In file included from >>> numpy/core/src/multiarray/multiarraymodule.c:3753:0: >>> >>> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: >>> error: ?PyArray_Partition? undeclared here (not in a function) >>> >>> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: >>> error: ?PyArray_ArgPartition? undeclared here (not in a function) >>> >>> In file included from >>> numpy/core/src/multiarray/multiarraymodule.c:3753:0: >>> >>> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:303:18: >>> error: ?PyArray_Partition? undeclared here (not in a function) >>> >>> build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.c:304:18: >>> error: ?PyArray_ArgPartition? undeclared here (not in a function) >>> >>> error: Command "gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 >>> -fPIC -DHAVE_NPY_CONFIG_H=1 -Inumpy/core/include >>> -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy >>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core >>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath >>> -Inumpy/core/src/npysort -Inumpy/core/include >>> -I/opt/lisa/os/epd-7.1.2/include/python2.7 -c >>> numpy/core/src/multiarray/multiarraymodule.c -o >>> build/temp.linux-x86_64-2.7/numpy/core/src/multiarray/multiarraymodule.o" >>> failed with exit status 1 >>> >>> >>> So it seam we have the same problem with those 2 functions. They are >>> defined in numpy/core/src/multiarray/item_selection.c, but not in the .h >>> file. >>> >>> >> I'm going to guess that there is something special about your virtualenv. >> >> >> >> > The prototypes should be in `arrayobject.h` and `ndarrayobject.h`. Perhaps > old versions are being used and cached somewhere. Are they precompiled > somewhere? > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexeftimiades at gmail.com Thu Sep 12 21:09:12 2013 From: alexeftimiades at gmail.com (Alex Eftimiades) Date: Thu, 12 Sep 2013 21:09:12 -0400 Subject: [Numpy-discussion] How to find number of bytes in a PyArrayObject without a segfault Message-ID: <2194E267-6671-4E5F-ADCC-A002AFB502CC@gmail.com> I ultimately want to access the total number of bytes in a PyArrayObject through a C++ program. I cannot seem to do it without sefaulting. All of the following yields a segfault: my_array->descr->subarray->shape my_array->dimensions PyArray_SHAPE(my_array) PyArray_DIMS(my_array) PyArray_ITEMSIZE(my_array) and of course... PyArray_NBYTES(my_array) I am working from within sage if that makes any difference. Is there a way to iterate over an array from within C++ without knowing the number of bytes in the array (and using strides--which oddly enough do not result in a segfault), or iterating over each dimension? Please point me in the right direction. There must be a way or else the entire software suite would not work. Thanks, Alex Eftimiades From markbak at gmail.com Fri Sep 13 04:07:47 2013 From: markbak at gmail.com (Mark Bakker) Date: Fri, 13 Sep 2013 10:07:47 +0200 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? Message-ID: Hello List, I am teaching a Python programming class where students use their own computer. When I create an array with 3 rows and 4 columns, a = zeros((3,4)), and ask for the shape, shape(a), then some students get (3,4), while others get (3L,4L). I guess the 3L and 4L are long integers. I wonder why they don't all get the same thing. They all installed numpy through Canopy Express. Any thoughts? Is this a setting in printoptions maybe? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Fri Sep 13 04:14:51 2013 From: markbak at gmail.com (Mark Bakker) Date: Fri, 13 Sep 2013 10:14:51 +0200 Subject: [Numpy-discussion] can argmax be used to return row and column indices? Message-ID: Hello list, I am trying to find the indices of the maximum value in a 2D array. argmax works fine, but returns the index in the flattened array. That is often not very convenient. Is there a function that returns the index of the row and column? Or can the index of the flattened array easily be converted to the row can column indices (I know, I can write a function for that, but I figure numpy already has one). Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregorio.bastardo at gmail.com Fri Sep 13 04:19:13 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Fri, 13 Sep 2013 10:19:13 +0200 Subject: [Numpy-discussion] can argmax be used to return row and column indices? In-Reply-To: References: Message-ID: Hi Mark, You're looking for "np.unravel_index" function. Cheers, Gregorio 2013/9/13 Mark Bakker : > Hello list, > > I am trying to find the indices of the maximum value in a 2D array. > argmax works fine, but returns the index in the flattened array. > That is often not very convenient. > Is there a function that returns the index of the row and column? > Or can the index of the flattened array easily be converted to the row can > column indices (I know, I can write a function for that, but I figure numpy > already has one). > > Thanks, > > Mark > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cournape at gmail.com Fri Sep 13 04:22:23 2013 From: cournape at gmail.com (David Cournapeau) Date: Fri, 13 Sep 2013 09:22:23 +0100 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? In-Reply-To: References: Message-ID: Are they all using the same platform ? I suspect you're seeing the (3L, 4L) on windows 64 bits, correct ? FWIW, the numpy provided in Canopy is straight up upstream numpy + few patches not merged in releases yet, and any divergence from upstream would be considered as a bug by the canopy packaging people (i.e. me :-) ). David On Fri, Sep 13, 2013 at 9:07 AM, Mark Bakker wrote: > Hello List, > > I am teaching a Python programming class where students use their own > computer. > > When I create an array with 3 rows and 4 columns, a = zeros((3,4)), and > ask for the shape, shape(a), then some students get (3,4), while others get > (3L,4L). > > I guess the 3L and 4L are long integers. I wonder why they don't all get > the same thing. They all installed numpy through Canopy Express. Any > thoughts? Is this a setting in printoptions maybe? > > Thanks, > > Mark > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Fri Sep 13 04:27:35 2013 From: markbak at gmail.com (Mark Bakker) Date: Fri, 13 Sep 2013 10:27:35 +0200 Subject: [Numpy-discussion] can argmax be used to return row and column indices? Message-ID: Thanks, Gregorio. I would like it if argmax had a keyword option to return the row,column index automatically (or whatever the dimension of the array). Afterall, argmax already knows the shape of the array. Calling np.unravel_index( np.argmax( A ) ) seems unnecessarily long. But it works well though! I am not sure that such a PR would get much support.... Thanks again, Mark Hi Mark, > > You're looking for "np.unravel_index" function. > > Cheers, > Gregorio > Hello list, > > I am trying to find the indices of the maximum value in a 2D array. > argmax works fine, but returns the index in the flattened array. > That is often not very convenient. > Is there a function that returns the index of the row and column? > Or can the index of the flattened array easily be converted to the row can > column indices (I know, I can write a function for that, but I figure numpy > already has one). > > Thanks, > > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Fri Sep 13 04:32:42 2013 From: markbak at gmail.com (Mark Bakker) Date: Fri, 13 Sep 2013 10:32:42 +0200 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? In-Reply-To: References: Message-ID: Now that you mention it, (3L,4L) probably indeed occurs on Windows 64 bit installations. Not sure about Mac 64 bit. I haven't tried that. So, is it desirable that the 32bit returns different integers than the 64bit? I would guess not. Thanks, Mark Are they all using the same platform ? I suspect you're seeing the (3L, 4L) > on windows 64 bits, correct ? > > FWIW, the numpy provided in Canopy is straight up upstream numpy + few > patches not merged in releases yet, and any divergence from upstream would > be considered as a bug by the canopy packaging people (i.e. me :-) ). > > David > > Hello List, > > I am teaching a Python programming class where students use their own > computer. > > When I create an array with 3 rows and 4 columns, a = zeros((3,4)), and > ask for the shape, shape(a), then some students get (3,4), while others get > (3L,4L). > > I guess the 3L and 4L are long integers. I wonder why they don't all get > the same thing. They all installed numpy through Canopy Express. Any > thoughts? Is this a setting in printoptions maybe? > > Thanks, > > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.haessig at crans.org Fri Sep 13 04:51:57 2013 From: pierre.haessig at crans.org (Pierre Haessig) Date: Fri, 13 Sep 2013 10:51:57 +0200 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? In-Reply-To: References: Message-ID: <5232D22D.1090608@crans.org> Hi, Le 13/09/2013 10:32, Mark Bakker a ?crit : > Now that you mention it, (3L,4L) probably indeed occurs on Windows 64 > bit installations. > Not sure about Mac 64 bit. I haven't tried that. > > So, is it desirable that the 32bit returns different integers than the > 64bit? I would guess not. What I find strange is that on my 64 bits Debian I get (3,4) and not (3L,4L)... Pierre -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From robert.kern at gmail.com Fri Sep 13 05:22:55 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 13 Sep 2013 10:22:55 +0100 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? In-Reply-To: <5232D22D.1090608@crans.org> References: <5232D22D.1090608@crans.org> Message-ID: On Fri, Sep 13, 2013 at 9:51 AM, Pierre Haessig wrote: > > Hi, > > Le 13/09/2013 10:32, Mark Bakker a ?crit : > > Now that you mention it, (3L,4L) probably indeed occurs on Windows 64 > > bit installations. > > Not sure about Mac 64 bit. I haven't tried that. > > > > So, is it desirable that the 32bit returns different integers than the > > 64bit? I would guess not. > What I find strange is that on my 64 bits Debian I get (3,4) and not > (3L,4L)... The Python `int` type represents a C `long` integer. On almost all 32-bit platforms, a C `long` is 32-bits, and memory addresses and offsets are also 32-bits. On 64-bit platforms, memory addresses and offsets are 64-bits, but nothing in the C standard forces the `long` type to be 64-bits. Most UNIXy 64-bit platforms choose to make the `long` type 64-bits, but Win64 decided to keep them 32-bits for binary compatibility with the Windows API. Since ndarray sizes measure things the size of memory offsets, they need to be 64-bit on 64-bit platforms. On most platforms, this fits into a Python `int`. On Win64, they don't, so they get promoted to the unbounded Python `long` types. Is it desirable? Not really, but it is typically harmless. If you have code that is supposed to take ndarray shapes and does not accept either integer type, it is broken even if ndarray.shape always gave out `long` integers on all platforms. The only place this is really a problem is in things like doctest, where the string representation is being compared. But even then, floating point is going to be a *much* bigger problem. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.haessig at crans.org Fri Sep 13 05:30:37 2013 From: pierre.haessig at crans.org (Pierre Haessig) Date: Fri, 13 Sep 2013 11:30:37 +0200 Subject: [Numpy-discussion] shape is (3,4) or (3L,4L) ? In-Reply-To: References: <5232D22D.1090608@crans.org> Message-ID: <5232DB3D.9070909@crans.org> Hi Robert, Le 13/09/2013 11:22, Robert Kern a ?crit : > > The Python `int` type represents a C `long` integer. On almost all > 32-bit platforms, a C `long` is 32-bits, and memory addresses and > offsets are also 32-bits. On 64-bit platforms, memory addresses and > offsets are 64-bits, but nothing in the C standard forces the `long` > type to be 64-bits. Most UNIXy 64-bit platforms choose to make the > `long` type 64-bits, but Win64 decided to keep them 32-bits for binary > compatibility with the Windows API. > > [...] Thanks for the nice explanation ! Pierre -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From gregorio.bastardo at gmail.com Fri Sep 13 06:03:47 2013 From: gregorio.bastardo at gmail.com (Gregorio Bastardo) Date: Fri, 13 Sep 2013 12:03:47 +0200 Subject: [Numpy-discussion] can argmax be used to return row and column indices? In-Reply-To: References: Message-ID: There's already been some dance around this topic, maybe you will find the concept behind it: http://search.gmane.org/?query=unravel_index&group=gmane.comp.python.numeric.general 2013/9/13 Mark Bakker : > Thanks, Gregorio. > I would like it if argmax had a keyword option to return the row,column > index automatically (or whatever the dimension of the array). > Afterall, argmax already knows the shape of the array. > Calling np.unravel_index( np.argmax( A ) ) seems unnecessarily long. But it > works well though! > I am not sure that such a PR would get much support.... > Thanks again, > Mark > > > >> Hi Mark, >> >> You're looking for "np.unravel_index" function. >> >> Cheers, >> Gregorio > > >> >> >> >> Hello list, >> >> I am trying to find the indices of the maximum value in a 2D array. >> argmax works fine, but returns the index in the flattened array. >> That is often not very convenient. >> Is there a function that returns the index of the row and column? >> Or can the index of the flattened array easily be converted to the row can >> column indices (I know, I can write a function for that, but I figure numpy >> already has one). >> >> Thanks, >> >> Mark > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From martin.raspaud at smhi.se Fri Sep 13 07:42:36 2013 From: martin.raspaud at smhi.se (Martin Raspaud) Date: Fri, 13 Sep 2013 13:42:36 +0200 Subject: [Numpy-discussion] [ANN] Introductory Pytroll Workshop, Nov. 25-26 2013, at SMHI, Sweden Message-ID: <5232FA2C.4000508@smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [sorry for crossposting] Hi all, After the success of last years workshop, we decided to hold the Pytroll workshop this year again at SMHI. Pytroll is a collection of free and open source python modules for the reading, interpretation, and writing of weather satellite data. The provided python packages are designed to be used both in R&D environments and in 24/7 operational production. More info at: www.pytroll.org This time, it will be a two days introductory workshop, during which we will present the different packages of Pytroll, and help you get started with the installation and the different applications you might have. Preliminary program: Day 1: - - Presentation of the different pytroll components - - Installation of pytroll on the participants' computer Day 2: - - Illustration with different test cases and/or user defined applications brought by the participant. Please tell us if you plan to attend, so we can plan effectively. For any more information, have a look at last year's workshop's webpage: http://www.pytroll.org/workshop2012.html or contact us with any questions directly or on the pytroll googlegroup. Best regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSMvosAAoJEBdvyODiyJI4LfkH/iisXrIEEIHhPgDdKPFtqOrJ q6pFq70RkgXwDVWC3AoRfeE9eggO0mE2RZOPL16I9a+1RE/HFKz6Az4sVI5UtC0Z nAU+pJd1KOzEBPBn59LQtU6+wX0rOI5UCA6yPMmR3DxkDDLhbrHLeebwLgVQb+gG gVMl4ADdRFbt5nRkAs2/I/bD3gs3AxMwrSuXgP8hgF8IDyE/w1ZmwHivhEHHFup8 /AB2411rPuNRxf1T121nj+st8i6+PDw43L+jIbqL4bRPflwXDA1yibX/WiN+aunt z3q6kCDfofnoM4vv0PlPcsYdVoqs0L3q7w3GwIz1e8tPm94sSrFMn/5zDWEya4c= =TOlz -----END PGP SIGNATURE----- From martin.raspaud at smhi.se Fri Sep 13 07:55:59 2013 From: martin.raspaud at smhi.se (Martin Raspaud) Date: Fri, 13 Sep 2013 13:55:59 +0200 Subject: [Numpy-discussion] [ANN] Introductory Pytroll Workshop, Nov. 25-26 2013, at SMHI, Sweden Message-ID: <5232FD4F.3050505@smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [sorry for crossposting] Hi all, After the success of last years workshop, we decided to hold the Pytroll workshop this year again at SMHI. Pytroll is a collection of free and open source python modules for the reading, interpretation, and writing of weather satellite data. The provided python packages are designed to be used both in R&D environments and in 24/7 operational production. More info at: www.pytroll.org This time, it will be a two days introductory workshop, during which we will present the different packages of Pytroll, and help you get started with the installation and the different applications you might have. Preliminary program: Day 1: - - Presentation of the different pytroll components - - Installation of pytroll on the participants' computer Day 2: - - Illustration with different test cases and/or user defined applications brought by the participant. Please tell us if you plan to attend, so we can plan effectively. For any more information, have a look at last year's workshop's webpage: http://www.pytroll.org/workshop2012.html or contact us with any questions directly or on the pytroll googlegroup. Best regards, Martin - -- You received this message because you are subscribed to the Google Groups "python meteorology" group. To unsubscribe from this group and stop receiving emails from it, send an email to python-meteorology+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSMv1PAAoJEBdvyODiyJI4oiUIALqh3tdOov2Ykh6BLpfQRYdL BMw7BcHGZ81CXyldgQHQF4OROskce2od/FC18L3vgQrqFEI6uZ8hLqlhk3LvXrnP ToJsERolpcicAdcsMcMK2l/0ePUYfOS+Yu4pck4H1+skNlbBcBg98uW8ECXwe7Ks mhtQkcdNNSiEcCIe8/Yxb4Tbb6udM7CBBedJ7Xa5CmV5ZsC7lPKysOlNUZppTiBY +w5bfDRlPKEQHV6AkLZhyv6eYcd9Ncv5utcnRKICFbmXWtvpSC1c+zlB7vgAU7Um SU4f9UgsQw/Y+cd6OzxuQtsOrPO11NYXBFm0v8IcodGcgCYVhjVU6asQ32UFmTM= =XOzN -----END PGP SIGNATURE----- From ben.root at ou.edu Fri Sep 13 10:28:20 2013 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 13 Sep 2013 10:28:20 -0400 Subject: [Numpy-discussion] can argmax be used to return row and column indices? In-Reply-To: References: Message-ID: On Fri, Sep 13, 2013 at 4:27 AM, Mark Bakker wrote: > Thanks, Gregorio. > I would like it if argmax had a keyword option to return the row,column > index automatically (or whatever the dimension of the array). > Afterall, argmax already knows the shape of the array. > Calling np.unravel_index( np.argmax( A ) ) seems unnecessarily long. But > it works well though! > I am not sure that such a PR would get much support.... > Thanks again, > Mark > > What should it do when np.argmax() gets "axis=1" argument? I see confusion occurring with parsing the returned results for arbitrary dimension inputs. -1 on any such PR. +1 on making sure all arg*() functions have unravel_index() very prominent in their documentation (which it does right now for argmax()). Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 13 22:10:37 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 13 Sep 2013 20:10:37 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: <522CEFE6.2050306@uci.edu> References: <522CEFE6.2050306@uci.edu> Message-ID: Hi Christolph, Could you debug this a bit? ERROR: test_record_no_hash (test_multiarray.TestRecord) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site- packages\numpy\core\tests\test_multiarray.py", line 2464, in test_record_no_hash self.assertRaises(TypeError, hash, a[0]) File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises return context.handle('assertRaises', callableObj, args, kwargs) File "X:\Python33\lib\unittest\case.py", line 135, in handle callable_obj(*args, **kwargs) File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ self.obj_name)) TypeError: unhashable type: 'writeable void-scalar' I don't understand that at all. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sat Sep 14 02:12:09 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 13 Sep 2013 23:12:09 -0700 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: <522CEFE6.2050306@uci.edu> Message-ID: <5233FE39.30300@uci.edu> On 9/13/2013 7:10 PM, Charles R Harris wrote: > Hi Christolph, > > Could you debug this a bit? > > ERROR: test_record_no_hash (test_multiarray.TestRecord) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "X:\Python33\lib\site- > packages\numpy\core\tests\test_multiarray.py", > line 2464, in test_record_no_hash > self.assertRaises(TypeError, hash, a[0]) > File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises > return context.handle('assertRaises', callableObj, args, kwargs) > File "X:\Python33\lib\unittest\case.py", line 135, in handle > callable_obj(*args, **kwargs) > File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ > self.obj_name)) > TypeError: unhashable type: 'writeable void-scalar' > > > I don't understand that at all. > > > > Chuck > I could reproduce the weirdness outside the testsuite on 64 bit Python 3.2, 3.3 and 3.4. It looks like the interpreter gets corrupted. It could be due to numpy 1.8 using a wrong signature for the hash function [1] on win-amd64-py3.x. Since Python 3.2 the return value must be of type `Py_hash_t` (defined as `Py_ssize_t`) [2], not `C long` as for Python <= 3.1. [1] https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/scalartypes.c.src#L3038 [2] http://docs.python.org/3.2/c-api/typeobj.html#PyTypeObject.tp_hash Christoph From ckkart at hoc.net Sat Sep 14 08:15:22 2013 From: ckkart at hoc.net (Christian K.) Date: Sat, 14 Sep 2013 09:15:22 -0300 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: Message-ID: Am 08.09.13 16:14, schrieb Charles R Harris: > Hi all, > > I'm happy to announce the second beta release of Numpy 1.8.0. This > release should solve the Windows problems encountered in the first beta. > Many thanks to Christolph Gohlke and Julian Taylor for their hard work > in getting those issues settled. > > It would be good if folks running OS X could try out this release and > report any issues on the numpy-dev mailing list. Unfortunately the files > still need to be installed from source as dmg files are not avalable at > this time. Build works (built with latest Anaconda python) but the tests fail on my machine (OSX 10.8.4): FAIL: test_mode_raw (test_linalg.TestQR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/linalg/tests/test_linalg.py", line 798, in test_mode_raw old_assert_almost_equal(h, h1, decimal=8) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/testing/utils.py", line 454, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/testing/utils.py", line 811, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 8 decimals (mismatch 83.3333333333%) x: array([[ 5.91607978, -0.61024233, -1.01707056], [ 7.43735744, 0.82807867, -3.21391305]]) y: array([[-5.91607978, 0.43377175, 0.72295291], [-7.43735744, 0.82807867, 0.89262383]]) ====================================================================== FAIL: test_linalg.test_xerbla ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/ck/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/linalg/tests/test_linalg.py", line 925, in test_xerbla assert_(False) File "/Users/ck/anaconda/lib/python2.7/site-packages/numpy/testing/utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError ---------------------------------------------------------------------- Ran 5235 tests in 264.167s In [7]: np.__config__.show() atlas_threads_info: NOT AVAILABLE blas_opt_info: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] define_macros = [('NO_ATLAS_INFO', 3)] atlas_blas_threads_info: NOT AVAILABLE openblas_info: NOT AVAILABLE lapack_opt_info: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] extra_compile_args = ['-msse3'] define_macros = [('NO_ATLAS_INFO', 3)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE compilers: GNU Fortran (GCC) 4.2.3 i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) Should I use another compiler? Regards, Christian From pav at iki.fi Sat Sep 14 08:37:26 2013 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 14 Sep 2013 15:37:26 +0300 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: References: Message-ID: 14.09.2013 15:15, Christian K. kirjoitti: [clip] > FAIL: test_mode_raw (test_linalg.TestQR) > FAIL: test_linalg.test_xerbla [clip] > Should I use another compiler? Thanks. Both of the above are known issues on OSX, with known fixes (I think fixed in master branch, not sure if backported to 1.8.x). -- Pauli Virtanen From charlesr.harris at gmail.com Sat Sep 14 09:09:39 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 Sep 2013 07:09:39 -0600 Subject: [Numpy-discussion] ANN: 1.8.0b2 release. In-Reply-To: <5233FE39.30300@uci.edu> References: <522CEFE6.2050306@uci.edu> <5233FE39.30300@uci.edu> Message-ID: On Sat, Sep 14, 2013 at 12:12 AM, Christoph Gohlke wrote: > On 9/13/2013 7:10 PM, Charles R Harris wrote: > > Hi Christolph, > > > > Could you debug this a bit? > > > > ERROR: test_record_no_hash (test_multiarray.TestRecord) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > "X:\Python33\lib\site- > > packages\numpy\core\tests\test_multiarray.py", > > line 2464, in test_record_no_hash > > self.assertRaises(TypeError, hash, a[0]) > > File "X:\Python33\lib\unittest\case.py", line 570, in assertRaises > > return context.handle('assertRaises', callableObj, args, kwargs) > > File "X:\Python33\lib\unittest\case.py", line 135, in handle > > callable_obj(*args, **kwargs) > > File "X:\Python33\lib\unittest\case.py", line 153, in __exit__ > > self.obj_name)) > > TypeError: unhashable type: 'writeable void-scalar' > > > > > > I don't understand that at all. > > > > > > > > Chuck > > > > I could reproduce the weirdness outside the testsuite on 64 bit Python > 3.2, 3.3 and 3.4. It looks like the interpreter gets corrupted. > > It could be due to numpy 1.8 using a wrong signature for the hash > function [1] on win-amd64-py3.x. Since Python 3.2 the return value must > be of type `Py_hash_t` (defined as `Py_ssize_t`) [2], not `C long` as > for Python <= 3.1. > > [1] > > https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/scalartypes.c.src#L3038 > [2] http://docs.python.org/3.2/c-api/typeobj.html#PyTypeObject.tp_hash > > That would do it. I'll bet you are right. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Sun Sep 15 08:00:38 2013 From: tmp50 at ukr.net (Dmitrey) Date: Sun, 15 Sep 2013 15:00:38 +0300 Subject: [Numpy-discussion] [ANN] OpenOpt suite v 0.51 Message-ID: <1379245965.95520674.hcvdp5xv@fmst-1.ukr.net> Hi all, new OpenOpt suite v 0.51 has been released: Some improvements for? FuncDesigner ? automatic differentiation and QP FuncDesigner now can model sparse (MI)(QC)QP Octave QP solver has been connected MATLAB solvers linprog ( LP ), quadprog ( QP ), lsqlin ( LLSP ), bintprog ( MILP ) New NLP solver: knitro Some elements of 2nd order interval analysis, mostly for interalg Some interalg improvements interalg can directly handle (MI)LP and (possibly nonconvex) (MI)(QC)QP New classes: knapsack problem ( KSP ), bin packing problem ( BPP ), dominating set problem ( DSP ) FuncDesigner can model SOCP SpaceFuncs ? has been adjusted for recent versions of Python and NumPy visit http://openopt.org for more details. Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 15 21:04:15 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 Sep 2013 19:04:15 -0600 Subject: [Numpy-discussion] Binary releases Message-ID: Hi All, Numpy 1.8 is about ready for an rc1, which brings up the question of which binary builds so put up on sourceforge. For Windows maybe 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with MKL I think these should be good for both windows 7 and window 8, correct me if I am wrong. For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X version. The 10.5 version is a bit harder to come by than 10.6 and up. It looks like 10.9 is coming up, but it isn't out yet. I have no idea what Python version to match up these, but assuming all of them, then 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 2. OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 3. OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. That seems like a lot. It is fairly easy to compile from source on the mac these days, are all those binary packages really needed? I don't know what I am doing with the binary stuff, so any suggestions are welcome. For building it is possible to set up a Macintosh with vmware fusion to handle all of these, but there is time and money involved in that. Any one who is already capable of doing some of these builds is welcome to volunteer. Note, Intel has offered MKL licenses to the open source projects under the NumFocus umbrella, but I don't know of anyone who has taken advantage of that yet or how much time it will take to go through the undoubtedly needed paper work, but I would like to get some of those licenses and move to MSVC so that we can stop rolling around gasping in pain whenever it comes to window builds. Then there is Fortran for windows... Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From smortaz at exchange.microsoft.com Sun Sep 15 21:26:54 2013 From: smortaz at exchange.microsoft.com (Shahrokh Mortazavi) Date: Mon, 16 Sep 2013 01:26:54 +0000 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: <07b40be4810e40e2aca0ee800693d27c@DFM-DB3MBX15-08.exchange.corp.microsoft.com> hi chuck, this is shahrokh mortazavi from microsoft (proj lead on PTVS http://pytools.codeplex.com . 1. 1st - thank you and everyone else involved for helping creating, maintain & deliver these libraries esp on windows. 2. you shouldn't have any issues w licenses on windows. contact me directly & i'll provide you msdn licenses. i chat w friends at intel to provide fortran if required as well. cheers, s ________________________________ From: numpy-discussion-bounces at scipy.org on behalf of Charles R Harris Sent: Sunday, September 15, 2013 6:04 PM To: numpy-discussion Subject: [Numpy-discussion] Binary releases Hi All, Numpy 1.8 is about ready for an rc1, which brings up the question of which binary builds so put up on sourceforge. For Windows maybe 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with MKL I think these should be good for both windows 7 and window 8, correct me if I am wrong. For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X version. The 10.5 version is a bit harder to come by than 10.6 and up. It looks like 10.9 is coming up, but it isn't out yet. I have no idea what Python version to match up these, but assuming all of them, then 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 2. OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 3. OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. That seems like a lot. It is fairly easy to compile from source on the mac these days, are all those binary packages really needed? I don't know what I am doing with the binary stuff, so any suggestions are welcome. For building it is possible to set up a Macintosh with vmware fusion to handle all of these, but there is time and money involved in that. Any one who is already capable of doing some of these builds is welcome to volunteer. Note, Intel has offered MKL licenses to the open source projects under the NumFocus umbrella, but I don't know of anyone who has taken advantage of that yet or how much time it will take to go through the undoubtedly needed paper work, but I would like to get some of those licenses and move to MSVC so that we can stop rolling around gasping in pain whenever it comes to window builds. Then there is Fortran for windows... Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Sep 15 21:54:41 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 Sep 2013 19:54:41 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: <07b40be4810e40e2aca0ee800693d27c@DFM-DB3MBX15-08.exchange.corp.microsoft.com> References: <07b40be4810e40e2aca0ee800693d27c@DFM-DB3MBX15-08.exchange.corp.microsoft.com> Message-ID: On Sun, Sep 15, 2013 at 7:26 PM, Shahrokh Mortazavi < smortaz at exchange.microsoft.com> wrote: > hi chuck, > > this is shahrokh mortazavi from microsoft (proj lead on PTVS > http://pytools.codeplex.com . > > 1. 1st - thank you and everyone else involved for helping creating, > maintain & deliver these libraries esp on windows. > 2. you shouldn't have any issues w licenses on windows. contact me > directly & i'll provide you msdn licenses. i chat w friends at intel to > provide fortran if required as well. > > cheers, > Wow, that's a great offer. Let's see how the discussion goes so we can get organized on our end to take you up on that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 15 22:45:11 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 15 Sep 2013 22:45:11 -0400 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris wrote: > Hi All, > > Numpy 1.8 is about ready for an rc1, which brings up the question of which > binary builds so put up on sourceforge. For Windows maybe > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with > MKL Are we not running into problems with scipy? scipy would need to use the same libraries, AFAIU (given Fortran and maybe C compatibilities) Josef > > I think these should be good for both windows 7 and window 8, correct me if > I am wrong. For Mac there is first the question of OS X versions, (10.5?), > 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X > version. The 10.5 version is a bit harder to come by than 10.6 and up. It > looks like 10.9 is coming up, but it isn't out yet. I have no idea what > Python version to match up these, but assuming all of them, then > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked > with Accelerate. > OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked > with Accelerate. > OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked > with Accelerate. > > That seems like a lot. It is fairly easy to compile from source on the mac > these days, are all those binary packages really needed? > > I don't know what I am doing with the binary stuff, so any suggestions are > welcome. > > For building it is possible to set up a Macintosh with vmware fusion to > handle all of these, but there is time and money involved in that. Any one > who is already capable of doing some of these builds is welcome to > volunteer. Note, Intel has offered MKL licenses to the open source projects > under the NumFocus umbrella, but I don't know of anyone who has taken > advantage of that yet or how much time it will take to go through the > undoubtedly needed paper work, but I would like to get some of those > licenses and move to MSVC so that we can stop rolling around gasping in pain > whenever it comes to window builds. Then there is Fortran for windows... > > Thoughts? > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at gmail.com Mon Sep 16 02:10:48 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 16 Sep 2013 06:10:48 +0000 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 2:45 AM, wrote: > On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris > wrote: > > Hi All, > > > > Numpy 1.8 is about ready for an rc1, which brings up the question of > which > > binary builds so put up on sourceforge. For Windows maybe > > > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked > with > > MKL > > Are we not running into problems with scipy? > scipy would need to use the same libraries, AFAIU (given Fortran and > maybe C compatibilities) > Indeed. If numpy goes MSVC + MKL, then scipy should go the same way. Some other options to go to MinGW 4.x are being discussed on https://github.com/scipy/scipy/issues/2829. > > Josef > > > > > I think these should be good for both windows 7 and window 8, correct me > if > > I am wrong. For Mac there is first the question of OS X versions, > (10.5?), > > 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X > > version. The 10.5 version is a bit harder to come by than 10.6 and up. It > > looks like 10.9 is coming up, but it isn't out yet. I have no idea what > > Python version to match up these, but assuming all of them, then > > > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > linked > > with Accelerate. > > OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > linked > > with Accelerate. > > OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > linked > > with Accelerate. > > > > That seems like a lot. It is fairly easy to compile from source on the > mac > > these days, are all those binary packages really needed? > That's not exactly the right list - the same installers built on 10.6 also work on 10.7 and 10.8. > > > > I don't know what I am doing with the binary stuff, so any suggestions > are > > welcome. > > > > For building it is possible to set up a Macintosh with vmware fusion to > > handle all of these, but there is time and money involved in that. Any > one > > who is already capable of doing some of these builds is welcome to > > volunteer. > I can do the OS X builds if needed. Let's focus on the Windows builds first, those are much more of a problem. Ralf > Note, Intel has offered MKL licenses to the open source projects > > under the NumFocus umbrella, but I don't know of anyone who has taken > > advantage of that yet or how much time it will take to go through the > > undoubtedly needed paper work, but I would like to get some of those > > licenses and move to MSVC so that we can stop rolling around gasping in > pain > > whenever it comes to window builds. Then there is Fortran for windows... > > > > Thoughts? > > > > Chuck > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 16 05:49:55 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 16 Sep 2013 10:49:55 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 7:10 AM, Ralf Gommers wrote: > On Mon, Sep 16, 2013 at 2:45 AM, wrote: >> >> On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris >> wrote: >> > Hi All, >> > >> > Numpy 1.8 is about ready for an rc1, which brings up the question of >> > which >> > binary builds so put up on sourceforge. For Windows maybe >> > >> > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC >> > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked >> > with >> > MKL >> >> Are we not running into problems with scipy? >> scipy would need to use the same libraries, AFAIU (given Fortran and >> maybe C compatibilities) > > Indeed. If numpy goes MSVC + MKL, then scipy should go the same way. Some > other options to go to MinGW 4.x are being discussed on > https://github.com/scipy/scipy/issues/2829. Is it actually legal to distribute scipy linked with MKL? Scipy still includes GPL code (umfpack), and shipping MKL+GPL code integrated into a single download is extremely dicey... (this goes also for any downstream users who might package precompiled numpy/scipy with other packages). (In either case I think we ought to just bite the bullet and get MinGW 4.x running as a supported option, even if we don't use it for the official binaries and even if this requires some unaesthetic hacks. I bet we'd have more windows developers if there was an accessible way to build on windows...) -n From njs at pobox.com Mon Sep 16 05:56:55 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 16 Sep 2013 10:56:55 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 2:04 AM, Charles R Harris wrote: > Hi All, > > Numpy 1.8 is about ready for an rc1, which brings up the question of which > binary builds so put up on sourceforge. For Windows maybe > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with > MKL Another question is, can we start distributing wheels yet? That would make 'pip install' realistic on Windows, but I don't know how much trouble they are to build. -n From cournape at gmail.com Mon Sep 16 06:18:48 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 16 Sep 2013 11:18:48 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 10:49 AM, Nathaniel Smith wrote: > On Mon, Sep 16, 2013 at 7:10 AM, Ralf Gommers > wrote: > > On Mon, Sep 16, 2013 at 2:45 AM, wrote: > >> > >> On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris > >> wrote: > >> > Hi All, > >> > > >> > Numpy 1.8 is about ready for an rc1, which brings up the question of > >> > which > >> > binary builds so put up on sourceforge. For Windows maybe > >> > > >> > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > >> > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked > >> > with > >> > MKL > >> > >> Are we not running into problems with scipy? > >> scipy would need to use the same libraries, AFAIU (given Fortran and > >> maybe C compatibilities) > > > > Indeed. If numpy goes MSVC + MKL, then scipy should go the same way. Some > > other options to go to MinGW 4.x are being discussed on > > https://github.com/scipy/scipy/issues/2829. > > Is it actually legal to distribute scipy linked with MKL? Scipy still > includes GPL code (umfpack), and shipping MKL+GPL code integrated into > a single download is extremely dicey... (this goes also for any > downstream users who might package precompiled numpy/scipy with other > packages). > Wait, we don't includes UMFPACK. We can optionally link against it, but that's not done for any of our binary (unless this was changed recently ?) > > (In either case I think we ought to just bite the bullet and get MinGW > 4.x running as a supported option, even if we don't use it for the > official binaries and even if this requires some unaesthetic hacks. I > bet we'd have more windows developers if there was an accessible way > to build on windows...) > Mingw 4 already works for compilation, no ? If not, that's definitely something to fix. The discussion around binary distribution should not preclude supporting it for people who want it. David > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Sep 16 06:20:00 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 16 Sep 2013 11:20:00 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 10:56 AM, Nathaniel Smith wrote: > On Mon, Sep 16, 2013 at 2:04 AM, Charles R Harris > wrote: > > Hi All, > > > > Numpy 1.8 is about ready for an rc1, which brings up the question of > which > > binary builds so put up on sourceforge. For Windows maybe > > > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked > with > > MKL > > Another question is, can we start distributing wheels yet? That would > make 'pip install' realistic on Windows, but I don't know how much > trouble they are to build. > I made a proof of concept with Olivier Grisel from scikit learn at EuroScipy. I submitted a talk to pycon.fr to show how to automate the whole thing with vagrant/packer, and if the talk is accepted, I will work on making all the scripts available. David > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wrw at mac.com Mon Sep 16 09:31:39 2013 From: wrw at mac.com (William Ray Wing) Date: Mon, 16 Sep 2013 09:31:39 -0400 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> On Sep 15, 2013, at 9:04 PM, Charles R Harris wrote: > Hi All, > > Numpy 1.8 is about ready for an rc1, which brings up the question of which binary builds so put up on sourceforge. For Windows maybe [byte] > For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X version. The 10.5 version is a bit harder to come by than 10.6 and up. It looks like 10.9 is coming up, but it isn't out yet. I have no idea what Python version to match up these, but assuming all of them, then > ? OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. > ? OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. > ? OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. > That seems like a lot. It is fairly easy to compile from source on the mac these days, are all those binary packages really needed? > > I don't know what I am doing with the binary stuff, so any suggestions are welcome. If you will forgive an observation from a Mac user and (amateur) developer. I have twice tried to build Numpy from source and both times failed. The problem was that I couldn't find a single comprehensive set of directions that started from a virgin system (nothing but Apple's python and Xcode) and proceed to working copies of Numpy (and of course Matplotlib). Long time users know all about the differences between SourceForge, Github, and such. But bootstrapping pip, homebrew, macports, and similar was totally opaque to me. Sorry for the rant, but what I'm trying to say is that if there were such a recipe and it was clearly pointed to, then the need for a lengthy list of binaries would be pretty much moot. Thanks for listening, Bill From charlesr.harris at gmail.com Mon Sep 16 11:36:27 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Sep 2013 09:36:27 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: New summary 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with MKL These should be good for both windows 7 and window 8. For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 and 10.8, so 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. The main question seems to be distribution and coordination with scipy. I was thinking we would link in MKL statically, which I think should be OK. Christoph does that and it should decouple Numpy from Scipy. It may not be the most efficient way to do things, but it would work. My impression is that if we wanted to distribute a dynamic library then every user would need an MKL license to use it. It would be good to get this settled soon as we can't afford to futz around with this forever waiting to release Numpy 1.8 and Scipy 0.13. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Sep 16 13:54:00 2013 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 16 Sep 2013 20:54:00 +0300 Subject: [Numpy-discussion] Binary releases In-Reply-To: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: 16.09.2013 16:31, William Ray Wing kirjoitti: [clip] > If you will forgive an observation from a Mac user and (amateur) developer. > I have twice tried to build Numpy from source and both times failed. > The problem was that I couldn't find a single comprehensive set of > directions that started from a virgin system (nothing but > Apple's python and Xcode) and proceed to working copies of > Numpy (and of course Matplotlib). The problem with OSX is that as a software development platform with open source tools, it is simply immature. You have homebrew, macports, fink, python.org binaries, Apple's own Python installation, proprietary BLAS/LAPACK with its own quirks, unified multiarch binaries, and whatnot in the mix. This makes everything more complicated to support, especially as the situation evolves with time and "recipes" that once worked stop working. -- Pauli Virtanen From warren.weckesser at gmail.com Mon Sep 16 13:54:29 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 16 Sep 2013 13:54:29 -0400 Subject: [Numpy-discussion] Unexpected casting result Message-ID: An unexpected casting result was just reported on stackoverflow: http://stackoverflow.com/questions/18833639/attributeerror-in-python-numpy-when-constructing-function-for-certain-values The following show the essence of the issue: In [1]: np.__version__ Out[1]: '1.9.0.dev-6ce65d8' In [2]: type(np.array(1.) * (2**64-1)) Out[2]: numpy.float64 In [3]: type(np.array(1.) * (2**64)) Out[3]: float Note that the result of `np.array(1.0) * 2**64` is a Python float, not a numpy float64. Is this intentional? (As pointed out in the stackoverflow question, the issue https://github.com/numpy/numpy/issues/3409 is at least tangentially related.) Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Sep 16 14:12:42 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 16 Sep 2013 19:12:42 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris wrote: > New summary > > 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, > linked with MKL > > These should be good for both windows 7 and window 8. > > Wait, when was it decided to move to MSVC for the official binaries ? Especially using ifort/MKL on windows means it will be difficult for other projects to produce packages on top of it. For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, > 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 > and 10.8, so > > 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native > compiler, linked with Accelerate. > > The main question seems to be distribution and coordination with scipy. I > was thinking we would link in MKL statically, which I think should be OK. > Christoph does that and it should decouple Numpy from Scipy. It may not be > the most efficient way to do things, but it would work. My impression is > that if we wanted to distribute a dynamic library then every user would > need an MKL license to use it. > > It would be good to get this settled soon as we can't afford to futz > around with this forever waiting to release Numpy 1.8 and Scipy 0.13. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 16 14:19:08 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Sep 2013 12:19:08 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 12:12 PM, David Cournapeau wrote: > > > > On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> New summary >> >> 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC >> 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, >> linked with MKL >> >> These should be good for both windows 7 and window 8. >> >> > Wait, when was it decided to move to MSVC for the official binaries ? > Especially using ifort/MKL on windows means it will be difficult for other > projects to produce packages on top of it. > > It hasn't been decided, this is just a modified version of the initial post. Why not use MSVC? python.org does. What is the problem with statically linked MKL? Do other packages need to link to lapack? The windows build problem has hung around for years. I'm tired of it. > For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, >> 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 >> and 10.8, so >> >> 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native >> compiler, linked with Accelerate. >> >> The main question seems to be distribution and coordination with scipy. I >> was thinking we would link in MKL statically, which I think should be OK. >> Christoph does that and it should decouple Numpy from Scipy. It may not be >> the most efficient way to do things, but it would work. My impression is >> that if we wanted to distribute a dynamic library then every user would >> need an MKL license to use it. >> >> It would be good to get this settled soon as we can't afford to futz >> around with this forever waiting to release Numpy 1.8 and Scipy 0.13. >> >> Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Sep 16 14:26:15 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 16 Sep 2013 14:26:15 -0400 Subject: [Numpy-discussion] Unexpected casting result In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 1:54 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > An unexpected casting result was just reported on stackoverflow: > > http://stackoverflow.com/questions/18833639/attributeerror-in-python-numpy-when-constructing-function-for-certain-values > > The following show the essence of the issue: > > In [1]: np.__version__ > Out[1]: '1.9.0.dev-6ce65d8' > > In [2]: type(np.array(1.) * (2**64-1)) > Out[2]: numpy.float64 > > In [3]: type(np.array(1.) * (2**64)) > Out[3]: float > > Note that the result of `np.array(1.0) * 2**64` is a Python float, not a > numpy float64. Is this intentional? > > (As pointed out in the stackoverflow question, the issue > https://github.com/numpy/numpy/issues/3409 is at least tangentially > related.) > > Warren > > The original poster of the stackoverflow question has reported the issue on github: https://github.com/numpy/numpy/issues/3756 Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Sep 16 14:31:56 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 16 Sep 2013 19:31:56 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 7:19 PM, Charles R Harris wrote: > > > > On Mon, Sep 16, 2013 at 12:12 PM, David Cournapeau wrote: > >> >> >> >> On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> New summary >>> >>> 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC >>> 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, >>> linked with MKL >>> >>> These should be good for both windows 7 and window 8. >>> >>> >> Wait, when was it decided to move to MSVC for the official binaries ? >> Especially using ifort/MKL on windows means it will be difficult for other >> projects to produce packages on top of it. >> >> > It hasn't been decided, this is just a modified version of the initial > post. > ok, sorry for the confusion > Why not use MSVC? python.org does. What is the problem with statically > linked MKL? Do other packages need to link to lapack? The windows build > problem has hung around for years. I'm tired of it. > Which build problem ? Being tired of it does not justify a decision in particular, otherwise we fall into the fallacy "there is a problem, therefore we must do something". There are multiple issues: - moving away from gcc 3.x on 32 bits. Those compilers are old, but it works well today. It is an untenable situation in the long term, though. I will look again at building numpy/scipy with mingw-w64 in 32 bits to see if the problems with -static-* are gone or not. - 64 bits support: linked to first point. If the first point is solved, I am relatively confident this one can too. - moving away from Atlas to MKL: this is much more problematic. First, MKL is *huge*. For info, the MKL package we provide @ Enthought is 70 MB zip compressed, and that's for the DLLs. The static version may be even bigger - using ifort for fortran: by doing this, we impose on *every* package downstream to use ifort as well (same for MKL BTW). There is also the issue of a blas/lapack for windows 64 bits. There the situation has changed a lot since I last looked into those issues: cygwin (required by atlas) now supports 64 bits natively, and openblas is relatively well supported. I would certainly be happy to get rid of ATLAS which is a PITA to maintain, and use openblas instead. Other packages *do* need to link into lapack (scikit learn, theano are examples I am aware of). David For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, >>> 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 >>> and 10.8, so >>> >>> 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native >>> compiler, linked with Accelerate. >>> >>> The main question seems to be distribution and coordination with scipy. >>> I was thinking we would link in MKL statically, which I think should be OK. >>> Christoph does that and it should decouple Numpy from Scipy. It may not be >>> the most efficient way to do things, but it would work. My impression is >>> that if we wanted to distribute a dynamic library then every user would >>> need an MKL license to use it. >>> >>> It would be good to get this settled soon as we can't afford to futz >>> around with this forever waiting to release Numpy 1.8 and Scipy 0.13. >>> >>> > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 16 14:48:52 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Sep 2013 12:48:52 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 12:31 PM, David Cournapeau wrote: > > > > On Mon, Sep 16, 2013 at 7:19 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Sep 16, 2013 at 12:12 PM, David Cournapeau wrote: >> >>> >>> >>> >>> On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> New summary >>>> >>>> 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC >>>> 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, >>>> linked with MKL >>>> >>>> These should be good for both windows 7 and window 8. >>>> >>>> >>> Wait, when was it decided to move to MSVC for the official binaries ? >>> Especially using ifort/MKL on windows means it will be difficult for other >>> projects to produce packages on top of it. >>> >>> >> It hasn't been decided, this is just a modified version of the initial >> post. >> > > ok, sorry for the confusion > > >> Why not use MSVC? python.org does. What is the problem with statically >> linked MKL? Do other packages need to link to lapack? The windows build >> problem has hung around for years. I'm tired of it. >> > > Which build problem ? Being tired of it does not justify a decision in > particular, otherwise we fall into the fallacy "there is a problem, > therefore we must do something". There are multiple issues: > If it isn't a good reason, what is? ;) > > - moving away from gcc 3.x on 32 bits. Those compilers are old, but it > works well today. It is an untenable situation in the long term, though. I > will look again at building numpy/scipy with mingw-w64 in 32 bits to see if > the problems with -static-* are gone or not. > - 64 bits support: linked to first point. If the first point is solved, I > am relatively confident this one can too. > OK. What is the timeline? Days, weeks, months? - moving away from Atlas to MKL: this is much more problematic. First, MKL > is *huge*. For info, the MKL package we provide @ Enthought is 70 MB zip > compressed, and that's for the DLLs. The static version may be even bigger > I don't think static linkage would bring in the whole library, just the parts needed. Christolph's packages are < 10MB. Redistribution using the offered licenses is a potential problem that we need to get clarification on. No redistribution, no MKL. > - using ifort for fortran: by doing this, we impose on *every* package > downstream to use ifort as well (same for MKL BTW). > How does that work? > > There is also the issue of a blas/lapack for windows 64 bits. There the > situation has changed a lot since I last looked into those issues: cygwin > (required by atlas) now supports 64 bits natively, and openblas is > relatively well supported. I would certainly be happy to get rid of ATLAS > which is a PITA to maintain, and use openblas instead. > > Other packages *do* need to link into lapack (scikit learn, theano are > examples I am aware of). > So we need to distribute an LAPACK DLL? That sounds like a separate problem. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wrw at mac.com Mon Sep 16 14:57:10 2013 From: wrw at mac.com (William Ray Wing) Date: Mon, 16 Sep 2013 14:57:10 -0400 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Sep 16, 2013, at 1:54 PM, Pauli Virtanen wrote: > 16.09.2013 16:31, William Ray Wing kirjoitti: > [clip] >> If you will forgive an observation from a Mac user and (amateur) developer. >> I have twice tried to build Numpy from source and both times failed. >> The problem was that I couldn't find a single comprehensive set of >> directions that started from a virgin system (nothing but >> Apple's python and Xcode) and proceed to working copies of >> Numpy (and of course Matplotlib). > > The problem with OSX is that as a software development platform with > open source tools, it is simply immature. You have homebrew, macports, > fink, python.org binaries, Apple's own Python installation, proprietary > BLAS/LAPACK with its own quirks, unified multiarch binaries, and whatnot > in the mix. This makes everything more complicated to support, > especially as the situation evolves with time and "recipes" that once > worked stop working. > > -- > Pauli Virtanen > I think I could make a good case that maintaining a recipe is a lot less work than maintaining a binary distribution. Doing the second implicitly requires doing the first anyway. Thanks, Bill From rowen at uw.edu Mon Sep 16 15:01:26 2013 From: rowen at uw.edu (Russell E. Owen) Date: Mon, 16 Sep 2013 12:01:26 -0700 Subject: [Numpy-discussion] Binary releases References: Message-ID: In article , Ralf Gommers wrote: > On Mon, Sep 16, 2013 at 2:45 AM, wrote: > > > On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris > > wrote: > > > Hi All, > > > > > > Numpy 1.8 is about ready for an rc1, which brings up the question of > > which > > > binary builds so put up on sourceforge. For Windows maybe > > >... > > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked > > > with Accelerate. > > > OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked > > > with Accelerate. > > > OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked > > > with Accelerate. > > > > > > That seems like a lot. It is fairly easy to compile from source on the > > mac > > > these days, are all those binary packages really needed? > > > > That's not exactly the right list - the same installers built on 10.6 also > work on 10.7 and 10.8. I agree. I'll chime in and give my recommendations, though Ralf is the expert: For MacOS X I suggest building binary installers for python.org's python 2.7, 3.2 and 3.3 (the 64-bit versions). The result will run on 10.6 and later. It is safest to build these on MacOS X 10.6; it may work to build on a later MacOS X, but it sure doesn't for some packages. You will have to update to the latest bdist_mpkg to build Mac binary installers for python 3. I've not tried it yet. I don't think users expect a binary installer for Apple's python; I don't recall ever seeing these for numpy, scipy, matplotlib.... But if you do want to supply one, Apple provides Python 2.5, 2.6 and 2.7 but no 3.x (at least in MacOS X 10.8). -- Russell From njs at pobox.com Mon Sep 16 15:05:50 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 16 Sep 2013 20:05:50 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris wrote: > New summary > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with > MKL > > These should be good for both windows 7 and window 8. > > For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, > 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 and > 10.8, so > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked > with Accelerate. > > The main question seems to be distribution and coordination with scipy. I > was thinking we would link in MKL statically, which I think should be OK. > Christoph does that and it should decouple Numpy from Scipy. It may not be > the most efficient way to do things, but it would work. My impression is > that if we wanted to distribute a dynamic library then every user would need > an MKL license to use it. > > It would be good to get this settled soon as we can't afford to futz around > with this forever waiting to release Numpy 1.8 and Scipy 0.13. Why not just release numpy 1.8 with the old and terrible system? As you know I'm 110% in favor of getting rid of it, but 1.8 is ready to go and 1.9 is coming soon enough, and the old and terrible system does work right now, today. None of the other options have this property. As you know, parallelization is the key to performance, and reducing serial data dependencies is the key to parallelization ;-). -n From rowen at uw.edu Mon Sep 16 15:09:47 2013 From: rowen at uw.edu (Russell E. Owen) Date: Mon, 16 Sep 2013 12:09:47 -0700 Subject: [Numpy-discussion] Binary releases References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: In article <8E95A257-3F06-43B7-8407-95916D284FE6 at mac.com>, William Ray Wing wrote: > On Sep 15, 2013, at 9:04 PM, Charles R Harris > wrote: > > > Hi All, > > > > Numpy 1.8 is about ready for an rc1, which brings up the question of which > > binary builds so put up on sourceforge. For Windows maybe > > [byte] > > > For Mac there is first the question of OS X versions, (10.5?), 10.6, > > 10.7, 10.8. I don't know if some builds work on more than one OS X version. > > The 10.5 version is a bit harder to come by than 10.6 and up. It looks like > > 10.9 is coming up, but it isn't out yet. I have no idea what Python version > > to match up these, but assuming all of them, then > > ? OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked with Accelerate. > > ? OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked with Accelerate. > > ? OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > linked with Accelerate. > > That seems like a lot. It is fairly easy to compile from source on the mac > > these days, are all those binary packages really needed? > > > > I don't know what I am doing with the binary stuff, so any suggestions are > > welcome. > > > If you will forgive an observation from a Mac user and (amateur) developer. > I have twice tried to build Numpy from source and both times failed. The > problem was that I couldn't find a single comprehensive set of directions > that started from a virgin system (nothing but Apple's python and Xcode) and > proceed to working copies of Numpy (and of course Matplotlib). > > Long time users know all about the differences between SourceForge, Github, > and such. But bootstrapping pip, homebrew, macports, and similar was totally > opaque to me. > > Sorry for the rant, but what I'm trying to say is that if there were such a > recipe and it was clearly pointed to, then the need for a lengthy list of > binaries would be pretty much moot. > > Thanks for listening, > Bill I sympathize. Unfortunately it changes all the time so it's hard to keep up to date. The usual suggestion is to either install a self-contained python distribution such as Anaconda, which contains all sorts of useful packages, or use the the binary installer (which requires python.org python). For the record: binary installers offer a very important additional benefit: the resulting package can be included in an application with some assurance about what versions of MacOS X it supports. If you build from source you probably have no idea what versions of MacOS X the package will support -- which is fine for personal use, but not for distribution. -- Russell From charlesr.harris at gmail.com Mon Sep 16 15:10:08 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Sep 2013 13:10:08 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 1:05 PM, Nathaniel Smith wrote: > On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris > wrote: > > New summary > > > > 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC > > 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked > with > > MKL > > > > These should be good for both windows 7 and window 8. > > > > For Mac there is first the question of OS X versions, (10.5?), 10.6, > 10.7, > > 10.8. If 10.5 is omitted, packages built on 10.6 should be good for 10.7 > and > > 10.8, so > > > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > linked > > with Accelerate. > > > > The main question seems to be distribution and coordination with scipy. I > > was thinking we would link in MKL statically, which I think should be OK. > > Christoph does that and it should decouple Numpy from Scipy. It may not > be > > the most efficient way to do things, but it would work. My impression is > > that if we wanted to distribute a dynamic library then every user would > need > > an MKL license to use it. > > > > It would be good to get this settled soon as we can't afford to futz > around > > with this forever waiting to release Numpy 1.8 and Scipy 0.13. > > Why not just release numpy 1.8 with the old and terrible system? As > you know I'm 110% in favor of getting rid of it, but 1.8 is ready to > go and 1.9 is coming soon enough, and the old and terrible system does > work right now, today. None of the other options have this property. > > As you know, parallelization is the key to performance, and reducing > serial data dependencies is the key to parallelization ;-). > And necessity is the mother of invention ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.j.a.cock at googlemail.com Mon Sep 16 15:15:06 2013 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 16 Sep 2013 20:15:06 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 8:05 PM, Nathaniel Smith wrote: > > Why not just release numpy 1.8 with the old and terrible system? As > you know I'm 110% in favor of getting rid of it, but 1.8 is ready to > go and 1.9 is coming soon enough, and the old and terrible system does > work right now, today. None of the other options have this property. On the down side, the "old and terrible system" does not cover providing pre-built binaries for 64 bit Windows. Doing that right is important not just for SciPy but for any other downstream package including C code compiled against the NumPy C API (and the people doing this probably will only have access to free compilers). Regards, Peter From njs at pobox.com Mon Sep 16 16:05:57 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 16 Sep 2013 21:05:57 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 8:15 PM, Peter Cock wrote: > On Mon, Sep 16, 2013 at 8:05 PM, Nathaniel Smith wrote: >> >> Why not just release numpy 1.8 with the old and terrible system? As >> you know I'm 110% in favor of getting rid of it, but 1.8 is ready to >> go and 1.9 is coming soon enough, and the old and terrible system does >> work right now, today. None of the other options have this property. > > On the down side, the "old and terrible system" does not > cover providing pre-built binaries for 64 bit Windows. > > Doing that right is important not just for SciPy but for any > other downstream package including C code compiled > against the NumPy C API (and the people doing this > probably will only have access to free compilers). That's not a downside -- that's the situation right now and will continue to be the situation for the immediate future, if we cut a 1.8rc1 tomorrow and also if we don't cut a 1.8rc1 tomorrow. Again, I'm absolutely behind getting this sorted out, but holding up the release on all platforms is not going to make win64 standalone binaries appear any faster, and in the mean time everyone seems to be getting along OK, either because they're using a distribution, are on another platform, or taking advantage of Cristoph's generosity (thank you Cristoph!). Worst case, if it all gets sorted out next week we could release an 1.8.1 to celebrate... -n From ralf.gommers at gmail.com Mon Sep 16 17:51:52 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 16 Sep 2013 21:51:52 +0000 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 7:01 PM, Russell E. Owen wrote: > In article > , > Ralf Gommers wrote: > > > On Mon, Sep 16, 2013 at 2:45 AM, wrote: > > > > > On Sun, Sep 15, 2013 at 9:04 PM, Charles R Harris > > > wrote: > > > > Hi All, > > > > > > > > Numpy 1.8 is about ready for an rc1, which brings up the question of > > > which > > > > binary builds so put up on sourceforge. For Windows maybe > > > >... > > > > OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > > linked > > > > with Accelerate. > > > > OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > > linked > > > > with Accelerate. > > > > OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, > > > linked > > > > with Accelerate. > > > > > > > > That seems like a lot. It is fairly easy to compile from source on > the > > > mac > > > > these days, are all those binary packages really needed? > > > > > > > That's not exactly the right list - the same installers built on 10.6 > also > > work on 10.7 and 10.8. > > I agree. I'll chime in and give my recommendations, though Ralf is the > expert: > David is our resident build/distribute guru actually. > For MacOS X I suggest building binary installers for python.org's python > 2.7, 3.2 and 3.3 (the 64-bit versions). The result will run on 10.6 and > later. It is safest to build these on MacOS X 10.6; it may work to build > on a later MacOS X, but it sure doesn't for some packages. > Agreed. I think we discussed before not providing OS X 10.5 and python 2.6 binaries, that would make sense imho. > > You will have to update to the latest bdist_mpkg to build Mac binary > installers for python 3. I've not tried it yet. > I just tried, it's now possible to build 3.x binaries with a simple ``paver dmg -p 3.3``. > > I don't think users expect a binary installer for Apple's python; I > don't recall ever seeing these for numpy, scipy, matplotlib.... But if > you do want to supply one, We don't I think. Just python.org, keeps it simple. Cheers, Ralf > Apple provides Python 2.5, 2.6 and 2.7 but no > 3.x (at least in MacOS X 10.8). > > -- Russell > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Sep 16 19:28:12 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Sep 2013 16:28:12 -0700 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 2:56 AM, Nathaniel Smith wrote: > Another question is, can we start distributing wheels yet? > yes, yes, yes -- though maybe not for the beta testing. wheels will never catch on , or even get debugged if none of us use them. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Sep 16 19:32:04 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Sep 2013 16:32:04 -0700 Subject: [Numpy-discussion] Binary releases In-Reply-To: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 6:31 AM, William Ray Wing wrote: > If you will forgive an observation from a Mac user and (amateur) > developer. I have twice tried to build Numpy from source and both times > failed. The problem was that I couldn't find a single comprehensive set of > directions that started from a virgin system (nothing but Apple's python > and Xcode) and proceed to working copies of Numpy > This is odd -- it could build out of the box with plain old: setup.py install or, for that matter. pip install. > (and of course Matplotlib). > > MPL is a whole different kettle of fish -- I know most folks need more of the "stack" than numpy, but are you sure you had trouble building numpy? Sorry for the rant, but what I'm trying to say is that if there were such a > recipe and it was clearly pointed to, then the need for a lengthy list of > binaries would be pretty much moot. > Actually, numpy should be easy, but MPL. scipy, etc, -- there is no such think as a single easy recipe -- too many versions of python, OS-X, systems for the dependencies, etc. sad, but true. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Sep 16 19:36:56 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Sep 2013 16:36:56 -0700 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 12:15 PM, Peter Cock wrote: > Doing that right is important not just for SciPy but for any > other downstream package including C code compiled > against the NumPy C API (and the people doing this > probably will only have access to free compilers). > > well, "free as in beer" anyway -- I"ve always used the MS "express" compilers for my work. And I'd like to continue. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 16 19:55:35 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 Sep 2013 17:55:35 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 12:31 PM, David Cournapeau wrote: > > > > On Mon, Sep 16, 2013 at 7:19 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Sep 16, 2013 at 12:12 PM, David Cournapeau wrote: >> >>> >>> >>> >>> On Mon, Sep 16, 2013 at 4:36 PM, Charles R Harris < >>> charlesr.harris at gmail.com> wrote: >>> >>>> New summary >>>> >>>> 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC >>>> 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, >>>> linked with MKL >>>> >>>> These should be good for both windows 7 and window 8. >>>> >>>> >>> Wait, when was it decided to move to MSVC for the official binaries ? >>> Especially using ifort/MKL on windows means it will be difficult for other >>> projects to produce packages on top of it. >>> >>> >> It hasn't been decided, this is just a modified version of the initial >> post. >> > > ok, sorry for the confusion > > >> Why not use MSVC? python.org does. What is the problem with statically >> linked MKL? Do other packages need to link to lapack? The windows build >> problem has hung around for years. I'm tired of it. >> > > Which build problem ? Being tired of it does not justify a decision in > particular, otherwise we fall into the fallacy "there is a problem, > therefore we must do something". There are multiple issues: > > - moving away from gcc 3.x on 32 bits. Those compilers are old, but it > works well today. It is an untenable situation in the long term, though. I > will look again at building numpy/scipy with mingw-w64 in 32 bits to see if > the problems with -static-* are gone or not. > - 64 bits support: linked to first point. If the first point is solved, I > am relatively confident this one can too. > - moving away from Atlas to MKL: this is much more problematic. First, > MKL is *huge*. For info, the MKL package we provide @ Enthought is 70 MB > zip compressed, and that's for the DLLs. The static version may be even > bigger > - using ifort for fortran: by doing this, we impose on *every* package > downstream to use ifort as well (same for MKL BTW). > > There is also the issue of a blas/lapack for windows 64 bits. There the > situation has changed a lot since I last looked into those issues: cygwin > (required by atlas) now supports 64 bits natively, and openblas is > relatively well supported. I would certainly be happy to get rid of ATLAS > which is a PITA to maintain, and use openblas instead. > > Openblas includes some of lapack and is available for x86_64 on windows. It isn't clear what is included nor what compiler was used. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron at ahmadia.net Tue Sep 17 08:24:47 2013 From: aron at ahmadia.net (Aron Ahmadia) Date: Tue, 17 Sep 2013 07:24:47 -0500 Subject: [Numpy-discussion] Handling warnings when solving nearly-singular matrices Message-ID: Hey folks, The mathematicians over at Walking Randomly have expressed some dissatisfaction with the lack of warnings when solving nearly-singular matrices: http://www.walkingrandomly.com/?p=5092 An issue has been raised here: https://github.com/numpy/numpy/issues/3755 The Julia developers are also discussing this here: https://github.com/JuliaLang/julia/issues/4286 I don't have the cycles to submit a patch immediately, but I can come up with something this weekend if somebody doesn't want to work on this sooner. Cheers, Aron -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Sep 17 16:20:45 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 17 Sep 2013 22:20:45 +0200 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 16, 2013 at 10:05 PM, Nathaniel Smith wrote: > On Mon, Sep 16, 2013 at 8:15 PM, Peter Cock > wrote: > > On Mon, Sep 16, 2013 at 8:05 PM, Nathaniel Smith wrote: > >> > >> Why not just release numpy 1.8 with the old and terrible system? As > >> you know I'm 110% in favor of getting rid of it, but 1.8 is ready to > >> go and 1.9 is coming soon enough, and the old and terrible system does > >> work right now, today. None of the other options have this property. > The above makes a lot of sense, so I decided to check that it actually does work. Unsurprisingly, it needs fixing: https://github.com/numpy/numpy/issues/3760 Ralf > > > > On the down side, the "old and terrible system" does not > > cover providing pre-built binaries for 64 bit Windows. > > > > Doing that right is important not just for SciPy but for any > > other downstream package including C code compiled > > against the NumPy C API (and the people doing this > > probably will only have access to free compilers). > > That's not a downside -- that's the situation right now and will > continue to be the situation for the immediate future, if we cut a > 1.8rc1 tomorrow and also if we don't cut a 1.8rc1 tomorrow. Again, I'm > absolutely behind getting this sorted out, but holding up the release > on all platforms is not going to make win64 standalone binaries appear > any faster, and in the mean time everyone seems to be getting along > OK, either because they're using a distribution, are on another > platform, or taking advantage of Cristoph's generosity (thank you > Cristoph!). > > Worst case, if it all gets sorted out next week we could release an > 1.8.1 to celebrate... > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Sep 17 19:16:02 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 17 Sep 2013 16:16:02 -0700 Subject: [Numpy-discussion] Using setuptools develop mode with numpy distutils Message-ID: Hi, I need to use numpy's distutils Extension to build a fortran / f2py extension. But I also really like setuptools' "develop" mode. However, if I do: from numpy.distutils.core import Extension and use that to define my Extension object, but use: from setuptools import setup And use that for the main setup, I get: error: each element of 'ext_modules' option must be an Extension instance or 2-tuple So apparently setuptools doesn't recognize numpy's Extension object as an Extension object. Is there a way around this? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Tue Sep 17 20:51:01 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Tue, 17 Sep 2013 17:51:01 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation Message-ID: Hi all, While I realize that this is certainly tweaking multiprocessing beyond its specifications, I would like to use it on Windows to start a 32-bit Python process from a 64-bit Python process (use case: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) In fact, this is "easy" to do by using multiprocessing.set_executable (...while that may not be its original role): import multiprocessing as mp import imp, site, sys if "32" in sys.executable: # checking for my 32-bit Python install del sys.path[1:] # recompute sys.path print(sys.path) site.main() print(sys.path) # now points to the 32bit site-packages import numpy if __name__ == '__main__': mp.set_executable(sys.executable.replace("33", "33-32")) # path of my 32-bit Python install mp.Process(target=lambda: None).start() The sys.path modifications are needed as otherwise the child process inherits the parent's sys.path and importing numpy (from the 64-bit path) fails as it is "not a valid Win32 application", complains Python (rightly). However, even after the sys.path modifications, the numpy import fails with the error message (that I had never seen before): from . import multiarray # <- numpy/core/__init__.py, line 5 SystemError: initialization of multiarray raised an unreported exception Any hints as to how this could be fixed would be most welcome. Thanks in advance, Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 18 05:19:15 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Sep 2013 10:19:15 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 1:51 AM, Antony Lee wrote: > > Hi all, > > While I realize that this is certainly tweaking multiprocessing beyond its specifications, I would like to use it on Windows to start a 32-bit Python process from a 64-bit Python process (use case: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) Just use subprocess to start up the 32-bit Python. If you want to use the multiprocessing tools for communicating data, use a Manager server in the 32-bit Python to communicate over a socket. http://docs.python.org/2/library/multiprocessing#managers http://docs.python.org/2/library/multiprocessing#using-a-remote-manager It is possible that this won't work if the protocol assumes that the bitness is the same between server and client (e.g. struct.pack('Q', ...)), but I suspect this is not the case. You may also consider writing a small server using pyzmq or similar. I am guessing that you are just calling one function from pyFFTW and getting the result back. A simple REQ/REP server is easy to write with pyzmq. Do you need to use pyFFTW for some specific functionality that is not available in numpy.fft or scipy.fftpack? -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 18 05:20:51 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 18 Sep 2013 10:20:51 +0100 Subject: [Numpy-discussion] Using setuptools develop mode with numpy distutils In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 12:16 AM, Chris Barker wrote: > > Hi, > > I need to use numpy's distutils Extension to build a fortran / f2py extension. > > But I also really like setuptools' "develop" mode. Just import setuptools before doing the numpy.distutils imports. You don't need to use setuptools.setup() in order to get access to the setuptools-added commands. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Wed Sep 18 10:20:27 2013 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 18 Sep 2013 16:20:27 +0200 Subject: [Numpy-discussion] ANN: SfePy 2013.3 Message-ID: <5239B6AB.3050208@ntc.zcu.cz> I am pleased to announce release 2013.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Downloads, mailing list, wiki: http://code.google.com/p/sfepy/ Git (source) repository, issue tracker: http://github.com/sfepy Highlights of this release -------------------------- - implementation of Mesh topology data structures in C - implementation of regions based on C Mesh (*) - MultiProblem solver for conjugate solution of subproblems - new advanced examples (vibro-acoustics, Stokes flow with slip conditions) (*) Warning: region selection syntax has been changed in a principal way, see [1]. Besides the simple renaming, all regions meant for boundary conditions or boundary/surface integrals need to have their kind set explicitly to 'facet' (or 'edge' in 2D, 'face' in 3D). [1] http://sfepy.org/doc-devel/users_guide.html#regions For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke? From joseluismietta at yahoo.com.ar Wed Sep 18 13:10:40 2013 From: joseluismietta at yahoo.com.ar (=?utf-8?B?Sm9zw6ggTHVpcyBNaWV0dGE=?=) Date: Wed, 18 Sep 2013 10:10:40 -0700 (PDT) Subject: [Numpy-discussion] ANN: SfePy 2013.3 In-Reply-To: <5239B6AB.3050208@ntc.zcu.cz> References: <5239B6AB.3050208@ntc.zcu.cz> Message-ID: <1379524240.71081.YahooMailNeo@web142303.mail.bf1.yahoo.com> Dear Robert. Im intresting in modeling mechanical deformation of magnetorheological elastomers (material formed by inorganic chains inserting in a polymeric matrix -see figure 2 in the attached file-). The inorganic chais are like pearl necklace (are formed by saligned pheres ). Can I use this tool (SfePy) for modeling the mechanical deformation (i.e.: shape deformation by appling a force in longitudinal and transversal direction respect to the inorganic chains)? How can I do that? Im a newby user of python and I dont know anything about simple finite elements analysis. What documentation (books or other) do yo recommend for my introduction in this topic? ? Best Regards, Jos? Luis Mietta ________________________________ De: Robert Cimrman Para: SciPy Developers List ; Discussion of Numerical Python ; scipy-user at scipy.org Enviado: mi?rcoles, 18 de septiembre de 2013 11:20 Asunto: [Numpy-discussion] ANN: SfePy 2013.3 I am pleased to announce release 2013.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Downloads, mailing list, wiki: http://code.google.com/p/sfepy/ Git (source) repository, issue tracker: http://github.com/sfepy Highlights of this release -------------------------- - implementation of Mesh topology data structures in C - implementation of regions based on C Mesh (*) - MultiProblem solver for conjugate solution of subproblems - new advanced examples (vibro-acoustics, Stokes flow with slip conditions) (*) Warning: region selection syntax has been changed in a principal way, see [1]. Besides the simple renaming, all regions meant for boundary conditions or boundary/surface integrals need to have their kind set explicitly to 'facet' (or 'edge' in 2D, 'face' in 3D). [1] http://sfepy.org/doc-devel/users_guide.html#regions For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke? _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Tomografia de MRE.pdf Type: application/pdf Size: 3931754 bytes Desc: not available URL: From bahtiyor_zohidov at mail.ru Wed Sep 18 14:11:42 2013 From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=) Date: Wed, 18 Sep 2013 22:11:42 +0400 Subject: [Numpy-discussion] =?utf-8?q?ValueError=3A_matrices_are_not_align?= =?utf-8?q?ed!!!?= Message-ID: <1379527902.575983897@f106.i.mail.ru> Hello, I am trying to solve linear Ax=b problem, but some error occurs in the process like:? ---------------------------------------------- Traceback (most recent call last): File "C:\Python27\Conjugate_Gradient.py", line 66, in x, iter_number = conjGrad(Av,A, x, b) File "C:\Python27\Conjugate_Gradient.py", line 51, in conjGrad u = Av(A,s) File "C:\Python27\Conjugate_Gradient.py", line 41, in Av return np.dot(A,v) ValueError: matrices are not aligned -----------------------------------------------? I put the code below to check it. import numpy as np import math def Av(A,v): ? ? ?return np.dot(A,v) def conjGrad(Av,A, x, b, tol=1.0e-9): ? ? n = len(x) ? ? r = b - Av(A,x) ? ? s = r.copy() ? ? for i in range(n): ? ? ? ? ?u = Av(A,s) ? ? ? ? ?alpha = np.dot(s,r)/np.dot(s,u) ? ? ? ? ?x = x + aplha*s ? ? ? ? ?r = b - Av(A,x) ? ? ? ? ?if (math.sqrt(np.dot(r,r))) < tol: ? ? ? ? ? ? ? ? ? ? ?break ? ? ? ? ?else: ? ? ? ? ? ? ? ? ? ? ?beta = - np.dot(r,u)/np.dot(s,u) ? ? ? ? ? ? ? ? ? ? ?s = r + beta * s ? ? return x,i if __name__ == '__main__': ? ? ?# Input values ? ? ?A = np.arange(32, dtype=float).reshape((4,8)) ? ? ?x = np.zeros(8) ? ? ?b = np.array([2.5, 4.5, 6.5, 8.0]) ? ? ?x, iter_number = conjGrad(Av,A, x, b) I would appreciate any solution to this problem... Thanks in advance -- happy man -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad.froehle at gmail.com Wed Sep 18 14:50:19 2013 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Wed, 18 Sep 2013 11:50:19 -0700 Subject: [Numpy-discussion] ValueError: matrices are not aligned!!! In-Reply-To: <1379527902.575983897@f106.i.mail.ru> References: <1379527902.575983897@f106.i.mail.ru> Message-ID: In [2]: %debug > (5)Av() 4 def Av(A,v): ----> 5 return np.dot(A,v) 6 ipdb> !A.shape (4, 8) ipdb> !v.shape (4,) In your code it looks like you are essentially computing A.dot(v) where A is a 4-by-8 matrix and v is vector of length 4. That's what the error is telling you --- that the matrix and vector have incompatible dimensions. I've never seen the conjugate gradient method used on non-square matrices... are you sure this is what you want to be doing? -Brad On Wed, Sep 18, 2013 at 11:11 AM, Happyman wrote: > Hello, > > I am trying to solve linear Ax=b problem, but some error occurs in the > process like: > ---------------------------------------------- > > Traceback (most recent call last): > File "C:\Python27\Conjugate_Gradient.py", line 66, in > x, iter_number = conjGrad(Av,A, x, b) > File "C:\Python27\Conjugate_Gradient.py", line 51, in conjGrad > u = Av(A,s) > File "C:\Python27\Conjugate_Gradient.py", line 41, in Av > return np.dot(A,v) > ValueError: matrices are not aligned > ----------------------------------------------- > > I put the code below to check it. > > import numpy as np > import math > > def Av(A,v): > return np.dot(A,v) > > def conjGrad(Av,A, x, b, tol=1.0e-9): > n = len(x) > r = b - Av(A,x) > s = r.copy() > for i in range(n): > u = Av(A,s) > alpha = np.dot(s,r)/np.dot(s,u) > x = x + aplha*s > r = b - Av(A,x) > if (math.sqrt(np.dot(r,r))) < tol: > break > else: > beta = - np.dot(r,u)/np.dot(s,u) > s = r + beta * s > return x,i > > if __name__ == '__main__': > # Input values > A = np.arange(32, dtype=float).reshape((4,8)) > x = np.zeros(8) > b = np.array([2.5, 4.5, 6.5, 8.0]) > x, iter_number = conjGrad(Av,A, x, b) > > I would appreciate any solution to this problem... > Thanks in advance > -- > happy man > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From cimrman3 at ntc.zcu.cz Wed Sep 18 15:21:01 2013 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 18 Sep 2013 21:21:01 +0200 Subject: [Numpy-discussion] ANN: SfePy 2013.3 In-Reply-To: <1379524240.71081.YahooMailNeo@web142303.mail.bf1.yahoo.com> References: <5239B6AB.3050208@ntc.zcu.cz> <1379524240.71081.YahooMailNeo@web142303.mail.bf1.yahoo.com> Message-ID: <5239FD1D.2070804@ntc.zcu.cz> Dear Jos?, On 09/18/2013 07:10 PM, Jos? Luis Mietta wrote: > Dear Robert. > > Im intresting in modeling mechanical deformation of magnetorheological > elastomers (material formed by inorganic chains inserting in a polymeric > matrix -see figure 2 in the attached file-). The inorganic chais are like > pearl necklace (are formed by saligned pheres ). > > Can I use this tool (SfePy) for modeling the mechanical deformation (i.e.: > shape deformation by appling a force in longitudinal and transversal > direction respect to the inorganic chains)? I guess so :) You are interested in modeling a single chain, not the whole microstructure, right? (BTW. if you are interested in modeling the whole "box" with many chains, check [1]). > How can I do that? These are general steps, not particular to sfepy: First you need to create a FE mesh out of the CT data. There are probably many packages that can do that. I am familiar (and know it works) with [2], which has been created by Vladimir Lukes, another sfepy developer. Then you need to define the problem - specify boundary conditions, choose FE approximation, apply loads and finally solve and visualize. > Im a newby user of python and I dont know anything about simple finite > elements analysis. What documentation (books or other) do yo recommend for > my introduction in this topic? Check the wiki page [3] and possibly the references given there. If you have other questions, we can discuss off-list, or on the sfepy mailing list [4]. Best regards, r. [1] http://parfe.sourceforge.net/ [2] http://sfepy.org/dicom2fem/ [3] http://en.wikipedia.org/wiki/Finite_element_method [4] https://groups.google.com/forum/#!forum/sfepy-devel From chris.barker at noaa.gov Wed Sep 18 15:51:55 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 18 Sep 2013 12:51:55 -0700 Subject: [Numpy-discussion] Using setuptools develop mode with numpy distutils In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 2:20 AM, Robert Kern wrote: > Just import setuptools before doing the numpy.distutils imports. You don't > need to use setuptools.setup() in order to get access to the > setuptools-added commands. > > Thanks --- that works fine -- I could have sworn I tried that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bahtiyor_zohidov at mail.ru Wed Sep 18 16:19:34 2013 From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=) Date: Thu, 19 Sep 2013 00:19:34 +0400 Subject: [Numpy-discussion] =?utf-8?q?ValueError=3A_matrices_are_not_align?= =?utf-8?q?ed!!!?= References: <1379527902.575983897@f106.i.mail.ru> Message-ID: <1379535574.815574822@f191.i.mail.ru> I understand ,sometimes, it is normal that number of equations are less or more than number of unknowns that means non square matrix appearance. If so,?how ?would you compose Av(v) function, because I have only constant dimensional values such as A matrix(MxN) and array b array (M)??? ?????, 18 ???????? 2013, 11:50 -07:00 ?? "Bradley M. Froehle" : >In [2]: %debug >> (5)Av() >??????4 def Av(A,v): >----> 5 return np.dot(A,v) >??????6 > >ipdb> !A.shape >(4, 8) >ipdb> !v.shape >(4,) > >In your code it looks like you are essentially computing A.dot(v) >where A is a 4-by-8 matrix and v is vector of length 4. That's what >the error is telling you --- that the matrix and vector have >incompatible dimensions. > >I've never seen the conjugate gradient method used on non-square >matrices... are you sure this is what you want to be doing? > >-Brad > >On Wed, Sep 18, 2013 at 11:11 AM, Happyman < bahtiyor_zohidov at mail.ru > wrote: >> Hello, >> >> I am trying to solve linear Ax=b problem, but some error occurs in the >> process like: >> ---------------------------------------------- >> >> Traceback (most recent call last): >> File "C:\Python27\Conjugate_Gradient.py", line 66, in >> x, iter_number = conjGrad(Av,A, x, b) >> File "C:\Python27\Conjugate_Gradient.py", line 51, in conjGrad >> u = Av(A,s) >> File "C:\Python27\Conjugate_Gradient.py", line 41, in Av >> return np.dot(A,v) >> ValueError: matrices are not aligned >> ----------------------------------------------- >> >> I put the code below to check it. >> >> import numpy as np >> import math >> >> def Av(A,v): >> return np.dot(A,v) >> >> def conjGrad(Av,A, x, b, tol=1.0e-9): >> n = len(x) >> r = b - Av(A,x) >> s = r.copy() >> for i in range(n): >> u = Av(A,s) >> alpha = np.dot(s,r)/np.dot(s,u) >> x = x + aplha*s >> r = b - Av(A,x) >> if (math.sqrt(np.dot(r,r))) < tol: >> break >> else: >> beta = - np.dot(r,u)/np.dot(s,u) >> s = r + beta * s >> return x,i >> >> if __name__ == '__main__': >> # Input values >> A = np.arange(32, dtype=float).reshape((4,8)) >> x = np.zeros(8) >> b = np.array([2.5, 4.5, 6.5, 8.0]) >> x, iter_number = conjGrad(Av,A, x, b) >> >> I would appreciate any solution to this problem... >> Thanks in advance >> -- >> happy man >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion -- happy Man -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Wed Sep 18 21:40:44 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Wed, 18 Sep 2013 18:40:44 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). To be honest I don't know yet if the FFTs are going to be the limiting step but I thought I may as well give pyFFTW a try and ran into that issue... Antony 2013/9/18 Robert Kern > On Wed, Sep 18, 2013 at 1:51 AM, Antony Lee > wrote: > > > > Hi all, > > > > While I realize that this is certainly tweaking multiprocessing beyond > its specifications, I would like to use it on Windows to start a 32-bit > Python process from a 64-bit Python process (use case: I need to interface > with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a > 32-bit compiled version (yes, I could try to install MSVC and compile it > myself but I'm trying to avoid that...)) > > Just use subprocess to start up the 32-bit Python. If you want to use the > multiprocessing tools for communicating data, use a Manager server in the > 32-bit Python to communicate over a socket. > > http://docs.python.org/2/library/multiprocessing#managers > http://docs.python.org/2/library/multiprocessing#using-a-remote-manager > > It is possible that this won't work if the protocol assumes that the > bitness is the same between server and client (e.g. struct.pack('Q', ...)), > but I suspect this is not the case. > > You may also consider writing a small server using pyzmq or similar. I am > guessing that you are just calling one function from pyFFTW and getting the > result back. A simple REQ/REP server is easy to write with pyzmq. Do you > need to use pyFFTW for some specific functionality that is not available in > numpy.fft or scipy.fftpack? > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Sep 19 02:20:21 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 19 Sep 2013 08:20:21 +0200 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: On Wed, Sep 18, 2013 at 2:51 AM, Antony Lee wrote: > While I realize that this is certainly tweaking multiprocessing beyond its > specifications, I would like to use it on Windows to start a 32-bit Python > process from a 64-bit Python process (use case: I need to interface with a > 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit > compiled version (yes, I could try to install MSVC and compile it myself but > I'm trying to avoid that...)) In the end, this may be the less painful solution, especially if speed is a requirement. St?fan From Jerome.Kieffer at esrf.fr Thu Sep 19 02:45:05 2013 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Thu, 19 Sep 2013 08:45:05 +0200 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: <20130919084505.28762714.Jerome.Kieffer@esrf.fr> On Wed, 18 Sep 2013 18:40:44 -0700 Antony Lee wrote: > Thanks, I didn't know that multiprocessing Managers could be used with > processes not started by multiprocessing itself... I will give them a try. > I just need to compute FFTs, but speed is a real issue for me (I am using > the results for real-time feedback). To be honest I don't know yet if the > FFTs are going to be the limiting step but I thought I may as well give > pyFFTW a try and ran into that issue... If FFTs are needed as well as speed, I got good results with PyCuda + CuFFT (and a GPU from Nvidia). -- J?r?me Kieffer On-Line Data analysis / Software Group ISDD / ESRF tel +33 476 882 445 From heng at cantab.net Thu Sep 19 04:05:35 2013 From: heng at cantab.net (Henry Gomersall) Date: Thu, 19 Sep 2013 09:05:35 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: <523AB04F.1040408@cantab.net> On 18/09/13 01:51, Antony Lee wrote: > I need to interface with a 64-bit DLL and use an extension (pyFFTW) > for which I can only find a 32-bit compiled version (yes, I could try > to install MSVC and compile it myself but I'm trying to avoid that...)) I'm now in a position that I might be able to release a 64-bit Windows version of pyFFTW (I actually have a 64-bit installation of windows) Rather stupidly, one of the problems at the moment is that I can't upload the .msi files created under win7 to pypi! (the magic bytes check is failing due to a version mismatch). I've had feedback that this is possible. Give me a few hours and I'll see what I can do... Cheers, Henry From heng at cantab.net Thu Sep 19 04:06:56 2013 From: heng at cantab.net (Henry Gomersall) Date: Thu, 19 Sep 2013 09:06:56 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: <523AB04F.1040408@cantab.net> References: <523AB04F.1040408@cantab.net> Message-ID: <523AB0A0.9020708@cantab.net> On 19/09/13 09:05, Henry Gomersall wrote: > I've had feedback that this is possible. Give me a few hours and I'll > see what I can do... I mean that it builds under win 64-bit. I'll prob push a .exe. Cheers, Henry From robert.kern at gmail.com Thu Sep 19 04:55:05 2013 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Sep 2013 09:55:05 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee wrote: > > Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. > I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. > To be honest I don't know yet if the FFTs are going to be the limiting step but I thought I may as well give pyFFTW a try and ran into that issue... In that case, thinking about multiprocessing or even pyFFTW is far too premature. Implement your code with numpy.fft and see what performance you actually get. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Thu Sep 19 12:58:37 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 19 Sep 2013 09:58:37 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. Other replies below. Antony 2013/9/19 Robert Kern > On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee > wrote: > > > > Thanks, I didn't know that multiprocessing Managers could be used with > processes not started by multiprocessing itself... I will give them a try. > > I just need to compute FFTs, but speed is a real issue for me (I am > using the results for real-time feedback). > > I am pretty sure that the overhead of communicating a large array from one > process to another will vastly overwhelm any speed gains you get by using > pyFFTW over numpy.fft. > I would have hoped that the large arrays are simply written (from the beginning) to shared memory (what multiprocessing.sharedctypes.Array seems to do(?)) and that interprocess communication would be cheap enough (but what do I know about that). > > > > To be honest I don't know yet if the FFTs are going to be the limiting > step but I thought I may as well give pyFFTW a try and ran into that > issue... > > In that case, thinking about multiprocessing or even pyFFTW is far too > premature. Implement your code with numpy.fft and see what performance you > actually get. > There is another (and, in fact, main) reason for me to use multiprocessing: the main app runs a GUI and running the data analysis in the same process just makes it painfully slow (I have tried that). Instead, running the data analysis in a separate process keeps the GUI responsive. Now whether the data analysis process should use numpy.fft or pyFFTW is a separate question; I realize that the gains from pyFFTW may probably be negligible compared to the other costs (... including the costs of tweaking multiprocessing beyond its specifications) but I was just giving it a try when I ran into the issue and was just puzzled by the error message I had never seen before. > -- > Robert Kern > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From heng at cantab.net Thu Sep 19 13:09:21 2013 From: heng at cantab.net (Henry Gomersall) Date: Thu, 19 Sep 2013 18:09:21 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: On 2013-09-19 17:58, Antony Lee wrote: > Henry: thanks a lot, that would be very appreciated regardless of > whether I end up using it in this specific project or not. > Other replies below. > I've actually spent rather too long fiddling with Windows on this one! I can't for the life of me get cl.exe to compile the code. It has a long list of cryptic errors that gcc seems to not find at all :( I'll keep at it! hen From robert.kern at gmail.com Thu Sep 19 13:39:08 2013 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 19 Sep 2013 18:39:08 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee wrote: > > Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. > Other replies below. > > Antony > > 2013/9/19 Robert Kern >> >> On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee wrote: >> > >> > Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. >> > I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). >> >> I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. > > I would have hoped that the large arrays are simply written (from the beginning) to shared memory (what multiprocessing.sharedctypes.Array seems to do(?)) and that interprocess communication would be cheap enough (but what do I know about that). It certainly won't be automatic just by passing a numpy array to the manager. You will have to manually create the shared memory, pass its handle to the other process, and copy into it. But even the copy of the array may overwhelm the speed gains between PyFFTW and numpy.fft. If you can set it up such that the subprocess owns the shared memory for both input and output and the GUI process always writes into the input shared array directly and reads out the output shared array, then might work out okay. This works well when the inputs/outputs are always the same size. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Thu Sep 19 14:29:29 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 19 Sep 2013 11:29:29 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: 2013/9/19 Robert Kern > On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee > wrote: > > > > Henry: thanks a lot, that would be very appreciated regardless of > whether I end up using it in this specific project or not. > > Other replies below. > > > > Antony > > > > 2013/9/19 Robert Kern > >> > >> On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee > wrote: > >> > > >> > Thanks, I didn't know that multiprocessing Managers could be used > with processes not started by multiprocessing itself... I will give them a > try. > >> > I just need to compute FFTs, but speed is a real issue for me (I am > using the results for real-time feedback). > >> > >> I am pretty sure that the overhead of communicating a large array from > one process to another will vastly overwhelm any speed gains you get by > using pyFFTW over numpy.fft. > > > > I would have hoped that the large arrays are simply written (from the > beginning) to shared memory (what multiprocessing.sharedctypes.Array seems > to do(?)) and that interprocess communication would be cheap enough (but > what do I know about that). > > It certainly won't be automatic just by passing a numpy array to the > manager. You will have to manually create the shared memory, pass its > handle to the other process, and copy into it. But even the copy of the > array may overwhelm the speed gains between PyFFTW and numpy.fft. If you > can set it up such that the subprocess owns the shared memory for both > input and output and the GUI process always writes into the input shared > array directly and reads out the output shared array, then might work out > okay. This works well when the inputs/outputs are always the same size. > The arrays would always be the same size, and there is no array copy involved, as (I think that) I can have the C dll directly write whatever data needs to be analyzed to the shared memory array -- basically what you're suggesting. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Thu Sep 19 18:12:14 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 19 Sep 2013 15:12:14 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: <523AB0A0.9020708@cantab.net> References: <523AB04F.1040408@cantab.net> <523AB0A0.9020708@cantab.net> Message-ID: <523B76BE.5070001@uci.edu> On 9/19/2013 1:06 AM, Henry Gomersall wrote: > On 19/09/13 09:05, Henry Gomersall wrote: >> I've had feedback that this is possible. Give me a few hours and I'll >> see what I can do... > > I mean that it builds under win 64-bit. I'll prob push a .exe. > > Cheers, > > Henry It's probbly the wrong place to suggest/discuss this... To build PyFFTW with MSVC one could replace the C99 headers and types with their numpy counterparts, e.g. `double complex`->`np.complex128`. There would be no need for `complex.h` or `stdint.h`. IIRC, with MSVC the `long double` type is identical to `double`. PyFFTW will likely not work with mingw compiled libfftw3l-3.dll and msvc compiled numpy. Maybe an option to map all fftwl functions to their fftw counterparts would work. That way it will also be possible to link PyFFTW to Intel's MKL (instead of libfftw3*-3.dll), which does not support `long double` types. Christoph From heng at cantab.net Fri Sep 20 05:10:07 2013 From: heng at cantab.net (Henry Gomersall) Date: Fri, 20 Sep 2013 10:10:07 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: <523B76BE.5070001@uci.edu> References: " " <523AB04F.1040408@cantab.net> <523AB0A0.9020708@cantab.net> <523B76BE.5070001@uci.edu> Message-ID: <83eb29abbb4d8ce909d2d3d09be6b816@cam.ac.uk> On 2013-09-19 23:12, Christoph Gohlke wrote: > On 9/19/2013 1:06 AM, Henry Gomersall wrote: >> On 19/09/13 09:05, Henry Gomersall wrote: >>> I've had feedback that this is possible. Give me a few hours and I'll >>> see what I can do... >> >> I mean that it builds under win 64-bit. I'll prob push a .exe. >> >> Cheers, >> >> Henry > > It's probbly the wrong place to suggest/discuss this... To build PyFFTW > with MSVC one could replace the C99 headers and types with their numpy > counterparts, e.g. `double complex`->`np.complex128`. There would be no > need for `complex.h` or `stdint.h`. IIRC, with MSVC the `long double` > type is identical to `double`. PyFFTW will likely not work with mingw > compiled libfftw3l-3.dll and msvc compiled numpy. Maybe an option to > map > all fftwl functions to their fftw counterparts would work. That way it > will also be possible to link PyFFTW to Intel's MKL (instead of > libfftw3*-3.dll), which does not support `long double` types. > Yeah, these are the problems I'm trying to sort out. One strategy is to simply leave out the long double transform schemes when numpy.dtype('longdouble') == numpy.dtype('float64'), which means that pyFFTW will work just fine with the mingw dlls as both numpy.longdouble and numpy.float64 arrays would map to the double precision fftw functions. This _should_ be seamless and transparent to the user. A remaining issue is that there would be an ABI incompatibility between pyFFTW and the (never called) fftwl binaries. It probably wouldn't matter in practice, but it would upset me. Perhaps using MSVC compiled FFTW is a better solution... Henry From heng at cantab.net Fri Sep 20 17:17:46 2013 From: heng at cantab.net (Henry Gomersall) Date: Fri, 20 Sep 2013 22:17:46 +0100 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: References: Message-ID: <523CBB7A.4020607@cantab.net> On 18/09/13 01:51, Antony Lee wrote: > While I realize that this is certainly tweaking multiprocessing beyond > its specifications, I would like to use it on Windows to start a > 32-bit Python process from a 64-bit Python process (use case: I need > to interface with a 64-bit DLL and use an extension (pyFFTW) for which > I can only find a 32-bit compiled version (yes, I could try to install > MSVC and compile it myself but I'm trying to avoid that...)) There is now a release on PyPI including installers for both 32- and 64-bit Python 2.7, 3.2 and 3.3. The long double schemes are ignored as on 64-bit windows that type simply maps to double (though it should be seamless from the Python/Numpy end). All tests satisfied :) (that was some work!) Cheers, Henry From antony.lee at berkeley.edu Fri Sep 20 20:38:19 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Fri, 20 Sep 2013 17:38:19 -0700 Subject: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation In-Reply-To: <523CBB7A.4020607@cantab.net> References: <523CBB7A.4020607@cantab.net> Message-ID: Thanks a lot! Antony 2013/9/20 Henry Gomersall > On 18/09/13 01:51, Antony Lee wrote: > > While I realize that this is certainly tweaking multiprocessing beyond > > its specifications, I would like to use it on Windows to start a > > 32-bit Python process from a 64-bit Python process (use case: I need > > to interface with a 64-bit DLL and use an extension (pyFFTW) for which > > I can only find a 32-bit compiled version (yes, I could try to install > > MSVC and compile it myself but I'm trying to avoid that...)) > > There is now a release on PyPI including installers for both 32- and > 64-bit Python 2.7, 3.2 and 3.3. > > The long double schemes are ignored as on 64-bit windows that type > simply maps to double (though it should be seamless from the > Python/Numpy end). > > All tests satisfied :) (that was some work!) > > Cheers, > > Henry > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jthetzel at gmail.com Sat Sep 21 13:55:36 2013 From: jthetzel at gmail.com (Jeremy Hetzel) Date: Sat, 21 Sep 2013 15:25:36 -0230 Subject: [Numpy-discussion] Pull request review #3770: Trapezoidal distribution Message-ID: I've added a trapezoidal distribution to numpy.random for consideration, pull request 3770: https://github.com/numpy/numpy/pull/3770 Similar to the triangular distribution, the trapezoidal distribution may be used where the underlying distribution is not known, but some knowledge of the limits and mode exists. The trapezoidal distribution generalizes the triangular distribution by allowing the modal values to be expressed as a range instead of a point estimate. The trapezoidal distribution implemented, known as the "generalized trapezoidal distribution," has three additional parameters: growth, decay, and boundary ratio. Adjusting these from the default values create trapezoidal-like distributions with non-linear behavior. Examples can be seen in an R vignette ( http://cran.r-project.org/web/packages/trapezoid/vignettes/trapezoid.pdf ), as well as these papers by J.R. van Dorp and colleagues: 1) van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal distributions. Metrika. 58(1):85?97. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf 2) van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. (2007) An elicitation procedure for the generalized trapezoidal distribution with a uniform central stage. Decision Analysis Journal. 4:156?166. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf The docstring for the proposed numpy.random.trapezoidal() is as follows: """ trapezoidal(left, mode1, mode2, right, size=None, m=2, n=2, alpha=1) Draw samples from the generalized trapezoidal distribution. The trapezoidal distribution is defined by minimum (``left``), lower mode (``mode1``), upper mode (``mode1``), and maximum (``right``) parameters. The generalized trapezoidal distribution adds three more parameters: the growth rate (``m``), decay rate (``n``), and boundary ratio (``alpha``) parameters. The generalized trapezoidal distribution simplifies to the trapezoidal distribution when ``m = n = 2`` and ``alpha = 1``. It further simplifies to a triangular distribution when ``mode1 == mode2``. Parameters ---------- left : scalar Lower limit. mode1 : scalar The value where the first peak of the distribution occurs. The value should fulfill the condition ``left <= mode1 <= mode2``. mode2 : scalar The value where the first peak of the distribution occurs. The value should fulfill the condition ``mode1 <= mode2 <= right``. right : scalar Upper limit, should be larger than or equal to `mode2`. size : int or tuple of ints, optional Output shape. Default is None, in which case a single value is returned. m : scalar, optional Growth parameter. n : scalar, optional Decay parameter. alpha : scalar, optional Boundary ratio parameter. Returns ------- samples : ndarray or scalar The returned samples all lie in the interval [left, right]. Notes ----- With ``left``, ``mode1``, ``mode2``, ``right``, ``m``, ``n``, and ``alpha`` parametrized as :math:`a, b, c, d, m, n, \\text{ and } \\alpha`, respectively, the probability density function for the generalized trapezoidal distribution is .. math:: f{\\scriptscriptstyle X}(x\mid\theta) = \\mathcal{C}(\\Theta) \\times \\begin{cases} \\alpha \\left(\\frac{x - \\alpha}{b - \\alpha} \\right)^{m - 1}, & \\text{for } a \\leq x < b \\\\ (1 - \\alpha) \\left(\frac{x - b}{c - b} \\right) + \\alpha, & \\text{for } b \\leq x < c \\\\ \\left(\\frac{d - x}{d - c} \\right)^{n-1}, & \\text{for } c \\leq x \\leq d \\end{cases} with the normalizing constant :math:`\\mathcal{C}(\\Theta)` defined as ..math:: \\mathcal{C}(\\Theta) = \\frac{2mn} {2 \\alpha \\left(b - a\\right) n + \\left(\\alpha + 1 \\right) \\left(c - b \\right)mn + 2 \\left(d - c \\right)m} and where the parameter vector :math:`\\Theta = \\{a, b, c, d, m, n, \\alpha \\}, \\text{ } a \\leq b \\leq c \\leq d, \\text{ and } m, n, \\alpha >0`. Similar to the triangular distribution, the trapezoidal distribution may be used where the underlying distribution is not known, but some knowledge of the limits and mode exists. The trapezoidal distribution generalizes the triangular distribution by allowing the modal values to be expressed as a range instead of a point estimate. The growth, decay, and boundary ratio parameters of the generalized trapezoidal distribution further allow for non-linear behavior to be specified. References ---------- .. [1] van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal distributions. Metrika. 58(1):85?97. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf .. [2] van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. (2007) An elicitation proce-dure for the generalized trapezoidal distribution with a uniform central stage. Decision AnalysisJournal. 4:156?166. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf Examples -------- Draw values from the distribution and plot the histogram: >>> import matplotlib.pyplot as plt >>> h = plt.hist(np.random.triangular(0, 0.25, 0.75, 1, 100000), bins=200, ... normed=True) >>> plt.show() """ I am unsure if NumPy encourages incorporation of new distributions into numpy.random or instead into separate modules, but found the exercise to be helpful regardless. Thanks, Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Sep 22 07:24:22 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Sep 2013 07:24:22 -0400 Subject: [Numpy-discussion] Pull request review #3770: Trapezoidal distribution In-Reply-To: References: Message-ID: On Sat, Sep 21, 2013 at 1:55 PM, Jeremy Hetzel wrote: > I've added a trapezoidal distribution to numpy.random for consideration, > pull request 3770: > https://github.com/numpy/numpy/pull/3770 > > Similar to the triangular distribution, the trapezoidal distribution may be > used where the underlying distribution is not known, but some knowledge of > the limits and mode exists. The trapezoidal distribution generalizes the > triangular distribution by allowing the modal values to be expressed as a > range instead of a point estimate. > > The trapezoidal distribution implemented, known as the "generalized > trapezoidal distribution," has three additional parameters: growth, decay, > and boundary ratio. Adjusting these from the default values create > trapezoidal-like distributions with non-linear behavior. Examples can be > seen in an R vignette ( > http://cran.r-project.org/web/packages/trapezoid/vignettes/trapezoid.pdf ), > as well as these papers by J.R. van Dorp and colleagues: > > 1) van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal > distributions. Metrika. 58(1):85?97. Preprint available: > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf > > 2) van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. > (2007) An elicitation procedure for the generalized trapezoidal distribution > with a uniform central stage. Decision Analysis Journal. 4:156?166. Preprint > available: > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf > > The docstring for the proposed numpy.random.trapezoidal() is as follows: > > """ > trapezoidal(left, mode1, mode2, right, size=None, m=2, n=2, alpha=1) > > Draw samples from the generalized trapezoidal distribution. > > The trapezoidal distribution is defined by minimum (``left``), lower > mode (``mode1``), upper > mode (``mode1``), and maximum (``right``) parameters. The > generalized trapezoidal distribution > adds three more parameters: the growth rate (``m``), decay rate > (``n``), and boundary > ratio (``alpha``) parameters. The generalized trapezoidal > distribution simplifies > to the trapezoidal distribution when ``m = n = 2`` and ``alpha = > 1``. It further > simplifies to a triangular distribution when ``mode1 == mode2``. > > Parameters > ---------- > left : scalar > Lower limit. > mode1 : scalar > The value where the first peak of the distribution occurs. > The value should fulfill the condition ``left <= mode1 <= > mode2``. > mode2 : scalar > The value where the first peak of the distribution occurs. > The value should fulfill the condition ``mode1 <= mode2 <= > right``. > right : scalar > Upper limit, should be larger than or equal to `mode2`. > size : int or tuple of ints, optional > Output shape. Default is None, in which case a single value is > returned. > m : scalar, optional > Growth parameter. > n : scalar, optional > Decay parameter. > alpha : scalar, optional > Boundary ratio parameter. > > Returns > ------- > samples : ndarray or scalar > The returned samples all lie in the interval [left, right]. > > Notes > ----- > With ``left``, ``mode1``, ``mode2``, ``right``, ``m``, ``n``, and > ``alpha`` parametrized as > :math:`a, b, c, d, m, n, \\text{ and } \\alpha`, respectively, > the probability density function for the generalized trapezoidal > distribution is > > .. math:: > f{\\scriptscriptstyle X}(x\mid\theta) = > \\mathcal{C}(\\Theta) \\times > \\begin{cases} > \\alpha \\left(\\frac{x - \\alpha}{b - \\alpha} > \\right)^{m - 1}, & \\text{for } a \\leq x < b \\\\ > (1 - \\alpha) \\left(\frac{x - b}{c - b} \\right) > + \\alpha, & \\text{for } b \\leq x < c \\\\ > \\left(\\frac{d - x}{d - c} \\right)^{n-1}, & > \\text{for } c \\leq x \\leq d > \\end{cases} > > with the normalizing constant :math:`\\mathcal{C}(\\Theta)` defined > as > > ..math:: > \\mathcal{C}(\\Theta) = > \\frac{2mn} > {2 \\alpha \\left(b - a\\right) n + > \\left(\\alpha + 1 \\right) \\left(c - b \\right)mn > + > 2 \\left(d - c \\right)m} > > and where the parameter vector :math:`\\Theta = \\{a, b, c, d, m, n, > \\alpha \\}, \\text{ } a \\leq b \\leq c \\leq d, \\text{ and } m, n, > \\alpha >0`. > > Similar to the triangular distribution, the trapezoidal distribution > may be used where the > underlying distribution is not known, but some knowledge of the > limits and > mode exists. The trapezoidal distribution generalizes the triangular > distribution by allowing > the modal values to be expressed as a range instead of a point > estimate. The growth, decay, and > boundary ratio parameters of the generalized trapezoidal > distribution further allow for non-linear > behavior to be specified. > > References > ---------- > .. [1] van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal > distributions. > Metrika. 58(1):85?97. > Preprint available: > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf > .. [2] van Dorp, J. R., Rambaud, S.C., Perez, J. G., and > Pleguezuelo, R. H. (2007) > An elicitation proce-dure for the generalized trapezoidal > distribution with a uniform central stage. > Decision AnalysisJournal. 4:156?166. > Preprint available: > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf > > Examples > -------- > Draw values from the distribution and plot the histogram: > > >>> import matplotlib.pyplot as plt > >>> h = plt.hist(np.random.triangular(0, 0.25, 0.75, 1, 100000), > bins=200, > ... normed=True) > >>> plt.show() > > """ > > I am unsure if NumPy encourages incorporation of new distributions into > numpy.random or instead into separate modules, but found the exercise to be > helpful regardless. I don't see a reason that numpy.random shouldn't get new distributions. It would also be useful to add the corresponding distribution to scipy.stats. I'm not familiar with the generalized trapezoidal distribution and don't know where it's used, neither have I ever used triangular. naming: n, m would indicate to me that they are integers, but it they can be floats (>0) alpha, beta ? about the parameterization - no problem here Is there a standard version, e.g. left=0, right=1, mode1=?, ... ? In scipy.stats.distribution we are required to use a location, scale parameterization, where loc shifts the distribution and scale stretches it. Is there a standard parameterization for that?, for example left = loc = 0 (default) or left = loc / scale = 0 right = scale = 1 (default) mode1_relative = mode1 / scale mode2_relative = mode2 / scale n, m unchanged no defaults just checked: your naming corresponds to triangular, and triang in scipy has the corresponding loc-scale parameterization. Josef > > Thanks, > Jeremy > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mszepien at gmail.com Sun Sep 22 08:17:39 2013 From: mszepien at gmail.com (Mark Szepieniec) Date: Sun, 22 Sep 2013 14:17:39 +0200 Subject: [Numpy-discussion] Pull request review #3770: Trapezoidal distribution In-Reply-To: References: Message-ID: On Sun, Sep 22, 2013 at 1:24 PM, wrote: > On Sat, Sep 21, 2013 at 1:55 PM, Jeremy Hetzel wrote: > > I've added a trapezoidal distribution to numpy.random for consideration, > > pull request 3770: > > https://github.com/numpy/numpy/pull/3770 > > > > Similar to the triangular distribution, the trapezoidal distribution may > be > > used where the underlying distribution is not known, but some knowledge > of > > the limits and mode exists. The trapezoidal distribution generalizes the > > triangular distribution by allowing the modal values to be expressed as a > > range instead of a point estimate. > > > > The trapezoidal distribution implemented, known as the "generalized > > trapezoidal distribution," has three additional parameters: growth, > decay, > > and boundary ratio. Adjusting these from the default values create > > trapezoidal-like distributions with non-linear behavior. Examples can be > > seen in an R vignette ( > > http://cran.r-project.org/web/packages/trapezoid/vignettes/trapezoid.pdf), > > as well as these papers by J.R. van Dorp and colleagues: > > > > 1) van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal > > distributions. Metrika. 58(1):85?97. Preprint available: > > > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf > > > > 2) van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. > > (2007) An elicitation procedure for the generalized trapezoidal > distribution > > with a uniform central stage. Decision Analysis Journal. 4:156?166. > Preprint > > available: > > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf > > > > The docstring for the proposed numpy.random.trapezoidal() is as follows: > > > > """ > > trapezoidal(left, mode1, mode2, right, size=None, m=2, n=2, > alpha=1) > > > > Draw samples from the generalized trapezoidal distribution. > > > > The trapezoidal distribution is defined by minimum (``left``), > lower > > mode (``mode1``), upper > > mode (``mode1``), and maximum (``right``) parameters. The > > generalized trapezoidal distribution > > adds three more parameters: the growth rate (``m``), decay rate > > (``n``), and boundary > > ratio (``alpha``) parameters. The generalized trapezoidal > > distribution simplifies > > to the trapezoidal distribution when ``m = n = 2`` and ``alpha = > > 1``. It further > > simplifies to a triangular distribution when ``mode1 == mode2``. > > > > Parameters > > ---------- > > left : scalar > > Lower limit. > > mode1 : scalar > > The value where the first peak of the distribution occurs. > > The value should fulfill the condition ``left <= mode1 <= > > mode2``. > > mode2 : scalar > > The value where the first peak of the distribution occurs. > > The value should fulfill the condition ``mode1 <= mode2 <= > > right``. > > right : scalar > > Upper limit, should be larger than or equal to `mode2`. > > size : int or tuple of ints, optional > > Output shape. Default is None, in which case a single value > is > > returned. > > m : scalar, optional > > Growth parameter. > > n : scalar, optional > > Decay parameter. > > alpha : scalar, optional > > Boundary ratio parameter. > > > > Returns > > ------- > > samples : ndarray or scalar > > The returned samples all lie in the interval [left, right]. > > > > Notes > > ----- > > With ``left``, ``mode1``, ``mode2``, ``right``, ``m``, ``n``, and > > ``alpha`` parametrized as > > :math:`a, b, c, d, m, n, \\text{ and } \\alpha`, respectively, > > the probability density function for the generalized trapezoidal > > distribution is > > > > .. math:: > > f{\\scriptscriptstyle X}(x\mid\theta) = > > \\mathcal{C}(\\Theta) \\times > > \\begin{cases} > > \\alpha \\left(\\frac{x - \\alpha}{b - \\alpha} > > \\right)^{m - 1}, & \\text{for } a \\leq x < b \\\\ > > (1 - \\alpha) \\left(\frac{x - b}{c - b} > \\right) > > + \\alpha, & \\text{for } b \\leq x < c \\\\ > > \\left(\\frac{d - x}{d - c} \\right)^{n-1}, & > > \\text{for } c \\leq x \\leq d > > \\end{cases} > > > > with the normalizing constant :math:`\\mathcal{C}(\\Theta)` > defined > > as > > > > ..math:: > > \\mathcal{C}(\\Theta) = > > \\frac{2mn} > > {2 \\alpha \\left(b - a\\right) n + > > \\left(\\alpha + 1 \\right) \\left(c - b > \\right)mn > > + > > 2 \\left(d - c \\right)m} > > > > and where the parameter vector :math:`\\Theta = \\{a, b, c, d, > m, n, > > \\alpha \\}, \\text{ } a \\leq b \\leq c \\leq d, \\text{ and } m, n, > > \\alpha >0`. > > > > Similar to the triangular distribution, the trapezoidal > distribution > > may be used where the > > underlying distribution is not known, but some knowledge of the > > limits and > > mode exists. The trapezoidal distribution generalizes the > triangular > > distribution by allowing > > the modal values to be expressed as a range instead of a point > > estimate. The growth, decay, and > > boundary ratio parameters of the generalized trapezoidal > > distribution further allow for non-linear > > behavior to be specified. > > > > References > > ---------- > > .. [1] van Dorp, J. R. and Kotz, S. (2003) Generalized > trapezoidal > > distributions. > > Metrika. 58(1):85?97. > > Preprint available: > > > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDorp.pdf > > .. [2] van Dorp, J. R., Rambaud, S.C., Perez, J. G., and > > Pleguezuelo, R. H. (2007) > > An elicitation proce-dure for the generalized trapezoidal > > distribution with a uniform central stage. > > Decision AnalysisJournal. 4:156?166. > > Preprint available: > > http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf > > > > Examples > > -------- > > Draw values from the distribution and plot the histogram: > > > > >>> import matplotlib.pyplot as plt > > >>> h = plt.hist(np.random.triangular(0, 0.25, 0.75, 1, 100000), > > bins=200, > > ... normed=True) > > >>> plt.show() > > > > """ > > > > I am unsure if NumPy encourages incorporation of new distributions into > > numpy.random or instead into separate modules, but found the exercise to > be > > helpful regardless. > > I don't see a reason that numpy.random shouldn't get new > distributions. It would also be useful to add the corresponding > distribution to scipy.stats. > > I'm not familiar with the generalized trapezoidal distribution and > don't know where it's used, neither have I ever used triangular. > > naming: n, m would indicate to me that they are integers, but it they > can be floats (>0) > alpha, beta ? > > > about the parameterization - no problem here > > Is there a standard version, e.g. left=0, right=1, mode1=?, ... ? > > In scipy.stats.distribution we are required to use a location, scale > parameterization, where loc shifts the distribution and scale > stretches it. > Is there a standard parameterization for that?, for example > left = loc = 0 (default) or left = loc / scale = 0 > right = scale = 1 (default) > mode1_relative = mode1 / scale > mode2_relative = mode2 / scale > n, m unchanged no defaults > > just checked: > your naming corresponds to triangular, and triang in scipy has the > corresponding loc-scale parameterization. > > > Josef > > > > > > Thanks, > > Jeremy > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion I think you need to s/first/second in the description of the mode2 parameter? -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.reed.c at gmail.com Sun Sep 22 10:21:18 2013 From: david.reed.c at gmail.com (David Reed) Date: Sun, 22 Sep 2013 10:21:18 -0400 Subject: [Numpy-discussion] Min Error Message-ID: Hi, I am getting a strange error when finding the minimum of a matrix. The weird thing is I get this while running within iPython shell, and if I do %debug and go to the line where this fails and run the command `a = np.min(D, axis=0)`, I get no error. Here is the trace: draw_lines/main.pyc in haus_distance(A, B) 28 D = D.reshape((len(A), None)) 29 ---> 30 a = np.min(D, axis=0) 31 32 # vp = np.max(np.min(D, axis=0)) anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in amin(a, axis, out, keepdims) 1985 else: 1986 return _methods._amin(a, axis=axis, -> 1987 out=out, keepdims=keepdims) 1988 1989 def alen(a): anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in _amin(a, axis, out, keepdims) 12 def _amin(a, axis=None, out=None, keepdims=False): 13 return um.minimum.reduce(a, axis=axis, ---> 14 out=out, keepdims=keepdims) 15 16 def _sum(a, axis=None, dtype=None, out=None, keepdims=False): TypeError: an integer is required -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sun Sep 22 10:42:26 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 22 Sep 2013 16:42:26 +0200 Subject: [Numpy-discussion] Min Error In-Reply-To: References: Message-ID: <1379860946.2834.4.camel@sebastian-laptop> On Sun, 2013-09-22 at 10:21 -0400, David Reed wrote: > Hi, > > > I am getting a strange error when finding the minimum of a matrix. > The weird thing is I get this while running within iPython shell, and > if I do %debug and go to the line where this fails and run the command > `a = np.min(D, axis=0)`, I get no error. > > > Here is the trace: > > > draw_lines/main.pyc in haus_distance(A, B) > 28 D = D.reshape((len(A), None)) > 29 > ---> 30 a = np.min(D, axis=0) > 31 > 32 # vp = np.max(np.min(D, axis=0)) > There is an error in the reshape code. The real error is the None in your reshape right above, it should be -1. This is not surfaced correctly due to a bug in Reshape, but instead gets surfaced (somewhat randomly) lateron. - Sebastian > > anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in > amin(a, axis, out, keepdims) > 1985 else: > 1986 return _methods._amin(a, axis=axis, > -> 1987 out=out, keepdims=keepdims) > 1988 > 1989 def alen(a): > > > anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in > _amin(a, axis, out, keepdims) > 12 def _amin(a, axis=None, out=None, keepdims=False): > 13 return um.minimum.reduce(a, axis=axis, > ---> 14 out=out, keepdims=keepdims) > 15 > 16 def _sum(a, axis=None, dtype=None, out=None, keepdims=False): > > > TypeError: an integer is required > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From david.reed.c at gmail.com Sun Sep 22 10:45:50 2013 From: david.reed.c at gmail.com (David Reed) Date: Sun, 22 Sep 2013 10:45:50 -0400 Subject: [Numpy-discussion] Min Error In-Reply-To: <1379860946.2834.4.camel@sebastian-laptop> References: <1379860946.2834.4.camel@sebastian-laptop> Message-ID: Thanks Sebastian, but still have same error. Also doesn't explain why it works when I run the same code at the debug prompt. Scratching my head on this one. Dave On Sun, Sep 22, 2013 at 10:42 AM, Sebastian Berg wrote: > On Sun, 2013-09-22 at 10:21 -0400, David Reed wrote: > > Hi, > > > > > > I am getting a strange error when finding the minimum of a matrix. > > The weird thing is I get this while running within iPython shell, and > > if I do %debug and go to the line where this fails and run the command > > `a = np.min(D, axis=0)`, I get no error. > > > > > > Here is the trace: > > > > > > draw_lines/main.pyc in haus_distance(A, B) > > 28 D = D.reshape((len(A), None)) > > 29 > > ---> 30 a = np.min(D, axis=0) > > 31 > > 32 # vp = np.max(np.min(D, axis=0)) > > > > There is an error in the reshape code. The real error is the None in > your reshape right above, it should be -1. > This is not surfaced correctly due to a bug in Reshape, but instead gets > surfaced (somewhat randomly) lateron. > > - Sebastian > > > > > anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in > > amin(a, axis, out, keepdims) > > 1985 else: > > 1986 return _methods._amin(a, axis=axis, > > -> 1987 out=out, keepdims=keepdims) > > 1988 > > 1989 def alen(a): > > > > > > anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in > > _amin(a, axis, out, keepdims) > > 12 def _amin(a, axis=None, out=None, keepdims=False): > > 13 return um.minimum.reduce(a, axis=axis, > > ---> 14 out=out, keepdims=keepdims) > > 15 > > 16 def _sum(a, axis=None, dtype=None, out=None, keepdims=False): > > > > > > TypeError: an integer is required > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.reed.c at gmail.com Sun Sep 22 10:54:31 2013 From: david.reed.c at gmail.com (David Reed) Date: Sun, 22 Sep 2013 10:54:31 -0400 Subject: [Numpy-discussion] Min Error In-Reply-To: References: <1379860946.2834.4.camel@sebastian-laptop> Message-ID: Sebastian, I apologize, that did work. I forgot to autoreload. Can you explain why the original code did work at the debug prompt? -Dave On Sun, Sep 22, 2013 at 10:45 AM, David Reed wrote: > Thanks Sebastian, but still have same error. Also doesn't explain why it > works when I run the same code at the debug prompt. Scratching my head on > this one. > > Dave > > > On Sun, Sep 22, 2013 at 10:42 AM, Sebastian Berg < > sebastian at sipsolutions.net> wrote: > >> On Sun, 2013-09-22 at 10:21 -0400, David Reed wrote: >> > Hi, >> > >> > >> > I am getting a strange error when finding the minimum of a matrix. >> > The weird thing is I get this while running within iPython shell, and >> > if I do %debug and go to the line where this fails and run the command >> > `a = np.min(D, axis=0)`, I get no error. >> > >> > >> > Here is the trace: >> > >> > >> > draw_lines/main.pyc in haus_distance(A, B) >> > 28 D = D.reshape((len(A), None)) >> > 29 >> > ---> 30 a = np.min(D, axis=0) >> > 31 >> > 32 # vp = np.max(np.min(D, axis=0)) >> > >> >> There is an error in the reshape code. The real error is the None in >> your reshape right above, it should be -1. >> This is not surfaced correctly due to a bug in Reshape, but instead gets >> surfaced (somewhat randomly) lateron. >> >> - Sebastian >> >> > >> > anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in >> > amin(a, axis, out, keepdims) >> > 1985 else: >> > 1986 return _methods._amin(a, axis=axis, >> > -> 1987 out=out, keepdims=keepdims) >> > 1988 >> > 1989 def alen(a): >> > >> > >> > anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in >> > _amin(a, axis, out, keepdims) >> > 12 def _amin(a, axis=None, out=None, keepdims=False): >> > 13 return um.minimum.reduce(a, axis=axis, >> > ---> 14 out=out, keepdims=keepdims) >> > 15 >> > 16 def _sum(a, axis=None, dtype=None, out=None, keepdims=False): >> > >> > >> > TypeError: an integer is required >> > >> > >> > >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Sun Sep 22 11:25:48 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 22 Sep 2013 17:25:48 +0200 Subject: [Numpy-discussion] Min Error In-Reply-To: References: <1379860946.2834.4.camel@sebastian-laptop> Message-ID: <1379863548.2834.10.camel@sebastian-laptop> On Sun, 2013-09-22 at 10:54 -0400, David Reed wrote: > Sebastian, > > > I apologize, that did work. I forgot to autoreload. Can you explain > why the original code did work at the debug prompt? > Probably the debug prompt clears the errors or such. There was a bug (it actually is already fixed in 1.8. but since it was part of a few larger changes, it didn't get backported). If you know the Python C-Api. In it, you set the error *and* return a magic value (NULL or -1) to indicate that there may have been an error. This is also commonly done in conversion functions. The error got set, but the magic value did not get returned, making everything continue as normal, *until* some other function gets a magic value, which triggers: if ((value == -1) && PyErr_Occurred()) because PyErr_Occurred returns something when it should not. - Sebastian > > -Dave > > > On Sun, Sep 22, 2013 at 10:45 AM, David Reed > wrote: > Thanks Sebastian, but still have same error. Also doesn't > explain why it works when I run the same code at the debug > prompt. Scratching my head on this one. > > > Dave > > > On Sun, Sep 22, 2013 at 10:42 AM, Sebastian Berg > wrote: > On Sun, 2013-09-22 at 10:21 -0400, David Reed wrote: > > Hi, > > > > > > I am getting a strange error when finding the > minimum of a matrix. > > The weird thing is I get this while running within > iPython shell, and > > if I do %debug and go to the line where this fails > and run the command > > `a = np.min(D, axis=0)`, I get no error. > > > > > > Here is the trace: > > > > > > draw_lines/main.pyc in haus_distance(A, B) > > 28 D = D.reshape((len(A), None)) > > 29 > > ---> 30 a = np.min(D, axis=0) > > 31 > > 32 # vp = np.max(np.min(D, axis=0)) > > > > > There is an error in the reshape code. The real error > is the None in > your reshape right above, it should be -1. > This is not surfaced correctly due to a bug in > Reshape, but instead gets > surfaced (somewhat randomly) lateron. > > - Sebastian > > > > > > anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in > > amin(a, axis, out, keepdims) > > 1985 else: > > 1986 return _methods._amin(a, axis=axis, > > -> 1987 out=out, > keepdims=keepdims) > > 1988 > > 1989 def alen(a): > > > > > > > anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in > > _amin(a, axis, out, keepdims) > > 12 def _amin(a, axis=None, out=None, > keepdims=False): > > 13 return um.minimum.reduce(a, axis=axis, > > ---> 14 out=out, > keepdims=keepdims) > > 15 > > 16 def _sum(a, axis=None, dtype=None, out=None, > keepdims=False): > > > > > > TypeError: an integer is required > > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From d.l.goldsmith at gmail.com Sun Sep 22 17:04:09 2013 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Sun, 22 Sep 2013 14:04:09 -0700 Subject: [Numpy-discussion] Generating a (uniformly distributed) random bit list of length N Message-ID: Is np.random.randint(2, size=N) the fastest way to do this? Thanks! DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Sep 22 21:22:06 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 23 Sep 2013 03:22:06 +0200 Subject: [Numpy-discussion] Generating a (uniformly distributed) random bit list of length N In-Reply-To: References: Message-ID: On 22 Sep 2013 23:04, "David Goldsmith" wrote: > > Is np.random.randint(2, size=N) the fastest way to do this? Thanks! Are you concerned about speed or memory use? The operation you show should already be quite fast. A more memory efficient approach would be to generate integers and use their binary representation. St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lorenzo.digregorio at gmail.com Mon Sep 23 05:43:33 2013 From: lorenzo.digregorio at gmail.com (Lorenzo Di Gregorio) Date: Mon, 23 Sep 2013 11:43:33 +0200 Subject: [Numpy-discussion] Runtime Warning by numpy.divide(0.0, 0.0) can't be ignored Message-ID: Hello, numpy.divide(0.0,0.0) raises a warning which doesn't appear to get caught by numpy.seterr(divide='ignore') This example: import numpy numpy.divide(0.0,0.0) numpy.divide(1.0,0.0) numpy.seterr(divide='ignore') numpy.divide(1.0,0.0) numpy.divide(0.0,0.0) produces this output: Warning (from warnings module): File "C:\Users\ldigrego\Desktop\tst.py", line 2 numpy.divide(0.0,0.0) RuntimeWarning: invalid value encountered in divide Warning (from warnings module): File "C:\Users\ldigrego\Desktop\tst.py", line 3 numpy.divide(1.0,0.0) RuntimeWarning: divide by zero encountered in divide Warning (from warnings module): File "C:\Users\ldigrego\Desktop\tst.py", line 6 numpy.divide(0.0,0.0) RuntimeWarning: invalid value encountered in divide Strangely, on Ipython the numpy.seterr(divide='ignore') seems to catch the warning: import numpy numpy.divide(0.0,0.0) Out[1]: nan C:\eclipse\plugins\org.python.pydev_2.8.1.2013072611\pysrc\pydevconsole.py:1: RuntimeWarning: invalid value encountered in divide from __future__ import nested_scopes #Jython 2.1 support numpy.divide(1.0,0.0) Out[1]: inf C:\eclipse\plugins\org.python.pydev_2.8.1.2013072611\pysrc\pydevconsole.py:1: RuntimeWarning: divide by zero encountered in divide from __future__ import nested_scopes #Jython 2.1 support numpy.seterr(divide='ignore') Out[1]: {'divide': 'warn', 'invalid': 'warn', 'over': 'warn', 'under': 'ignore'} numpy.divide(1.0,0.0) Out[1]: inf numpy.divide(0.0,0.0) Out[1]: nan I could not find information on Google: is this a known problem? Is there a way to suppress this warning? I'm working on a 64b Win7 machine employing numpy-MKL-1.7.0.win-amd64-py2.7.exe. Best Regards, Lorenzo -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 23 06:28:09 2013 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 23 Sep 2013 11:28:09 +0100 Subject: [Numpy-discussion] Runtime Warning by numpy.divide(0.0, 0.0) can't be ignored In-Reply-To: References: Message-ID: On Monday, September 23, 2013, Lorenzo Di Gregorio wrote: > Hello, > > numpy.divide(0.0,0.0) raises a warning which doesn't appear to get caught > by numpy.seterr(divide='ignore') > > This example: > > import numpy > numpy.divide(0.0,0.0) > numpy.divide(1.0,0.0) > numpy.seterr(divide='ignore') > numpy.divide(1.0,0.0) > numpy.divide(0.0,0.0) > > produces this output: > > Warning (from warnings module): > File "C:\Users\ldigrego\Desktop\tst.py", line 2 > numpy.divide(0.0,0.0) > RuntimeWarning: invalid value encountered in divide > > Warning (from warnings module): > File "C:\Users\ldigrego\Desktop\tst.py", line 3 > numpy.divide(1.0,0.0) > RuntimeWarning: divide by zero encountered in divide > > Warning (from warnings module): > File "C:\Users\ldigrego\Desktop\tst.py", line 6 > numpy.divide(0.0,0.0) > RuntimeWarning: invalid value encountered in divide > > Strangely, on Ipython the numpy.seterr(divide='ignore') seems to catch the > warning: > > import numpy > numpy.divide(0.0,0.0) > Out[1]: nan > C:\eclipse\plugins\org.python.pydev_2.8.1.2013072611\pysrc\pydevconsole.py:1: > RuntimeWarning: invalid value encountered in divide > from __future__ import nested_scopes #Jython 2.1 support > numpy.divide(1.0,0.0) > Out[1]: inf > C:\eclipse\plugins\org.python.pydev_2.8.1.2013072611\pysrc\pydevconsole.py:1: > RuntimeWarning: divide by zero encountered in divide > from __future__ import nested_scopes #Jython 2.1 support > numpy.seterr(divide='ignore') > Out[1]: {'divide': 'warn', 'invalid': 'warn', 'over': 'warn', 'under': > 'ignore'} > numpy.divide(1.0,0.0) > Out[1]: inf > numpy.divide(0.0,0.0) > Out[1]: nan > > I could not find information on Google: is this a known problem? Is there > a way to suppress this warning > This is not a division warning. It is warning you that a NaN (an "invalid value") was generated. It can be controlled using "invalid='ignore'". The reason that it looks like you suppressed it in the IPython example is that you triggered the warning earlier. The behavior of warnings is that they only get issued once for each code location. So the second time you attempted it after setting "divide='ignore'" it wasn't going to show up anyways. > I'm working on a 64b Win7 machine employing > numpy-MKL-1.7.0.win-amd64-py2.7.exe. > > Best Regards, > Lorenzo > -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Sep 23 08:27:54 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 23 Sep 2013 08:27:54 -0400 Subject: [Numpy-discussion] openblas? Message-ID: Does numpy/scipy support building with openblas for blas,lapack instead of atlas? From d.s.seljebotn at astro.uio.no Mon Sep 23 10:23:57 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Sep 2013 16:23:57 +0200 Subject: [Numpy-discussion] openblas? In-Reply-To: References: Message-ID: <52404EFD.7000607@astro.uio.no> On 09/23/2013 02:27 PM, Neal Becker wrote: > Does numpy/scipy support building with openblas for blas,lapack instead of > atlas? I have that working myself. I think it was simply a matter of using the configuration below; NumPy will think it is ATLAS, but that doesn't matter. Once NumPy is built with that configuration, just rebuild SciPy and it will use information from NumPy to link with the same... $ cat site.cfg [atlas] atlas_libs = openblas library_dirs = /mn/stornext/u2/dagss/opt/OpenBLAS.mt Dag Sverre From charlesr.harris at gmail.com Mon Sep 23 13:03:07 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 11:03:07 -0600 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. Message-ID: Hi All, I have gotten no feedback on the removal of the numarray and oldnumeric packages. Consequently the removal will take place on 9/28. Scream now or never... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Mon Sep 23 13:21:44 2013 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Mon, 23 Sep 2013 19:21:44 +0200 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On 23.09.2013, at 7:03PM, Charles R Harris wrote: > I have gotten no feedback on the removal of the numarray and oldnumeric packages. Consequently the removal will take place on 9/28. Scream now or never... The only thing I'd care about is the nd_image subpackage, but as far as I can see, that's already just a wrapper to import scipy.ndimage. I take it there are no pure numpy implementations for the likes of map_coordinates, right? Cheers, Derek From stsci.perry at gmail.com Mon Sep 23 13:32:55 2013 From: stsci.perry at gmail.com (Perry Greenfield) Date: Mon, 23 Sep 2013 13:32:55 -0400 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: <4D9ED2A3-FFF9-4A46-A389-827E642305F2@gmail.com> We at STScI have no objection. If we have any residual dependencies, we'll then find out and fix them. Perry On Sep 23, 2013, at 1:03 PM, Charles R Harris wrote: > Hi All, > > I have gotten no feedback on the removal of the numarray and oldnumeric packages. Consequently the removal will take place on 9/28. Scream now or never... > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jthetzel at gmail.com Mon Sep 23 13:40:30 2013 From: jthetzel at gmail.com (Jeremy Hetzel) Date: Mon, 23 Sep 2013 15:10:30 -0230 Subject: [Numpy-discussion] Pull request review #3770: Trapezoidal distribution In-Reply-To: References: Message-ID: On Sun, Sep 22, 2013 at 9:47 AM, Mark Szepieniec wrote: > > On Sun, Sep 22, 2013 at 1:24 PM, wrote: > >> >> I don't see a reason that numpy.random shouldn't get new >> distributions. It would also be useful to add the corresponding >> distribution to scipy.stats. >> > I have the pdf, cdf, and inverse cdf for the generalized trapezoidal. I've looked through the other distributions at scipy.stats and adding this one should not be difficult. I'll work on it next. > >> naming: n, m would indicate to me that they are integers, but it they >> can be floats (>0) >> alpha, beta ? >> > The three additional parameters for growth rate, decay rate, and boundary ratio are floats > 0. I renamed them from `m`, `n`, and `alpha` (which is how they're parameterized in the published probability density function) to simply `growth`, `decay`, and `ratio`. Does that fit into the NumPy style? It feels intuitive to me. > > >> Is there a standard version, e.g. left=0, right=1, mode1=?, ... ? >> >> In scipy.stats.distribution we are required to use a location, scale >> parameterization, where loc shifts the distribution and scale >> stretches it. >> Is there a standard parameterization for that?, for example >> left = loc = 0 (default) or left = loc / scale = 0 >> right = scale = 1 (default) >> mode1_relative = mode1 / scale >> mode2_relative = mode2 / scale >> n, m unchanged no defaults >> >> just checked: >> your naming corresponds to triangular, and triang in scipy has the >> corresponding loc-scale parameterization. >> > Thanks. There is no standard version of the distribution that I'm aware of, but for the purposes of scipy.stats, left=0, right=1 and mode1, mode2 being either 0.25, 0.75 or 1/3, 2/3, seem reasonable. I'll give more thought to the location and scale and send an email to scipy-dev if I need guidance. Looking at scipy.stats.triang, my initial thought is: left_relative = loc mode1_relative = loc + mode1*scale mode2_relative = loc + mode2*scale right_relative = loc + scale growth, decay, and ratio are unchanged. > > I think you need to s/first/second in the description of the mode2 > parameter? > Thanks for catching that. Fixed in a recent commit. mode2 should be the second peak of the distribution. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 23 13:58:31 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 11:58:31 -0600 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 11:21 AM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > On 23.09.2013, at 7:03PM, Charles R Harris > wrote: > > > I have gotten no feedback on the removal of the numarray and oldnumeric > packages. Consequently the removal will take place on 9/28. Scream now or > never... > > The only thing I'd care about is the nd_image subpackage, but as far as I > can see, that's already > just a wrapper to import scipy.ndimage. I take it there are no pure numpy > implementations for > the likes of map_coordinates, right? > As far as I know, that is correct. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Mon Sep 23 15:25:12 2013 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Mon, 23 Sep 2013 12:25:12 -0700 Subject: [Numpy-discussion] Generating a (uniformly distributed) random bit list of length N Message-ID: Thanks, St?fan, speed: N ~ 1e9. Thanks again. DG ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 22 Sep 2013 14:04:09 -0700 > From: David Goldsmith > Subject: [Numpy-discussion] Generating a (uniformly distributed) > random bit list of length N > To: numpy-discussion at scipy.org > Message-ID: > < > CAFtPsZqG7Upjy8s04NPwkn8gJVDVGORu3rPjCVLN6ZKRWnPuTQ at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Is np.random.randint(2, size=N) the fastest way to do this? Thanks! > > DG > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20130922/b3cebeaa/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Mon, 23 Sep 2013 03:22:06 +0200 > From: St?fan van der Walt > Subject: Re: [Numpy-discussion] Generating a (uniformly distributed) > random bit list of length N > To: Discussion of Numerical Python > Message-ID: > < > CABDkGQkq829A_nJ7xxCS5Xauz-pTWcyf_zBsLt+0gUFNzWyDaA at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On 22 Sep 2013 23:04, "David Goldsmith" wrote: > > > > Is np.random.randint(2, size=N) the fastest way to do this? Thanks! > > Are you concerned about speed or memory use? The operation you show should > already be quite fast. A more memory efficient approach would be to > generate integers and use their binary representation. > > St?fan > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/numpy-discussion/attachments/20130923/e4f56af4/attachment-0001.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.j.a.cock at googlemail.com Mon Sep 23 16:42:26 2013 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 23 Sep 2013 21:42:26 +0100 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 6:03 PM, Charles R Harris wrote: > Hi All, > > I have gotten no feedback on the removal of the numarray and oldnumeric > packages. Consequently the removal will take place on 9/28. Scream now or > never... > > Chuck Hi Chuck, Could you clarify how we'd know if this is a problem in a large package? i.e. Is it just Python imports I need to double check, or also C level? Thanks! Peter From charlesr.harris at gmail.com Mon Sep 23 17:18:31 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 15:18:31 -0600 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 2:42 PM, Peter Cock wrote: > On Mon, Sep 23, 2013 at 6:03 PM, Charles R Harris > wrote: > > Hi All, > > > > I have gotten no feedback on the removal of the numarray and oldnumeric > > packages. Consequently the removal will take place on 9/28. Scream now or > > never... > > > > Chuck > > Hi Chuck, > > Could you clarify how we'd know if this is a problem in a large package? > i.e. Is it just Python imports I need to double check, or also C level? > > Just Python level unless you are calling python from C. The packages are not normally imported, so you should be able to just grep through. You could also apply the patch and see what happens. That might be the best test. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 23 17:27:44 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 15:27:44 -0600 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 3:18 PM, Charles R Harris wrote: > > > > On Mon, Sep 23, 2013 at 2:42 PM, Peter Cock wrote: > >> On Mon, Sep 23, 2013 at 6:03 PM, Charles R Harris >> wrote: >> > Hi All, >> > >> > I have gotten no feedback on the removal of the numarray and oldnumeric >> > packages. Consequently the removal will take place on 9/28. Scream now >> or >> > never... >> > >> > Chuck >> >> Hi Chuck, >> >> Could you clarify how we'd know if this is a problem in a large package? >> i.e. Is it just Python imports I need to double check, or also C level? >> >> > Just Python level unless you are calling python from C. The packages are > not normally imported, so you should be able to just grep through. You > could also apply the patch and > see what happens. That might be the best test. > > Also, the 1.8 release, when it comes, should issue a ModuleDeprecationWarning warning when the packages are imported. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 23 17:53:14 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 15:53:14 -0600 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 3:18 PM, Charles R Harris wrote: > > > > On Mon, Sep 23, 2013 at 2:42 PM, Peter Cock wrote: > >> On Mon, Sep 23, 2013 at 6:03 PM, Charles R Harris >> wrote: >> > Hi All, >> > >> > I have gotten no feedback on the removal of the numarray and oldnumeric >> > packages. Consequently the removal will take place on 9/28. Scream now >> or >> > never... >> > >> > Chuck >> >> Hi Chuck, >> >> Could you clarify how we'd know if this is a problem in a large package? >> i.e. Is it just Python imports I need to double check, or also C level? >> >> > Just Python level unless you are calling python from C. The packages are > not normally imported, so you should be able to just grep through. You > could also apply the patch and > see what happens. That might be the best test. > I take that back. There is a _capi.c and include files for numarray. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Sep 23 20:22:53 2013 From: cournape at gmail.com (David Cournapeau) Date: Tue, 24 Sep 2013 01:22:53 +0100 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: Ok, so I've looked a bit into it tonight: - used mingw-w64 4.8.1 (32 bits host) - openblas binaries available on the official website (seem to be built with mingw w64) - used -static-libgcc, -static-libstdc++ and -static-libgfortran - building numpy went ok, test suite almost passes, nothing too alarming. - scipy is still a bit trouble some, I need to look more into it. It definitely looks better than last time I've tried (where it crashed right away). David On Tue, Sep 17, 2013 at 9:20 PM, Ralf Gommers wrote: > > > > On Mon, Sep 16, 2013 at 10:05 PM, Nathaniel Smith wrote: > >> On Mon, Sep 16, 2013 at 8:15 PM, Peter Cock >> wrote: >> > On Mon, Sep 16, 2013 at 8:05 PM, Nathaniel Smith wrote: >> >> >> >> Why not just release numpy 1.8 with the old and terrible system? As >> >> you know I'm 110% in favor of getting rid of it, but 1.8 is ready to >> >> go and 1.9 is coming soon enough, and the old and terrible system does >> >> work right now, today. None of the other options have this property. >> > > The above makes a lot of sense, so I decided to check that it actually > does work. Unsurprisingly, it needs fixing: > https://github.com/numpy/numpy/issues/3760 > > Ralf > > >> > >> > On the down side, the "old and terrible system" does not >> > cover providing pre-built binaries for 64 bit Windows. >> > >> > Doing that right is important not just for SciPy but for any >> > other downstream package including C code compiled >> > against the NumPy C API (and the people doing this >> > probably will only have access to free compilers). >> >> That's not a downside -- that's the situation right now and will >> continue to be the situation for the immediate future, if we cut a >> 1.8rc1 tomorrow and also if we don't cut a 1.8rc1 tomorrow. Again, I'm >> absolutely behind getting this sorted out, but holding up the release >> on all platforms is not going to make win64 standalone binaries appear >> any faster, and in the mean time everyone seems to be getting along >> OK, either because they're using a distribution, are on another >> platform, or taking advantage of Cristoph's generosity (thank you >> Cristoph!). >> >> Worst case, if it all gets sorted out next week we could release an >> 1.8.1 to celebrate... >> >> -n >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 24 00:05:37 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Sep 2013 00:05:37 -0400 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 23, 2013 at 8:22 PM, David Cournapeau wrote: > Ok, so I've looked a bit into it tonight: > > - used mingw-w64 4.8.1 (32 bits host) > - openblas binaries available on the official website (seem to be built > with mingw w64) > - used -static-libgcc, -static-libstdc++ and -static-libgfortran > - building numpy went ok, test suite almost passes, nothing too alarming. > - scipy is still a bit trouble some, I need to look more into it. It > definitely looks better than last time I've tried (where it crashed right > away). Sounds encouraging Thank you, Josef > > David > > > On Tue, Sep 17, 2013 at 9:20 PM, Ralf Gommers > wrote: >> >> >> >> >> On Mon, Sep 16, 2013 at 10:05 PM, Nathaniel Smith wrote: >>> >>> On Mon, Sep 16, 2013 at 8:15 PM, Peter Cock >>> wrote: >>> > On Mon, Sep 16, 2013 at 8:05 PM, Nathaniel Smith wrote: >>> >> >>> >> Why not just release numpy 1.8 with the old and terrible system? As >>> >> you know I'm 110% in favor of getting rid of it, but 1.8 is ready to >>> >> go and 1.9 is coming soon enough, and the old and terrible system does >>> >> work right now, today. None of the other options have this property. >> >> >> The above makes a lot of sense, so I decided to check that it actually >> does work. Unsurprisingly, it needs fixing: >> https://github.com/numpy/numpy/issues/3760 >> >> Ralf >> >>> >>> > >>> > On the down side, the "old and terrible system" does not >>> > cover providing pre-built binaries for 64 bit Windows. >>> > >>> > Doing that right is important not just for SciPy but for any >>> > other downstream package including C code compiled >>> > against the NumPy C API (and the people doing this >>> > probably will only have access to free compilers). >>> >>> That's not a downside -- that's the situation right now and will >>> continue to be the situation for the immediate future, if we cut a >>> 1.8rc1 tomorrow and also if we don't cut a 1.8rc1 tomorrow. Again, I'm >>> absolutely behind getting this sorted out, but holding up the release >>> on all platforms is not going to make win64 standalone binaries appear >>> any faster, and in the mean time everyone seems to be getting along >>> OK, either because they're using a distribution, are on another >>> platform, or taking advantage of Cristoph's generosity (thank you >>> Cristoph!). >>> >>> Worst case, if it all gets sorted out next week we could release an >>> 1.8.1 to celebrate... >>> >>> -n >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Sep 24 00:35:55 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 23 Sep 2013 22:35:55 -0600 Subject: [Numpy-discussion] Binary releases In-Reply-To: References: <8E95A257-3F06-43B7-8407-95916D284FE6@mac.com> Message-ID: On Mon, Sep 23, 2013 at 6:22 PM, David Cournapeau wrote: > Ok, so I've looked a bit into it tonight: > > - used mingw-w64 4.8.1 (32 bits host) > - openblas binaries available on the official website (seem to be built > with mingw w64) > - used -static-libgcc, -static-libstdc++ and -static-libgfortran > - building numpy went ok, test suite almost passes, nothing too alarming. > - scipy is still a bit trouble some, I need to look more into it. It > definitely looks better than last time I've tried (where it crashed right > away). > > Sounds promising. What sort of test failures did you see for numpy? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhattersley at gmail.com Tue Sep 24 06:18:02 2013 From: rhattersley at gmail.com (Richard Hattersley) Date: Tue, 24 Sep 2013 11:18:02 +0100 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On 23 September 2013 18:03, Charles R Harris wrote: > I have gotten no feedback on the removal of the numarray and oldnumeric > packages. Consequently the removal will take place on 9/28. Scream now or > never... > I know I always like to get feedback either way ... so +1 for removal. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 24 08:34:21 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 24 Sep 2013 08:34:21 -0400 Subject: [Numpy-discussion] Pull request review #3770: Trapezoidal distribution In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 1:40 PM, Jeremy Hetzel wrote: > On Sun, Sep 22, 2013 at 9:47 AM, Mark Szepieniec wrote: >> >> >> On Sun, Sep 22, 2013 at 1:24 PM, wrote: >>> >>> >>> I don't see a reason that numpy.random shouldn't get new >>> distributions. It would also be useful to add the corresponding >>> distribution to scipy.stats. > > > I have the pdf, cdf, and inverse cdf for the generalized trapezoidal. I've > looked through the other distributions at scipy.stats and adding this one > should not be difficult. I'll work on it next. Thank you > >>> >>> >>> naming: n, m would indicate to me that they are integers, but it they >>> can be floats (>0) >>> alpha, beta ? > > > The three additional parameters for growth rate, decay rate, and boundary > ratio are floats > 0. I renamed them from `m`, `n`, and `alpha` (which is > how they're parameterized in the published probability density function) to > simply `growth`, `decay`, and `ratio`. Does that fit into the NumPy style? > It feels intuitive to me. `growth`, `decay`, and `ratio` sounds much better we try also in scipy.stats to move away from some of the one letter argument names. > > >>> >>> >>> Is there a standard version, e.g. left=0, right=1, mode1=?, ... ? >>> >>> In scipy.stats.distribution we are required to use a location, scale >>> parameterization, where loc shifts the distribution and scale >>> stretches it. >>> Is there a standard parameterization for that?, for example >>> left = loc = 0 (default) or left = loc / scale = 0 >>> right = scale = 1 (default) >>> mode1_relative = mode1 / scale >>> mode2_relative = mode2 / scale >>> n, m unchanged no defaults >>> >>> just checked: >>> your naming corresponds to triangular, and triang in scipy has the >>> corresponding loc-scale parameterization. > > > Thanks. There is no standard version of the distribution that I'm aware of, > but for the purposes of scipy.stats, left=0, right=1 and mode1, mode2 being > either 0.25, 0.75 or 1/3, 2/3, seem reasonable. I'll give more thought to > the location and scale and send an email to scipy-dev if I need guidance. > Looking at scipy.stats.triang, my initial thought is: > left_relative = loc > mode1_relative = loc + mode1*scale > mode2_relative = loc + mode2*scale > right_relative = loc + scale > growth, decay, and ratio are unchanged. mode1 and mode2 don't need a default, they can be shape parameters which don't have defaults in scipy.stats. with left=0, right=1 hard coded in the formulas, we have a "standard" version and get the transformation with loc and scale The implied parameterization looks good, terminology ? mode1, mode2 are "relative" to right - left, based on 0,1 interval (in fractions of left - right length) your `xxx_relative` are the actual values on the real line, i.e. not relative to loc and scale (It's actually the same as with triang, which I had forgotten to look at initially.) Josef > >> >> >> I think you need to s/first/second in the description of the mode2 >> parameter? > > > Thanks for catching that. Fixed in a recent commit. mode2 should be the > second peak of the distribution. > > > Jeremy > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From p.j.a.cock at googlemail.com Tue Sep 24 10:55:33 2013 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 24 Sep 2013 15:55:33 +0100 Subject: [Numpy-discussion] Removal of numarray and oldnumeric packages. In-Reply-To: References: Message-ID: On Mon, Sep 23, 2013 at 10:53 PM, Charles R Harris wrote: > On Mon, Sep 23, 2013 at 3:18 PM, Charles R Harris wrote: >> On Mon, Sep 23, 2013 at 2:42 PM, Peter Cock wrote: >>> >>> Hi Chuck, >>> >>> Could you clarify how we'd know if this is a problem in a large package? >>> i.e. Is it just Python imports I need to double check, or also C level? >>> >> >> Just Python level unless you are calling python from C. The packages are >> not normally imported, so you should be able to just grep through. You could >> also apply the patch and see what happens. That might be the best test. > > I take that back. There is a _capi.c and include files for numarray. > > Chuck Thanks - I just ran our test suite against numpy compiled from that commit: https://github.com/numpy/numpy/pull/3638 We seem to be OK, so I have no objection to removing this. Peter From d.l.goldsmith at gmail.com Wed Sep 25 02:59:40 2013 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Tue, 24 Sep 2013 23:59:40 -0700 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? Message-ID: Is this a valid algorithm for generating a 3D Wiener process? (When I graph the results, they certainly look like potential Brownian motion tracks.) def Wiener3D(incr, N): r = incr*(R.randint(3, size=(N,))-1) r[0] = 0 r = r.cumsum() t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) t[0] = 0 t = t.cumsum() p = np.pi*incr*(R.randint(3, size=(N,))-1) p[0] = 0 p = p.cumsum() x = r*np.cos(t)*np.sin(p) y = r*np.sin(t)*np.sin(p) z = r*np.cos(p) return np.array((x,y,z)).T Thanks! DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From edmondo.porcu at gmail.com Wed Sep 25 03:06:33 2013 From: edmondo.porcu at gmail.com (Edmondo Porcu) Date: Wed, 25 Sep 2013 09:06:33 +0200 Subject: [Numpy-discussion] Numpy newbie question: matrix creation Message-ID: Dear all, I am a Newbie with Numpy and I would need some advice on how to create a matrix with certain characteristics : - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal values are 0,0.1,0.2,0.3 etc) - The number of columns of the matrix is a parameter of this matrix creation algorithm - Only the rows where the sum is 1 must be kept Would great appreciate your advice and suggestions Best Regards Edmondo -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Sep 25 08:04:41 2013 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 25 Sep 2013 14:04:41 +0200 Subject: [Numpy-discussion] Numpy newbie question: matrix creation In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 9:06 AM, Edmondo Porcu wrote: > I am a Newbie with Numpy and I would need some advice on how to create a > matrix with certain characteristics : > > - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal > values are 0,0.1,0.2,0.3 etc) You can generate random integers between 0 and 10, and divide them by 10: np.random.random_integers(0, 10, (5, 5)) / 10. > - Only the rows where the sum is 1 must be kept I'm not sure I understand the exact structure you describe. Perhaps an example would help. St?fan From robert.kern at gmail.com Wed Sep 25 08:10:16 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 25 Sep 2013 13:10:16 +0100 Subject: [Numpy-discussion] Numpy newbie question: matrix creation In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 8:06 AM, Edmondo Porcu wrote: > > Dear all, > > I am a Newbie with Numpy and I would need some advice on how to create a matrix with certain characteristics : > > - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal values are 0,0.1,0.2,0.3 etc) > > - The number of columns of the matrix is a parameter of this matrix creation algorithm > > - Only the rows where the sum is 1 must be kept > > Would great appreciate your advice and suggestions Use a multinomial distribution with n=10, then divide by 10.0. For example, with 5 columns and 20 samples: [~] |4> np.random.multinomial(10, np.ones(5)/5.0, size=20) / 10.0 array([[ 0.1, 0.1, 0.4, 0.1, 0.3], [ 0.2, 0.4, 0.1, 0.3, 0. ], [ 0.3, 0.2, 0. , 0.2, 0.3], [ 0. , 0.4, 0.4, 0. , 0.2], [ 0.2, 0.3, 0.1, 0.2, 0.2], [ 0.1, 0.5, 0.1, 0.1, 0.2], [ 0.4, 0. , 0.2, 0.3, 0.1], [ 0.3, 0.3, 0.1, 0. , 0.3], [ 0. , 0.2, 0.5, 0.2, 0.1], [ 0.5, 0.1, 0. , 0.2, 0.2], [ 0.1, 0.3, 0.2, 0.2, 0.2], [ 0.3, 0.1, 0.2, 0.2, 0.2], [ 0.2, 0.3, 0. , 0.3, 0.2], [ 0.2, 0.3, 0.2, 0.3, 0. ], [ 0.1, 0.3, 0.2, 0. , 0.4], [ 0.1, 0.1, 0.2, 0.5, 0.1], [ 0.1, 0.6, 0. , 0. , 0.3], [ 0.1, 0.2, 0.4, 0.1, 0.2], [ 0.3, 0.1, 0.2, 0.2, 0.2], [ 0.2, 0.1, 0.4, 0.2, 0.1]]) -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From edmondo.porcu at gmail.com Wed Sep 25 08:12:15 2013 From: edmondo.porcu at gmail.com (Edmondo Porcu) Date: Wed, 25 Sep 2013 14:12:15 +0200 Subject: [Numpy-discussion] Numpy newbie question: matrix creation In-Reply-To: References: Message-ID: That's what I was looking for, except that I want to be sure to generate all the possible combinations, and to have no repeated values. Thanks Edmondo 2013/9/25 Robert Kern > On Wed, Sep 25, 2013 at 8:06 AM, Edmondo Porcu > wrote: > > > > Dear all, > > > > I am a Newbie with Numpy and I would need some advice on how to create a > matrix with certain characteristics : > > > > - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal > values are 0,0.1,0.2,0.3 etc) > > > > - The number of columns of the matrix is a parameter of this matrix > creation algorithm > > > > - Only the rows where the sum is 1 must be kept > > > > Would great appreciate your advice and suggestions > > Use a multinomial distribution with n=10, then divide by 10.0. > > For example, with 5 columns and 20 samples: > > [~] > |4> np.random.multinomial(10, np.ones(5)/5.0, size=20) / 10.0 > array([[ 0.1, 0.1, 0.4, 0.1, 0.3], > [ 0.2, 0.4, 0.1, 0.3, 0. ], > [ 0.3, 0.2, 0. , 0.2, 0.3], > [ 0. , 0.4, 0.4, 0. , 0.2], > [ 0.2, 0.3, 0.1, 0.2, 0.2], > [ 0.1, 0.5, 0.1, 0.1, 0.2], > [ 0.4, 0. , 0.2, 0.3, 0.1], > [ 0.3, 0.3, 0.1, 0. , 0.3], > [ 0. , 0.2, 0.5, 0.2, 0.1], > [ 0.5, 0.1, 0. , 0.2, 0.2], > [ 0.1, 0.3, 0.2, 0.2, 0.2], > [ 0.3, 0.1, 0.2, 0.2, 0.2], > [ 0.2, 0.3, 0. , 0.3, 0.2], > [ 0.2, 0.3, 0.2, 0.3, 0. ], > [ 0.1, 0.3, 0.2, 0. , 0.4], > [ 0.1, 0.1, 0.2, 0.5, 0.1], > [ 0.1, 0.6, 0. , 0. , 0.3], > [ 0.1, 0.2, 0.4, 0.1, 0.2], > [ 0.3, 0.1, 0.2, 0.2, 0.2], > [ 0.2, 0.1, 0.4, 0.2, 0.1]]) > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Sep 25 08:38:22 2013 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 25 Sep 2013 13:38:22 +0100 Subject: [Numpy-discussion] Numpy newbie question: matrix creation In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 1:12 PM, Edmondo Porcu wrote: > > That's what I was looking for, except that I want to be sure to generate all the possible combinations, and to have no repeated values. Okay, then you need to find all of the integer partitions of 10 with `ncols` elements (padding with 0s for those partitions with fewer than `ncols` elements), finding the permutations of each padded partition, then eliminating duplicates. import itertools import numpy as np def gen_partitions(n): """ Generate integer partitions for `n`. http://homepages.ed.ac.uk/jkellehe/partitions.php """ a = [0 for i in range(n + 1)] k = 1 y = n - 1 while k != 0: x = a[k - 1] + 1 k -= 1 while 2*x <= y: a[k] = x y -= x k += 1 l = k + 1 while x <= y: a[k] = x a[l] = y yield a[:k + 2] x += 1 y -= 1 a[k] = x + y y = x + y - 1 yield a[:k + 1] def gen_permuted_partitions(ncols, n=10): for partition in gen_partitions(n): partition = list(partition) if len(partition) > ncols: continue partition.extend([0] * (ncols - len(partition))) for x in itertools.permutations(partition): yield x def compute_matrix(ncols): return np.array(sorted(set(gen_permuted_partitions(ncols)))) / 10.0 -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.isaac at gmail.com Wed Sep 25 09:00:36 2013 From: alan.isaac at gmail.com (Alan G Isaac) Date: Wed, 25 Sep 2013 09:00:36 -0400 Subject: [Numpy-discussion] Numpy newbie question: matrix creation In-Reply-To: References: Message-ID: <5242DE74.2070707@gmail.com> On 9/25/2013 3:06 AM, Edmondo Porcu wrote: > advice on how to create a matrix with certain characteristics : > - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal values are 0,0.1,0.2,0.3 etc) > - The number of columns of the matrix is a parameter of this matrix creation algorithm > - Only the rows where the sum is 1 must be kept This is not nearly enough information. But since it is all you offer, here is an answer for a square matrix with parameter n: np.identity(n) Alan Isaac From ndbecker2 at gmail.com Wed Sep 25 09:36:50 2013 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 25 Sep 2013 09:36:50 -0400 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? References: Message-ID: David Goldsmith wrote: > Is this a valid algorithm for generating a 3D Wiener process? (When I > graph the results, they certainly look like potential Brownian motion > tracks.) > > def Wiener3D(incr, N): > r = incr*(R.randint(3, size=(N,))-1) > r[0] = 0 > r = r.cumsum() > t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) > t[0] = 0 > t = t.cumsum() > p = np.pi*incr*(R.randint(3, size=(N,))-1) > p[0] = 0 > p = p.cumsum() > x = r*np.cos(t)*np.sin(p) > y = r*np.sin(t)*np.sin(p) > z = r*np.cos(p) > return np.array((x,y,z)).T > > Thanks! > > DG Not the kind of Wiener process I learned of. This would be the integral of white noise. Here you have used: 1. discrete increments 2. spherical coordinates From njs at pobox.com Wed Sep 25 09:44:48 2013 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 25 Sep 2013 14:44:48 +0100 Subject: [Numpy-discussion] Heads up re: new memory tracking infrastructure in python 3.4 Message-ID: Hi all, Possibly of interest to some folks here, it looks like in Python 3.4 there'll be a generic system for hooking and tracking memory allocations: http://www.python.org/dev/peps/pep-0445/ I'm not planning to do anything with this information myself, but if anyone's been thinking about numpy memory profiling, the new numpy malloc hooks, etc., then it might make sense to see how we could integrate with this more general system. -n From warren.weckesser at gmail.com Wed Sep 25 12:51:01 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 25 Sep 2013 12:51:01 -0400 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 9:36 AM, Neal Becker wrote: > David Goldsmith wrote: > > > Is this a valid algorithm for generating a 3D Wiener process? (When I > > graph the results, they certainly look like potential Brownian motion > > tracks.) > > > > def Wiener3D(incr, N): > > r = incr*(R.randint(3, size=(N,))-1) > > r[0] = 0 > > r = r.cumsum() > > t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) > > t[0] = 0 > > t = t.cumsum() > > p = np.pi*incr*(R.randint(3, size=(N,))-1) > > p[0] = 0 > > p = p.cumsum() > > x = r*np.cos(t)*np.sin(p) > > y = r*np.sin(t)*np.sin(p) > > z = r*np.cos(p) > > return np.array((x,y,z)).T > > > > Thanks! > > > > DG > > Not the kind of Wiener process I learned of. This would be the integral of > white noise. Here you have used: > > 1. discrete increments > 2. spherical coordinates > > I agree with Neal: that is not a Wiener process. In your process, the *angles* that describe the position undergo a random walk, so the particle tends to move in the same direction over short intervals. To simulate a Wiener process (i.e. Brownian motion) in 3D, you can simply evolve each coordinate independently as a 1D process. Here's a simple function to generate a sample from a Wiener process. The dimension is determined by the shape of the starting point x0. import numpy as np def wiener(x0, n, dt, delta): """Generate an n-dimensional random walk. The array of values generated by this function simulate a Wiener process. Arguments --------- x0 : float or array The starting point of the random walk. n : int The number of steps to take. dt : float The time step. delta : float delta determines the "speed" of the random walk. The random variable of the position at time t, X(t), has a normal distribution whose mean is the position at time t=0 and whose variance is delta**2*t. Returns ------- x : numpy array The shape of `x` is (n+1,) + x0.shape. The first element in the array is x0. """ x0 = np.asfarray(x0) shp = (n+1,) + x0.shape # Generate a sample numbers from a normal distribution. r = np.random.normal(size=shp, scale=delta*np.sqrt(dt)) # Replace the first element with 0.0, so that x0 + r.cumsum() results # in the first element being x0. r[0] = 0.0 # This computes the random walk by forming the cumulative sum of # the random sample. x = r.cumsum(axis=0) x += x0 return x Warren _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Sep 25 13:39:03 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 25 Sep 2013 13:39:03 -0400 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 12:51 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > > On Wed, Sep 25, 2013 at 9:36 AM, Neal Becker wrote: > >> David Goldsmith wrote: >> >> > Is this a valid algorithm for generating a 3D Wiener process? (When I >> > graph the results, they certainly look like potential Brownian motion >> > tracks.) >> > >> > def Wiener3D(incr, N): >> > r = incr*(R.randint(3, size=(N,))-1) >> > r[0] = 0 >> > r = r.cumsum() >> > t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) >> > t[0] = 0 >> > t = t.cumsum() >> > p = np.pi*incr*(R.randint(3, size=(N,))-1) >> > p[0] = 0 >> > p = p.cumsum() >> > x = r*np.cos(t)*np.sin(p) >> > y = r*np.sin(t)*np.sin(p) >> > z = r*np.cos(p) >> > return np.array((x,y,z)).T >> > >> > Thanks! >> > >> > DG >> >> Not the kind of Wiener process I learned of. This would be the integral >> of >> white noise. Here you have used: >> >> 1. discrete increments >> 2. spherical coordinates >> >> > > I agree with Neal: that is not a Wiener process. In your process, the > *angles* that describe the position undergo a random walk, so the particle > tends to move in the same direction over short intervals. > > To simulate a Wiener process (i.e. Brownian motion) in 3D, you can simply > evolve each coordinate independently as a 1D process. > > Here's a simple function to generate a sample from a Wiener process. The > dimension is determined by the shape of the starting point x0. > > > import numpy as np > > > def wiener(x0, n, dt, delta): > """Generate an n-dimensional random walk. > > Whoops--that's a misleading docstring. The `n` in "an n-dimensional random walk" is not the same `n` that is the second argument of the function (which is the number of steps to compute). Warren > The array of values generated by this function simulate a Wiener > process. > > Arguments > --------- > x0 : float or array > The starting point of the random walk. > n : int > The number of steps to take. > dt : float > The time step. > delta : float > delta determines the "speed" of the random walk. The random > variable > of the position at time t, X(t), has a normal distribution whose > mean > is the position at time t=0 and whose variance is delta**2*t. > > Returns > ------- > x : numpy array > The shape of `x` is (n+1,) + x0.shape. > The first element in the array is x0. > """ > x0 = np.asfarray(x0) > shp = (n+1,) + x0.shape > > # Generate a sample numbers from a normal distribution. > r = np.random.normal(size=shp, scale=delta*np.sqrt(dt)) > > # Replace the first element with 0.0, so that x0 + r.cumsum() results > # in the first element being x0. > r[0] = 0.0 > > # This computes the random walk by forming the cumulative sum of > # the random sample. > x = r.cumsum(axis=0) > x += x0 > > return x > > > > > Warren > > > _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Wed Sep 25 13:41:17 2013 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Wed, 25 Sep 2013 10:41:17 -0700 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? Message-ID: Thanks, guys. Yeah, I realized the problem w/ the uniform-increment-variable-direction approach this morning: physically, it ignores the fact that the particles hitting the particle being tracked are going to have a distribution of momentum, not all the same, just varying in direction. But I don't quite understand Warren's observation: "the 'angles' that describe the position undergo a random walk [actually, it would seem that they don't, since they too fail the varying-as-white-noise test], so the particle tends to move in the same direction over short intervals"--is this just another way of saying that, since I was varying the angles by -1, 0, or 1 unit each time, the simulation is susceptible to "unnaturally" long strings of -1, 0, or 1 increments? Thanks again, DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Sep 25 14:02:27 2013 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 25 Sep 2013 14:02:27 -0400 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 1:41 PM, David Goldsmith wrote: > Thanks, guys. Yeah, I realized the problem w/ the > uniform-increment-variable-direction approach this morning: physically, it > ignores the fact that the particles hitting the particle being tracked are > going to have a distribution of momentum, not all the same, just varying in > direction. But I don't quite understand Warren's observation: "the > 'angles' that describe the position undergo a random walk [actually, it > would seem that they don't, since they too fail the varying-as-white-noise > test], so the particle tends to move in the same direction over short > intervals"--is this just another way of saying that, since I was varying > the angles by -1, 0, or 1 unit each time, the simulation is susceptible to > "unnaturally" long strings of -1, 0, or 1 increments? Thanks again, > Note: I was interpreting your code as the discretization of a stochastic process, and I was experimenting with values of `incr` that were small, e.g. `incr = 0.01`. This code t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) t[0] = 0 t = t.cumsum() makes `t` a (discrete) random walk. At each time step, t either remains the same, or changes by +/- 2*np.pi*incr. If `incr` is small, then `t[1]` is a small step from `t[0]`. Similarly, `p[1]` will be close to `p[0]`. So the particle "remembers" its direction. A particle undergoing Brownian motion does not have this memory. Warren > DG > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fmmirzaei at gmail.com Wed Sep 25 20:26:34 2013 From: fmmirzaei at gmail.com (Faraz Mirzaei) Date: Wed, 25 Sep 2013 17:26:34 -0700 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef Message-ID: Hi everyone, I'm using np.ma.corrcoef to compute the correlation coefficients among rows of a masked matrix, where the masked elements are the missing data. I've observed that in some cases, the np.ma.corrcoef gives invalid coefficients that are greater than 1 or less than -1. Here's an example: x = array([[ 7, -4, -1, -7, -3, -2], [ 6, -3, 0, 4, 0, 5], [-4, -5, 7, 5, -7, -7], [-5, 5, -8, 0, 1, 4]]) x_ma = np.ma.masked_less_equal(x , -5) C = np.round(np.ma.corrcoef(x_ma), 2) print C [[1.0 0.73 -- -1.68] [0.73 1.0 -0.86 -0.38] [-- -0.86 1.0 --] [-1.68 -0.38 -- 1.0]] As you can see, the [0,3] element is -1.68 which is not a valid correlation coefficient. (Valid correlation coefficients should be between -1 and 1). I looked at the code for np.ma.corrcoef, and this behavior seems to be due to the way that mean values of the rows of the input matrix are computed and subtracted from them. Apparently, the mean value is individually computed for each row, without masking the elements corresponding to the masked elements of the other row of the matrix, with respect to which the correlation coefficient is being computed. I guess the right way should be to recompute the mean value for each row every time that a correlation coefficient is being computed for two rows after propagating pair-wise masked values. Please let me know what you think. Thanks, Faraz -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 25 23:05:49 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Sep 2013 23:05:49 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei wrote: > Hi everyone, > > I'm using np.ma.corrcoef to compute the correlation coefficients among rows > of a masked matrix, where the masked elements are the missing data. I've > observed that in some cases, the np.ma.corrcoef gives invalid coefficients > that are greater than 1 or less than -1. > > Here's an example: > > x = array([[ 7, -4, -1, -7, -3, -2], > [ 6, -3, 0, 4, 0, 5], > [-4, -5, 7, 5, -7, -7], > [-5, 5, -8, 0, 1, 4]]) > > x_ma = np.ma.masked_less_equal(x , -5) > > C = np.round(np.ma.corrcoef(x_ma), 2) > > print C > > [[1.0 0.73 -- -1.68] > [0.73 1.0 -0.86 -0.38] > [-- -0.86 1.0 --] > [-1.68 -0.38 -- 1.0]] > > As you can see, the [0,3] element is -1.68 which is not a valid correlation > coefficient. (Valid correlation coefficients should be between -1 and 1). > > I looked at the code for np.ma.corrcoef, and this behavior seems to be due > to the way that mean values of the rows of the input matrix are computed and > subtracted from them. Apparently, the mean value is individually computed > for each row, without masking the elements corresponding to the masked > elements of the other row of the matrix, with respect to which the > correlation coefficient is being computed. > > I guess the right way should be to recompute the mean value for each row > every time that a correlation coefficient is being computed for two rows > after propagating pair-wise masked values. > > Please let me know what you think. just general comments, I have no experience here >From what you are saying it sounds like np.ma is not doing pairwise deletion in calculating the mean (which only requires ignoring missings in one array), however it does (correctly) do pairwise deletion in calculating the cross product. covariance or correlation matrices with pairwise deletion are not necessarily "proper" covariance or correlation matrices. I've read that they don't need to be positive semi-definite, but I've never heard of values outside of [-1, 1]. It might only be a problem if you have a large fraction of missing values.. I think the current behavior in np.ma makes sense in that it uses all the information available in estimating the mean, which should be more accurate if we use more information. But it makes cov and corrcoef even weirder than they already are with pairwise deletion. Row-wise deletion (deleting observations that have at least one missing), which would create "proper" correlation matrices, wouldn't produce much in your example. I would check what R or other packages are doing and follow their lead, or add another option. (similar: we had a case in statsmodels where I used initially all information for calculating the mean, but then we dropped some observations to match the behavior of Stata, and to use the same observations for calculating the mean and the follow up statistics.) looks like pandas might be truncating the correlations to [-1, 1] (I didn't check) >>> import pandas as pd >>> x_pd = pd.DataFrame(x_ma.T) >>> x_pd.corr() 0 1 2 3 0 1.000000 0.734367 -1.000000 -0.240192 1 0.734367 1.000000 -0.856565 -0.378777 2 -1.000000 -0.856565 1.000000 NaN 3 -0.240192 -0.378777 NaN 1.000000 >>> np.round(np.ma.corrcoef(x_ma), 6) masked_array(data = [[1.0 0.734367 -1.190909 -1.681346] [0.734367 1.0 -0.856565 -0.378777] [-1.190909 -0.856565 1.0 --] [-1.681346 -0.378777 -- 1.0]], mask = [[False False False False] [False False False False] [False False False True] [False False True False]], fill_value = 1e+20) Josef > > Thanks, > > Faraz > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Wed Sep 25 23:19:30 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 25 Sep 2013 23:19:30 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 11:05 PM, wrote: > On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei wrote: >> Hi everyone, >> >> I'm using np.ma.corrcoef to compute the correlation coefficients among rows >> of a masked matrix, where the masked elements are the missing data. I've >> observed that in some cases, the np.ma.corrcoef gives invalid coefficients >> that are greater than 1 or less than -1. >> >> Here's an example: >> >> x = array([[ 7, -4, -1, -7, -3, -2], >> [ 6, -3, 0, 4, 0, 5], >> [-4, -5, 7, 5, -7, -7], >> [-5, 5, -8, 0, 1, 4]]) >> >> x_ma = np.ma.masked_less_equal(x , -5) >> >> C = np.round(np.ma.corrcoef(x_ma), 2) >> >> print C >> >> [[1.0 0.73 -- -1.68] >> [0.73 1.0 -0.86 -0.38] >> [-- -0.86 1.0 --] >> [-1.68 -0.38 -- 1.0]] >> >> As you can see, the [0,3] element is -1.68 which is not a valid correlation >> coefficient. (Valid correlation coefficients should be between -1 and 1). >> >> I looked at the code for np.ma.corrcoef, and this behavior seems to be due >> to the way that mean values of the rows of the input matrix are computed and >> subtracted from them. Apparently, the mean value is individually computed >> for each row, without masking the elements corresponding to the masked >> elements of the other row of the matrix, with respect to which the >> correlation coefficient is being computed. >> >> I guess the right way should be to recompute the mean value for each row >> every time that a correlation coefficient is being computed for two rows >> after propagating pair-wise masked values. >> >> Please let me know what you think. > > just general comments, I have no experience here > > From what you are saying it sounds like np.ma is not doing pairwise > deletion in calculating the mean (which only requires ignoring > missings in one array), however it does (correctly) do pairwise > deletion in calculating the cross product. Actually, I think the calculation of the mean is not relevant for having weird correlation coefficients without clipping. With pairwise deletion you use different samples, subsets of the data, for the variances and the covariances. It should be easy (?) to construct examples where the pairwise deletion for the covariance produces a large positive or negative number, and both variances and standard deviations are small, using two different subsamples. Once you calculate the correlation coefficient, it could be all over the place, independent of the mean calculations. conclusion: pairwise deletion requires post-processing if you want a proper correlation matrix. Josef > > covariance or correlation matrices with pairwise deletion are not > necessarily "proper" covariance or correlation matrices. > I've read that they don't need to be positive semi-definite, but I've > never heard of values outside of [-1, 1]. It might only be a problem > if you have a large fraction of missing values.. > > I think the current behavior in np.ma makes sense in that it uses all > the information available in estimating the mean, which should be more > accurate if we use more information. But it makes cov and corrcoef > even weirder than they already are with pairwise deletion. > > Row-wise deletion (deleting observations that have at least one > missing), which would create "proper" correlation matrices, wouldn't > produce much in your example. > > I would check what R or other packages are doing and follow their > lead, or add another option. > (similar: we had a case in statsmodels where I used initially all > information for calculating the mean, but then we dropped some > observations to match the behavior of Stata, and to use the same > observations for calculating the mean and the follow up statistics.) > > > looks like pandas might be truncating the correlations to [-1, 1] (I > didn't check) > >>>> import pandas as pd >>>> x_pd = pd.DataFrame(x_ma.T) >>>> x_pd.corr() > 0 1 2 3 > 0 1.000000 0.734367 -1.000000 -0.240192 > 1 0.734367 1.000000 -0.856565 -0.378777 > 2 -1.000000 -0.856565 1.000000 NaN > 3 -0.240192 -0.378777 NaN 1.000000 > >>>> np.round(np.ma.corrcoef(x_ma), 6) > masked_array(data = > [[1.0 0.734367 -1.190909 -1.681346] > [0.734367 1.0 -0.856565 -0.378777] > [-1.190909 -0.856565 1.0 --] > [-1.681346 -0.378777 -- 1.0]], > mask = > [[False False False False] > [False False False False] > [False False False True] > [False False True False]], > fill_value = 1e+20) > > > Josef > > >> >> Thanks, >> >> Faraz >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> From davidmenhur at gmail.com Thu Sep 26 04:02:30 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 26 Sep 2013 10:02:30 +0200 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On 25 September 2013 19:41, David Goldsmith wrote: > "the 'angles' that describe the position undergo a random walk [actually, > it would seem that they don't, since they too fail the > varying-as-white-noise test], so the particle tends to move in the same > direction over short intervals"--is this just another way of saying that, > since I was varying the angles by -1, 0, or 1 unit each time, the > simulation is susceptible to "unnaturally" long strings of -1, 0, or 1 > increments? In the 1D case, the white noise has a gaussian probability distribution of being positive or negative. Translated to the Wiener process, it means you would have to sum normally distributed values. When you go 3D you can do the same thing, taking a random displacement from a N(0,1) and two random angles. The issue here is that the polar angles cannot be taken uniformly, but instead they have to be distributed proportionally to the jacobian. As you have it now, your particle will tend to move towards the poles. If you want to visualize it: take a sphere and imagine dots spaced evenly at angles (intersection of meridians and parallels, for example): they are much more dense at the poles. The simplest way is to do it in cartesian coordinates: take x, y, and z independently from N(0,1). If you want to generate only one normal number per step, consider the jacobian in the angles. David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 26 04:21:59 2013 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2013 09:21:59 +0100 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: If you want a proper self-consistent correlation/covariance matrix, then pairwise deletion just makes no sense period, I don't see how postprocessing can help. If you want a matrix of correlations, then pairwise deletion makes sense. It's an interesting point that arguably the current ma.corrcoef code may actually give you a better estimator of the individual correlation coefficients than doing full pairwise deletion, but it's pretty surprising and unexpected... when people call corrcoef I think they are asking "please compute the textbook formula for 'sample correlation'" not "please give me some arbitrary good estimator for the population correlation", so we probably have to change it. (Hopefully no-one has published anything based on the current code.) -n On 26 Sep 2013 04:19, wrote: > On Wed, Sep 25, 2013 at 11:05 PM, wrote: > > On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei > wrote: > >> Hi everyone, > >> > >> I'm using np.ma.corrcoef to compute the correlation coefficients among > rows > >> of a masked matrix, where the masked elements are the missing data. I've > >> observed that in some cases, the np.ma.corrcoef gives invalid > coefficients > >> that are greater than 1 or less than -1. > >> > >> Here's an example: > >> > >> x = array([[ 7, -4, -1, -7, -3, -2], > >> [ 6, -3, 0, 4, 0, 5], > >> [-4, -5, 7, 5, -7, -7], > >> [-5, 5, -8, 0, 1, 4]]) > >> > >> x_ma = np.ma.masked_less_equal(x , -5) > >> > >> C = np.round(np.ma.corrcoef(x_ma), 2) > >> > >> print C > >> > >> [[1.0 0.73 -- -1.68] > >> [0.73 1.0 -0.86 -0.38] > >> [-- -0.86 1.0 --] > >> [-1.68 -0.38 -- 1.0]] > >> > >> As you can see, the [0,3] element is -1.68 which is not a valid > correlation > >> coefficient. (Valid correlation coefficients should be between -1 and > 1). > >> > >> I looked at the code for np.ma.corrcoef, and this behavior seems to be > due > >> to the way that mean values of the rows of the input matrix are > computed and > >> subtracted from them. Apparently, the mean value is individually > computed > >> for each row, without masking the elements corresponding to the masked > >> elements of the other row of the matrix, with respect to which the > >> correlation coefficient is being computed. > >> > >> I guess the right way should be to recompute the mean value for each row > >> every time that a correlation coefficient is being computed for two rows > >> after propagating pair-wise masked values. > >> > >> Please let me know what you think. > > > > just general comments, I have no experience here > > > > From what you are saying it sounds like np.ma is not doing pairwise > > deletion in calculating the mean (which only requires ignoring > > missings in one array), however it does (correctly) do pairwise > > deletion in calculating the cross product. > > Actually, I think the calculation of the mean is not relevant for > having weird correlation coefficients without clipping. > > With pairwise deletion you use different samples, subsets of the data, > for the variances and the covariances. > It should be easy (?) to construct examples where the pairwise > deletion for the covariance produces a large positive or negative > number, and both variances and standard deviations are small, using > two different subsamples. > Once you calculate the correlation coefficient, it could be all over > the place, independent of the mean calculations. > > conclusion: pairwise deletion requires post-processing if you want a > proper correlation matrix. > > Josef > > > > > covariance or correlation matrices with pairwise deletion are not > > necessarily "proper" covariance or correlation matrices. > > I've read that they don't need to be positive semi-definite, but I've > > never heard of values outside of [-1, 1]. It might only be a problem > > if you have a large fraction of missing values.. > > > > I think the current behavior in np.ma makes sense in that it uses all > > the information available in estimating the mean, which should be more > > accurate if we use more information. But it makes cov and corrcoef > > even weirder than they already are with pairwise deletion. > > > > Row-wise deletion (deleting observations that have at least one > > missing), which would create "proper" correlation matrices, wouldn't > > produce much in your example. > > > > I would check what R or other packages are doing and follow their > > lead, or add another option. > > (similar: we had a case in statsmodels where I used initially all > > information for calculating the mean, but then we dropped some > > observations to match the behavior of Stata, and to use the same > > observations for calculating the mean and the follow up statistics.) > > > > > > looks like pandas might be truncating the correlations to [-1, 1] (I > > didn't check) > > > >>>> import pandas as pd > >>>> x_pd = pd.DataFrame(x_ma.T) > >>>> x_pd.corr() > > 0 1 2 3 > > 0 1.000000 0.734367 -1.000000 -0.240192 > > 1 0.734367 1.000000 -0.856565 -0.378777 > > 2 -1.000000 -0.856565 1.000000 NaN > > 3 -0.240192 -0.378777 NaN 1.000000 > > > >>>> np.round(np.ma.corrcoef(x_ma), 6) > > masked_array(data = > > [[1.0 0.734367 -1.190909 -1.681346] > > [0.734367 1.0 -0.856565 -0.378777] > > [-1.190909 -0.856565 1.0 --] > > [-1.681346 -0.378777 -- 1.0]], > > mask = > > [[False False False False] > > [False False False False] > > [False False False True] > > [False False True False]], > > fill_value = 1e+20) > > > > > > Josef > > > > > >> > >> Thanks, > >> > >> Faraz > >> > >> > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Sep 26 06:51:34 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Sep 2013 06:51:34 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: > If you want a proper self-consistent correlation/covariance matrix, then > pairwise deletion just makes no sense period, I don't see how postprocessing > can help. clipping to [-1, 1] and finding the nearest positive semi-definite matrix. For the latter there is some code in statsmodels, and several newer algorithms that I haven't looked at. It's a quite common problem in finance, but usually longer time series with not a large number of missing values. > > If you want a matrix of correlations, then pairwise deletion makes sense. > It's an interesting point that arguably the current ma.corrcoef code may > actually give you a better estimator of the individual correlation > coefficients than doing full pairwise deletion, but it's pretty surprising > and unexpected... when people call corrcoef I think they are asking "please > compute the textbook formula for 'sample correlation'" not "please give me > some arbitrary good estimator for the population correlation", so we > probably have to change it. > > (Hopefully no-one has published anything based on the current code.) I haven't seen a textbook version of this yet. Calculating every mean (n + 1) * n / 2 times sounds a bit excessive, especially if it doesn't really solve the problem. Josef > > -n > > On 26 Sep 2013 04:19, wrote: >> >> On Wed, Sep 25, 2013 at 11:05 PM, wrote: >> > On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei >> > wrote: >> >> Hi everyone, >> >> >> >> I'm using np.ma.corrcoef to compute the correlation coefficients among >> >> rows >> >> of a masked matrix, where the masked elements are the missing data. >> >> I've >> >> observed that in some cases, the np.ma.corrcoef gives invalid >> >> coefficients >> >> that are greater than 1 or less than -1. >> >> >> >> Here's an example: >> >> >> >> x = array([[ 7, -4, -1, -7, -3, -2], >> >> [ 6, -3, 0, 4, 0, 5], >> >> [-4, -5, 7, 5, -7, -7], >> >> [-5, 5, -8, 0, 1, 4]]) >> >> >> >> x_ma = np.ma.masked_less_equal(x , -5) >> >> >> >> C = np.round(np.ma.corrcoef(x_ma), 2) >> >> >> >> print C >> >> >> >> [[1.0 0.73 -- -1.68] >> >> [0.73 1.0 -0.86 -0.38] >> >> [-- -0.86 1.0 --] >> >> [-1.68 -0.38 -- 1.0]] >> >> >> >> As you can see, the [0,3] element is -1.68 which is not a valid >> >> correlation >> >> coefficient. (Valid correlation coefficients should be between -1 and >> >> 1). >> >> >> >> I looked at the code for np.ma.corrcoef, and this behavior seems to be >> >> due >> >> to the way that mean values of the rows of the input matrix are >> >> computed and >> >> subtracted from them. Apparently, the mean value is individually >> >> computed >> >> for each row, without masking the elements corresponding to the masked >> >> elements of the other row of the matrix, with respect to which the >> >> correlation coefficient is being computed. >> >> >> >> I guess the right way should be to recompute the mean value for each >> >> row >> >> every time that a correlation coefficient is being computed for two >> >> rows >> >> after propagating pair-wise masked values. >> >> >> >> Please let me know what you think. >> > >> > just general comments, I have no experience here >> > >> > From what you are saying it sounds like np.ma is not doing pairwise >> > deletion in calculating the mean (which only requires ignoring >> > missings in one array), however it does (correctly) do pairwise >> > deletion in calculating the cross product. >> >> Actually, I think the calculation of the mean is not relevant for >> having weird correlation coefficients without clipping. >> >> With pairwise deletion you use different samples, subsets of the data, >> for the variances and the covariances. >> It should be easy (?) to construct examples where the pairwise >> deletion for the covariance produces a large positive or negative >> number, and both variances and standard deviations are small, using >> two different subsamples. >> Once you calculate the correlation coefficient, it could be all over >> the place, independent of the mean calculations. >> >> conclusion: pairwise deletion requires post-processing if you want a >> proper correlation matrix. >> >> Josef >> >> > >> > covariance or correlation matrices with pairwise deletion are not >> > necessarily "proper" covariance or correlation matrices. >> > I've read that they don't need to be positive semi-definite, but I've >> > never heard of values outside of [-1, 1]. It might only be a problem >> > if you have a large fraction of missing values.. >> > >> > I think the current behavior in np.ma makes sense in that it uses all >> > the information available in estimating the mean, which should be more >> > accurate if we use more information. But it makes cov and corrcoef >> > even weirder than they already are with pairwise deletion. >> > >> > Row-wise deletion (deleting observations that have at least one >> > missing), which would create "proper" correlation matrices, wouldn't >> > produce much in your example. >> > >> > I would check what R or other packages are doing and follow their >> > lead, or add another option. >> > (similar: we had a case in statsmodels where I used initially all >> > information for calculating the mean, but then we dropped some >> > observations to match the behavior of Stata, and to use the same >> > observations for calculating the mean and the follow up statistics.) >> > >> > >> > looks like pandas might be truncating the correlations to [-1, 1] (I >> > didn't check) >> > >> >>>> import pandas as pd >> >>>> x_pd = pd.DataFrame(x_ma.T) >> >>>> x_pd.corr() >> > 0 1 2 3 >> > 0 1.000000 0.734367 -1.000000 -0.240192 >> > 1 0.734367 1.000000 -0.856565 -0.378777 >> > 2 -1.000000 -0.856565 1.000000 NaN >> > 3 -0.240192 -0.378777 NaN 1.000000 >> > >> >>>> np.round(np.ma.corrcoef(x_ma), 6) >> > masked_array(data = >> > [[1.0 0.734367 -1.190909 -1.681346] >> > [0.734367 1.0 -0.856565 -0.378777] >> > [-1.190909 -0.856565 1.0 --] >> > [-1.681346 -0.378777 -- 1.0]], >> > mask = >> > [[False False False False] >> > [False False False False] >> > [False False False True] >> > [False False True False]], >> > fill_value = 1e+20) >> > >> > >> > Josef >> > >> > >> >> >> >> Thanks, >> >> >> >> Faraz >> >> >> >> >> >> >> >> _______________________________________________ >> >> NumPy-Discussion mailing list >> >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Thu Sep 26 06:56:21 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Sep 2013 06:56:21 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 6:51 AM, wrote: > On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >> If you want a proper self-consistent correlation/covariance matrix, then >> pairwise deletion just makes no sense period, I don't see how postprocessing >> can help. > > clipping to [-1, 1] and finding the nearest positive semi-definite matrix. > For the latter there is some code in statsmodels, and several newer > algorithms that I haven't looked at. > > It's a quite common problem in finance, but usually longer time series > with not a large number of missing values. > >> >> If you want a matrix of correlations, then pairwise deletion makes sense. >> It's an interesting point that arguably the current ma.corrcoef code may >> actually give you a better estimator of the individual correlation >> coefficients than doing full pairwise deletion, but it's pretty surprising >> and unexpected... when people call corrcoef I think they are asking "please >> compute the textbook formula for 'sample correlation'" not "please give me >> some arbitrary good estimator for the population correlation", so we >> probably have to change it. >> >> (Hopefully no-one has published anything based on the current code.) > > I haven't seen a textbook version of this yet. > > Calculating every mean (n + 1) * n / 2 times sounds a bit excessive, > especially if it doesn't really solve the problem. unless you also calculate each standard deviation (n + 1) * n / 2 times. But then you loose the relationship between cov and corrcoeff. Josef > > Josef > >> >> -n >> >> On 26 Sep 2013 04:19, wrote: >>> >>> On Wed, Sep 25, 2013 at 11:05 PM, wrote: >>> > On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei >>> > wrote: >>> >> Hi everyone, >>> >> >>> >> I'm using np.ma.corrcoef to compute the correlation coefficients among >>> >> rows >>> >> of a masked matrix, where the masked elements are the missing data. >>> >> I've >>> >> observed that in some cases, the np.ma.corrcoef gives invalid >>> >> coefficients >>> >> that are greater than 1 or less than -1. >>> >> >>> >> Here's an example: >>> >> >>> >> x = array([[ 7, -4, -1, -7, -3, -2], >>> >> [ 6, -3, 0, 4, 0, 5], >>> >> [-4, -5, 7, 5, -7, -7], >>> >> [-5, 5, -8, 0, 1, 4]]) >>> >> >>> >> x_ma = np.ma.masked_less_equal(x , -5) >>> >> >>> >> C = np.round(np.ma.corrcoef(x_ma), 2) >>> >> >>> >> print C >>> >> >>> >> [[1.0 0.73 -- -1.68] >>> >> [0.73 1.0 -0.86 -0.38] >>> >> [-- -0.86 1.0 --] >>> >> [-1.68 -0.38 -- 1.0]] >>> >> >>> >> As you can see, the [0,3] element is -1.68 which is not a valid >>> >> correlation >>> >> coefficient. (Valid correlation coefficients should be between -1 and >>> >> 1). >>> >> >>> >> I looked at the code for np.ma.corrcoef, and this behavior seems to be >>> >> due >>> >> to the way that mean values of the rows of the input matrix are >>> >> computed and >>> >> subtracted from them. Apparently, the mean value is individually >>> >> computed >>> >> for each row, without masking the elements corresponding to the masked >>> >> elements of the other row of the matrix, with respect to which the >>> >> correlation coefficient is being computed. >>> >> >>> >> I guess the right way should be to recompute the mean value for each >>> >> row >>> >> every time that a correlation coefficient is being computed for two >>> >> rows >>> >> after propagating pair-wise masked values. >>> >> >>> >> Please let me know what you think. >>> > >>> > just general comments, I have no experience here >>> > >>> > From what you are saying it sounds like np.ma is not doing pairwise >>> > deletion in calculating the mean (which only requires ignoring >>> > missings in one array), however it does (correctly) do pairwise >>> > deletion in calculating the cross product. >>> >>> Actually, I think the calculation of the mean is not relevant for >>> having weird correlation coefficients without clipping. >>> >>> With pairwise deletion you use different samples, subsets of the data, >>> for the variances and the covariances. >>> It should be easy (?) to construct examples where the pairwise >>> deletion for the covariance produces a large positive or negative >>> number, and both variances and standard deviations are small, using >>> two different subsamples. >>> Once you calculate the correlation coefficient, it could be all over >>> the place, independent of the mean calculations. >>> >>> conclusion: pairwise deletion requires post-processing if you want a >>> proper correlation matrix. >>> >>> Josef >>> >>> > >>> > covariance or correlation matrices with pairwise deletion are not >>> > necessarily "proper" covariance or correlation matrices. >>> > I've read that they don't need to be positive semi-definite, but I've >>> > never heard of values outside of [-1, 1]. It might only be a problem >>> > if you have a large fraction of missing values.. >>> > >>> > I think the current behavior in np.ma makes sense in that it uses all >>> > the information available in estimating the mean, which should be more >>> > accurate if we use more information. But it makes cov and corrcoef >>> > even weirder than they already are with pairwise deletion. >>> > >>> > Row-wise deletion (deleting observations that have at least one >>> > missing), which would create "proper" correlation matrices, wouldn't >>> > produce much in your example. >>> > >>> > I would check what R or other packages are doing and follow their >>> > lead, or add another option. >>> > (similar: we had a case in statsmodels where I used initially all >>> > information for calculating the mean, but then we dropped some >>> > observations to match the behavior of Stata, and to use the same >>> > observations for calculating the mean and the follow up statistics.) >>> > >>> > >>> > looks like pandas might be truncating the correlations to [-1, 1] (I >>> > didn't check) >>> > >>> >>>> import pandas as pd >>> >>>> x_pd = pd.DataFrame(x_ma.T) >>> >>>> x_pd.corr() >>> > 0 1 2 3 >>> > 0 1.000000 0.734367 -1.000000 -0.240192 >>> > 1 0.734367 1.000000 -0.856565 -0.378777 >>> > 2 -1.000000 -0.856565 1.000000 NaN >>> > 3 -0.240192 -0.378777 NaN 1.000000 >>> > >>> >>>> np.round(np.ma.corrcoef(x_ma), 6) >>> > masked_array(data = >>> > [[1.0 0.734367 -1.190909 -1.681346] >>> > [0.734367 1.0 -0.856565 -0.378777] >>> > [-1.190909 -0.856565 1.0 --] >>> > [-1.681346 -0.378777 -- 1.0]], >>> > mask = >>> > [[False False False False] >>> > [False False False False] >>> > [False False False True] >>> > [False False True False]], >>> > fill_value = 1e+20) >>> > >>> > >>> > Josef >>> > >>> > >>> >> >>> >> Thanks, >>> >> >>> >> Faraz >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> NumPy-Discussion mailing list >>> >> NumPy-Discussion at scipy.org >>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> From njs at pobox.com Thu Sep 26 07:35:56 2013 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2013 12:35:56 +0100 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 11:51 AM, wrote: > On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >> If you want a proper self-consistent correlation/covariance matrix, then >> pairwise deletion just makes no sense period, I don't see how postprocessing >> can help. > > clipping to [-1, 1] and finding the nearest positive semi-definite matrix. > For the latter there is some code in statsmodels, and several newer > algorithms that I haven't looked at. While in general there's an interesting problem for how to best estimate a joint covariance matrix in the presence of missing data, this sounds way beyond the scope of lowly corrcoef(). The very fact that there is active research on the problem means that we shouldn't be trying to pick one algorithm and declare that it's *the* correct one to build in as a basic tool for unsophisticated users. It's not clear that this is even what people want corrcoef to do in general. I always use it as just a convenient way to estimate lots of pairwise correlations, not as a way to estimate a joint correlation matrix... > It's a quite common problem in finance, but usually longer time series > with not a large number of missing values. > >> >> If you want a matrix of correlations, then pairwise deletion makes sense. >> It's an interesting point that arguably the current ma.corrcoef code may >> actually give you a better estimator of the individual correlation >> coefficients than doing full pairwise deletion, but it's pretty surprising >> and unexpected... when people call corrcoef I think they are asking "please >> compute the textbook formula for 'sample correlation'" not "please give me >> some arbitrary good estimator for the population correlation", so we >> probably have to change it. >> >> (Hopefully no-one has published anything based on the current code.) > > I haven't seen a textbook version of this yet. By textbook I mean, users expect corrcoef to use this formula, which is printed in every textbook: https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#For_a_sample The vast majority of people using correlations think that "sample correlation" justs mean this number, not "some arbitrary finite-sample estimator of the underlying population correlation". So the obvious interpretation of pairwise correlations is that you apply that formula to each set of pairwise complete observations. > Calculating every mean (n + 1) * n / 2 times sounds a bit excessive, > especially if it doesn't really solve the problem. I don't understand what's "excessive" about calculating every mean/stddev (n + 1)*n/2 times. By that logic one should never call corrcoef at all, because calculating (n + 1)*n/2 covariances is also excessive :-). It's not like we're talking about some massive slowdown. >>> > looks like pandas might be truncating the correlations to [-1, 1] (I >>> > didn't check) >>> > >>> >>>> import pandas as pd >>> >>>> x_pd = pd.DataFrame(x_ma.T) >>> >>>> x_pd.corr() >>> > 0 1 2 3 >>> > 0 1.000000 0.734367 -1.000000 -0.240192 >>> > 1 0.734367 1.000000 -0.856565 -0.378777 >>> > 2 -1.000000 -0.856565 1.000000 NaN >>> > 3 -0.240192 -0.378777 NaN 1.000000 >>> > >>> >>>> np.round(np.ma.corrcoef(x_ma), 6) >>> > masked_array(data = >>> > [[1.0 0.734367 -1.190909 -1.681346] >>> > [0.734367 1.0 -0.856565 -0.378777] >>> > [-1.190909 -0.856565 1.0 --] >>> > [-1.681346 -0.378777 -- 1.0]], That can't be truncation -- where corrcoef has -1.68, pandas has -0.24, not -1.0. R gives the same result as pandas (except that by default it propagates NA and give a range of options for handling NAs, so you have to explicitly request pairwise complete if you want it, and they make this the most annoying option to type ;-), and the help page explicitly warns that this "can result in covariance or correlation matrices which are not positive semi-definite"): > x <- c(7, -4, -1, -7, -3, -2, 6, -3, 0, 4, 0, 5, -4, -5, 7, 5, -7, -7, -5, 5, -8, 0, 1, 4) > x <- matrix(x, nrow=4, byrow=T) > cor(t(x), use="pairwise.complete.obs") [,1] [,2] [,3] [,4] [1,] 1.0000000 0.43330535 -0.22749669 -0.5246782 [2,] 0.4333053 1.00000000 -0.01829124 -0.2120425 [3,] -0.2274967 -0.01829124 1.00000000 -0.6423032 [4,] -0.5246782 -0.21204248 -0.64230323 1.0000000 -n From josef.pktd at gmail.com Thu Sep 26 12:28:54 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Sep 2013 12:28:54 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 7:35 AM, Nathaniel Smith wrote: > On Thu, Sep 26, 2013 at 11:51 AM, wrote: >> On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >>> If you want a proper self-consistent correlation/covariance matrix, then >>> pairwise deletion just makes no sense period, I don't see how postprocessing >>> can help. >> >> clipping to [-1, 1] and finding the nearest positive semi-definite matrix. >> For the latter there is some code in statsmodels, and several newer >> algorithms that I haven't looked at. > > While in general there's an interesting problem for how to best > estimate a joint covariance matrix in the presence of missing data, > this sounds way beyond the scope of lowly corrcoef(). The very fact > that there is active research on the problem means that we shouldn't > be trying to pick one algorithm and declare that it's *the* correct > one to build in as a basic tool for unsophisticated users. > > It's not clear that this is even what people want corrcoef to do in > general. I always use it as just a convenient way to estimate lots of > pairwise correlations, not as a way to estimate a joint correlation > matrix... I don't expect that corrcoef or cov should provide a positive semidefinite approximation. I just wanted to point out what users can do with the return from corrcoef or cov if they want a proper correlation or covariance matrix. (same discussion with pandas, post-processing is a separate step.) > >> It's a quite common problem in finance, but usually longer time series >> with not a large number of missing values. >> >>> >>> If you want a matrix of correlations, then pairwise deletion makes sense. >>> It's an interesting point that arguably the current ma.corrcoef code may >>> actually give you a better estimator of the individual correlation >>> coefficients than doing full pairwise deletion, but it's pretty surprising >>> and unexpected... when people call corrcoef I think they are asking "please >>> compute the textbook formula for 'sample correlation'" not "please give me >>> some arbitrary good estimator for the population correlation", so we >>> probably have to change it. >>> >>> (Hopefully no-one has published anything based on the current code.) >> >> I haven't seen a textbook version of this yet. > > By textbook I mean, users expect corrcoef to use this formula, which > is printed in every textbook: > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#For_a_sample > The vast majority of people using correlations think that "sample > correlation" justs mean this number, not "some arbitrary finite-sample > estimator of the underlying population correlation". So the obvious > interpretation of pairwise correlations is that you apply that formula > to each set of pairwise complete observations. This textbook version **assumes** that we have the same observations for all/both variables, and doesn't say what to do if not. I'm usually mainly interested the covariance/correlation matrix for estimating some underlying population or model parameters or do hypothesis tests with them. I just wanted to point out that there is no "obvious" ("There should be one-- ...") way to define pairwise deletion correlation matrices. But maybe just doing a loop [corrcoef(x, y) for x in data for y in data] still makes the most sense. Dunno > >> Calculating every mean (n + 1) * n / 2 times sounds a bit excessive, >> especially if it doesn't really solve the problem. > > I don't understand what's "excessive" about calculating every > mean/stddev (n + 1)*n/2 times. By that logic one should never call > corrcoef at all, because calculating (n + 1)*n/2 covariances is also > excessive :-). It's not like we're talking about some massive > slowdown. > >>>> > looks like pandas might be truncating the correlations to [-1, 1] (I >>>> > didn't check) >>>> > >>>> >>>> import pandas as pd >>>> >>>> x_pd = pd.DataFrame(x_ma.T) >>>> >>>> x_pd.corr() >>>> > 0 1 2 3 >>>> > 0 1.000000 0.734367 -1.000000 -0.240192 >>>> > 1 0.734367 1.000000 -0.856565 -0.378777 >>>> > 2 -1.000000 -0.856565 1.000000 NaN >>>> > 3 -0.240192 -0.378777 NaN 1.000000 >>>> > >>>> >>>> np.round(np.ma.corrcoef(x_ma), 6) >>>> > masked_array(data = >>>> > [[1.0 0.734367 -1.190909 -1.681346] >>>> > [0.734367 1.0 -0.856565 -0.378777] >>>> > [-1.190909 -0.856565 1.0 --] >>>> > [-1.681346 -0.378777 -- 1.0]], > > That can't be truncation -- where corrcoef has -1.68, pandas has > -0.24, not -1.0. my mistake, for not reading carefully enough Josef > > R gives the same result as pandas (except that by default it > propagates NA and give a range of options for handling NAs, so you > have to explicitly request pairwise complete if you want it, and they > make this the most annoying option to type ;-), and the help page > explicitly warns that this "can result in covariance or correlation > matrices which are not positive semi-definite"): > >> x <- c(7, -4, -1, -7, -3, -2, 6, -3, 0, 4, 0, 5, -4, -5, 7, 5, -7, -7, -5, 5, -8, 0, 1, 4) >> x <- matrix(x, nrow=4, byrow=T) >> cor(t(x), use="pairwise.complete.obs") > [,1] [,2] [,3] [,4] > [1,] 1.0000000 0.43330535 -0.22749669 -0.5246782 > [2,] 0.4333053 1.00000000 -0.01829124 -0.2120425 > [3,] -0.2274967 -0.01829124 1.00000000 -0.6423032 > [4,] -0.5246782 -0.21204248 -0.64230323 1.0000000 > > -n > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From njs at pobox.com Thu Sep 26 13:21:42 2013 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2013 18:21:42 +0100 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On 26 Sep 2013 17:32, wrote: > > On Thu, Sep 26, 2013 at 7:35 AM, Nathaniel Smith wrote: > > By textbook I mean, users expect corrcoef to use this formula, which > > is printed in every textbook: > > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#For_a_sample > > The vast majority of people using correlations think that "sample > > correlation" justs mean this number, not "some arbitrary finite-sample > > estimator of the underlying population correlation". So the obvious > > interpretation of pairwise correlations is that you apply that formula > > to each set of pairwise complete observations. > > This textbook version **assumes** that we have the same observations > for all/both variables, and doesn't say what to do if not. > I'm usually mainly interested the covariance/correlation matrix for > estimating some underlying population or model parameters or do > hypothesis tests with them. > > I just wanted to point out that there is no "obvious" ("There should > be one-- ...") way to define pairwise deletion correlation matrices. Yeah, fair enough. > But maybe just doing a loop [corrcoef(x, y) for x in data for y in > data] still makes the most sense. Dunno I'm not 100% sure what the best answer is either, but it seems we agree that these are the only reasonable options: (1) refuse to give correlations if there are missing values (2) the pairwise version pandas/R do (3) maybe something in between (like only including fully complete rows, or giving an option to pick between these) But the key thing here is that the current behaviour is definitely *wrong* and misleading people, so we better do something about that. (And if no one pops up to fix it maybe we should just remove the function entirely from 1.8, because numerically wrong answers are Serious Business?) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseluismietta at yahoo.com.ar Thu Sep 26 14:19:38 2013 From: joseluismietta at yahoo.com.ar (=?iso-8859-1?Q?Jos=E8_Luis_Mietta?=) Date: Thu, 26 Sep 2013 11:19:38 -0700 (PDT) Subject: [Numpy-discussion] At.: use less RAM memory and increase execution speed Message-ID: <1380219578.46758.YahooMailNeo@web142302.mail.bf1.yahoo.com> Hi experts! I wanna use less RAM memory in my Monte Carlo simulations. In my algorithm I use numpy arrays and xrange() function. I hear that I can reduce RAM used in my lagorithm if I do the next: 1) replace xrange() for range(). 2) replace numpya arrays for python lists 3) use reset() function for deleting useless arrays. Is that true? In adition, I wanna? increase execution speed of my code (I use numpy and SciPy functions). How can I apply Cython? Will it help? Please help. Thanks a lot!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 26 14:39:33 2013 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 Sep 2013 19:39:33 +0100 Subject: [Numpy-discussion] At.: use less RAM memory and increase execution speed In-Reply-To: <1380219578.46758.YahooMailNeo@web142302.mail.bf1.yahoo.com> References: <1380219578.46758.YahooMailNeo@web142302.mail.bf1.yahoo.com> Message-ID: On Thu, Sep 26, 2013 at 7:19 PM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > > Hi experts! > > I wanna use less RAM memory in my Monte Carlo simulations. In my algorithm I use numpy arrays and xrange() function. > I hear that I can reduce RAM used in my lagorithm if I do the next: > > 1) replace xrange() for range(). range() will pretty much always use *more* memory than xrange(). > 2) replace numpya arrays for python lists It depends. I recommended that to you because you were using np.append() a *lot* on large arrays, and this can cause memory fragmentation issues as these large arrays need to be reallocated every time. A list of float objects of length N will use *more* memory than an equivalent float64 array of length N. I also recommended that you simply preallocate arrays of the right size and fill them in. That would be the ideal solution. > 3) use reset() function for deleting useless arrays. There is no reset() function. You may have heard about the %reset magic command in IPython which clears IPython's interactive namespace. You would not use it in your code. > Is that true? > > In adition, I wanna increase execution speed of my code (I use numpy and SciPy functions). How can I apply Cython? Will it help? There is no way to tell without knowing what is taking all of the execution time in your code. You will need to profile your code to find out. Cython *can* help quite frequently. Sometimes it won't, or at least there may be much easier things you can do to speed up your code. For example, I am willing to bet that fixing your code to avoid np.append() will make your code much faster. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From msarahan at gmail.com Thu Sep 26 14:40:39 2013 From: msarahan at gmail.com (Michael Sarahan) Date: Thu, 26 Sep 2013 11:40:39 -0700 Subject: [Numpy-discussion] At.: use less RAM memory and increase execution speed In-Reply-To: <1380219578.46758.YahooMailNeo@web142302.mail.bf1.yahoo.com> References: <1380219578.46758.YahooMailNeo@web142302.mail.bf1.yahoo.com> Message-ID: xrange should be more memory efficient than range: http://stackoverflow.com/questions/135041/should-you-always-favor-xrange-over-range Replacing arrays with lists is probably a bad idea for a lot of reasons. You'll lose nice vectorization of simple operations, and all of numpy's other benefits. To be more parsimonious with memory, you probably want to pay close attention to array data types and make sure that things aren't being automatically upconverted to higher precision data types. I've never considered using reset. Perhaps you need to take a look at your program's structure and make sure that useless arrays can be garbage collected properly. Preallocation of arrays can give you tons of benefits with regards to array size and program speed. If you aren't using preallocation, now's a great time to start. You can pass numpy arrays into Cython functions, and you can also call numpy/scipy functions within Cython functions. Identify your bottlenecks using some kind of profiling, then work on optimizing those with Cython. HTH, Mike On Thu, Sep 26, 2013 at 11:19 AM, Jos? Luis Mietta < joseluismietta at yahoo.com.ar> wrote: > Hi experts! > > I wanna use less RAM memory in my Monte Carlo simulations. In my > algorithm I use numpy arrays and xrange() function. > I hear that I can reduce RAM used in my lagorithm if I do the next: > > 1) replace xrange() for range(). > 2) replace numpya arrays for python lists > 3) use reset() function for deleting useless arrays. > Is that true? > > In adition, I wanna increase execution speed of my code (I use numpy and > SciPy functions). How can I apply Cython? Will it help? > > Please help. > > Thanks a lot!! > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fmmirzaei at gmail.com Thu Sep 26 16:59:46 2013 From: fmmirzaei at gmail.com (Faraz Mirzaei) Date: Thu, 26 Sep 2013 13:59:46 -0700 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef Message-ID: Thanks Josef and Nathaniel for your responses. In the application that I have, I don't use the correlation coefficient matrix as a whole (so I don't care if it is PSD or not). I simply read the off-diagonal elements for pair-wise correlation coefficients. I use the pairwise correlation coefficient to test if the data from various sources (i.e., rows of the matrix), agree with each other when present. Right now, I use, ma.corrcoef( x[ i, :] , x[ j, :]) and read the off-diagonal element in a loop over i and j. It is just a bit uglier than calling ma.corrcoef(x). At least for my application, truncation to -1 or +1 (or scaling such that largest values becomes 1 etc) is completely wrong, since it would imply that the two sources completely agree with each other (factoring out a minus sign), which may not the case. For example, consider the first and last rows of the example I provided: >>> print x_ma [[ 7 -4 -1 -- -3 -2] [ 6 -3 0 4 0 5] [-4 -- 7 5 -- --] [-- 5 -- 0 1 4]] >>> np.ma.corrcoef(x_ma)[0,3] -1.6813456149534147 On the other hand, if we supply only the first and third row to the function, we get: >>> np.ma.corrcoef(x_ma[0,:], x_ma[3,:]) masked_array(data = [[1.0 -0.240192230708] [-0.240192230708 1.0]], mask = [[False False] [False False]], fill_value = 1e+20) Interestingly, this is the same as what pandas results as the [3,0] element of the correlation coefficient matrix, and it is equal to pair-wise deletion result: >>> np.corrcoef([-4, -3, -2], [5, 1, 4]) #Note that this is NOT np.ma.corrcoef >>> array([[ 1. , -0.24019223], [-0.24019223, 1. ]]) Also, I don't know why the ma.corrcoef results Josef has mentioned are different than mine. In particular, Josef reports element [2, 0] of the ma.corrcoef result to be -1.19, but I get -- (i.e., missing and masked, probably due to too few samples available). Josef: are you sure that you have entered the example values correctly into python? Along the same lines, the results that Nathaniel has posted from R are different, since the input is not a masked matrix I guess (please note that in the original example, I had masked values less than or equal to -5). In any case, I think the correlation coefficient between two rows of a matrix should not depend on what other rows are supplied. In other words, np.ma.corrcoef(x_ma)[0,3] should be equal to np.ma.corrcoef(x_ma[0,:], x_ma[3,:])[0,1] (which apparently happens to be what pandas reports). This change would need recomputing the mean for every pair-wise coefficient calculation, but since we are computing cross products O(n^2) times, the overall big-O complexity won't change. And please don't remove this functionality. I will volunteer to fix it however we decide :) We can just clarify the behavior in the documentation. Thanks, Faraz -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 26 18:42:02 2013 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Sep 2013 23:42:02 +0100 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On 26 Sep 2013 21:59, "Faraz Mirzaei" wrote: > > Thanks Josef and Nathaniel for your responses. > > In the application that I have, I don't use the correlation coefficient matrix as a whole (so I don't care if it is PSD or not). I simply read the off-diagonal elements for pair-wise correlation coefficients. I use the pairwise correlation coefficient to test if the data from various sources (i.e., rows of the matrix), agree with each other when present. > > Right now, I use, ma.corrcoef( x[ i, :] , x[ j, :]) and read the off-diagonal element in a loop over i and j. It is just a bit uglier than calling ma.corrcoef(x). > > At least for my application, truncation to -1 or +1 (or scaling such that largest values becomes 1 etc) is completely wrong, since it would imply that the two sources completely agree with each other (factoring out a minus sign), which may not the case. For example, consider the first and last rows of the example I provided: > > >>> print x_ma > [[ 7 -4 -1 -- -3 -2] > [ 6 -3 0 4 0 5] > [-4 -- 7 5 -- --] > [-- 5 -- 0 1 4]] > > >>> np.ma.corrcoef(x_ma)[0,3] > -1.6813456149534147 > > > On the other hand, if we supply only the first and third row to the function, we get: > > >>> np.ma.corrcoef(x_ma[0,:], x_ma[3,:]) > masked_array(data = > [[1.0 -0.240192230708] > [-0.240192230708 1.0]], > mask = > [[False False] > [False False]], > fill_value = 1e+20) > > Interestingly, this is the same as what pandas results as the [3,0] element of the correlation coefficient matrix, and it is equal to pair-wise deletion result: > > >>> np.corrcoef([-4, -3, -2], [5, 1, 4]) #Note that this is NOT np.ma.corrcoef > >>> > array([[ 1. , -0.24019223], > [-0.24019223, 1. ]]) > > > Also, I don't know why the ma.corrcoef results Josef has mentioned are different than mine. In particular, Josef reports element [2, 0] of the ma.corrcoef result to be -1.19, but I get -- (i.e., missing and masked, probably due to too few samples available). Josef: are you sure that you have entered the example values correctly into python? Along the same lines, the results that Nathaniel has posted from R are different, since the input is not a masked matrix I guess (please note that in the original example, I had masked values less than or equal to -5). Yes, sorry, this is just a cut and paste error - in fact the result I posted is what R gives for the stay with values <= -5 replaced by NA, but I left this line out of the email. I think the only difference is that R and pandas give a correlation of 1.0 when there are only 1 or 2 data points, and ma.corrcoef returns masked in this case. Not sure which makes more sense. > > In any case, I think the correlation coefficient between two rows of a matrix should not depend on what other rows are supplied. In other words, np.ma.corrcoef(x_ma)[0,3] should be equal to np.ma.corrcoef(x_ma[0,:], x_ma[3,:])[0,1] (which apparently happens to be what pandas reports). > > This change would need recomputing the mean for every pair-wise coefficient calculation, but since we are computing cross products O(n^2) times, the overall big-O complexity won't change. > > And please don't remove this functionality. I will volunteer to fix it however we decide :) We can just clarify the behavior in the documentation. In the long run I prefer R's behaviour of requiring the user to specify before skipping anything, but I tend to agree that in the short term pairwise deletion is what ma.corrcoef users expect and what we should do. Maybe you could implement the fix and we could move the discussion to the PR? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Sep 26 19:01:14 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 26 Sep 2013 19:01:14 -0400 Subject: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 6:42 PM, Nathaniel Smith wrote: > On 26 Sep 2013 21:59, "Faraz Mirzaei" wrote: >> >> Thanks Josef and Nathaniel for your responses. >> >> In the application that I have, I don't use the correlation coefficient >> matrix as a whole (so I don't care if it is PSD or not). I simply read the >> off-diagonal elements for pair-wise correlation coefficients. I use the >> pairwise correlation coefficient to test if the data from various sources >> (i.e., rows of the matrix), agree with each other when present. >> >> Right now, I use, ma.corrcoef( x[ i, :] , x[ j, :]) and read the >> off-diagonal element in a loop over i and j. It is just a bit uglier than >> calling ma.corrcoef(x). >> >> At least for my application, truncation to -1 or +1 (or scaling such that >> largest values becomes 1 etc) is completely wrong, since it would imply that >> the two sources completely agree with each other (factoring out a minus >> sign), which may not the case. For example, consider the first and last rows >> of the example I provided: >> >> >>> print x_ma >> [[ 7 -4 -1 -- -3 -2] >> [ 6 -3 0 4 0 5] >> [-4 -- 7 5 -- --] >> [-- 5 -- 0 1 4]] >> >> >>> np.ma.corrcoef(x_ma)[0,3] >> -1.6813456149534147 >> >> >> On the other hand, if we supply only the first and third row to the >> function, we get: >> >> >>> np.ma.corrcoef(x_ma[0,:], x_ma[3,:]) >> masked_array(data = >> [[1.0 -0.240192230708] >> [-0.240192230708 1.0]], >> mask = >> [[False False] >> [False False]], >> fill_value = 1e+20) >> >> Interestingly, this is the same as what pandas results as the [3,0] >> element of the correlation coefficient matrix, and it is equal to pair-wise >> deletion result: >> >> >>> np.corrcoef([-4, -3, -2], [5, 1, 4]) #Note that this is NOT >> >>> np.ma.corrcoef >> >>> >> array([[ 1. , -0.24019223], >> [-0.24019223, 1. ]]) >> >> >> Also, I don't know why the ma.corrcoef results Josef has mentioned are >> different than mine. In particular, Josef reports element [2, 0] of the >> ma.corrcoef result to be -1.19, but I get -- (i.e., missing and masked, >> probably due to too few samples available). Josef: are you sure that you >> have entered the example values correctly into python? Along the same lines, >> the results that Nathaniel has posted from R are different, since the input >> is not a masked matrix I guess (please note that in the original example, I >> had masked values less than or equal to -5). > > Yes, sorry, this is just a cut and paste error - in fact the result I posted > is what R gives for the stay with values <= -5 replaced by NA, but I left > this line out of the email. > > I think the only difference is that R and pandas give a correlation of 1.0 > when there are only 1 or 2 data points, and ma.corrcoef returns masked in > this case. Not sure which makes more sense. > >> >> In any case, I think the correlation coefficient between two rows of a >> matrix should not depend on what other rows are supplied. In other words, >> np.ma.corrcoef(x_ma)[0,3] should be equal to np.ma.corrcoef(x_ma[0,:], >> x_ma[3,:])[0,1] (which apparently happens to be what pandas reports). >> >> This change would need recomputing the mean for every pair-wise >> coefficient calculation, but since we are computing cross products O(n^2) >> times, the overall big-O complexity won't change. >> >> And please don't remove this functionality. I will volunteer to fix it >> however we decide :) We can just clarify the behavior in the documentation. > > In the long run I prefer R's behaviour of requiring the user to specify > before skipping anything, but I tend to agree that in the short term > pairwise deletion is what ma.corrcoef users expect and what we should do. > Maybe you could implement the fix and we could move the discussion to the > PR? pandas has a cython function in algos that loops over all pairs and calculates mean, cross product and standard deviation for each pair separately. I agree that that would be the best choice for pairwise deletion for np.ma.corrcoef, and cov Josef > > -n > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From davidmenhur at gmail.com Fri Sep 27 04:00:53 2013 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 27 Sep 2013 10:00:53 +0200 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On 26 September 2013 10:02, Da?id wrote: > The simplest way is to do it in cartesian coordinates: take x, y, and z > independently from N(0,1). If you want to generate only one normal number > per step, consider the jacobian in the angles. Actually, this is wrong, as it would allow displacements (at 1 sigma) of 1 along the axis, but up to sqrt(3) along diagonals. What you actually want is a multivariate normal distribution with covariance proportional to the identity (uncorrelation between axis and isotropy). See formulae here: http://en.wikipedia.org/wiki/Multivariate_normal_distribution -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Sep 27 04:27:47 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 27 Sep 2013 09:27:47 +0100 Subject: [Numpy-discussion] Valid algorithm for generating a 3D Wiener Process? In-Reply-To: References: Message-ID: On Fri, Sep 27, 2013 at 9:00 AM, Da?id wrote: > > > On 26 September 2013 10:02, Da?id wrote: >> >> The simplest way is to do it in cartesian coordinates: take x, y, and z independently from N(0,1). If you want to generate only one normal number per step, consider the jacobian in the angles. > > Actually, this is wrong, as it would allow displacements (at 1 sigma) of 1 along the axis, but up to sqrt(3) along diagonals. What you actually want is a multivariate normal distribution with covariance proportional to the identity (uncorrelation between axis and isotropy). No, you were right the first time. Sampling 3 independent N(0,1) variates is equivalent to an isotropic 3D multivariate normal. This is a special property of the normal distribution because of the behavior of exp(-x**2). The multivariate normal PDF can be decomposed into a product of univariate normals. exp(-(x**2 + y**2 + z**2)) = exp(-x**2) * exp(-y**2) * exp(-z**2) -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Sep 27 08:27:31 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 27 Sep 2013 14:27:31 +0200 Subject: [Numpy-discussion] Indexing changes/deprecations Message-ID: <1380284851.16880.27.camel@sebastian-laptop> Hey, since I am working on the indexing. I was wondering about a few smaller things: * 0-d boolean array, `np.array(0)[True]` (will work now) would give np.array([0]) as a copy, instead of the original array. I guess I could add a FutureWarning or so, but I am not sure and overall the chance of creating bugs seems low. (The boolean index should always add 1 dimension and here, remove 0 dimensions -> 1-d result.) * All index operations return a view; never the object. This means that `v = arr[...]` is slightly slower. But since it does not affect `arr[...] = vals`, I think the speed implications are negligible. * Does anyone have an idea if there is a way to change the subclass logic that view based item setting is implemented as: np.asarray(subclass[index]) = vals I somewhat think the subclass should rather implement `__setitem__` instead of relying on numpy calling its `__getitem__`, but I don't see how it can be changed. * Still thinking a bit about implementing a keepdims keyword or function, to handle matrix type logic mostly in the C-code. And most importantly, is there any behaviour thing in the index machinery that is bugging you, which I may have forgotten until now? - Sebastian From ben.root at ou.edu Fri Sep 27 09:26:56 2013 From: ben.root at ou.edu (Benjamin Root) Date: Fri, 27 Sep 2013 09:26:56 -0400 Subject: [Numpy-discussion] Indexing changes/deprecations In-Reply-To: <1380284851.16880.27.camel@sebastian-laptop> References: <1380284851.16880.27.camel@sebastian-laptop> Message-ID: On Fri, Sep 27, 2013 at 8:27 AM, Sebastian Berg wrote: > Hey, > > since I am working on the indexing. I was wondering about a few smaller > things: > > * 0-d boolean array, `np.array(0)[True]` (will work now) would > give np.array([0]) as a copy, instead of the original array. > I guess I could add a FutureWarning or so, but I am not sure > and overall the chance of creating bugs seems low. > > (The boolean index should always add 1 dimension and here, > remove 0 dimensions -> 1-d result.) > > * All index operations return a view; never the object. This > means that `v = arr[...]` is slightly slower. But since it > does not affect `arr[...] = vals`, I think the speed > implications are negligible. > > * Does anyone have an idea if there is a way to change the subclass > logic that view based item setting is implemented as: > np.asarray(subclass[index]) = vals > > I somewhat think the subclass should rather implement `__setitem__` > instead of relying on numpy calling its `__getitem__`, but I > don't see how it can be changed. > > * Still thinking a bit about implementing a keepdims keyword or > function, to handle matrix type logic mostly in the C-code. > > And most importantly, is there any behaviour thing in the index > machinery that is bugging you, which I may have forgotten until now? > > - Sebastian > > Boolean indexing could use a facelift. First, consider the following (albeit minor) annoyance: >>> import numpy as np >>> a = np.arange(5) >>> a[[True, False, True, False, True]] array([1, 0, 1, 0, 1]) >>> b = np.array([True, False, True, False, True]) >>> a[b] array([0, 2, 4]) Next, it would be nice if boolean indexing returned a view (wishful thinking, I know): >>> c = a[b] >>> c array([0, 2, 4]) >>> c[1] = 7 >>> c array([0, 7, 4]) >>> a array([0, 1, 2, 3, 4]) Cheers! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Sep 27 09:35:10 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 27 Sep 2013 15:35:10 +0200 Subject: [Numpy-discussion] Indexing changes/deprecations In-Reply-To: References: <1380284851.16880.27.camel@sebastian-laptop> Message-ID: <1380288910.6733.4.camel@sebastian-laptop> On Fri, 2013-09-27 at 09:26 -0400, Benjamin Root wrote: > > Boolean indexing could use a facelift. First, consider the following > (albeit minor) annoyance: > Done. Well will be deprecation warnings for the time being, though. > > Next, it would be nice if boolean indexing returned a view (wishful > thinking, I know): > Yeah, that is impossible unless you create some intermediate non-array object, so that is out of the scope of things for now I think. > >>> c = a[b] > >>> c > array([0, 2, 4]) > >>> c[1] = 7 > >>> c > array([0, 7, 4]) > >>> a > array([0, 1, 2, 3, 4]) > > > Cheers! > Ben Root > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jaime.frio at gmail.com Fri Sep 27 11:45:57 2013 From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=) Date: Fri, 27 Sep 2013 08:45:57 -0700 Subject: [Numpy-discussion] Indexing changes/deprecations In-Reply-To: <1380284851.16880.27.camel@sebastian-laptop> References: <1380284851.16880.27.camel@sebastian-laptop> Message-ID: On Fri, Sep 27, 2013 at 5:27 AM, Sebastian Berg wrote: > > And most importantly, is there any behaviour thing in the index > machinery that is bugging you, which I may have forgotten until now? > I find this behavior of boolean indexing a little bit annoying: >>> a = np.arange(12).reshape(3, 4) >>> a array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) >>> row_idx = np.array([True, True, False]) >>> col_idx = np.array([False, True, True, False]) This shouldn't work, but it does, because there are the same number of Trues in both indexing arrays. Do we really want this to happen?: >>> a[row_idx, col_idx] array([1, 6]) This shouldn't work, and it doesn't: >>> col_idx = np.array([False, True, True, True]) >>> a[row_idx, col_idx] Traceback (most recent call last): File "", line 1, in ValueError: shape mismatch: objects cannot be broadcast to a single shape It would be nice if something like this worked, or at least it should raise a different error, because those arrays **can** be broadcast to a single shape: >>> a[row_idx[:, np.newaxis], col_idx] Traceback (most recent call last): File "", line 1, in ValueError: shape mismatch: objects cannot be broadcast to a single shape For this there is the following workaround, although it does creation of a fully expanded boolean indexing array, which I was hoping the previous non-working code would avoid: >>> a[row_idx[:, np.newaxis] & col_idx] array([1, 2, 3, 5, 6, 7]) Jaime -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Sep 27 12:36:44 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 27 Sep 2013 18:36:44 +0200 Subject: [Numpy-discussion] Indexing changes/deprecations In-Reply-To: References: <1380284851.16880.27.camel@sebastian-laptop> Message-ID: <1380299804.6733.52.camel@sebastian-laptop> On Fri, 2013-09-27 at 08:45 -0700, Jaime Fern?ndez del R?o wrote: > > On Fri, Sep 27, 2013 at 5:27 AM, Sebastian Berg > wrote: > > And most importantly, is there any behaviour thing in the > index > machinery that is bugging you, which I may have forgotten > until now? > > > I find this behavior of boolean indexing a little bit annoying: > > > >>> a = np.arange(12).reshape(3, 4) > >>> a > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]) > >>> row_idx = np.array([True, True, False]) > > >>> col_idx = np.array([False, True, True, False]) > > > This shouldn't work, but it does, because there are the same number of > Trues in both indexing arrays. Do we really want this to happen?: > >>> a[row_idx, col_idx] > array([1, 6]) > I agree that this can be confusing, but I think the fancy indexing logic (plus the "boolean is much like a nonzero(boolean) call") dictates this. One could think about doing something here, but it would basically require a secondary indexing mechanism like `arr.no_broadcast_fancy[index]`. `np.ix_` currently implements a conversion logic for this, though it cannot support slices. > > This shouldn't work, and it doesn't: > > >>> col_idx = np.array([False, True, True, True]) > > >>> a[row_idx, col_idx] > Traceback (most recent call last): > File "", line 1, in > ValueError: shape mismatch: objects cannot be broadcast to a single > shape In [11]: a[np.ix_(row_idx, col_idx)] Out[11]: array([[1, 2, 3], [5, 6, 7]]) > However, there is one further point here that I think is likely worth changing. And that is adding a check that the boolean array has the correct shape. The `nonzero` logic works good, but it allows things like: np.array([1, 2])[np.array([True, False, False, False])] Which is a bit weird, though maybe not harmful. - Sebastian > Jaime > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From rhattersley at gmail.com Fri Sep 27 14:14:46 2013 From: rhattersley at gmail.com (Richard Hattersley) Date: Fri, 27 Sep 2013 19:14:46 +0100 Subject: [Numpy-discussion] Indexing changes/deprecations In-Reply-To: <1380284851.16880.27.camel@sebastian-laptop> References: <1380284851.16880.27.camel@sebastian-laptop> Message-ID: On 27 September 2013 13:27, Sebastian Berg wrote: > And most importantly, is there any behaviour thing in the index > machinery that is bugging you, which I may have forgotten until now? > Well, since you asked... I'd *love* to see the fancy indexing behaviour moved to a separate method(s). Yes, I know! I'm not realistically expecting that to be tackled right now. And it sometimes seems like something of a sacred idol that one is not supposed to question. But I've kept quiet on the issue for too long and would love to know if anyone else thinks the same. It confuses people. Actually, it confuses the hell out of people. I'm *still* finding out new quirks of its behaviour and I've been using NumPy in a professional role for years... although you should bear in mind I could just be a slow learner. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Sep 27 14:34:18 2013 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Sep 2013 21:34:18 +0300 Subject: [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence" In-Reply-To: References: Message-ID: 27.09.2013 19:33, Nathaniel Smith kirjoitti: [clip] > I really don't understand what arcane magic is used to make ndarray += > csc_matrix work at all, but my question is, is it going to break when > we complete the casting transition described above? It was just > supposed to catch things like int += float. This maybe clarifies it: >>> import numpy >>> import scipy.sparse >>> x = numpy.ones((2,2)) >>> y = scipy.sparse.csr_matrix(x) >>> z = x >>> z += y >>> x array([[ 1., 1.], [ 1., 1.]]) >>> z matrix([[ 2., 2.], [ 2., 2.]]) The execution flows like this: ndarray.__iadd__(arr, sparr) np.add(arr, sparr, out=???) return NotImplemented # wtf return NotImplemented Python does arr = sparr.__radd__(arr) Since Scipy master sparse matrices now have __numpy_ufunc__, but it doesn't handle out= arguments, the second step currently raises a TypeError (for Numpy master + Scipy master). And this is actually the correct thing to do, as having np.add return NotImplemented is just broken. Only ndarray.__iadd__ has the authority to return the NotImplemented. To make the in-place ops work again, it seems Numpy needs some additional fixes in its binary op machinery, before __numpy_ufunc__ business works fully as intended. Namely, the binary op routines will need to catch TypeErrors and convert them to NotImplemented. The code paths where Numpy ufuncs currently return NotImplemented could also be changed to raise TypeErrors, but I'm not sure if someone somewhere relies on this behavior (I hope not). -- Pauli Virtanen From njs at pobox.com Fri Sep 27 15:15:37 2013 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 27 Sep 2013 20:15:37 +0100 Subject: [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence" In-Reply-To: References: Message-ID: On Fri, Sep 27, 2013 at 7:34 PM, Pauli Virtanen wrote: > 27.09.2013 19:33, Nathaniel Smith kirjoitti: > [clip] >> I really don't understand what arcane magic is used to make ndarray += >> csc_matrix work at all, but my question is, is it going to break when >> we complete the casting transition described above? It was just >> supposed to catch things like int += float. > > This maybe clarifies it: > >>>> import numpy >>>> import scipy.sparse >>>> x = numpy.ones((2,2)) >>>> y = scipy.sparse.csr_matrix(x) >>>> z = x >>>> z += y >>>> x > array([[ 1., 1.], > [ 1., 1.]]) >>>> z > matrix([[ 2., 2.], > [ 2., 2.]]) > > The execution flows like this: > > ndarray.__iadd__(arr, sparr) > np.add(arr, sparr, out=???) > return NotImplemented # wtf > return NotImplemented > Python does arr = sparr.__radd__(arr) > > Since Scipy master sparse matrices now have __numpy_ufunc__, but it > doesn't handle out= arguments, the second step currently raises a > TypeError (for Numpy master + Scipy master). > > And this is actually the correct thing to do, as having np.add return > NotImplemented is just broken. Only ndarray.__iadd__ has the authority > to return the NotImplemented. > > To make the in-place ops work again, it seems Numpy needs some > additional fixes in its binary op machinery, before __numpy_ufunc__ > business works fully as intended. Namely, the binary op routines will > need to catch TypeErrors and convert them to NotImplemented. > > The code paths where Numpy ufuncs currently return NotImplemented could > also be changed to raise TypeErrors, but I'm not sure if someone > somewhere relies on this behavior (I hope not). Okay, so I see three separate issues: 1) My original concern, that the upcoming casting change for in-place operations will cause some horrible interaction. Tentatively this seems like it might be okay since even after the "cast" succeeds, np.add is still just refusing to do the operation, so hopefully we can set it up so that it will continue to fail once the casting rule becomes more strict. 2) The issue that ufuncs return NotImplemented and it makes baby Guido cry. This is completely broken, agreed. Not sure when someone will get around to clearing this stuff up. 3) The issue of how to make an in-place like ndarray += sparse continue to work in the brave new __numpy_ufunc__ world. For this last issue, I think we disagree. It seems to me that the right answer is that csc_matrix.__numpy_ufunc__ needs to step up and start supporting out=! If I have a large dense ndarray and I try to += a sparse array to it, this operation should take no temporary memory and nnz time. Right now it sounds like it actually copies the large dense ndarray, which takes time and space proportional to its size. AFAICT the only way to avoid that is for scipy.sparse to implement out=. It shouldn't be that hard...? -n From pav at iki.fi Fri Sep 27 15:27:03 2013 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 Sep 2013 22:27:03 +0300 Subject: [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence" In-Reply-To: References: Message-ID: 27.09.2013 22:15, Nathaniel Smith kirjoitti: [clip] > 3) The issue of how to make an in-place like ndarray += sparse > continue to work in the brave new __numpy_ufunc__ world. > > For this last issue, I think we disagree. It seems to me that the > right answer is that csc_matrix.__numpy_ufunc__ needs to step up and > start supporting out=! If I have a large dense ndarray and I try to += > a sparse array to it, this operation should take no temporary memory > and nnz time. Right now it sounds like it actually copies the large > dense ndarray, which takes time and space proportional to its size. > AFAICT the only way to avoid that is for scipy.sparse to implement > out=. It shouldn't be that hard...? Sure, scipy.sparse can easily support also the output argument. But I still maintain that the implementation of __iadd__ in Numpy is wrong. What it does now is: def __iadd__(self, other): return np.add(self, other, out=self) But since we know this raises a TypeError if the input is of a type that cannot be dealt with, it should be: def __iadd__(self, other): try: return np.add(self, other, out=self) except TypeError: return NotImplemented Of course, it's written in C so it's a bit more painful to write this. I think this will have little performance impact, since the check would be only a NULL check in the inplace op methods + subsequent handling. I can take a look at some point at this... -- Pauli Virtanen From njs at pobox.com Fri Sep 27 16:33:24 2013 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 27 Sep 2013 21:33:24 +0100 Subject: [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence" In-Reply-To: References: Message-ID: On Fri, Sep 27, 2013 at 8:27 PM, Pauli Virtanen wrote: > 27.09.2013 22:15, Nathaniel Smith kirjoitti: > [clip] >> 3) The issue of how to make an in-place like ndarray += sparse >> continue to work in the brave new __numpy_ufunc__ world. >> >> For this last issue, I think we disagree. It seems to me that the >> right answer is that csc_matrix.__numpy_ufunc__ needs to step up and >> start supporting out=! If I have a large dense ndarray and I try to += >> a sparse array to it, this operation should take no temporary memory >> and nnz time. Right now it sounds like it actually copies the large >> dense ndarray, which takes time and space proportional to its size. >> AFAICT the only way to avoid that is for scipy.sparse to implement >> out=. It shouldn't be that hard...? > > Sure, scipy.sparse can easily support also the output argument. Great! I guess solving this somehow will be release-critical, to avoid a regression in this case when __numpy_ufunc__ gets released. If the solution should be in scipy, I guess we should file the bug there? > But I still maintain that the implementation of __iadd__ in Numpy is > wrong. Oh yeah totally. > What it does now is: > > def __iadd__(self, other): > return np.add(self, other, out=self) > > But since we know this raises a TypeError if the input is of a type that > cannot be dealt with, it should be: > > def __iadd__(self, other): > try: > return np.add(self, other, out=self) > except TypeError: > return NotImplemented > > Of course, it's written in C so it's a bit more painful to write this. > > I think this will have little performance impact, since the check would > be only a NULL check in the inplace op methods + subsequent handling. I > can take a look at some point at this... I'm a little uncertain about the "swallow all TypeErrors" aspect of this -- e.g. this could have really weird effects for object arrays, where ufuncs may raise arbitrary user exceptions. One possibility in the long run is to just say, if you want to override ndarray __iadd__ or whatever, then you have to use __numpy_ufunc__. Not really much point in having *two* sets of implementations of the NotImplemented dance for the same operation. -n From charlesr.harris at gmail.com Mon Sep 30 11:17:14 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 09:17:14 -0600 Subject: [Numpy-discussion] 1.8.0rc1 Message-ID: Hi All, NumPy 1.8.0rc1 is up now on sourceforge.The binary builds are included except for Python 3.3 on windows, which will arrive later. Many thanks to Ralf for the binaries, and to those who found and fixed the bugs in the last beta. Any remaining bugs are all my fault ;) I hope this will be the last release before final, so please test it thoroughly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Sep 30 12:21:26 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 30 Sep 2013 18:21:26 +0200 Subject: [Numpy-discussion] 1.8.0rc1 In-Reply-To: References: Message-ID: <5249A506.6060200@googlemail.com> On 30.09.2013 17:17, Charles R Harris wrote: > Hi All, > > NumPy 1.8.0rc1 is up now on sourceforge > .The > binary builds are included except for Python 3.3 on windows, which will > arrive later. Many thanks to Ralf for the binaries, and to those who > found and fixed the bugs in the last beta. Any remaining bugs are all my > fault ;) I hope this will be the last release before final, so please > test it thoroughly. > Can we also do a rc for 1.7.2? The number of fixes that went in is not small and they should be tested by some users before release. From charlesr.harris at gmail.com Mon Sep 30 12:28:55 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 10:28:55 -0600 Subject: [Numpy-discussion] 1.8.0rc1 In-Reply-To: <5249A506.6060200@googlemail.com> References: <5249A506.6060200@googlemail.com> Message-ID: On Mon, Sep 30, 2013 at 10:21 AM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 30.09.2013 17:17, Charles R Harris wrote: > > Hi All, > > > > NumPy 1.8.0rc1 is up now on sourceforge > > .The > > binary builds are included except for Python 3.3 on windows, which will > > arrive later. Many thanks to Ralf for the binaries, and to those who > > found and fixed the bugs in the last beta. Any remaining bugs are all my > > fault ;) I hope this will be the last release before final, so please > > test it thoroughly. > > > > Can we also do a rc for 1.7.2? > The number of fixes that went in is not small and they should be tested > by some users before release. > I was planning on that right after 1.8 is done. I still need to backport your little cpp fixes, the changes since 1.7 make that a bit tricky. Then there is checking for other missing backports... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 30 14:02:38 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 30 Sep 2013 19:02:38 +0100 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: Message-ID: Everyone please do actually test this! It is really in your best interest, and I think people don't always realize this. Here's how it works: - If you test it *now*, and it breaks your code that worked with 1.7, and you *tell* us this now, then it's *our* problem and we hold up the release to fix the bug. - If you test it *after* we release, and it breaks your code, then we are sad but you have to work around it (because we can't magically make that release not have happened, your users will be using it anyway), and we put it on the stack with all the other bugs. All of which we care about but it's a large enough stack that it's not going to get any special priority, because, see above about how at this point you'll have had to work around it anyway. -n On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris wrote: > Hi All, > > NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are included > except for Python 3.3 on windows, which will arrive later. Many thanks to > Ralf for the binaries, and to those who found and fixed the bugs in the last > beta. Any remaining bugs are all my fault ;) I hope this will be the last > release before final, so please test it thoroughly. > > Chuck > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From tmp50 at ukr.net Mon Sep 30 16:32:41 2013 From: tmp50 at ukr.net (Dmitrey) Date: Mon, 30 Sep 2013 23:32:41 +0300 Subject: [Numpy-discussion] [ANN] MATLAB fmincon now available in Python2 Message-ID: <1380572275.326208946.d9mh0o59@frv43.ukr.net> Hi all, current state of Python <-> MATLAB connection soft doesn't allow passing of function handlers, however, a walkaround has been implemented via some tricks, so now MATLAB function fmincon is available in Python-written OpenOpt and FuncDesigner frameworks (with possibility of automatic differentiation,? example ). Future plans? include MATLAB fsolve, ode23, ode45 (unlike scipy fsolve and ode they can handle sparse matrices), fgoalattain, maybe global optimization toolbox solvers. I intend to post the message to several forums, so to keep discussion in a single place use OpenOpt forum thread http://forum.openopt.org/viewtopic.php?id=769 Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Mon Sep 30 17:51:39 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 30 Sep 2013 14:51:39 -0700 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: Message-ID: <5249F26B.2040109@uci.edu> On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do actually test this! It is really in your best > interest, and I think people don't always realize this. > > Here's how it works: > - If you test it *now*, and it breaks your code that worked with 1.7, > and you *tell* us this now, then it's *our* problem and we hold up the > release to fix the bug. > - If you test it *after* we release, and it breaks your code, then we > are sad but you have to work around it (because we can't magically > make that release not have happened, your users will be using it > anyway), and we put it on the stack with all the other bugs. All of > which we care about but it's a large enough stack that it's not going > to get any special priority, because, see above about how at this > point you'll have had to work around it anyway. > > -n > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > wrote: >> Hi All, >> >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are included >> except for Python 3.3 on windows, which will arrive later. Many thanks to >> Ralf for the binaries, and to those who found and fixed the bugs in the last >> beta. Any remaining bugs are all my fault ;) I hope this will be the last >> release before final, so please test it thoroughly. >> >> Chuck Hello, NumPy 1.8.0rc1 looks good. All tests pass on Windows and most 3rd party packages test OK now. Thank you. A few tests still fail in the following packages when run with numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to numpy-MKL-1.7.1-win-amd64-py3.3: 1) Pandas 0.12.0 ``` ====================================================================== FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\pandas\tests\test_series.py", line 254, in test_nansum_buglet assert_almost_equal(result, 1) File "X:\Python33\lib\site-packages\pandas\util\testing.py", line 134, in assert_almost_equal np.testing.assert_(isiterable(b)) File "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line 44, in assert_ raise AssertionError(msg) AssertionError ``` Possibly related: ``` >>> import numpy as np >>> from pandas import Series >>> s = Series([0.0]) >>> result = np.nansum(s) >>> print(result) Traceback (most recent call last): File "", line 1, in File "X:\Python33\lib\site-packages\pandas\core\base.py", line 19, in __str__ return self.__unicode__() File "X:\Python33\lib\site-packages\pandas\core\series.py", line 1115, in __unicode__ length=len(self) > 50, TypeError: len() of unsized object ``` 2) Bottleneck 0.7.0 https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 3) skimage 0.8.2 These tests passed with numpy 1.8.0b2: ``` ====================================================================== FAIL: test_grey.test_non_square_image ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest self.test(*self.arg) File "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", line 162, in test_non_square_image testing.assert_array_equal(binary_res, grey_res) File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 718, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 50.6328125%) x: array([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False],... y: array([[ True, True, True, ..., True, False, False], [ True, True, True, ..., False, False, False], [ True, True, True, ..., False, False, False],... ====================================================================== FAIL: test_grey.test_binary_erosion ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest self.test(*self.arg) File "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", line 169, in test_binary_erosion testing.assert_array_equal(binary_res, grey_res) File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 718, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 48.260498046875%) x: array([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False],... y: array([[ True, True, True, ..., True, False, False], [ True, True, True, ..., False, False, False], [ True, True, True, ..., False, False, False],... ====================================================================== FAIL: test_grey.test_binary_closing ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest self.test(*self.arg) File "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", line 183, in test_binary_closing testing.assert_array_equal(binary_res, grey_res) File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 718, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 66.302490234375%) x: array([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False],... y: array([[ True, True, True, ..., True, True, True], [ True, True, True, ..., True, True, True], [ True, True, True, ..., False, False, False],... ====================================================================== FAIL: test_grey.test_binary_opening ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest self.test(*self.arg) File "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", line 190, in test_binary_opening testing.assert_array_equal(binary_res, grey_res) File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 718, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line 644, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 58.465576171875%) x: array([[False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False], [False, False, False, ..., False, False, False],... y: array([[ True, True, True, ..., True, True, False], [ True, True, True, ..., True, True, False], [ True, True, True, ..., False, False, False],... ``` Christoph From charlesr.harris at gmail.com Mon Sep 30 18:45:47 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 16:45:47 -0600 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: <5249F26B.2040109@uci.edu> References: <5249F26B.2040109@uci.edu> Message-ID: On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke wrote: > > > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do > actually test this! It is really in your best > > interest, and I think people don't always realize this. > > > > Here's how it works: > > - If you test it *now*, and it breaks your code that worked with 1.7, > > and you *tell* us this now, then it's *our* problem and we hold up the > > release to fix the bug. > > - If you test it *after* we release, and it breaks your code, then we > > are sad but you have to work around it (because we can't magically > > make that release not have happened, your users will be using it > > anyway), and we put it on the stack with all the other bugs. All of > > which we care about but it's a large enough stack that it's not going > > to get any special priority, because, see above about how at this > > point you'll have had to work around it anyway. > > > > -n > > > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > > wrote: > >> Hi All, > >> > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are included > >> except for Python 3.3 on windows, which will arrive later. Many thanks > to > >> Ralf for the binaries, and to those who found and fixed the bugs in the > last > >> beta. Any remaining bugs are all my fault ;) I hope this will be the > last > >> release before final, so please test it thoroughly. > >> > >> Chuck > > > Hello, > > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most 3rd party > packages test OK now. Thank you. > > A few tests still fail in the following packages when run with > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to > numpy-MKL-1.7.1-win-amd64-py3.3: > > 1) Pandas 0.12.0 > > ``` > ====================================================================== > FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\pandas\tests\test_series.py", > line 254, in test_nansum_buglet > assert_almost_equal(result, 1) > File "X:\Python33\lib\site-packages\pandas\util\testing.py", line > 134, in assert_almost_equal > np.testing.assert_(isiterable(b)) > File "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line > 44, in assert_ > raise AssertionError(msg) > AssertionError > ``` > > Possibly related: > > ``` > >>> import numpy as np > >>> from pandas import Series > >>> s = Series([0.0]) > >>> result = np.nansum(s) > >>> print(result) > Traceback (most recent call last): > File "", line 1, in > File "X:\Python33\lib\site-packages\pandas\core\base.py", line 19, in > __str__ > return self.__unicode__() > File "X:\Python33\lib\site-packages\pandas\core\series.py", line > 1115, in __unicode__ > length=len(self) > 50, > TypeError: len() of unsized object > ``` > > 2) Bottleneck 0.7.0 > > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 > > 3) skimage 0.8.2 > > These tests passed with numpy 1.8.0b2: > > ``` > ====================================================================== > FAIL: test_grey.test_non_square_image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 162, in test_non_square_image > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 50.6328125%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, False, False], > [ True, True, True, ..., False, False, False], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_erosion > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 169, in test_binary_erosion > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 48.260498046875%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, False, False], > [ True, True, True, ..., False, False, False], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_closing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 183, in test_binary_closing > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 66.302490234375%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, True, True], > [ True, True, True, ..., True, True, True], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_opening > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 190, in test_binary_opening > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 58.465576171875%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, True, False], > [ True, True, True, ..., True, True, False], > [ True, True, True, ..., False, False, False],... > ``` > I'll bet the skimage problems come from https://github.com/numpy/numpy/pull/3811. They may be doing something naughty... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 30 19:07:06 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 17:07:06 -0600 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: <5249F26B.2040109@uci.edu> Message-ID: On Mon, Sep 30, 2013 at 4:45 PM, Charles R Harris wrote: > > > > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke wrote: > >> >> >> On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do >> actually test this! It is really in your best >> > interest, and I think people don't always realize this. >> > >> > Here's how it works: >> > - If you test it *now*, and it breaks your code that worked with 1.7, >> > and you *tell* us this now, then it's *our* problem and we hold up the >> > release to fix the bug. >> > - If you test it *after* we release, and it breaks your code, then we >> > are sad but you have to work around it (because we can't magically >> > make that release not have happened, your users will be using it >> > anyway), and we put it on the stack with all the other bugs. All of >> > which we care about but it's a large enough stack that it's not going >> > to get any special priority, because, see above about how at this >> > point you'll have had to work around it anyway. >> > >> > -n >> > >> > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris >> > wrote: >> >> Hi All, >> >> >> >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are included >> >> except for Python 3.3 on windows, which will arrive later. Many thanks >> to >> >> Ralf for the binaries, and to those who found and fixed the bugs in >> the last >> >> beta. Any remaining bugs are all my fault ;) I hope this will be the >> last >> >> release before final, so please test it thoroughly. >> >> >> >> Chuck >> >> >> Hello, >> >> NumPy 1.8.0rc1 looks good. All tests pass on Windows and most 3rd party >> packages test OK now. Thank you. >> >> A few tests still fail in the following packages when run with >> numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to >> numpy-MKL-1.7.1-win-amd64-py3.3: >> >> 1) Pandas 0.12.0 >> >> ``` >> ====================================================================== >> FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "X:\Python33\lib\site-packages\pandas\tests\test_series.py", >> line 254, in test_nansum_buglet >> assert_almost_equal(result, 1) >> File "X:\Python33\lib\site-packages\pandas\util\testing.py", line >> 134, in assert_almost_equal >> np.testing.assert_(isiterable(b)) >> File "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line >> 44, in assert_ >> raise AssertionError(msg) >> AssertionError >> ``` >> >> Possibly related: >> >> ``` >> >>> import numpy as np >> >>> from pandas import Series >> >>> s = Series([0.0]) >> >>> result = np.nansum(s) >> >>> print(result) >> Traceback (most recent call last): >> File "", line 1, in >> File "X:\Python33\lib\site-packages\pandas\core\base.py", line 19, in >> __str__ >> return self.__unicode__() >> File "X:\Python33\lib\site-packages\pandas\core\series.py", line >> 1115, in __unicode__ >> length=len(self) > 50, >> TypeError: len() of unsized object >> ``` >> >> 2) Bottleneck 0.7.0 >> >> https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 >> >> 3) skimage 0.8.2 >> >> These tests passed with numpy 1.8.0b2: >> >> ``` >> ====================================================================== >> FAIL: test_grey.test_non_square_image >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest >> self.test(*self.arg) >> File >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> line 162, in test_non_square_image >> testing.assert_array_equal(binary_res, grey_res) >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 718, in assert_array_equal >> verbose=verbose, header='Arrays are not equal') >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 644, in assert_array_compare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not equal >> >> (mismatch 50.6328125%) >> x: array([[False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False],... >> y: array([[ True, True, True, ..., True, False, False], >> [ True, True, True, ..., False, False, False], >> [ True, True, True, ..., False, False, False],... >> >> ====================================================================== >> FAIL: test_grey.test_binary_erosion >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest >> self.test(*self.arg) >> File >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> line 169, in test_binary_erosion >> testing.assert_array_equal(binary_res, grey_res) >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 718, in assert_array_equal >> verbose=verbose, header='Arrays are not equal') >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 644, in assert_array_compare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not equal >> >> (mismatch 48.260498046875%) >> x: array([[False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False],... >> y: array([[ True, True, True, ..., True, False, False], >> [ True, True, True, ..., False, False, False], >> [ True, True, True, ..., False, False, False],... >> >> ====================================================================== >> FAIL: test_grey.test_binary_closing >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest >> self.test(*self.arg) >> File >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> line 183, in test_binary_closing >> testing.assert_array_equal(binary_res, grey_res) >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 718, in assert_array_equal >> verbose=verbose, header='Arrays are not equal') >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 644, in assert_array_compare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not equal >> >> (mismatch 66.302490234375%) >> x: array([[False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False],... >> y: array([[ True, True, True, ..., True, True, True], >> [ True, True, True, ..., True, True, True], >> [ True, True, True, ..., False, False, False],... >> >> ====================================================================== >> FAIL: test_grey.test_binary_opening >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "X:\Python33\lib\site-packages\nose\case.py", line 198, in runTest >> self.test(*self.arg) >> File >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> line 190, in test_binary_opening >> testing.assert_array_equal(binary_res, grey_res) >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 718, in assert_array_equal >> verbose=verbose, header='Arrays are not equal') >> File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> 644, in assert_array_compare >> raise AssertionError(msg) >> AssertionError: >> Arrays are not equal >> >> (mismatch 58.465576171875%) >> x: array([[False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False], >> [False, False, False, ..., False, False, False],... >> y: array([[ True, True, True, ..., True, True, False], >> [ True, True, True, ..., True, True, False], >> [ True, True, True, ..., False, False, False],... >> ``` >> > > I'll bet the skimage problems come from > https://github.com/numpy/numpy/pull/3811. They may be doing something > naughty... > > Reverting that commit fixes those skimage failures. However, there are a number of python2.7 failures that look pretty strange. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Mon Sep 30 19:12:33 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 30 Sep 2013 16:12:33 -0700 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: <5249F26B.2040109@uci.edu> Message-ID: <524A0561.4000808@uci.edu> On 9/30/2013 3:45 PM, Charles R Harris wrote: > > > > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke > wrote: > > > > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do > actually test this! It is really in your best > > interest, and I think people don't always realize this. > > > > Here's how it works: > > - If you test it *now*, and it breaks your code that worked with 1.7, > > and you *tell* us this now, then it's *our* problem and we hold up the > > release to fix the bug. > > - If you test it *after* we release, and it breaks your code, then we > > are sad but you have to work around it (because we can't magically > > make that release not have happened, your users will be using it > > anyway), and we put it on the stack with all the other bugs. All of > > which we care about but it's a large enough stack that it's not going > > to get any special priority, because, see above about how at this > > point you'll have had to work around it anyway. > > > > -n > > > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > > > wrote: > >> Hi All, > >> > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are included > >> except for Python 3.3 on windows, which will arrive later. Many thanks to > >> Ralf for the binaries, and to those who found and fixed the bugs in the last > >> beta. Any remaining bugs are all my fault ;) I hope this will be the last > >> release before final, so please test it thoroughly. > >> > >> Chuck > > > Hello, > > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most 3rd party > packages test OK now. Thank you. > > A few tests still fail in the following packages when run with > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to > numpy-MKL-1.7.1-win-amd64-py3.3: > > 1) Pandas 0.12.0 > > ``` > ====================================================================== > FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\pandas\tests\test_series.py", > line 254, in test_nansum_buglet > assert_almost_equal(result, 1) > File "X:\Python33\lib\site-packages\pandas\util\testing.py", line > 134, in assert_almost_equal > np.testing.assert_(isiterable(b)) > File "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line > 44, in assert_ > raise AssertionError(msg) > AssertionError > ``` > > Possibly related: > > ``` > >>> import numpy as np > >>> from pandas import Series > >>> s = Series([0.0]) > >>> result = np.nansum(s) > >>> print(result) > Traceback (most recent call last): > File "", line 1, in > File "X:\Python33\lib\site-packages\pandas\core\base.py", line > 19, in > __str__ > return self.__unicode__() > File "X:\Python33\lib\site-packages\pandas\core\series.py", line > 1115, in __unicode__ > length=len(self) > 50, > TypeError: len() of unsized object > ``` > > 2) Bottleneck 0.7.0 > > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 > > 3) skimage 0.8.2 > > These tests passed with numpy 1.8.0b2: > > ``` > ====================================================================== > FAIL: test_grey.test_non_square_image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 162, in test_non_square_image > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 50.6328125%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, False, False], > [ True, True, True, ..., False, False, False], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_erosion > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 169, in test_binary_erosion > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 48.260498046875%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, False, False], > [ True, True, True, ..., False, False, False], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_closing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 183, in test_binary_closing > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 66.302490234375%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, True, True], > [ True, True, True, ..., True, True, True], > [ True, True, True, ..., False, False, False],... > > ====================================================================== > FAIL: test_grey.test_binary_opening > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > runTest > self.test(*self.arg) > File > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > line 190, in test_binary_opening > testing.assert_array_equal(binary_res, grey_res) > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 718, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > 644, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 58.465576171875%) > x: array([[False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False], > [False, False, False, ..., False, False, False],... > y: array([[ True, True, True, ..., True, True, False], > [ True, True, True, ..., True, True, False], > [ True, True, True, ..., False, False, False],... > ``` > > I'll bet the skimage problems come from > https://github.com/numpy/numpy/pull/3811. They may be doing something > naughty... > > Chuck > A bool image is convolved with a uint8 kernel and the result compared for equality with an uint32 scalar... https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 Christoph From charlesr.harris at gmail.com Mon Sep 30 19:30:58 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 17:30:58 -0600 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: <524A0561.4000808@uci.edu> References: <5249F26B.2040109@uci.edu> <524A0561.4000808@uci.edu> Message-ID: On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke wrote: > On 9/30/2013 3:45 PM, Charles R Harris wrote: > > > > > > > > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke > > wrote: > > > > > > > > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do > > actually test this! It is really in your best > > > interest, and I think people don't always realize this. > > > > > > Here's how it works: > > > - If you test it *now*, and it breaks your code that worked with > 1.7, > > > and you *tell* us this now, then it's *our* problem and we hold up > the > > > release to fix the bug. > > > - If you test it *after* we release, and it breaks your code, then > we > > > are sad but you have to work around it (because we can't magically > > > make that release not have happened, your users will be using it > > > anyway), and we put it on the stack with all the other bugs. All of > > > which we care about but it's a large enough stack that it's not > going > > > to get any special priority, because, see above about how at this > > > point you'll have had to work around it anyway. > > > > > > -n > > > > > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > > > > > wrote: > > >> Hi All, > > >> > > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds are > included > > >> except for Python 3.3 on windows, which will arrive later. Many > thanks to > > >> Ralf for the binaries, and to those who found and fixed the bugs > in the last > > >> beta. Any remaining bugs are all my fault ;) I hope this will be > the last > > >> release before final, so please test it thoroughly. > > >> > > >> Chuck > > > > > > Hello, > > > > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most 3rd > party > > packages test OK now. Thank you. > > > > A few tests still fail in the following packages when run with > > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to > > numpy-MKL-1.7.1-win-amd64-py3.3: > > > > 1) Pandas 0.12.0 > > > > ``` > > > ====================================================================== > > FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\pandas\tests\test_series.py", > > line 254, in test_nansum_buglet > > assert_almost_equal(result, 1) > > File "X:\Python33\lib\site-packages\pandas\util\testing.py", line > > 134, in assert_almost_equal > > np.testing.assert_(isiterable(b)) > > File "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", > line > > 44, in assert_ > > raise AssertionError(msg) > > AssertionError > > ``` > > > > Possibly related: > > > > ``` > > >>> import numpy as np > > >>> from pandas import Series > > >>> s = Series([0.0]) > > >>> result = np.nansum(s) > > >>> print(result) > > Traceback (most recent call last): > > File "", line 1, in > > File "X:\Python33\lib\site-packages\pandas\core\base.py", line > > 19, in > > __str__ > > return self.__unicode__() > > File "X:\Python33\lib\site-packages\pandas\core\series.py", line > > 1115, in __unicode__ > > length=len(self) > 50, > > TypeError: len() of unsized object > > ``` > > > > 2) Bottleneck 0.7.0 > > > > > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 > > > > 3) skimage 0.8.2 > > > > These tests passed with numpy 1.8.0b2: > > > > ``` > > > ====================================================================== > > FAIL: test_grey.test_non_square_image > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 162, in test_non_square_image > > testing.assert_array_equal(binary_res, grey_res) > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 50.6328125%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, False, False], > > [ True, True, True, ..., False, False, False], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_erosion > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 169, in test_binary_erosion > > testing.assert_array_equal(binary_res, grey_res) > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 48.260498046875%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, False, False], > > [ True, True, True, ..., False, False, False], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_closing > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 183, in test_binary_closing > > testing.assert_array_equal(binary_res, grey_res) > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 66.302490234375%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, True, True], > > [ True, True, True, ..., True, True, True], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_opening > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 190, in test_binary_opening > > testing.assert_array_equal(binary_res, grey_res) > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 58.465576171875%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, True, False], > > [ True, True, True, ..., True, True, False], > > [ True, True, True, ..., False, False, False],... > > ``` > > > > I'll bet the skimage problems come from > > https://github.com/numpy/numpy/pull/3811. They may be doing something > > naughty... > > > > Chuck > > > > A bool image is convolved with a uint8 kernel and the result compared > for equality with an uint32 scalar... > > > https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 > > Looks like the result of the convolution is probably output as a bool, which now means 0,1, which does not work when checking equality with the number of pixels in the kernel. I'd call expressing the result of a convolution as a boolean very naughty. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Sep 30 19:44:35 2013 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 01 Oct 2013 01:44:35 +0200 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: <5249F26B.2040109@uci.edu> <524A0561.4000808@uci.edu> Message-ID: <524A0CE3.3080502@googlemail.com> On 01.10.2013 01:30, Charles R Harris wrote: > > > > On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke > wrote: > > On 9/30/2013 3:45 PM, Charles R Harris wrote: > > > > > > > > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke > > >> wrote: > > > > > > > > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do > > actually test this! It is really in your best > > > interest, and I think people don't always realize this. > > > > > > Here's how it works: > > > - If you test it *now*, and it breaks your code that worked > with 1.7, > > > and you *tell* us this now, then it's *our* problem and we > hold up the > > > release to fix the bug. > > > - If you test it *after* we release, and it breaks your > code, then we > > > are sad but you have to work around it (because we can't > magically > > > make that release not have happened, your users will be using it > > > anyway), and we put it on the stack with all the other bugs. > All of > > > which we care about but it's a large enough stack that it's > not going > > > to get any special priority, because, see above about how at > this > > > point you'll have had to work around it anyway. > > > > > > -n > > > > > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > > > >> wrote: > > >> Hi All, > > >> > > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds > are included > > >> except for Python 3.3 on windows, which will arrive later. > Many thanks to > > >> Ralf for the binaries, and to those who found and fixed the > bugs in the last > > >> beta. Any remaining bugs are all my fault ;) I hope this > will be the last > > >> release before final, so please test it thoroughly. > > >> > > >> Chuck > > > > > > Hello, > > > > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most > 3rd party > > packages test OK now. Thank you. > > > > A few tests still fail in the following packages when run with > > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to > > numpy-MKL-1.7.1-win-amd64-py3.3: > > > > 1) Pandas 0.12.0 > > > > ``` > > > ====================================================================== > > FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > "X:\Python33\lib\site-packages\pandas\tests\test_series.py", > > line 254, in test_nansum_buglet > > assert_almost_equal(result, 1) > > File > "X:\Python33\lib\site-packages\pandas\util\testing.py", line > > 134, in assert_almost_equal > > np.testing.assert_(isiterable(b)) > > File > "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line > > 44, in assert_ > > raise AssertionError(msg) > > AssertionError > > ``` > > > > Possibly related: > > > > ``` > > >>> import numpy as np > > >>> from pandas import Series > > >>> s = Series([0.0]) > > >>> result = np.nansum(s) > > >>> print(result) > > Traceback (most recent call last): > > File "", line 1, in > > File "X:\Python33\lib\site-packages\pandas\core\base.py", line > > 19, in > > __str__ > > return self.__unicode__() > > File > "X:\Python33\lib\site-packages\pandas\core\series.py", line > > 1115, in __unicode__ > > length=len(self) > 50, > > TypeError: len() of unsized object > > ``` > > > > 2) Bottleneck 0.7.0 > > > > > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 > > > > 3) skimage 0.8.2 > > > > These tests passed with numpy 1.8.0b2: > > > > ``` > > > ====================================================================== > > FAIL: test_grey.test_non_square_image > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line > 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 162, in test_non_square_image > > testing.assert_array_equal(binary_res, grey_res) > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 50.6328125%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, False, False], > > [ True, True, True, ..., False, False, False], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_erosion > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line > 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 169, in test_binary_erosion > > testing.assert_array_equal(binary_res, grey_res) > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 48.260498046875%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, False, False], > > [ True, True, True, ..., False, False, False], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_closing > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line > 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 183, in test_binary_closing > > testing.assert_array_equal(binary_res, grey_res) > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 66.302490234375%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, True, True], > > [ True, True, True, ..., True, True, True], > > [ True, True, True, ..., False, False, False],... > > > > > ====================================================================== > > FAIL: test_grey.test_binary_opening > > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "X:\Python33\lib\site-packages\nose\case.py", line > 198, in > > runTest > > self.test(*self.arg) > > File > > > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > > line 190, in test_binary_opening > > testing.assert_array_equal(binary_res, grey_res) > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 718, in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File > "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > > 644, in assert_array_compare > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > (mismatch 58.465576171875%) > > x: array([[False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False], > > [False, False, False, ..., False, False, False],... > > y: array([[ True, True, True, ..., True, True, False], > > [ True, True, True, ..., True, True, False], > > [ True, True, True, ..., False, False, False],... > > ``` > > > > I'll bet the skimage problems come from > > https://github.com/numpy/numpy/pull/3811. They may be doing something > > naughty... > > > > Chuck > > > > A bool image is convolved with a uint8 kernel and the result compared > for equality with an uint32 scalar... > > https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 > > > Looks like the result of the convolution is probably output as a bool, > which now means 0,1, which does not work when checking equality with the > number of pixels in the kernel. I'd call expressing the result of a > convolution as a boolean very naughty. using a view should fix it: conv = ndimage.convolve((image > 0).view(np.uint8), selem, output=out, mode='constant', cval=1) but it needs check for sum(selem) < 255 too. From cgohlke at uci.edu Mon Sep 30 20:13:07 2013 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 30 Sep 2013 17:13:07 -0700 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: <524A0CE3.3080502@googlemail.com> References: <5249F26B.2040109@uci.edu> <524A0561.4000808@uci.edu> <524A0CE3.3080502@googlemail.com> Message-ID: <524A1393.1040701@uci.edu> On 9/30/2013 4:44 PM, Julian Taylor wrote: > On 01.10.2013 01:30, Charles R Harris wrote: >> >> >> >> On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke > > wrote: >> >> On 9/30/2013 3:45 PM, Charles R Harris wrote: >> > >> > >> > >> > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke > >> > >> wrote: >> > >> > >> > >> > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone please do >> > actually test this! It is really in your best >> > > interest, and I think people don't always realize this. >> > > >> > > Here's how it works: >> > > - If you test it *now*, and it breaks your code that worked >> with 1.7, >> > > and you *tell* us this now, then it's *our* problem and we >> hold up the >> > > release to fix the bug. >> > > - If you test it *after* we release, and it breaks your >> code, then we >> > > are sad but you have to work around it (because we can't >> magically >> > > make that release not have happened, your users will be using it >> > > anyway), and we put it on the stack with all the other bugs. >> All of >> > > which we care about but it's a large enough stack that it's >> not going >> > > to get any special priority, because, see above about how at >> this >> > > point you'll have had to work around it anyway. >> > > >> > > -n >> > > >> > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris >> > > > > >> wrote: >> > >> Hi All, >> > >> >> > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds >> are included >> > >> except for Python 3.3 on windows, which will arrive later. >> Many thanks to >> > >> Ralf for the binaries, and to those who found and fixed the >> bugs in the last >> > >> beta. Any remaining bugs are all my fault ;) I hope this >> will be the last >> > >> release before final, so please test it thoroughly. >> > >> >> > >> Chuck >> > >> > >> > Hello, >> > >> > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most >> 3rd party >> > packages test OK now. Thank you. >> > >> > A few tests still fail in the following packages when run with >> > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to >> > numpy-MKL-1.7.1-win-amd64-py3.3: >> > >> > 1) Pandas 0.12.0 >> > >> > ``` >> > >> ====================================================================== >> > FAIL: test_nansum_buglet (pandas.tests.test_series.TestNanops) >> > >> ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File >> "X:\Python33\lib\site-packages\pandas\tests\test_series.py", >> > line 254, in test_nansum_buglet >> > assert_almost_equal(result, 1) >> > File >> "X:\Python33\lib\site-packages\pandas\util\testing.py", line >> > 134, in assert_almost_equal >> > np.testing.assert_(isiterable(b)) >> > File >> "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line >> > 44, in assert_ >> > raise AssertionError(msg) >> > AssertionError >> > ``` >> > >> > Possibly related: >> > >> > ``` >> > >>> import numpy as np >> > >>> from pandas import Series >> > >>> s = Series([0.0]) >> > >>> result = np.nansum(s) >> > >>> print(result) >> > Traceback (most recent call last): >> > File "", line 1, in >> > File "X:\Python33\lib\site-packages\pandas\core\base.py", line >> > 19, in >> > __str__ >> > return self.__unicode__() >> > File >> "X:\Python33\lib\site-packages\pandas\core\series.py", line >> > 1115, in __unicode__ >> > length=len(self) > 50, >> > TypeError: len() of unsized object >> > ``` >> > >> > 2) Bottleneck 0.7.0 >> > >> > >> https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 >> > >> > 3) skimage 0.8.2 >> > >> > These tests passed with numpy 1.8.0b2: >> > >> > ``` >> > >> ====================================================================== >> > FAIL: test_grey.test_non_square_image >> > >> ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> 198, in >> > runTest >> > self.test(*self.arg) >> > File >> > >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> > line 162, in test_non_square_image >> > testing.assert_array_equal(binary_res, grey_res) >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 718, in assert_array_equal >> > verbose=verbose, header='Arrays are not equal') >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 644, in assert_array_compare >> > raise AssertionError(msg) >> > AssertionError: >> > Arrays are not equal >> > >> > (mismatch 50.6328125%) >> > x: array([[False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False],... >> > y: array([[ True, True, True, ..., True, False, False], >> > [ True, True, True, ..., False, False, False], >> > [ True, True, True, ..., False, False, False],... >> > >> > >> ====================================================================== >> > FAIL: test_grey.test_binary_erosion >> > >> ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> 198, in >> > runTest >> > self.test(*self.arg) >> > File >> > >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> > line 169, in test_binary_erosion >> > testing.assert_array_equal(binary_res, grey_res) >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 718, in assert_array_equal >> > verbose=verbose, header='Arrays are not equal') >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 644, in assert_array_compare >> > raise AssertionError(msg) >> > AssertionError: >> > Arrays are not equal >> > >> > (mismatch 48.260498046875%) >> > x: array([[False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False],... >> > y: array([[ True, True, True, ..., True, False, False], >> > [ True, True, True, ..., False, False, False], >> > [ True, True, True, ..., False, False, False],... >> > >> > >> ====================================================================== >> > FAIL: test_grey.test_binary_closing >> > >> ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> 198, in >> > runTest >> > self.test(*self.arg) >> > File >> > >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> > line 183, in test_binary_closing >> > testing.assert_array_equal(binary_res, grey_res) >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 718, in assert_array_equal >> > verbose=verbose, header='Arrays are not equal') >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 644, in assert_array_compare >> > raise AssertionError(msg) >> > AssertionError: >> > Arrays are not equal >> > >> > (mismatch 66.302490234375%) >> > x: array([[False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False],... >> > y: array([[ True, True, True, ..., True, True, True], >> > [ True, True, True, ..., True, True, True], >> > [ True, True, True, ..., False, False, False],... >> > >> > >> ====================================================================== >> > FAIL: test_grey.test_binary_opening >> > >> ---------------------------------------------------------------------- >> > Traceback (most recent call last): >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> 198, in >> > runTest >> > self.test(*self.arg) >> > File >> > >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> > line 190, in test_binary_opening >> > testing.assert_array_equal(binary_res, grey_res) >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 718, in assert_array_equal >> > verbose=verbose, header='Arrays are not equal') >> > File >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> > 644, in assert_array_compare >> > raise AssertionError(msg) >> > AssertionError: >> > Arrays are not equal >> > >> > (mismatch 58.465576171875%) >> > x: array([[False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False], >> > [False, False, False, ..., False, False, False],... >> > y: array([[ True, True, True, ..., True, True, False], >> > [ True, True, True, ..., True, True, False], >> > [ True, True, True, ..., False, False, False],... >> > ``` >> > >> > I'll bet the skimage problems come from >> > https://github.com/numpy/numpy/pull/3811. They may be doing something >> > naughty... >> > >> > Chuck >> > >> >> A bool image is convolved with a uint8 kernel and the result compared >> for equality with an uint32 scalar... >> >> https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 >> >> >> Looks like the result of the convolution is probably output as a bool, >> which now means 0,1, which does not work when checking equality with the >> number of pixels in the kernel. I'd call expressing the result of a >> convolution as a boolean very naughty. > > using a view should fix it: > conv = ndimage.convolve((image > 0).view(np.uint8), selem, output=out, > mode='constant', cval=1) > but it needs check for sum(selem) < 255 too. I opened an issue at Christoph From charlesr.harris at gmail.com Mon Sep 30 20:37:25 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 18:37:25 -0600 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: <524A1393.1040701@uci.edu> References: <5249F26B.2040109@uci.edu> <524A0561.4000808@uci.edu> <524A0CE3.3080502@googlemail.com> <524A1393.1040701@uci.edu> Message-ID: On Mon, Sep 30, 2013 at 6:13 PM, Christoph Gohlke wrote: > On 9/30/2013 4:44 PM, Julian Taylor wrote: > > On 01.10.2013 01:30, Charles R Harris wrote: > >> > >> > >> > >> On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke >> > wrote: > >> > >> On 9/30/2013 3:45 PM, Charles R Harris wrote: > >> > > >> > > >> > > >> > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke < > cgohlke at uci.edu > >> > >> > >> wrote: > >> > > >> > > >> > > >> > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone > please do > >> > actually test this! It is really in your best > >> > > interest, and I think people don't always realize this. > >> > > > >> > > Here's how it works: > >> > > - If you test it *now*, and it breaks your code that worked > >> with 1.7, > >> > > and you *tell* us this now, then it's *our* problem and we > >> hold up the > >> > > release to fix the bug. > >> > > - If you test it *after* we release, and it breaks your > >> code, then we > >> > > are sad but you have to work around it (because we can't > >> magically > >> > > make that release not have happened, your users will be > using it > >> > > anyway), and we put it on the stack with all the other > bugs. > >> All of > >> > > which we care about but it's a large enough stack that it's > >> not going > >> > > to get any special priority, because, see above about how > at > >> this > >> > > point you'll have had to work around it anyway. > >> > > > >> > > -n > >> > > > >> > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris > >> > > >> charlesr.harris at gmail.com > >> >> wrote: > >> > >> Hi All, > >> > >> > >> > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary builds > >> are included > >> > >> except for Python 3.3 on windows, which will arrive later. > >> Many thanks to > >> > >> Ralf for the binaries, and to those who found and fixed > the > >> bugs in the last > >> > >> beta. Any remaining bugs are all my fault ;) I hope this > >> will be the last > >> > >> release before final, so please test it thoroughly. > >> > >> > >> > >> Chuck > >> > > >> > > >> > Hello, > >> > > >> > NumPy 1.8.0rc1 looks good. All tests pass on Windows and most > >> 3rd party > >> > packages test OK now. Thank you. > >> > > >> > A few tests still fail in the following packages when run > with > >> > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to > >> > numpy-MKL-1.7.1-win-amd64-py3.3: > >> > > >> > 1) Pandas 0.12.0 > >> > > >> > ``` > >> > > >> > ====================================================================== > >> > FAIL: test_nansum_buglet > (pandas.tests.test_series.TestNanops) > >> > > >> > ---------------------------------------------------------------------- > >> > Traceback (most recent call last): > >> > File > >> "X:\Python33\lib\site-packages\pandas\tests\test_series.py", > >> > line 254, in test_nansum_buglet > >> > assert_almost_equal(result, 1) > >> > File > >> "X:\Python33\lib\site-packages\pandas\util\testing.py", line > >> > 134, in assert_almost_equal > >> > np.testing.assert_(isiterable(b)) > >> > File > >> "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line > >> > 44, in assert_ > >> > raise AssertionError(msg) > >> > AssertionError > >> > ``` > >> > > >> > Possibly related: > >> > > >> > ``` > >> > >>> import numpy as np > >> > >>> from pandas import Series > >> > >>> s = Series([0.0]) > >> > >>> result = np.nansum(s) > >> > >>> print(result) > >> > Traceback (most recent call last): > >> > File "", line 1, in > >> > File > "X:\Python33\lib\site-packages\pandas\core\base.py", line > >> > 19, in > >> > __str__ > >> > return self.__unicode__() > >> > File > >> "X:\Python33\lib\site-packages\pandas\core\series.py", line > >> > 1115, in __unicode__ > >> > length=len(self) > 50, > >> > TypeError: len() of unsized object > >> > ``` > >> > > >> > 2) Bottleneck 0.7.0 > >> > > >> > > >> > https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 > >> > > >> > 3) skimage 0.8.2 > >> > > >> > These tests passed with numpy 1.8.0b2: > >> > > >> > ``` > >> > > >> > ====================================================================== > >> > FAIL: test_grey.test_non_square_image > >> > > >> > ---------------------------------------------------------------------- > >> > Traceback (most recent call last): > >> > File "X:\Python33\lib\site-packages\nose\case.py", line > >> 198, in > >> > runTest > >> > self.test(*self.arg) > >> > File > >> > > >> > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > >> > line 162, in test_non_square_image > >> > testing.assert_array_equal(binary_res, grey_res) > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 718, in assert_array_equal > >> > verbose=verbose, header='Arrays are not equal') > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 644, in assert_array_compare > >> > raise AssertionError(msg) > >> > AssertionError: > >> > Arrays are not equal > >> > > >> > (mismatch 50.6328125%) > >> > x: array([[False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False],... > >> > y: array([[ True, True, True, ..., True, False, False], > >> > [ True, True, True, ..., False, False, False], > >> > [ True, True, True, ..., False, False, False],... > >> > > >> > > >> > ====================================================================== > >> > FAIL: test_grey.test_binary_erosion > >> > > >> > ---------------------------------------------------------------------- > >> > Traceback (most recent call last): > >> > File "X:\Python33\lib\site-packages\nose\case.py", line > >> 198, in > >> > runTest > >> > self.test(*self.arg) > >> > File > >> > > >> > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > >> > line 169, in test_binary_erosion > >> > testing.assert_array_equal(binary_res, grey_res) > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 718, in assert_array_equal > >> > verbose=verbose, header='Arrays are not equal') > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 644, in assert_array_compare > >> > raise AssertionError(msg) > >> > AssertionError: > >> > Arrays are not equal > >> > > >> > (mismatch 48.260498046875%) > >> > x: array([[False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False],... > >> > y: array([[ True, True, True, ..., True, False, False], > >> > [ True, True, True, ..., False, False, False], > >> > [ True, True, True, ..., False, False, False],... > >> > > >> > > >> > ====================================================================== > >> > FAIL: test_grey.test_binary_closing > >> > > >> > ---------------------------------------------------------------------- > >> > Traceback (most recent call last): > >> > File "X:\Python33\lib\site-packages\nose\case.py", line > >> 198, in > >> > runTest > >> > self.test(*self.arg) > >> > File > >> > > >> > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > >> > line 183, in test_binary_closing > >> > testing.assert_array_equal(binary_res, grey_res) > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 718, in assert_array_equal > >> > verbose=verbose, header='Arrays are not equal') > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 644, in assert_array_compare > >> > raise AssertionError(msg) > >> > AssertionError: > >> > Arrays are not equal > >> > > >> > (mismatch 66.302490234375%) > >> > x: array([[False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False],... > >> > y: array([[ True, True, True, ..., True, True, True], > >> > [ True, True, True, ..., True, True, True], > >> > [ True, True, True, ..., False, False, False],... > >> > > >> > > >> > ====================================================================== > >> > FAIL: test_grey.test_binary_opening > >> > > >> > ---------------------------------------------------------------------- > >> > Traceback (most recent call last): > >> > File "X:\Python33\lib\site-packages\nose\case.py", line > >> 198, in > >> > runTest > >> > self.test(*self.arg) > >> > File > >> > > >> > "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", > >> > line 190, in test_binary_opening > >> > testing.assert_array_equal(binary_res, grey_res) > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 718, in assert_array_equal > >> > verbose=verbose, header='Arrays are not equal') > >> > File > >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line > >> > 644, in assert_array_compare > >> > raise AssertionError(msg) > >> > AssertionError: > >> > Arrays are not equal > >> > > >> > (mismatch 58.465576171875%) > >> > x: array([[False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False], > >> > [False, False, False, ..., False, False, False],... > >> > y: array([[ True, True, True, ..., True, True, False], > >> > [ True, True, True, ..., True, True, False], > >> > [ True, True, True, ..., False, False, False],... > >> > ``` > >> > > >> > I'll bet the skimage problems come from > >> > https://github.com/numpy/numpy/pull/3811. They may be doing > something > >> > naughty... > >> > > >> > Chuck > >> > > >> > >> A bool image is convolved with a uint8 kernel and the result > compared > >> for equality with an uint32 scalar... > >> > >> > https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 > >> > >> > >> Looks like the result of the convolution is probably output as a bool, > >> which now means 0,1, which does not work when checking equality with the > >> number of pixels in the kernel. I'd call expressing the result of a > >> convolution as a boolean very naughty. > > > > using a view should fix it: > > conv = ndimage.convolve((image > 0).view(np.uint8), selem, output=out, > > mode='constant', cval=1) > > but it needs check for sum(selem) < 255 too. > > I opened an issue at > > > The errors in skimage for Python 2.7 are also present for numpy 1.7.1. Maybe I'm missing some dependencies? A bunch of tests are skipped for Python 3.3 and I suspect the failing tests are among them. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej.certik at gmail.com Mon Sep 30 21:05:31 2013 From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=) Date: Mon, 30 Sep 2013 19:05:31 -0600 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" Message-ID: Hi, What is the rationale for using False in 'mask' for elements that should be included? http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html As opposed to using True for elements that should be included, which is what I was intuitively expecting when I started using the masked arrays. This "True convention" also happens to be the one used in Fortran, see e.g.: http://gcc.gnu.org/onlinedocs/gfortran/SUM.html So it's confusing why NumPy would chose a "False convention". Could it be, that NumPy views 'mask' as opacity? Then it would make sense to use True to make a value 'opaque'. Ondrej From charlesr.harris at gmail.com Mon Sep 30 21:38:58 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 19:38:58 -0600 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" In-Reply-To: References: Message-ID: On Mon, Sep 30, 2013 at 7:05 PM, Ond?ej ?ert?k wrote: > Hi, > > What is the rationale for using False in 'mask' for elements that > should be included? > > http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html > > As opposed to using True for elements that should be included, which > is what I was intuitively expecting when I started using the masked > arrays. This "True convention" also happens to be the one used in > Fortran, see e.g.: > > http://gcc.gnu.org/onlinedocs/gfortran/SUM.html > > So it's confusing why NumPy would chose a "False convention". Could it > be, that NumPy views 'mask' as opacity? Then it would make sense to > use True to make a value 'opaque'. > There was a lengthy discussion of this point back when the NA work was done. You might be able to find the thread with a search. As to why it is as it is, I suspect it is historical consistency. Pierre wrote the masked array package for numpy, but it may very well go back to the masked array package implemented for Numeric. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Sep 30 22:05:35 2013 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 30 Sep 2013 22:05:35 -0400 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" In-Reply-To: References: Message-ID: On Mon, Sep 30, 2013 at 9:38 PM, Charles R Harris wrote: > > > > On Mon, Sep 30, 2013 at 7:05 PM, Ond?ej ?ert?k > wrote: >> >> Hi, >> >> What is the rationale for using False in 'mask' for elements that >> should be included? >> >> http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html >> >> As opposed to using True for elements that should be included, which >> is what I was intuitively expecting when I started using the masked >> arrays. This "True convention" also happens to be the one used in >> Fortran, see e.g.: >> >> http://gcc.gnu.org/onlinedocs/gfortran/SUM.html >> >> So it's confusing why NumPy would chose a "False convention". Could it >> be, that NumPy views 'mask' as opacity? Then it would make sense to >> use True to make a value 'opaque'. > > > There was a lengthy discussion of this point back when the NA work was done. > You might be able to find the thread with a search. > > As to why it is as it is, I suspect it is historical consistency. Pierre > wrote the masked array package for numpy, but it may very well go back to > the masked array package implemented for Numeric. I don't know ancient history, but I thought it's "natural". (Actually, I never thought about it.) I always thought `mask` indicates the "masked" (invalid, hidden) values, and masked arrays mask the missing values. http://en.wikipedia.org/wiki/Masking_tape Josef > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Mon Sep 30 22:27:42 2013 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 30 Sep 2013 20:27:42 -0600 Subject: [Numpy-discussion] [SciPy-Dev] 1.8.0rc1 In-Reply-To: References: <5249F26B.2040109@uci.edu> <524A0561.4000808@uci.edu> <524A0CE3.3080502@googlemail.com> <524A1393.1040701@uci.edu> Message-ID: On Mon, Sep 30, 2013 at 6:37 PM, Charles R Harris wrote: > > > > On Mon, Sep 30, 2013 at 6:13 PM, Christoph Gohlke wrote: > >> On 9/30/2013 4:44 PM, Julian Taylor wrote: >> > On 01.10.2013 01:30, Charles R Harris wrote: >> >> >> >> >> >> >> >> On Mon, Sep 30, 2013 at 5:12 PM, Christoph Gohlke > >> > wrote: >> >> >> >> On 9/30/2013 3:45 PM, Charles R Harris wrote: >> >> > >> >> > >> >> > >> >> > On Mon, Sep 30, 2013 at 3:51 PM, Christoph Gohlke < >> cgohlke at uci.edu >> >> >> >> > >> wrote: >> >> > >> >> > >> >> > >> >> > On 9/30/2013 11:02 AM, Nathaniel Smith wrote:> Everyone >> please do >> >> > actually test this! It is really in your best >> >> > > interest, and I think people don't always realize this. >> >> > > >> >> > > Here's how it works: >> >> > > - If you test it *now*, and it breaks your code that >> worked >> >> with 1.7, >> >> > > and you *tell* us this now, then it's *our* problem and we >> >> hold up the >> >> > > release to fix the bug. >> >> > > - If you test it *after* we release, and it breaks your >> >> code, then we >> >> > > are sad but you have to work around it (because we can't >> >> magically >> >> > > make that release not have happened, your users will be >> using it >> >> > > anyway), and we put it on the stack with all the other >> bugs. >> >> All of >> >> > > which we care about but it's a large enough stack that >> it's >> >> not going >> >> > > to get any special priority, because, see above about how >> at >> >> this >> >> > > point you'll have had to work around it anyway. >> >> > > >> >> > > -n >> >> > > >> >> > > On Mon, Sep 30, 2013 at 4:17 PM, Charles R Harris >> >> > > > >> > charlesr.harris at gmail.com >> >> >> wrote: >> >> > >> Hi All, >> >> > >> >> >> > >> NumPy 1.8.0rc1 is up now on sourceforge .The binary >> builds >> >> are included >> >> > >> except for Python 3.3 on windows, which will arrive >> later. >> >> Many thanks to >> >> > >> Ralf for the binaries, and to those who found and fixed >> the >> >> bugs in the last >> >> > >> beta. Any remaining bugs are all my fault ;) I hope this >> >> will be the last >> >> > >> release before final, so please test it thoroughly. >> >> > >> >> >> > >> Chuck >> >> > >> >> > >> >> > Hello, >> >> > >> >> > NumPy 1.8.0rc1 looks good. All tests pass on Windows and >> most >> >> 3rd party >> >> > packages test OK now. Thank you. >> >> > >> >> > A few tests still fail in the following packages when run >> with >> >> > numpy-MKL-1.8.0rc1-win-amd64-py3.3 compared to >> >> > numpy-MKL-1.7.1-win-amd64-py3.3: >> >> > >> >> > 1) Pandas 0.12.0 >> >> > >> >> > ``` >> >> > >> >> >> ====================================================================== >> >> > FAIL: test_nansum_buglet >> (pandas.tests.test_series.TestNanops) >> >> > >> >> >> ---------------------------------------------------------------------- >> >> > Traceback (most recent call last): >> >> > File >> >> "X:\Python33\lib\site-packages\pandas\tests\test_series.py", >> >> > line 254, in test_nansum_buglet >> >> > assert_almost_equal(result, 1) >> >> > File >> >> "X:\Python33\lib\site-packages\pandas\util\testing.py", line >> >> > 134, in assert_almost_equal >> >> > np.testing.assert_(isiterable(b)) >> >> > File >> >> "D:\Dev\Compile\Test\numpy-build\numpy\testing\utils.py", line >> >> > 44, in assert_ >> >> > raise AssertionError(msg) >> >> > AssertionError >> >> > ``` >> >> > >> >> > Possibly related: >> >> > >> >> > ``` >> >> > >>> import numpy as np >> >> > >>> from pandas import Series >> >> > >>> s = Series([0.0]) >> >> > >>> result = np.nansum(s) >> >> > >>> print(result) >> >> > Traceback (most recent call last): >> >> > File "", line 1, in >> >> > File >> "X:\Python33\lib\site-packages\pandas\core\base.py", line >> >> > 19, in >> >> > __str__ >> >> > return self.__unicode__() >> >> > File >> >> "X:\Python33\lib\site-packages\pandas\core\series.py", line >> >> > 1115, in __unicode__ >> >> > length=len(self) > 50, >> >> > TypeError: len() of unsized object >> >> > ``` >> >> > >> >> > 2) Bottleneck 0.7.0 >> >> > >> >> > >> >> >> https://github.com/kwgoodman/bottleneck/issues/71#issuecomment-25331701 >> >> > >> >> > 3) skimage 0.8.2 >> >> > >> >> > These tests passed with numpy 1.8.0b2: >> >> > >> >> > ``` >> >> > >> >> >> ====================================================================== >> >> > FAIL: test_grey.test_non_square_image >> >> > >> >> >> ---------------------------------------------------------------------- >> >> > Traceback (most recent call last): >> >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> >> 198, in >> >> > runTest >> >> > self.test(*self.arg) >> >> > File >> >> > >> >> >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> >> > line 162, in test_non_square_image >> >> > testing.assert_array_equal(binary_res, grey_res) >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 718, in assert_array_equal >> >> > verbose=verbose, header='Arrays are not equal') >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 644, in assert_array_compare >> >> > raise AssertionError(msg) >> >> > AssertionError: >> >> > Arrays are not equal >> >> > >> >> > (mismatch 50.6328125%) >> >> > x: array([[False, False, False, ..., False, False, >> False], >> >> > [False, False, False, ..., False, False, False], >> >> > [False, False, False, ..., False, False, False],... >> >> > y: array([[ True, True, True, ..., True, False, >> False], >> >> > [ True, True, True, ..., False, False, False], >> >> > [ True, True, True, ..., False, False, False],... >> >> > >> >> > >> >> >> ====================================================================== >> >> > FAIL: test_grey.test_binary_erosion >> >> > >> >> >> ---------------------------------------------------------------------- >> >> > Traceback (most recent call last): >> >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> >> 198, in >> >> > runTest >> >> > self.test(*self.arg) >> >> > File >> >> > >> >> >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> >> > line 169, in test_binary_erosion >> >> > testing.assert_array_equal(binary_res, grey_res) >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 718, in assert_array_equal >> >> > verbose=verbose, header='Arrays are not equal') >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 644, in assert_array_compare >> >> > raise AssertionError(msg) >> >> > AssertionError: >> >> > Arrays are not equal >> >> > >> >> > (mismatch 48.260498046875%) >> >> > x: array([[False, False, False, ..., False, False, >> False], >> >> > [False, False, False, ..., False, False, False], >> >> > [False, False, False, ..., False, False, False],... >> >> > y: array([[ True, True, True, ..., True, False, >> False], >> >> > [ True, True, True, ..., False, False, False], >> >> > [ True, True, True, ..., False, False, False],... >> >> > >> >> > >> >> >> ====================================================================== >> >> > FAIL: test_grey.test_binary_closing >> >> > >> >> >> ---------------------------------------------------------------------- >> >> > Traceback (most recent call last): >> >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> >> 198, in >> >> > runTest >> >> > self.test(*self.arg) >> >> > File >> >> > >> >> >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> >> > line 183, in test_binary_closing >> >> > testing.assert_array_equal(binary_res, grey_res) >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 718, in assert_array_equal >> >> > verbose=verbose, header='Arrays are not equal') >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 644, in assert_array_compare >> >> > raise AssertionError(msg) >> >> > AssertionError: >> >> > Arrays are not equal >> >> > >> >> > (mismatch 66.302490234375%) >> >> > x: array([[False, False, False, ..., False, False, >> False], >> >> > [False, False, False, ..., False, False, False], >> >> > [False, False, False, ..., False, False, False],... >> >> > y: array([[ True, True, True, ..., True, True, >> True], >> >> > [ True, True, True, ..., True, True, True], >> >> > [ True, True, True, ..., False, False, False],... >> >> > >> >> > >> >> >> ====================================================================== >> >> > FAIL: test_grey.test_binary_opening >> >> > >> >> >> ---------------------------------------------------------------------- >> >> > Traceback (most recent call last): >> >> > File "X:\Python33\lib\site-packages\nose\case.py", line >> >> 198, in >> >> > runTest >> >> > self.test(*self.arg) >> >> > File >> >> > >> >> >> "X:\Python33\lib\site-packages\skimage\morphology\tests\test_grey.py", >> >> > line 190, in test_binary_opening >> >> > testing.assert_array_equal(binary_res, grey_res) >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 718, in assert_array_equal >> >> > verbose=verbose, header='Arrays are not equal') >> >> > File >> >> "X:\Python33\lib\site-packages\numpy\testing\utils.py", line >> >> > 644, in assert_array_compare >> >> > raise AssertionError(msg) >> >> > AssertionError: >> >> > Arrays are not equal >> >> > >> >> > (mismatch 58.465576171875%) >> >> > x: array([[False, False, False, ..., False, False, >> False], >> >> > [False, False, False, ..., False, False, False], >> >> > [False, False, False, ..., False, False, False],... >> >> > y: array([[ True, True, True, ..., True, True, >> False], >> >> > [ True, True, True, ..., True, True, False], >> >> > [ True, True, True, ..., False, False, False],... >> >> > ``` >> >> > >> >> > I'll bet the skimage problems come from >> >> > https://github.com/numpy/numpy/pull/3811. They may be doing >> something >> >> > naughty... >> >> > >> >> > Chuck >> >> > >> >> >> >> A bool image is convolved with a uint8 kernel and the result >> compared >> >> for equality with an uint32 scalar... >> >> >> >> >> https://github.com/scikit-image/scikit-image/blob/master/skimage/morphology/binary.py#L32 >> >> >> >> >> >> Looks like the result of the convolution is probably output as a bool, >> >> which now means 0,1, which does not work when checking equality with >> the >> >> number of pixels in the kernel. I'd call expressing the result of a >> >> convolution as a boolean very naughty. >> > >> > using a view should fix it: >> > conv = ndimage.convolve((image > 0).view(np.uint8), selem, output=out, >> > mode='constant', cval=1) >> > but it needs check for sum(selem) < 255 too. >> >> I opened an issue at >> >> >> > The errors in skimage for Python 2.7 are also present for numpy 1.7.1. > Maybe I'm missing some dependencies? A bunch of tests are skipped for > Python 3.3 and I suspect the failing tests are among them. > > The pandas test passes for current pandas dev, so it looks like a bug on their end that has been taken care of. test_nansum_buglet (__main__.TestNanops) ... ok Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Mon Sep 30 22:29:25 2013 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 30 Sep 2013 16:29:25 -1000 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" In-Reply-To: References: Message-ID: <524A3385.1030101@hawaii.edu> On 2013/09/30 4:05 PM, josef.pktd at gmail.com wrote: > On Mon, Sep 30, 2013 at 9:38 PM, Charles R Harris > wrote: >> >> >> >> On Mon, Sep 30, 2013 at 7:05 PM, Ond?ej ?ert?k >> wrote: >>> >>> Hi, >>> >>> What is the rationale for using False in 'mask' for elements that >>> should be included? >>> >>> http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html >>> >>> As opposed to using True for elements that should be included, which >>> is what I was intuitively expecting when I started using the masked >>> arrays. This "True convention" also happens to be the one used in >>> Fortran, see e.g.: >>> >>> http://gcc.gnu.org/onlinedocs/gfortran/SUM.html >>> >>> So it's confusing why NumPy would chose a "False convention". Could it >>> be, that NumPy views 'mask' as opacity? Then it would make sense to >>> use True to make a value 'opaque'. >> >> >> There was a lengthy discussion of this point back when the NA work was done. >> You might be able to find the thread with a search. >> >> As to why it is as it is, I suspect it is historical consistency. Pierre >> wrote the masked array package for numpy, but it may very well go back to >> the masked array package implemented for Numeric. > > I don't know ancient history, but I thought it's "natural". (Actually, > I never thought about it.) > > I always thought `mask` indicates the "masked" (invalid, hidden) > values, and masked arrays mask the missing values. Exactly. It is also consistent with the C and Unix convention of returning 0 on success and 1, or a non-zero error code on failure. In a similar vein, it works nicely with bit-mapped quality control flags, etc. When nothing is flagged, the value is good, and consequently not masked out. Eric > > http://en.wikipedia.org/wiki/Masking_tape > > Josef > >> >> Chuck >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ondrej.certik at gmail.com Mon Sep 30 22:57:18 2013 From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=) Date: Mon, 30 Sep 2013 20:57:18 -0600 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" In-Reply-To: <524A3385.1030101@hawaii.edu> References: <524A3385.1030101@hawaii.edu> Message-ID: On Mon, Sep 30, 2013 at 8:29 PM, Eric Firing wrote: > On 2013/09/30 4:05 PM, josef.pktd at gmail.com wrote: >> On Mon, Sep 30, 2013 at 9:38 PM, Charles R Harris >> wrote: >>> >>> >>> >>> On Mon, Sep 30, 2013 at 7:05 PM, Ond?ej ?ert?k >>> wrote: >>>> >>>> Hi, >>>> >>>> What is the rationale for using False in 'mask' for elements that >>>> should be included? >>>> >>>> http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html >>>> >>>> As opposed to using True for elements that should be included, which >>>> is what I was intuitively expecting when I started using the masked >>>> arrays. This "True convention" also happens to be the one used in >>>> Fortran, see e.g.: >>>> >>>> http://gcc.gnu.org/onlinedocs/gfortran/SUM.html >>>> >>>> So it's confusing why NumPy would chose a "False convention". Could it >>>> be, that NumPy views 'mask' as opacity? Then it would make sense to >>>> use True to make a value 'opaque'. >>> >>> >>> There was a lengthy discussion of this point back when the NA work was done. >>> You might be able to find the thread with a search. >>> >>> As to why it is as it is, I suspect it is historical consistency. Pierre >>> wrote the masked array package for numpy, but it may very well go back to >>> the masked array package implemented for Numeric. >> >> I don't know ancient history, but I thought it's "natural". (Actually, >> I never thought about it.) >> >> I always thought `mask` indicates the "masked" (invalid, hidden) >> values, and masked arrays mask the missing values. > > Exactly. It is also consistent with the C and Unix convention of > returning 0 on success and 1, or a non-zero error code on failure. In a > similar vein, it works nicely with bit-mapped quality control flags, > etc. When nothing is flagged, the value is good, and consequently not > masked out. I see, that makes sense. So to remember this, the rule is: "Specify elements that you want to get masked using True in 'mask'". But why do I need to invert the mask when I want to see the valid elements: In [1]: from numpy import ma In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False]) In [3]: a Out[3]: masked_array(data = [1 2 -- 4], mask = [False False True False], fill_value = 999999) In [4]: a[~a.mask] Out[4]: masked_array(data = [1 2 4], mask = [False False False], fill_value = 999999) I would find natural to write [4] as a[a.mask]. This is when it gets confusing. For example in Fortran, one does: integer :: a(4) = [1, 2, 3, 4] logical :: mask(4) = [.true., .true., .false., .true.] print *, a print *, pack(a, mask) and it prints: 1 2 3 4 1 2 4 So the behavior of mask when used as an index to select elements from an array is identical to NumPy --- True means include the element, False means exclude it. Ondrej From efiring at hawaii.edu Mon Sep 30 23:05:00 2013 From: efiring at hawaii.edu (Eric Firing) Date: Mon, 30 Sep 2013 17:05:00 -1000 Subject: [Numpy-discussion] Masked arrays: Rationale for "False convention" In-Reply-To: References: <524A3385.1030101@hawaii.edu> Message-ID: <524A3BDC.4070007@hawaii.edu> On 2013/09/30 4:57 PM, Ond?ej ?ert?k wrote: > On Mon, Sep 30, 2013 at 8:29 PM, Eric Firing wrote: >> On 2013/09/30 4:05 PM, josef.pktd at gmail.com wrote: >>> On Mon, Sep 30, 2013 at 9:38 PM, Charles R Harris >>> wrote: >>>> >>>> >>>> >>>> On Mon, Sep 30, 2013 at 7:05 PM, Ond?ej ?ert?k >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> What is the rationale for using False in 'mask' for elements that >>>>> should be included? >>>>> >>>>> http://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html >>>>> >>>>> As opposed to using True for elements that should be included, which >>>>> is what I was intuitively expecting when I started using the masked >>>>> arrays. This "True convention" also happens to be the one used in >>>>> Fortran, see e.g.: >>>>> >>>>> http://gcc.gnu.org/onlinedocs/gfortran/SUM.html >>>>> >>>>> So it's confusing why NumPy would chose a "False convention". Could it >>>>> be, that NumPy views 'mask' as opacity? Then it would make sense to >>>>> use True to make a value 'opaque'. >>>> >>>> >>>> There was a lengthy discussion of this point back when the NA work was done. >>>> You might be able to find the thread with a search. >>>> >>>> As to why it is as it is, I suspect it is historical consistency. Pierre >>>> wrote the masked array package for numpy, but it may very well go back to >>>> the masked array package implemented for Numeric. >>> >>> I don't know ancient history, but I thought it's "natural". (Actually, >>> I never thought about it.) >>> >>> I always thought `mask` indicates the "masked" (invalid, hidden) >>> values, and masked arrays mask the missing values. >> >> Exactly. It is also consistent with the C and Unix convention of >> returning 0 on success and 1, or a non-zero error code on failure. In a >> similar vein, it works nicely with bit-mapped quality control flags, >> etc. When nothing is flagged, the value is good, and consequently not >> masked out. > > I see, that makes sense. So to remember this, the rule is: > > "Specify elements that you want to get masked using True in 'mask'". > > But why do I need to invert the mask when I want to see the valid elements: > > In [1]: from numpy import ma > > In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False]) > > In [3]: a > Out[3]: > masked_array(data = [1 2 -- 4], > mask = [False False True False], > fill_value = 999999) > > > In [4]: a[~a.mask] > Out[4]: > masked_array(data = [1 2 4], > mask = [False False False], > fill_value = 999999) > > > I would find natural to write [4] as a[a.mask]. This is when it gets confusing. There is no getting around it; each of the two possible conventions has its advantages. But try this instead: In [2]: a = ma.array([1, 2, 3, 4], mask=[False, False, True, False]) In [3]: a.compressed() Out[3]: array([1, 2, 4]) I do occasionally need a "goodmask" which is the inverse of a.mask, but not very often; and when I do, needing to invert a.mask doesn't bother me. Eric > > For example in Fortran, one does: > > integer :: a(4) = [1, 2, 3, 4] > logical :: mask(4) = [.true., .true., .false., .true.] > print *, a > print *, pack(a, mask) > > and it prints: > > 1 2 3 4 > 1 2 4 > > So the behavior of mask when used as an index to select elements from > an array is identical to NumPy --- True means include the element, > False means exclude it. > > Ondrej > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >