Mailman 3 July 2011 - NumPy-Discussion

[ANN] Bento 0.0.6, a packaging solution for python software
by David Cournapeau July 15, 2011

July 15, 2011

Hi, I am pleased to announce a new release of bento, a packaging solution for python which aims at reproducibility, extensibility and simplicity. It supports every python version from 2.4 to 3.2. You can take a look at its main features on Bento's main page (http://cournape.github.com/Bento). The main features of this 0.0.6 release are: - Completely revamped distutils compatibility layer: it is now a thin layer around bento infrastructure, so that most bento packages should be pip-installable, while still keeping bento customization capabilities. - Build directory is now customizable through bentomaker with --build-directory option - Out of tree builds support (i.e. running bento in a directory which does not contain bento.info), with global --bento-info option - Hook File can now be specified in recursed bento.info - Preliminary support for .mpkg (Mac OS X native packaging) - More consistent API for extension/compiled library build registration - Both numpy and scipy can now be built with bento + waf as a build backend Bento is discussed on the bento mailing list (http://librelist.com/browser/bento). cheers, David

1 0

Masking entries in structured arrays
by Marcin Wlodarczak July 15, 2011

July 15, 2011

Hi, I was wondering whether it is possible to mask specific entries in a structured array. If I try to do the following: x = ma.masked_array([(2, 1.), (8, 2.)], dtype=[('a',int), ('b', float)]) x_masked = ma.masked_equal(x, 2) I get "AttributeError: 'NotImplementedType' object has no attribute 'ndim'", which actually makes sense since x.shape returns (2,). I really can't think of any way around this problem. Best regards, Marcin

2 2

ANN: NumPy 1.6.1 release candidate 3
by Ralf Gommers July 15, 2011

July 15, 2011

Hi, I am pleased to announce the availability of the third release candidate of NumPy 1.6.1. This is a bugfix release, list of fixed bugs: #1834 einsum fails for specific shapes #1837 einsum throws nan or freezes python for specific array shapes #1838 object <-> structured type arrays regression #1851 regression for SWIG based code in 1.6.0 #1863 Buggy results when operating on array copied with astype() #1870 Fix corner case of object array assignment #1843 Py3k: fix error with recarray #1885 nditer: Error in detecting double reduction loop #1874 f2py: fix --include_paths bug #1749 Fix ctypes.load_library() #1895/1896 iter: writeonly operands weren't always being buffered correctly This third RC has only a single change compared to RC2 (for #1895/1896), which fixes a serious regression in the iterator. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.6.1rc2/ Enjoy, Ralf

3 3

type-casting differences for comparisons
by Benjamin Root July 14, 2011

July 14, 2011

I just came across a real-head scratcher that took me a bit to figure out. I don't know if it counts as a bug or not. I have an array with dtype "f4" and a separate python float. Some elements of this array gets assigned this numpy float64 scalar value. (I know, I should be better off with a mask, but bear with me, this is just a demonstration code to isolate the core problem from a much more complicated program...) import numpy as np a = np.empty((500,500), dtype='f4') a[:] = np.random.random(a.shape) bad_val = 10*a.max() b = np.where(a > 0.8, bad_val, a) Now, the following seems to always evaluate to False, as expected: >>> np.any(b > bad_val) but, if I am (un-)lucky enough, this will sometimes evaluate to True: >>> any([(c > bad_val) for c in b.flat]) What it seems to me is that for the first comparison test, bad_val is casted down to float32 (or maybe b is casted up to float64?), but for the second example, the opposite is true. This can lead to some unexpected behaviors. Is there some sort of difference between type-casting of numpy scalars and numpy arrays? I would expect both to be the same. Ben Root

2 1

Can not compile documentation with python3.2
by Nadav Horesh July 14, 2011

July 14, 2011

I installed numpy-1.6.1-rc3 on python3.2, and used the python3 sphinx port (version 1.1pre) to compile the documentation and got this error: ==================================================================================================== nadav@nadav /dev/shm/numpy-1.6.1rc3/doc $ make latex mkdir -p build touch build/generate-stamp mkdir -p build/latex build/doctrees LANG=C sphinx-build -b latex -d build/doctrees source build/latex Running Sphinx v1.1pre 1.6rc3 1.6.1rc3 Exception occurred: File "/usr/lib64/python3.2/site-packages/Sphinx-1.1predev_20110713-py3.2.egg/sphinx/application.py", line 247, in setup_extension mod = __import__(extension, None, None, ['setup']) File "/dev/shm/numpy-1.6.1rc3/doc/sphinxext/numpydoc.py", line 37 title_re = re.compile(ur'^\s*[#*=]{4,}\n[a-z0-9 -]+\n[#*=]{4,}\s*', ^ SyntaxError: invalid syntax ==================================================================================================== Platform: 64bit gentoo linux Nadav.

1 0

Re: [Numpy-discussion] named ndarray axes
by Sam Quinan July 14, 2011

July 14, 2011

I'm currently working on interfacing ndarrays with a custom C-representation for n-dimensional arrays. My custom C code provides additional per-axis information (labeling, spacing between samples / range of sample positions along the axis, axis direction, cell vs.node centering, etc.) Subclassing ndarray to hold onto this info is fairly simple, but getting numpy's methods to intelligently modify that information when the array is sliced is something that I'm still trying to figure out. A robust way to attach per-axis info to a given ndarray, whether it just be a label or some more complex structure, would definitely be something I (and likely others) would find useful... That said, I'd love to know more about how the idx_axes() structure in your workaround works... - Sam On 7/13/11 12:00 PM, "numpy-discussion-request(a)scipy.org" <numpy-discussion-request(a)scipy.org> wrote: > Date: Tue, 12 Jul 2011 16:39:47 -0700 > From: Craig Yoshioka <craigyk(a)me.com> > Subject: [Numpy-discussion] named ndarray axes > To: NumPy-Discussion(a)scipy.org > Message-ID: <0FC8B43E-26CD-40ED-A6FA-59DD8D641998(a)me.com> > Content-Type: text/plain; CHARSET=US-ASCII > > I brought up a while ago about how it would be nice if numpy arrays could have > their axes 'labeled'. = I got an implementation that works pretty well for > me and in the process learned quite a few things, and was hoping to foster > some more discussion on this topic, as I think I have found a simple/flexible > solution to support this at the numpy level. > > Here are *some* examples code snippets from my unit tests on 'Array': > > a = Array((4,5,6)) > > # you can assign data to all axes by attribute: > a.Axes.Label = (0:'z',1:'y',2:'x'} > > # or add metadata to each individually: > a.Axes[1].Vector = [0,1,0] > a.Axes[2].Vector = [0,0,1] > > # simple case int indexing > b = a[0] > assert b.shape == (5,6) > assert b.Axes.Label == {0:'y',1:'x'} > assert b.Axes.Vector == {0:[0,1,0],1:[0,0,1]} > > # indexing with slices > b = a[:,0,:] > assert b.shape == (4,6) > assert b.Axes.Label == {0:'z',1:'x'} > assert b.Axes.Vector == {1:[0,0,1]} > > # indexing with ellipsis > b = a[...,0] > assert b.shape == (4,5) > assert b.Axes.Label == {0:'z',1:'y'} > > # indexing with ellipsis, newaxis, etc. > b = a[newaxis,...,2,newaxis] > assert b.shape == (1,4,5,1) > assert b.Axes.Label == {1:'z',2:'y'} > > # indexing with lists > b = a[[1,2],:,[1,2]] > assert b.shape == (2,5) > assert b.Axes.Label == {0:'z',1:'y'} > > # most interesting examples, indexing with axes labels---------------- > # I was a bit confused about how to handle indexing with mixed > axes/non-axes indexes > # IE: what does a['x',2:4] mean? on what axis is the 2:4 slice being > applied, the first? the first after 'x'? > # One option is to disallow mixing (simpler to implement, > understand?) > # Instead I chose to treat the axis indexing as a forced assignment > of an axis to a position. > > # axis indexing that transposes the first two dimensions, but otherwise > does nothing > b = a['y','z'] > assert b.shape == (5,4,6) > assert b.Axes.Label == {0:'y',1:'z',2:'x'} > > # abusing slices to allow specifying indexes for axes > b = a['y':0,'z'] > assert b.shape == (4,6) > assert b.Axes.Label == {0:'z',1:'x'} > > # unfortunately that means a slice index on an axis must be written like > so: > b = a['y':slice(0,2),'x','z'] > assert b.shape == (2,6,4) > assert b.Axes.Label == {0:'y',1:'x',2:'z'} > > b = a['y':[1,2,3],'x','z':slice(0:1)] > # or due to the forced transposition, this is the same as: > c = a['y','x','z'][[1,2,3],:,0:1] > > assert b.shape == (3,6,1) > assert b.Axes.Label == {0:'y',1:'x',2:'z'} > assert b.shape == c.shape > assert b.Axes == c.Axes > > > #----------------------------------------------------------------------------- > ----------- > > > To do all this I essentially had to recreate the effects of numpy indexing on > axes.... This is not ideal, but so far I seem to have addressed most of the > indexing I've used, at least. Here is what __getitem__ looks like: > > def __getitem__(self,idxs): > filtered_idxs,transposed_axes,kept_axes = self.idx_axes(idxs) > array = self.view(ndarray).transpose(transposed_axes) > array = array[filtered_idxs] > if isinstance(array,ndarray): > array = array.view(Array) > array.Axes = self.Axes.keep(kept_axes) > return array > > As you can see idx_axes() essentially recreates a lot of ndarray indexing > behavior, so that its effects can be explicitly handled. > > Having done all this, I think the best way for numpy to support 'labeled' axes > in the future is by having numpy itself keep track of a very simple tuple > attribute, like shape, and leave more complex axis naming/labeling to > subclasses on the python side. As an example, upon creating a new dimension > in an array, numpy assigns that dimension a semi-unique id, and this tuple > could be used in __array_finalize__. > > For example my __array_finalize__ could look like: > > def __array_finalize__(self,obj): > if hasattr(obj,'axesdata'): > for axesid in self.axes: > if axesid in obj.axes: > self.axesdata[axesid] = obj.axesdata[axesid] > > > This would cover a lot more situations and lead to much simpler code since the > work required on the C side would be minimal, but still allow robust and > custom tracking and propagation of axes information. > Subclasses that tap into this data would react to the result of numpy > operations vs. having to predict/anticipate. > > For example, my __getitem__, relying on the __array_finalize__ above, could > look like: > > def __getitem__(self,idxs): > filtered_idxs,transposed_axes= self.idx_axes(idxs) > array = self.transpose(transposed_axes) > return array[filtered_idxs] > > Not shown is how much simpler and robust the code for idx_axes would then be. > I estimate it would go from 130 loc to < 20 loc. > > Sorry for the extra long e-mail, > -Craig

3 4

Scipy 2011 Convore thread now open
by Peter Wang July 13, 2011

July 13, 2011

Hi folks, I have gone ahead and created a Convore group for the SciPy 2011 conference: https://convore.com/scipy-2011/ I have already created threads for each of the tutorial topics, and once the conference is underway, we'll create threads for each talk, so that audience can interact and post questions. Everyone is welcome to create topics of their own, in addition to the "official" conference topics. For those who are unfamiliar with Convore, it is a cross between a mailing list and a very souped-up IRC. It's usable for aynchronous discussion, but great for realtime, topical chats. Those of you who were at PyCon this year probably saw what a wonderful tool Convore proved to be for a tech conference. People used it for everything from BoF planning to dinner coordination to good-natured heckling of lightning talk speakers. I'm hoping that it will be used to similarly good effect for the SciPy Cheers, Peter

3 2

named ndarray axes
by Craig Yoshioka July 12, 2011

July 12, 2011

I brought up a while ago about how it would be nice if numpy arrays could have their axes 'labeled'. = I got an implementation that works pretty well for me and in the process learned quite a few things, and was hoping to foster some more discussion on this topic, as I think I have found a simple/flexible solution to support this at the numpy level. Here are *some* examples code snippets from my unit tests on 'Array': a = Array((4,5,6)) # you can assign data to all axes by attribute: a.Axes.Label = (0:'z',1:'y',2:'x'} # or add metadata to each individually: a.Axes[1].Vector = [0,1,0] a.Axes[2].Vector = [0,0,1] # simple case int indexing b = a[0] assert b.shape == (5,6) assert b.Axes.Label == {0:'y',1:'x'} assert b.Axes.Vector == {0:[0,1,0],1:[0,0,1]} # indexing with slices b = a[:,0,:] assert b.shape == (4,6) assert b.Axes.Label == {0:'z',1:'x'} assert b.Axes.Vector == {1:[0,0,1]} # indexing with ellipsis b = a[...,0] assert b.shape == (4,5) assert b.Axes.Label == {0:'z',1:'y'} # indexing with ellipsis, newaxis, etc. b = a[newaxis,...,2,newaxis] assert b.shape == (1,4,5,1) assert b.Axes.Label == {1:'z',2:'y'} # indexing with lists b = a[[1,2],:,[1,2]] assert b.shape == (2,5) assert b.Axes.Label == {0:'z',1:'y'} # most interesting examples, indexing with axes labels---------------- # I was a bit confused about how to handle indexing with mixed axes/non-axes indexes # IE: what does a['x',2:4] mean? on what axis is the 2:4 slice being applied, the first? the first after 'x'? # One option is to disallow mixing (simpler to implement, understand?) # Instead I chose to treat the axis indexing as a forced assignment of an axis to a position. # axis indexing that transposes the first two dimensions, but otherwise does nothing b = a['y','z'] assert b.shape == (5,4,6) assert b.Axes.Label == {0:'y',1:'z',2:'x'} # abusing slices to allow specifying indexes for axes b = a['y':0,'z'] assert b.shape == (4,6) assert b.Axes.Label == {0:'z',1:'x'} # unfortunately that means a slice index on an axis must be written like so: b = a['y':slice(0,2),'x','z'] assert b.shape == (2,6,4) assert b.Axes.Label == {0:'y',1:'x',2:'z'} b = a['y':[1,2,3],'x','z':slice(0:1)] # or due to the forced transposition, this is the same as: c = a['y','x','z'][[1,2,3],:,0:1] assert b.shape == (3,6,1) assert b.Axes.Label == {0:'y',1:'x',2:'z'} assert b.shape == c.shape assert b.Axes == c.Axes #---------------------------------------------------------------------------------------- To do all this I essentially had to recreate the effects of numpy indexing on axes.... This is not ideal, but so far I seem to have addressed most of the indexing I've used, at least. Here is what __getitem__ looks like: def __getitem__(self,idxs): filtered_idxs,transposed_axes,kept_axes = self.idx_axes(idxs) array = self.view(ndarray).transpose(transposed_axes) array = array[filtered_idxs] if isinstance(array,ndarray): array = array.view(Array) array.Axes = self.Axes.keep(kept_axes) return array As you can see idx_axes() essentially recreates a lot of ndarray indexing behavior, so that its effects can be explicitly handled. Having done all this, I think the best way for numpy to support 'labeled' axes in the future is by having numpy itself keep track of a very simple tuple attribute, like shape, and leave more complex axis naming/labeling to subclasses on the python side. As an example, upon creating a new dimension in an array, numpy assigns that dimension a semi-unique id, and this tuple could be used in __array_finalize__. For example my __array_finalize__ could look like: def __array_finalize__(self,obj): if hasattr(obj,'axesdata'): for axesid in self.axes: if axesid in obj.axes: self.axesdata[axesid] = obj.axesdata[axesid] This would cover a lot more situations and lead to much simpler code since the work required on the C side would be minimal, but still allow robust and custom tracking and propagation of axes information. Subclasses that tap into this data would react to the result of numpy operations vs. having to predict/anticipate. For example, my __getitem__, relying on the __array_finalize__ above, could look like: def __getitem__(self,idxs): filtered_idxs,transposed_axes= self.idx_axes(idxs) array = self.transpose(transposed_axes) return array[filtered_idxs] Not shown is how much simpler and robust the code for idx_axes would then be. I estimate it would go from 130 loc to < 20 loc. Sorry for the extra long e-mail, -Craig

1 0

object scalars
by Johann Hibschman July 12, 2011

July 12, 2011

Is there any way to wrap a sequence (in particular a python list) as a numpy object scalar, without it being promoted to an object array? In particular, np.object_([1, 2]).shape == (2,) np.array([1,2], dtype='O').shape == (2,) while I want some_call([1,2]).shape = () Thanks, Johann

2 2

New arrays in 1.6 not always C-contiguous
by Jens Jørgen Mortensen July 12, 2011

July 12, 2011

Hi! With numpy 1.5, I get this: >>> a = np.ones((2, 2)) >>> (2 * a.T).strides (16, 8) With 1.6, I get this: >>> (2 * a.T).strides (8, 16) So, this means I can't count on new arrays being C-contiguous any more. I guess there is a good reason for this. Anyway, I just thought I would mention it here - maybe I'm not the only one making this assumption when passing ndarrays to C code. Jens Jørgen

6 10