Mailman 3 March 2012 - NumPy-Discussion

More SPARC pain
by Matthew Brett March 8, 2012

March 8, 2012

Hi, I found this test caused a bus error on current trunk: <pre> import numpy as np from StringIO import StringIO as BytesIO from numpy.testing import assert_array_equal def test_2d_buf(): dtt = np.complex64 arr = np.arange(10, dtype=dtt) # 2D array arr2 = np.reshape(arr, (2, 5)) # Fortran write followed by (C or F) read caused bus error data_str = arr2.tostring('F') data_back = np.ndarray(arr2.shape, arr2.dtype, buffer=data_str, order='F') assert_array_equal(arr2, data_back) </pre> gdb run gives ... test_me3.test_2d_buf ... Program received signal SIGBUS, Bus error. 0xf78f5458 in _aligned_strided_to_contig_size8 ( dst=0xdc0e08 "\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\313\373\373\373\373", dst_stride=8, src=0xcdfc44 "", src_stride=16, N=5, __NPY_UNUSED_TAGGEDsrc_itemsize=8, __NPY_UNUSED_TAGGEDdata=0x0) at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:137 137 (*((@type@ *)dst)) = @swap@@elsize@(*((@type@ *)src)); Debug log attached. Shall I make an issue? Best, Matthew

1 1

use for missing (ignored) data?
by Neal Becker March 7, 2012

March 7, 2012

I'm wondering what is the use for the ignored data feature? I can use: A[valid_A_indexes] = whatever to process only the 'non-ignored' portions of A. So at least some simple cases of ignored data are already supported without introducing a new type. OTOH: w = A[valid_A_indexes] will copy A's data, and subsequent use of w[:] = something will not update A. Is this the reason for wanting the ignored data feature?

4 5

Github key audit.
by Charles R Harris March 7, 2012

March 7, 2012

Hi All, Many here have probably received the message from github about push/pull access being blocked until you have auditied your ssh keys. To generate a key fingerprint on fedora, I did the following: $charris@f16 ~$ ssh-keygen -l -f .ssh/id_dsa.pub I don't how this looks for those of you using windows. The compromise looks to be a 'whitehat' attack telling the rails developers to "FIX THE G*DD*MNED PROBLEM". More here, Github compromized<http://lwn.net/Articles/485162/> . Chuck

2 1

checking for c compiler during build
by Skipper Seabold March 7, 2012

March 7, 2012

Is there a way to use numpy.distuils to programmatically check for a C compiler at build time in a platform independent way?

1 1

Fixing PyArray_Descr flags member size, ABI vs pickling issue
by David Cournapeau March 7, 2012

March 7, 2012

Hi, This is following the discussion on bug http://projects.scipy.org/numpy/ticket/2017 Essentially, there is a discrepency between the actual type of the flags member in the dtype C structure (char) and how it is declared in the descriptor table (T_INT). The problem is that we are damned if we fix it, damned if we are not: - fixing T_INT to T_BYTE. flag in python is now fixed, but it breaks pickled numpy arrays - not fixing it means that the value is broken at the python level. Changing the flag to be an actual int would break the ABI. I currently have this: https://github.com/numpy/numpy/pull/228 Which one is the more appropriate ? Maybe a deprecation warning for 1.7 and change it in 1.8 ? Note that we can fix the hash issue with either solution, David

3 6

C++ Example
by Luis Pedro Coelho March 6, 2012

March 6, 2012

Hi, I sort of missed the big C++ discussion, but I'd like to give some examples of how writing code can become much simpler if you are based on C++. This is from my mahotas package, which has a thin C++ wrapper around numpy's C API https://github.com/luispedro/mahotas/blob/master/mahotas/_morph.cpp and it implements multi-type greyscale erosion. // numpy::aligned_array wraps PyArrayObject* template<typename T> void erode(numpy::aligned_array<T> res, numpy::aligned_array<T> array, numpy::aligned_array<T> Bc) { // Release the GIL using RAII gil_release nogil; const int N = res.size(); typename numpy::aligned_array<T>::iterator iter = array.begin(); // this is adapted from scipy.ndimage. // it implements the convolution-like filtering. filter_iterator<T> filter(res.raw_array(), Bc.raw_array(), EXTEND_NEAREST, is_bool(T())); const int N2 = filter.size(); T* rpos = res.data(); for (int i = 0; i != N; ++i, ++rpos, filter.iterate_both(iter)) { T value = std::numeric_limits<T>::max(); for (int j = 0; j != N2; ++j) { T arr_val = T(); filter.retrieve(iter, j, arr_val); value = std::min<T>(value, erode_sub(arr_val, filter[j])); } *rpos = value; } } If you compare this with the equivalent scipy.ndimage function, which is very good C code (but mostly write-only—in fact, ndimage has not been maintainable because it is so hard [at least for me, I've tried]): int NI_BinaryErosion(PyArrayObject* input, PyArrayObject* strct, PyArrayObject* mask, PyArrayObject* output, int bdr_value, npy_intp *origins, int invert, int center_is_true, int* changed, NI_CoordinateList **coordinate_list) { npy_intp struct_size = 0, *offsets = NULL, size, *oo, jj; npy_intp ssize, block_size = 0, *current = NULL, border_flag_value; int kk, true, false, msk_value; NI_Iterator ii, io, mi; NI_FilterIterator fi; Bool *ps, out = 0; char *pi, *po, *pm = NULL; NI_CoordinateBlock *block = NULL; ps = (Bool*)PyArray_DATA(strct); ssize = 1; for(kk = 0; kk < strct->nd; kk++) ssize *= strct->dimensions[kk]; for(jj = 0; jj < ssize; jj++) if (ps[jj]) ++struct_size; if (mask) { if (!NI_InitPointIterator(mask, &mi)) return 0; pm = (void *)PyArray_DATA(mask); } /* calculate the filter offsets: */ if (!NI_InitFilterOffsets(input, ps, strct->dimensions, origins, NI_EXTEND_CONSTANT, &offsets, &border_flag_value, NULL)) goto exit; /* initialize input element iterator: */ if (!NI_InitPointIterator(input, &ii)) goto exit; /* initialize output element iterator: */ if (!NI_InitPointIterator(output, &io)) goto exit; /* initialize filter iterator: */ if (!NI_InitFilterIterator(input->nd, strct->dimensions, struct_size, input->dimensions, origins, &fi)) goto exit; /* get data pointers an size: */ pi = (void *)PyArray_DATA(input); po = (void *)PyArray_DATA(output); size = 1; for(kk = 0; kk < input->nd; kk++) size *= input->dimensions[kk]; if (invert) { bdr_value = bdr_value ? 0 : 1; true = 0; false = 1; } else { bdr_value = bdr_value ? 1 : 0; true = 1; false = 0; } if (coordinate_list) { block_size = LIST_SIZE / input->nd / sizeof(int); if (block_size < 1) block_size = 1; if (block_size > size) block_size = size; *coordinate_list = NI_InitCoordinateList(block_size, input->nd); if (!*coordinate_list) goto exit; } /* iterator over the elements: */ oo = offsets; *changed = 0; msk_value = 1; for(jj = 0; jj < size; jj++) { int pchange = 0; if (mask) { switch(mask->descr->type_num) { CASE_GET_MASK(msk_value, pm, Bool); CASE_GET_MASK(msk_value, pm, UInt8); CASE_GET_MASK(msk_value, pm, UInt16); CASE_GET_MASK(msk_value, pm, UInt32); #if HAS_UINT64 CASE_GET_MASK(msk_value, pm, UInt64); #endif CASE_GET_MASK(msk_value, pm, Int8); CASE_GET_MASK(msk_value, pm, Int16); CASE_GET_MASK(msk_value, pm, Int32); CASE_GET_MASK(msk_value, pm, Int64); CASE_GET_MASK(msk_value, pm, Float32); CASE_GET_MASK(msk_value, pm, Float64); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); return 0; } } switch (input->descr->type_num) { CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Bool, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt8, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt16, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); #if HAS_UINT64 CASE_NI_ERODE_POINT(pi, out, oo, struct_size, UInt64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); #endif CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int8, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int16, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Int64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Float32, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); CASE_NI_ERODE_POINT(pi, out, oo, struct_size, Float64, msk_value, bdr_value, border_flag_value, center_is_true, true, false, pchange); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); goto exit; } switch (output->descr->type_num) { CASE_OUTPUT(po, out, Bool); CASE_OUTPUT(po, out, UInt8); CASE_OUTPUT(po, out, UInt16); CASE_OUTPUT(po, out, UInt32); #if HAS_UINT64 CASE_OUTPUT(po, out, UInt64); #endif CASE_OUTPUT(po, out, Int8); CASE_OUTPUT(po, out, Int16); CASE_OUTPUT(po, out, Int32); CASE_OUTPUT(po, out, Int64); CASE_OUTPUT(po, out, Float32); CASE_OUTPUT(po, out, Float64); default: PyErr_SetString(PyExc_RuntimeError, "data type not supported"); goto exit; } if (pchange) { *changed = 1; if (coordinate_list) { if (block == NULL || block->size == block_size) { block = NI_CoordinateListAddBlock(*coordinate_list); current = block->coordinates; } for(kk = 0; kk < input->nd; kk++) *current++ = ii.coordinates[kk]; block->size++; } } if (mask) { NI_FILTER_NEXT3(fi, ii, io, mi, oo, pi, po, pm); } else { NI_FILTER_NEXT2(fi, ii, io, oo, pi, po); } } exit: if (offsets) free(offsets); if (PyErr_Occurred()) { if (coordinate_list) { NI_FreeCoordinateList(*coordinate_list); *coordinate_list = NULL; } return 0; } else { return 1; } return PyErr_Occurred() ? 0 : 1; } HTH -- Luis Pedro Coelho | Institute for Molecular Medicine | http://luispedro.org

10 17

Bus error for Debian / SPARC on current trunk
by Matthew Brett March 6, 2012

March 6, 2012

Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Cheers, Matthew

5 18

Data cube optimization for combination
by Jose Miguel Ibáñez March 6, 2012

March 6, 2012

Hello everyone, does anyone know of an efficient implementation (maybe using numpy.where statement) of the next code for data cube (3d array) combining ? import numpy as np def combine( ) cube = np.random.rand(32,2048,2048) result = np.zeros([2048,2048], np.float32) for ii in range(2048): for jj in range(2048): result[, ii, jj] = np.sqrt((cube[:,ii, jj])).sum() It takes long time to run, however, >> result = np.median(cube,0) only around one second ! where is the point ? any suggestions ? Thanks !

3 2

dtype comparison, hash
by Andreas Kloeckner March 5, 2012

March 5, 2012

Hi all, Two questions: - Are dtypes supposed to be comparable (i.e. implement '==', '!=')? - Are dtypes supposed to be hashable? PyCUDA and PyOpenCL assume both in a few places, but at least hashability doesn't seem to be true. (If so, __hash__ should be implemented to throw an error. If not, we found a bug in the hash implementation.) Thanks! Andreas

3 6

all elements equal
by Neal Becker March 5, 2012

March 5, 2012

What is a simple, efficient way to determine if all elements in an array (in my case, 1D) are equal? How about close?

8 14

verbose output when running python script?
by Chao YUE March 5, 2012

March 5, 2012

Dear all, Sorry this is not the good place to ask but I think there must be someone who has done this before. Is there any way to see the execution of python script line by line as the verbose model of shell script? this can be done either in ipython or by running the python script in a shell script? thanks to all, cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 4

test errors on deprecation/runtime warnings
by Ralf Gommers March 4, 2012

March 4, 2012

Hi, Last week we merged https://github.com/numpy/numpy/pull/201, which causes DeprecationWarning's and RuntimeWarning's to be converted to errors if they occur when running the test suite. The purpose of that is to make sure that code that still uses other deprecated code (or code that for some reason generates warnings) to be cleaned up. In principle this is a good idea IMHO, but after merging we quickly found some problems with failing tests in scipy. Because this potentially affects any other projects or users that use the numpy NoseTester test runner, here's a proposal to deal with this issue: - make this behavior configurable by a keyword in `NoseTester.__init__()` - default to raising an error in numpy master - when making a branch for release, immediately set the default to not raise. Do this not only for the 1.7 release, but for any future release (at least until the oldest numpy version still in use has the keyword). The reason for this is that otherwise a new numpy release will trigger test failures in older releases of scipy or other packages. The pros: - We find issues that otherwise get ignored (see the issues Christoph just found when compiling with MSVC). - We're forced to clean up code when submitting PRs, instead of letting the warnings accumulate and having to deal with them just before release time. The con: - You may see test errors if you run a released version of scipy with a development version of numpy. Opinions? Concerns? Ralf

2 3

copy mask from existing masked array?
by Chao YUE March 4, 2012

March 4, 2012

Dear all, I have a matrix with dimension of (360,720) but with all global data. I have another land-sea mask matrix with only 2 unique values in it (land=1, sea=-1). So I can easily create transform the second array to a masked array. the problem is, how can I quickly transform the first one to a masked array using the same mask as the land-sea mask array? I hope my question is clear. If not, here is an example: In [93]: a=np.arange(10).reshape(2,5) In [95]: a=np.ma.masked_equal(a,2 In [96]: a=np.ma.masked_equal(a,8) In [97]: a Out[97]: masked_array(data = [[0 1 -- 3 4] [5 6 7 -- 9]], mask = [[False False True False False] [False False False True False]], fill_value = 8) In [100]: b=np.random.normal(0,2,size=(2,5)) I want to convert b to a masked array using exactly the same mask as a. thanks to all, cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 1

Floating point "close" function?
by Joe Kington March 3, 2012

March 3, 2012

Is there a numpy function for testing floating point equality that returns a boolean array? I'm aware of np.allclose, but I need a boolean array. Properly handling NaN's and Inf's (as allclose does) would be a nice bonus. I wrote the function below to do this, but I suspect there's a method in numpy that I missed. import numpy as np def close(a, b, rtol=1.e-5, atol=1.e-8, check_invalid=True): """Similar to numpy.allclose, but returns a boolean array. See numpy.allclose for an explanation of *rtol* and *atol*.""" def within_tol(x, y, atol, rtol): return np.less_equal(np.abs(x-y), atol + rtol * np.abs(y)) x = np.array(a, copy=False) y = np.array(b, copy=False) if not check_invalid: return within_tol(x, y, atol, rtol) xfin = np.isfinite(x) yfin = np.isfinite(y) if np.all(xfin) and np.all(yfin): return within_tol(x, y, atol, rtol) else: # Avoid subtraction with infinite/nan values... cond = np.zeros(np.broadcast(x, y).shape, dtype=np.bool) mask = xfin & yfin cond[mask] = within_tol(x[mask], y[mask], atol, rtol) # Inf and -Inf equality... cond[~mask] = (x[~mask] == y[~mask]) # NaN equality... cond[np.isnan(x) & np.isnan(y)] = True return cond # A few quick tests... assert np.any(close(0.300001, np.array([0.1, 0.2, 0.3, 0.4]))) x = np.array([0.1, np.nan, np.inf, -np.inf]) y = np.array([0.1000001, np.nan, np.inf, -np.inf]) assert np.all(close(x, y)) x = np.array([0.1, 0.2, np.inf]) y = np.array([0.101, np.nan, 0.2]) assert not np.all(close(x, y)) Thanks, -Joe

5 15

Where did Numpy-svn come from?
by Charles R Harris March 2, 2012

March 2, 2012

Github commits now send out mail: reply-toGitHub <noreply(a)github.com> tonumpy-svn(a)scipy.org This is a recent change, did we change something? I liked getting mail directly from github better. Chuck

1 0

Possible roadmap addendum: building better text file readers
by Wes McKinney March 2, 2012

March 2, 2012

dear all, I haven't read all 180 e-mails, but I didn't see this on Travis's initial list. All of the existing flat file reading solutions I have seen are not suitable for many applications, and they compare very unfavorably to tools present in other languages, like R. Here are some of the main issues I see: - Memory usage: creating millions of Python objects when reading a large file results in horrendously bad memory utilization, which the Python interpreter is loathe to return to the operating system. Any solution using the CSV module (like pandas's parsers-- which are a lot faster than anything else I know of in Python) suffers from this problem because the data come out boxed in tuples of PyObjects. Try loading a 1,000,000 x 20 CSV file into a structured array using np.genfromtxt or into a DataFrame using pandas.read_csv and you will immediately see the problem. R, by contrast, uses very little memory. - Performance: post-processing of Python objects results in poor performance. Also, for the actual parsing, anything regular expression based (like the loadtable effort over the summer, all apologies to those who worked on it), is doomed to failure. I think having a tool with a high degree of compatibility and intelligence for parsing unruly small files does make sense though, but it's not appropriate for large, well-behaved files. - Need to "factorize": as soon as there is an enum dtype in NumPy, we will want to enable the file parsers for structured arrays and DataFrame to be able to "factorize" / convert to enum certain columns (for example, all string columns) during the parsing process, and not afterward. This is very important for enabling fast groupby on large datasets and reducing unnecessary memory usage up front (imagine a column with a million values, with only 10 unique values occurring). This would be trivial to implement using a C hash table implementation like khash.h To be clear: I'm going to do this eventually whether or not it happens in NumPy because it's an existing problem for heavy pandas users. I see no reason why the code can't emit structured arrays, too, so we might as well have a common library component that I can use in pandas and specialize to the DataFrame internal structure. It seems clear to me that this work needs to be done at the lowest level possible, probably all in C (or C++?) or maybe Cython plus C utilities. If anyone wants to get involved in this particular problem right now, let me know! best, Wes

21 57

[Numpy] quadruple precision
by Paweł Biernat March 2, 2012

March 2, 2012

I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding.

8 10

Jaccard & Hamming Problem
by Zayd YAKOUBI March 1, 2012

March 1, 2012

Hello, I use the similarity measure "Jaccard" and "Hamming" of pckage Scipy.spacial.cdist (Python) in a clustering context, I applied to given typs of real and integer (0.6 0.2 1.7 May 8 ). They gave good results. But I just know that they normally only applies to binary data. The function of these two similarity measures are not specified in the documentation: http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.… . Does anyone of you can help me find these functions? Thank you in advance Regards, Zayd

3 3

Numpy fitting
by Pierre Barthelemy March 1, 2012

March 1, 2012

Dear all, i am writing a program for data analysis. One of the functions of this program gives the possibility to fit the functions. I therefore use the recipe described in : http://www.scipy.org/Cookbook/FittingData<http://www.scipy.org/Cookbook/FittingData> under the section "Simplifying the syntax". The code looks like this: class Parameter: def __init__(self, value): self.value = value self.fixed=False def set(self, value): if not self.fixed: self.value = value def __call__(self): return self.value def fit(function, parameters, y, x = None): def f(params): i = 0 for p in parameters: p.set(params[i]) i += 1 return y - function(x) if x is None: x = arange(y.shape[0]) p = [param() for param in parameters] out=optimize.leastsq(f, p, full_output=1) One thing that i would like to know is how can i get the error on the parameters ? From what i understood from the "Cookbook" page, and from the scipy manual ( http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.…), the second argument returned by the leastsq function gives access to these errors. std_error=std(y-function(x)) param_error=sqrt(diagonal(out[1])*std_error) The param_errors that i get in this case are extremely small. Much smaller than what i expected, and much smaller than what i can get fitting the function with matlab. So i guess i made an error here. Can someone tell me how i should do to retrieve the parameter errors ? Bests, Pierre

3 3

Numpy interpolate: cut through 2D matrix
by Pierre Barthelemy March 1, 2012

March 1, 2012

Hello, for a data analysis tool i am programming, i need to plot a cut through a 2D graph. I then have a 2D array, and the indices start=(start_x,start_y) and stop=(stop_x,stop_y) that are the position of the starting point and stop point of the cut. The code i programmed is placed on the bottom. This code returns only value existing in the original array: if the cut should pass between the column index i and column index i+1, it returns anyway the value at column index i. Is it possible to use the numpy.interpolate library to make such that when passing between i and i+1, the function returns an interpolation of the graph between the points [row,column] and [row,column+1] ? def cut_matrix(array,start,stop,shift=0): ''' Draws a cut through a 2D array, between the positions start=(row,column) and stop=(row,column) ''' n_row=array.shape[0] n_col=array.shape[1] if abs(start[1]-stop[1])>abs(start[0]-stop[0]): if stop[1]<start[1]: start,stop= stop,start col_index=arange(start[1],stop[1]+1).astype(int) row_index=round_(linspace(start[0],stop[0],len(col_index))).astype(int)+int(shift) row=(linspace(start[0],stop[0],len(col_index))) else: if stop[0]<start[0]: start,stop= stop,start row_index=arange(start[0],stop[0]+1).astype(int) col_index=round_(linspace(start[1],stop[1],len(row_index))).astype(int)+int(shift) if max(col_index)>n_col or min(col_index)<0: print 'Error: column index not in range' raise IndexError return if max(row_index)>n_row or min(row_index)<0: print 'Error: row index not in range' raise IndexError return return array[row_index,col_index]

2 1

Re: [Numpy-discussion] Proposed Roadmap Overview
by Richard Hattersley March 1, 2012

March 1, 2012

+1 on the NEP guideline As part of a team building a scientific analysis library, I'm attempting to understand the current state of NumPy development and its likely future (with a view to contributing if appropriate). The proposed NEP process would make that a whole lot easier. And if nothing else, it would reduce the chance of me posting questions about topics that had already been discussed/decided! Without the process the NEPs become another potential source of confusion and mixed messages. On 1 March 2012 03:02, Travis Oliphant wrote: > I Would like to hear the opinions of others on that point, > but yes, I think that is an appropriate procedure. > > Travis > > -- > Travis Oliphant > (on a mobile) > 512-826-7480 > > > On Feb 29, 2012, at 10:54 AM, Matthew Brett > <matthew.brett(a)gmail.com> wrote: > > > Hi, > > > > On Wed, Feb 29, 2012 at 1:46 AM, Travis Oliphant > <travis(a)continuum.io> wrote: > >> We already use the NEP process for such decisions. This > discussion came from simply from the *idea* of writing such a NEP. > >> > >> Nothing has been decided. Only opinions have been shared > that might influence the NEP. This is all pretty premature, > though --- migration to C++ features on a trial branch is > some months away were it to happen. > > > > Fernando can correct me if I'm wrong, but I think he was asking a > > governance question. That is: would you (as BDF$N) consider the > > following guideline: > > > > "As a condition for accepting significant changes to Numpy, for each > > significant change, there will be a NEP. The NEP shall follow the > > same model as the Python PEPs - that is - there will be a summary of > > the changes, the issues arising, the for / against opinions and > > alternatives offered. There will usually be a draft implementation. > > The NEP will contain the resolution of the discussion as it > relates to > > the code" > > > > For example, the masked array NEP, although very > substantial, contains > > little discussion of the controversy arising, or the intended > > resolution of the controversy: > > > > > https://github.com/numpy/numpy/blob/3f685a1a990f7b6e5149c80b52 > 436fb4207e49f5/doc/neps/missing-data.rst > > > > I mean, although it is useful, it is not in the form of a PEP, as > > Fernando has described it. > > > > Would you accept extending the guidelines to the NEP format? > > > > Best, > > > > Matthew > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > >

1 0

IPython 0.12 just entered Debian Testing
by Pierre Haessig March 1, 2012

March 1, 2012

Hi, Just to start the new month on a light & happy topic : IPython 0.12 has entered Debian Testing ! -- Pierre (I'm not at all involved in the process that enabled IPython make its way to Testing. I've been watching this quite closely however. I "suspect" there was a decent amount of work on packaging all the dependencies to make this possible and I'm thankful to the guys who did it)

2 1

Special matrices with structure?
by Jaakko Luttinen March 1, 2012

March 1, 2012

Hi! I was wondering whether it would be easy/possible/reasonable to have classes for arrays that have special structure in order to use less memory and speed up some computations? For instance: - symmetric matrix could be stored in almost half the memory required by a non-symmetric matrix - diagonal matrix only needs to store the diagonal vector - Toeplitz matrix only needs to store one or two vectors - sparse matrix only needs to store non-zero elements (some implementations in scipy.sparse) - and so on If such classes were implemented, it would be nice if they worked with numpy functions (dot, diag, ...) and operations (+, *, +=, ...) easily. I believe this has been discussed before but google didn't help a lot.. Regards, Jaakko

2 3

Proposed Roadmap Overview
by Travis Oliphant Feb. 29, 2012

Feb. 29, 2012

Mark Wiebe and I have been discussing off and on (as well as talking with Charles) a good way forward to balance two competing desires: * addition of new features that are needed in NumPy * improving the code-base generally and moving towards a more maintainable NumPy I know there are load voices for just focusing on the second of these and avoiding the first until we have finished that. I recognize the need to improve the code base, but I will also be pushing for improvements to the feature-set and user experience in the process. As a result, I am proposing a rough outline for releases over the next year: * NumPy 1.7 to come out as soon as the serious bugs can be eliminated. Bryan, Francesc, Mark, and I are able to help triage some of those. * NumPy 1.8 to come out in July which will have as many ABI-compatible feature enhancements as we can add while improving test coverage and code cleanup. I will post to this list more details of what we plan to address with it later. Included for possible inclusion are: * resolving the NA/missing-data issues * finishing group-by * incorporating the start of label arrays * incorporating a meta-object * a few new dtypes (variable-length string, varialbe-length unicode and an enum type) * adding ufunc support for flexible dtypes and possibly structured arrays * allowing generalized ufuncs to work on more kinds of arrays besides just contiguous * improving the ability for NumPy to receive JIT-generated function pointers for ufuncs and other calculation opportunities * adding "filters" to Input and Output * simple computed fields for dtypes * accepting a Data-Type specification as a class or JSON file * work towards improving the dtype-addition mechanism * re-factoring of code so that it can compile with a C++ compiler and be minimally dependent on Python data-structures. * NumPy 2.0 to come out in January of 2013. Mark Wiebe and I will post to this list a document that explains some of it's proposed features and enhancements. I won't steal his thunder for some of the things he is working on. If there are code issues people would like to see addressed, it would be a great time to speak up and/or propose something that you would like to see. In general NumPy 1.8 will have new features that need to be explored in order that NumPy 2.0 has enough code "experience" in order to be as useful as possible. I recognize that NumPy 1.8 has quite a few proposed features. These have been building up and are the big reason I've committed so many resources to NumPy. The feature-list did not just come out of my head. They are the result of talking and interacting with many NumPy users and watching the code get used (and not used) in the real world. This will be a faster pace of development. But, all of this will be in the open. If the NumPy 2.0 schedule is too aggressive, then we will have a NumPy 1.9 release in order to allow features to come out. Thanks, -Travis

37 196