Mailman 3 February 2009 - NumPy-Discussion

Re: [Numpy-discussion] Sphinx custom extension mess, and patches
by Gael Varoquaux Feb. 23, 2009

Feb. 23, 2009

On Tue, Feb 17, 2009 at 12:17:17AM +0100, Georg Brandl wrote: > I'm all for it. In the case of autosummary, I'm guilty of not getting it > in sooner. This will change soon. In other cases, I don't even know of > the extension, probably because those who write it deem it as too > project-specific to contribute it. > I don't ask for too much if an extension is contributed, so by all means > do at least post about your extensions! I am not blaming anyone, just pointing out a non ideal situation. It has already improved a lot with the matplotlib guys and the scipy guys merging some changes in extensions and publishing the extensions in an importable part of their source tree. It is true that I'll be happier when I will be able to import the only_directive, and the auto_summary from sphinx :). Thanks for your great work, Gaël

3 2

Import NumPy in Self-defined function script
by Tim Feb. 23, 2009

Feb. 23, 2009

Hi, I am defining a function in file "trainer.py". It requires a module called numpy as following: [code] from numpy import * import string def trainer(train_seqs): nul_dict = {"A":0,"C":1,"G":2,"T":3}# nucleotide to index of array init_prob = zeros(4,double)# initial prob at 1st col tran_prob = zeros([5,4,4],double)# transition prob indexed by ( col, nul in previous col, nul in current col) for line in train_seqs: # counts if line != '\n': z = line.split() z = z[0] init_prob[nul_dict[z[0]]] += 1 for i in range(0,5): tran_prob[i][nul_dict[z[i]]][nul_dict[z[i+1]]] += 1 init_prob /= init_prob.sum() # normalize transition initial prob [/code] Another script "controller.py" calls this function as: [code]#! /usr/bin/env python # from numpy import * import string import trainer import pdb; pdb.set_trace() ESE_file = file("./ESE.txt",'r') ESE_seqs = ESE_file.readlines() ESE_file.close() ESE_model=trainer.trainer(ESE_seqs) [/code] When I try to tun "controller.py", I get the error: [quote] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "controller.py", line 12, in <module> ESE_model=trainer.trainer(ESE_seqs) File "trainer.py", line 4, in trainer NameError: global name 'zeros' is not defined [/quote] I think zeros belongs to Module numpy, and I import it in my function file "trainer.py". So I was wonering what's the mistake I am making? Thanks for help! -Tim

4 6

How to generate equivalent "random" numbers in matlab and numpy?
by Nicolas Pinto Feb. 23, 2009

Feb. 23, 2009

Dear all, I'd like to generate equivalent sequences of 'random' numbers in matlab and numpy, is there any way I can do that? I tried to fix the seed (see below) but it doesn't work. # numpy In [29]: np.random.seed(1); np.random.permutation(5)+1 Out[29]: array([3, 2, 5, 1, 4]) % matlab >> rand('seed', 1); randperm(5) ans = 4 3 5 2 1 Thanks for your time. Best regards, -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto

2 1

small suggestion for numpy.testing utils
by Darren Dale Feb. 23, 2009

Feb. 23, 2009

Hello, I am using numpy's assert_array_equal and assert_array_almost_equal to unit test my physical quantities package. I made a single minor change to assert_array_compare that I think might make these functions more useful to ndarray subclasses, and thought maybe they could be useful to numpy itself. I tried applying this diff to numpy and running the test suite, and instead of 9 known failures I got 1 known failure, 11 skips, 2 errors and 2 failures. Perhaps it is possible that by not forcing the input arrays to be ndarray instances, some additional numpy features are exposed. Thanks, Darren $ svn diff Index: numpy/testing/utils.py =================================================================== --- numpy/testing/utils.py (revision 6370) +++ numpy/testing/utils.py (working copy) @@ -240,9 +240,9 @@ def assert_array_compare(comparison, x, y, err_msg='', verbose=True, header=''): - from numpy.core import asarray, isnan, any - x = asarray(x) - y = asarray(y) + from numpy.core import array, isnan, any + x = array(x, copy=False, subok=True) + y = array(y, copy=False, subok=True) def isnumber(x): return x.dtype.char in '?bhilqpBHILQPfdgFDG'

3 10

RFR: 991 - Make savez able to write ZIP64 files
by Stéfan van der Walt Feb. 22, 2009

Feb. 22, 2009

Hi all, Please review the patch attached to http://scipy.org/scipy/numpy/ticket/991 which enables ZIP64 extensions when saving and loading zipped data under Python >= 2.5 Thanks, Stéfan

2 4

RFR: 995 - numpy.load can't handle gzip file handles
by Stéfan van der Walt Feb. 22, 2009

Feb. 22, 2009

Hi, Based on an example on Effbot, I implemented a workaround for reverse seeking in gzip files. I need someone with Python 2.4 to review: http://www.scipy.org/scipy/numpy/ticket/995 Thanks! Stéfan

3 6

Efficient numpy slicing for a "sliding window approach".
by Nicolas Pinto Feb. 21, 2009

Feb. 21, 2009

Dear all, I'm trying to optimize the code below and I was wondering if there is an efficient method that could reduce the numpy slicing overheard without going with cython. Is there anyway I could use mgrid to get a matrix with all my "windows" and then do a large matrix multiply instead? Any idea? Thanks in advance. Best regards, -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto ======================================== import numpy as np from numpy import dot arrh, arrw, arrd = 480,640,96 arr = np.random.randn(arrh, arrw, arrd).astype("float32") stride = 16 winh, winw, wind = 128,64,96 limit = 100 clas_w = np.random.randn(8,4,96).astype("float32").ravel() @profile def func(): nh, nw = arrh-winh+1, arrw-winw+1 rngh = np.arange(nh) rngw = np.arange(nw) np.random.shuffle(rngh) np.random.shuffle(rngw) rngh = rngh[:limit] rngw = rngw[:limit] rngh = np.sort(rngh) rngw = np.sort(rngw) resps = np.empty((nh,nw), dtype="float32") for j in rngh: for i in rngw: win = arr[j:j+winh:stride, i:i+winw:stride].ravel() resp = dot(win, clas_w) resps[j,i] = resp resps = resps.ravel() # ... func() ======================================== % python kernprof.py -l -v sliding_win.py Wrote profile results to sliding_win.py.lprof Timer unit: 1e-06 s File: sliding_win.py Function: func at line 14 Total time: 0.224315 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 14 @profile 15 def func(): 16 1 7 7.0 0.0 nh, nw = arrh-winh+1, arrw-winw+1 17 1 14 14.0 0.0 rngh = np.arange(nh) 18 1 4 4.0 0.0 rngw = np.arange(nw) 19 20 1 89 89.0 0.0 np.random.shuffle(rngh) 21 1 124 124.0 0.1 np.random.shuffle(rngw) 22 23 1 3 3.0 0.0 rngh = rngh[:limit] 24 1 2 2.0 0.0 rngw = rngw[:limit] 25 26 1 39 39.0 0.0 rngh = np.sort(rngh) 27 1 19 19.0 0.0 rngw = np.sort(rngw) 28 1 24 24.0 0.0 resps = np.empty((nh,nw), dtype="float32") 29 101 127 1.3 0.1 for j in rngh: 30 10100 12636 1.3 5.6 for i in rngw: 31 10000 118190 11.8 52.7 win = arr[j:j+winh:stride, i:i+winw:stride].ravel() 32 10000 73854 7.4 32.9 resp = dot(win, clas_w) 33 10000 19180 1.9 8.6 resps[j,i] = resp 34 35 1 3 3.0 0.0 resps = resps.ravel()

3 4

Multiple draw text or paste image using numpy
by Frank Peacock Feb. 20, 2009

Feb. 20, 2009

Hello I am familiar with the fromarray function in the pil library Image module to plot pixels. Is it possible to do the same for drawtext for text or paste for images without looping with lists? Thanks Frank

2 1

inplace dot products (Robert Kern) (Re: Numpy-discussion Digest, Vol 29, Issue 69)
by David Henderson Feb. 20, 2009

Feb. 20, 2009

Hello all, I've been toying with the idea of an extended precision accumulator for the dot product written in numpy/core/src/multiarraymodule.c Once the modification is being performed, there is no reason not to allow the specification of an output array. The functions that exist now: The routine cumsum already exists with an accumulator of a specific type. The output location is an optional argument. numpy.cumsum(a, axis=None, dtype=None, out=None) Various forms of inner product (dot product) also exist: numpy.tensordot(a, b, axes=2) Returns the tensor dot product for (ndim >= 1) arrays along an axes. The first element of the sequence determines the axis or axes in `a` to sum over, and the second element in `axes` argument sequence determines the axis or axes in `b` to sum over. numpy.vdot(a,b) Returns the dot product of a and b for scalars and vectors of floating point and complex types. The first argument, a, is conjugated. numpy.innerproduct(a,b) Returns the inner product of a and b for arrays of floating point types. Like the generic NumPy equivalent the product sum is over the last dimension of a and b. NB: The first argument is not conjugated. The proposed extensions: numpy.tensordot(a, b, axes=2, dtype=None, out=None) numpy.vdot(a,b, dtype=None, out=None) numpy.innerproduct(a,b, dtype=None, out=None) (found in numpy/core/ src/multiarraymodule.c) Is this so difficult an extension to implement? Is there a better functional specification to be made? Granted, these routines do not exist in blas. Therefore, they wouldn't be speedy, at least without blas extensions. But... the key routines to make the generic extension already exist in numpy without blas. from numeric.py: When Numpy is built with an accelerated BLAS like ATLAS, these functions are replaced to make use of the faster implementations. The faster implementations only affect float32, float64, complex64, and complex128 arrays. Furthermore, the BLAS API only includes matrix-matrix, matrix-vector, and vector-vector products. Products of arrays with larger dimensionalities use the built in functions and are not accelerated. Looking at numpy/core/src/multiarraymodule.c: The existing routine that allows an accumulator specification and output array: static PyObject * PyArray_Sum(PyArrayObject *self, int axis, int rtype, PyArrayObject *out) The routine that would be modified and its new function prototypes: static PyObject * PyArray_InnerProduct(PyObject *op1, PyObject *op2, int rtype, PyArrayObject *out) Other C code that might be modified to support the new functionality is in arraytypes.inc.src. Routines here perform a dot product along one dimension for various argument dtypes and accumulator dtypes. It looks like the accumulator type is encoded in the #out signature. As I read it, there is no change needed to this source as all the different accumulator types are present in the output signature. /**begin repeat #name=BYTE, UBYTE, SHORT, USHORT, INT, UINT, LONG, ULONG, LONGLONG, ULONGLONG, FLOAT, DOUBLE, LONGDOUBLE# #type= byte, ubyte, short, ushort, int, uint, long, ulong, longlong, ulonglong, float, double, longdouble# #out= long, ulong, long, ulong, long, ulong, long, ulong, longlong, ulonglong, float, double, longdouble# */ static void @name@_dot(char *ip1, intp is1, char *ip2, intp is2, char *op, intp n, So far, the changes dont seem to be too messy. Am I missing anything here? David On Feb 20, 2009, at 10:00 AM, numpy-discussion-request(a)scipy.org wrote: > > Message: 2 > Date: Fri, 20 Feb 2009 09:52:03 -0600 > From: Robert Kern <robert.kern(a)gmail.com> > Subject: Re: [Numpy-discussion] inplace dot products > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <3d375d730902200752k6b71efc9gb1e0ac1d260d65d(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Feb 20, 2009 at 05:18, David Warde-Farley > <dwf(a)cs.toronto.edu> wrote: >> Hi Olivier, >> >> There was this idea posted on the Scipy-user list a while back: >> >> http://projects.scipy.org/pipermail/scipy-user/2008-August/ >> 017954.html >> >> but it doesn't look like he got anywhere with it, or even got a >> response. >> >> I just tried it and I observe the same behaviour. A quick look at the >> SciPy sources tells me there is something fishy. >> >> subroutine >> < >> tchar=s,d,c,z>gemm >> (m,n,k,alpha,a,b,beta,c,trans_a,trans_b,lda,ka,ldb,kb) >> ! c = gemm(alpha,a,b,beta=0,c=0,trans_a=0,trans_b=0,overwrite_c=0) >> ! Calculate C <- alpha * op(A) * op(B) + beta * C >> >> I don't read Fortran very well, but it seems to me as though the >> Fortran prototype doesn't match the python prototype. > > What do you mean? Based on the rest of the argument information in > that block, f2py creates the Python prototype. For example, all of the > m,n,k,lda,ka,ldb,kb dimensions are found from the input arrays > themselves, optional arguments are given defaults, etc. > >> I'll poke around a little more, but in summary: there's no numpy- >> sanctioned way to specify an output array for a dot(), AFAIK. This is >> a bit of an annoyance, I agree, though I seem to remember Robert Kern >> offering a fairly compelling argument why it's hard. I just don't >> know > > I believe I was only talking about why it would be hard to use a > higher-precision accumulator. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > > > ------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > End of Numpy-discussion Digest, Vol 29, Issue 69 > ************************************************

1 0

Initialize numpy array with other numpy arrays
by Frank Peacock Feb. 20, 2009

Feb. 20, 2009

Hello I would like to know whether I can do the following in some other way as this fails with setting an array with a sequence on each of the colour arrays: h,w=720,410 img = ones((h,w,3), uint8)*255 img[ngridn,ngride]=(ncolour[0],ncolour[1],ncolour[2]) pilImage = Image.fromarray(img, 'RGB') where ngridn,ngride,ncolour[m] are all 1-D with the same dimension (effectively ngridn and ngride values map within the bounds of the image) The following works fine: h,w=720,410 img = ones((h,w,3), uint8)*255 img[ngridn,ngride]=(255,0,0) pilImage = Image.fromarray(img, 'RGB') I would prefer not to use indices to solve the problem like the following: (as it is a lot slower) h,w=720,410 img = ones((h,w,3), uint8)*255 for n in range(len(gride)): img[ngridn[n],ngride[n]]=(ncolour[0][n],ncolour[1][n],ncolour[n][2]) pilImage = Image.fromarray(img, 'RGB') It is possible to not use indices and use numpy functions instead. Thanks Frank

3 3