Mailman 3 February 2006 - NumPy-Discussion

Re: [Numpy-discussion] Tests fail in SVN r2165
by Sasha 24 Feb '06

24 Feb '06

I've located the culprit. Once I revert numpy/core/__init__.py to 2164, tests pass. The change appears to postpone the import of NumpyTest: > svn diff -r 2164:2165 numpy/core/__init__.py Index: numpy/core/__init__.py =================================================================== --- numpy/core/__init__.py (revision 2164) +++ numpy/core/__init__.py (revision 2165) @@ -25,5 +25,6 @@ __all__ += rec.__all__ __all__ += char.__all__ -from numpy.testing import ScipyTest -test = ScipyTest().test +def test(level=1, verbosity=1): + from numpy.testing import NumpyTest + return NumpyTest().test(level, verbosity) <http://projects.scipy.org/scipy/numpy/changeset/2165#file2> I don't understand why this causes problems on my system, but it does. Pearu? On 2/24/06, Sasha <ndarray(a)mac.com> wrote: > I've reverted to r2164 and everything is back to normal. This > suggests that r2165 changes are at fault and not my setup. That > change deals with imports and the failure that I see happens during > import. > > I've tried NUMPY_IMPORT_DEBUG=1 and got this: > > Executing 'import testing' ... ok > Executing 'from testing import ScipyTest' ... ok > Executing 'from testing import NumpyTest' ... ok > Executing 'import core' ... ok > Executing 'from core import *' ... ok > Executing 'import random' ... ok > Executing 'from random import rand' ... ok > Executing 'from random import randn' ... ok > Executing 'import lib' ... ok > Executing 'from lib import *' ... ok > Executing 'import linalg' ... ok > Executing 'import dft' ... ok > Executing 'from dft import fft' ... ok > Executing 'from dft import ifft' ... ok > Found 4 tests for numpy.lib.getlimits > Found 30 tests for numpy.core.numerictypes > Found 11 tests for numpy.core.umath > Found 8 tests for numpy.lib.arraysetops > Found 42 tests for numpy.lib.type_check > Found 83 tests for numpy.core.multiarray > Found 3 tests for numpy.dft.helper > Found 27 tests for numpy.core.ma > Traceback (most recent call last): > ... > > Any suggestions for further diagnostic? > > > On 2/24/06, Albert Strasheim <fullung(a)gmail.com> wrote: > > Works for me. Just built revision 2165 on Windows with Visual Studio. > > > > Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] > > > > In [1]: import numpy > > > > In [2]: numpy.test() > > Found 83 tests for numpy.core.multiarray > > Found 3 tests for numpy.lib.getlimits > > Found 11 tests for numpy.core.umath > > Found 8 tests for numpy.lib.arraysetops > > Found 42 tests for numpy.lib.type_check > > Found 4 tests for numpy.lib.index_tricks > > Found 30 tests for numpy.core.numerictypes > > Found 27 tests for numpy.core.ma > > Found 1 tests for numpy.core.oldnumeric > > Found 9 tests for numpy.lib.twodim_base > > Found 8 tests for numpy.core.defmatrix > > Found 1 tests for numpy.lib.ufunclike > > Found 33 tests for numpy.lib.function_base > > Found 3 tests for numpy.dft.helper > > Found 1 tests for numpy.lib.polynomial > > Found 6 tests for numpy.core.records > > Found 14 tests for numpy.core.numeric > > Found 44 tests for numpy.lib.shape_base > > Found 0 tests for __main__ > > ............................................................................ > > ........................ > > ............................................................................ > > ........................ > > ............................................................................ > > ........................ > > ............................ > > ---------------------------------------------------------------------- > > Ran 328 tests in 0.781s > > > > OK > > Out[2]: <unittest.TextTestRunner object at 0x00F56190> > > > > In [3]: numpy.__version__ > > Out[3]: '0.9.6.2165' > > > > Cheers > > > > Albert > > > > >

2 2

unsuccessful install on 64bit machine
by Jose Borreguero 24 Feb '06

24 Feb '06

While installing numpy under a non-conventional directory: *python setup.py install --prefix==/gpfs1/active/jose/code/python *I get two classes of warnings: (1) blas_mkl_info: /tmp/numpy-0.9.5/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE (and others warnings similar to this) (2)numpy.core - nothing done with h_files= ['build/src/numpy/core/src/scalartypes.inc', 'build/src/numpy/core/src/arraytypes.inc', 'build/src/numpy/core/config.h', 'build/src/numpy/core/__multiarray_api.h'] (and others warnings similar to this) I created a very simple site.cfg file under numpy/distutils, which goes like this [blas] library_dirs = /usr/lib64 [lapack] library_dirs = /usr/lib64 I have no Atlas library installed in the system. My compiler: gcc version 3.4.4 20050721 (Red Hat 3.4.4-2) (I also have pg compilers) After setup.py has run, I have no */gpfs1/active/jose/code/python/lib64/python2.3/site-packages/numpy *created. So numpy is not installed. Any ideas, please? -- Jose M. Borreguero jmborr(a)gatech.edu, www.borreguero.com phone: 404 407 8980 Fax: 404 385 7478 Center for the Study of Systems Biology, School of Biology, Georgia Institute of Technology, 250 14St NW, Atlanta GA 30318

2 1

Build on Windows: system_info and site.cfg
by Albert Strasheim 24 Feb '06

24 Feb '06

Hello all I'm trying to build NumPy on Windows using optimized ATLAS and CLAPACK libraries. The system_info functions are currently doing things like: p = self.combine_paths(lib_dir, 'lib'+l+ext) and if self.search_static_first: exts = ['.a',so_ext] else: exts = [so_ext,'.a'] if sys.platform=='cygwin': exts.append('.dll.a') which generally isn't going work on Windows, where library names don't always start with 'lib' and always end in '.lib', for static libraries and DLL import libraries. It might be worth it to have users explicitly specify their build settings (compiler, flags, BLAS libraries, LAPACK libraries, etc.) in a site.cfg instead of trying to detect all the possible combinations of C and/or FORTRAN BLAS, LAPACK, FFTW and whatnot. A few default configurations could be provided for common configurations. Any thoughts? Is anyone interested in fixing the build on Windows? Thanks! Albert

1 0

Shapes and sizes
by Albert Strasheim 24 Feb '06

24 Feb '06

Hello all I'm trying to write a function that takes a scalar, 0d-, 1d- or 2d array and returns a scalar or an array obtained by performing some computations on this input argument. When the output of such a function is the same size as input, one can do the following to preallocate the output array: def f(a): arra = asarray(a) if a.ndim > 2: raise ValueError('invalid dimensions') b = empty_like(arra) # do some operations on arra and put the values in b return b However, when the output array only depends on the size of a, but isn't exactly the same, things seem to get more complicated. Consider a function that operates on the rows of a (or the "row" if a is 1d or 0d or a scalar). For every row of length n, the function might return a row of length (n/2 + 1) if n is even or a row of length (n + 1)/2 if n is odd. Thus, depending on the shape of a, empty must be called with one of three different shapes. def outsize(n): if n % 2 == 0: return n/2+1 return (n+1)/2 if arra.ndim == 0: b = empty(()) elif arra.ndim == 1: b = empty((outsize(arra.shape[0]),)) else: b = empty((arra.shape[0],outsize(arra.shape[1])) To me this seems like a lot of code that can could be simpler if there was a function to get the size of an array that returns a useful value even if a particular dimension doesn't exist, much like MATLAB's size, where one can write: b = zeros(size(a,1),outsize(size(a,2))); function [m]=outsize(n) if mod(n, 2) == 0; m = n/2+1; else m =(n+1)/2; end and still have it work with scalars, 1d arrays and 2d arrays. Even if there were such a function for NumPy, this still leaves the problem that the output is going to have the wrong shape for 0d and 1d arrays, specifically (1,1) and (outsize(n),1) instead of () and (outsize(n),). This problem is solved in MATLAB where there is no distinction between 1 and [1]. How do you guys deal with this problem in your functions? Regars Albert

3 4

Who is maintaining the SVN repository?
by Sasha 24 Feb '06

24 Feb '06

Where should I report problems with the SVN repository? I've tried to edit the log entry of my commit and got this: > svn propedit -r 2166 --revprop svn:log svn: DAV request failed; it's possible that the repository's pre-revprop-change hook either failed or is non-existent svn: At least one property change failed; repository is unchanged Is that right?

1 0

Tests fail in SVN r2165
by Sasha 24 Feb '06

24 Feb '06

Usin python 2.4.2 and latest SVN version I get the following: > python Python 2.4.2 (#3, Jan 13 2006, 13:52:39) [GCC 3.4.4 20050721 (Red Hat 3.4.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Found 3 tests for numpy.lib.getlimits Found 30 tests for numpy.core.numerictypes Found 11 tests for numpy.core.umath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 83 tests for numpy.core.multiarray Found 3 tests for numpy.dft.helper Found 27 tests for numpy.core.ma Found 1 tests for numpy.core.oldnumeric Traceback (most recent call last): File "<stdin>", line 1, in ? File "/.../lib/python2.4/site-packages/numpy/__init__.py", line 46, in test return NumpyTest('numpy').test(level, verbosity) File "/.../lib/python2.4/site-packages/numpy/testing/numpytest.py", line 422, in test suites.extend(self._get_module_tests(module, abs(level), verbosity)) File "/.../lib/python2.4/site-packages/numpy/testing/numpytest.py", line 355, in _get_module_tests self.warn('FAILURE importing tests for %s' % (mstr(module))) File "/.../lib/python2.4/site-packages/numpy/testing/numpytest.py", line 469, in warn from numpy.distutils.misc_util import yellow_text File "/.../lib/python2.4/site-packages/numpy/distutils/__init__.py", line 5, in ? import ccompiler File "/.../lib/python2.4/site-packages/numpy/distutils/ccompiler.py", line 12, in ? from exec_command import exec_command File "/.../lib/python2.4/site-packages/numpy/distutils/exec_command.py", line 54, in ? import tempfile File "/.../lib/python2.4/tempfile.py", line 33, in ? from random import Random as _Random ImportError: cannot import name Random Does anyone see the same? It looks like it fails while loading test_misc_util, but running that alone works: > python lib/python2.4/site-packages/numpy/distutils/tests/test_misc_util.py Found 4 tests for __main__ .... ---------------------------------------------------------------------- Ran 4 tests in 0.001s

1 0

Re: [Numpy-discussion] Visual Studio build broken due to array types changes
by Albert Strasheim 24 Feb '06

24 Feb '06

Hello Thanks! Looks like this solves the problems with Visual Studio. I did a python setup.py clean between builds. Shouldn't this clean up the source tree sufficiently? Regards Albert On 2/24/06, Travis Oliphant <oliphant.travis(a)ieee.org> wrote: > Albert Strasheim wrote: > > >Hello all > > > > > Please remove the build directory and try with a fresh build. > > It seems many of your errors come from a new version of the C-API. > > Thanks, > > -Travis

1 0

Simple NumPy-compatible vector w/C++ & SWIG?
by Magnus Lie Hetland 24 Feb '06

24 Feb '06

Hi! I'm working on a data structure library where one of the element types most likely will be a vector type (i.e., points in a multidimensional space, with the dimensionality set by the user). In the data structure (which is disk-based) I have work with raw bytes that I'd like to copy around as little as possible. The library itself is (being) written in C++, but I'm wrapping it with SWIG so I can drive and test it with Python. It seems to me that something NumPy-compatible might be the best choice for the vector type, but I'm not sure how I should do that. I've been thinking about simply implementing a minimal compatibility layer for the NumPy Array Interface; is it then possible to construct a NumPy array using this custom array, and get full support for the various array operations without actually copying the data? And: Any ideas on what to do on the C++ side? Is there any code/ library out there for a vector-thing that works well in C++ *and* that has wrapping code for NumPy? (I know the STL vector is wrapped to a Python list by default -- I'm just thinking that including those things in the equation would lead to lots of copied data...) -- Magnus Lie Hetland http://hetland.org

3 2

RE: [Numpy-discussion] algorithm, optimization, or other problem?
by Nadav Horesh 24 Feb '06

24 Feb '06

You may get a significant boost by replacing the line: w=w+ eta * (y*x - y**2*w) with w *= 1.0 - eta*y*y w += eta*y*x I ran a test on a similar expression and got 5 fold speed increase. The dot() function runs faster if you compile with dotblas. Nadav. -----Original Message----- From: numpy-discussion-admin(a)lists.sourceforge.net on behalf of Bruce Southey Sent: Tue 21-Feb-06 17:15 To: Brian Blais Cc: python-list(a)python.org; numpy-discussion(a)lists.sourceforge.net; scipy-user(a)scipy.net Subject: Re: [Numpy-discussion] algorithm, optimization, or other problem? Hi, In the current version, note that Y is scalar so replace the squaring (Y**2) with Y*Y as you do in the dohebb function. On my system without blas etc removing the squaring removes a few seconds (16.28 to 12.4). It did not seem to help factorizing Y. Also, eta and tau are constants so define them only once as scalars outside the loops and do the division outside the loop. It only saves about 0.2 seconds but these add up. The inner loop probably can be vectorized because it is just vector operations on a matrix. You are just computing over the ith dimension of X. I think that you could be able to find the matrix version on the net. Regards Bruce On 2/21/06, Brian Blais <bblais(a)bryant.edu> wrote: > Hello, > > I am trying to translate some Matlab/mex code to Python, for doing neural > simulations. This application is definitely computing-time limited, and I need to > optimize at least one inner loop of the code, or perhaps even rethink the algorithm. > The procedure is very simple, after initializing any variables: > > 1) select a random input vector, which I will call "x". right now I have it as an > array, and I choose columns from that array randomly. in other cases, I may need to > take an image, select a patch, and then make that a column vector. > > 2) calculate an output value, which is the dot product of the "x" and a weight > vector, "w", so > > y=dot(x,w) > > 3) modify the weight vector based on a matrix equation, like: > > w=w+ eta * (y*x - y**2*w) > ^ > | > +---- learning rate constant > > 4) repeat steps 1-3 many times > > I've organized it like: > > for e in 100: # outer loop > for i in 1000: # inner loop > (steps 1-3) > > display things. > > so that the bulk of the computation is in the inner loop, and is amenable to > converting to a faster language. This is my issue: > > straight python, in the example posted below for 250000 inner-loop steps, takes 20 > seconds for each outer-loop step. I tried Pyrex, which should work very fast on such > a problem, takes about 8.5 seconds per outer-loop step. The same code as a C-mex > file in matlab takes 1.5 seconds per outer-loop step. > > Given the huge difference between the Pyrex and the Mex, I feel that there is > something I am doing wrong, because the C-code for both should run comparably. > Perhaps the approach is wrong? I'm willing to take any suggestions! I don't mind > coding some in C, but the Python API seemed a bit challenging to me. > > One note: I am using the Numeric package, not numpy, only because I want to be able > to use the Enthought version for Windows. I develop on Linux, and haven't had a > chance to see if I can compile numpy using the Enthought Python for Windows. > > If there is anything else anyone needs to know, I'll post it. I put the main script, > and a dohebb.pyx code below. > > > thanks! > > Brian Blais > > -- > ----------------- > > bblais(a)bryant.edu > http://web.bryant.edu/~bblais > > > > > # Main script: > > from dohebb import * > import pylab as p > from Numeric import * > from RandomArray import * > import time > > x=random((100,1000)) # 1000 input vectors > > numpats=x.shape[0] > w=random((numpats,1)); > > th=random((1,1)) > > params={} > params['eta']=0.001; > params['tau']=100.0; > old_mx=0; > for e in range(100): > > rnd=randint(0,numpats,250000) > t1=time.time() > if 0: # straight python > for i in range(len(rnd)): > pat=rnd[i] > xx=reshape(x[:,pat],(1,-1)) > y=matrixmultiply(xx,w) > w=w+params['eta']*(y*transpose(xx)-y**2*w); > th=th+(1.0/params['tau'])*(y**2-th); > else: # pyrex > dohebb(params,w,th,x,rnd) > print time.time()-t1 > > > p.plot(w,'o-') > p.xlabel('weights') > p.show() > > > #============================================= > > # dohebb.pyx > > cdef extern from "Numeric/arrayobject.h": > > struct PyArray_Descr: > int type_num, elsize > char type > > ctypedef class Numeric.ArrayType [object PyArrayObject]: > cdef char *data > cdef int nd > cdef int *dimensions, *strides > cdef object base > cdef PyArray_Descr *descr > cdef int flags > > > def dohebb(params,ArrayType w,ArrayType th,ArrayType X,ArrayType rnd): > > > cdef int num_iterations > cdef int num_inputs > cdef int offset > cdef double *wp,*xp,*thp > cdef int *rndp > cdef double eta,tau > > eta=params['eta'] # learning rate > tau=params['tau'] # used for variance estimate > > cdef double y > num_iterations=rnd.dimensions[0] > num_inputs=w.dimensions[0] > > # get the pointers > wp=<double *>w.data > xp=<double *>X.data > rndp=<int *>rnd.data > thp=<double *>th.data > > for it from 0 <= it < num_iterations: > > offset=rndp[it]*num_inputs > > # calculate the output > y=0.0 > for i from 0 <= i < num_inputs: > y=y+wp[i]*xp[i+offset] > > # change in the weights > for i from 0 <= i < num_inputs: > wp[i]=wp[i]+eta*(y*xp[i+offset] - y*y*wp[i]) > > # estimate the variance > thp[0]=thp[0]+(1.0/tau)*(y**2-thp[0]) > > > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=k&kid3432&bid#0486&dat1642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

5 4

Visual Studio build broken due to array types changes
by Albert Strasheim 24 Feb '06

24 Feb '06

Hello all Recent changes to multiarraymodule.c has broken the build with Visual Studio for revision 2164 of numpy. I'd fix the problem, but I'm still a bit new to the NumPy sources. I've attached the build log in case someone can come up with a quick fix. Meanwhile, the SciPy build is also broken. Is Visual Studio considered to be a supported compiler, or is Mingw's GCC the only supported compiler for Windows builds? Regards Albert

1 2