Mailman 3 March 2009 - NumPy-Discussion

Re: [Numpy-discussion] Built Lapack, Atlas from source.... now numpy.linalg.eig() hangs at 100% CPU
by Chris Colbert March 28, 2009

March 28, 2009

David, The log was too big for the list, so I sent it to your email address directly. Chris 2009/3/27 Chris Colbert <sccolbert(a)gmail.com> > David, > > The log is attached. > > Thanks for giving me the bash command. I would have never figured that one > out.... > > Chris > > > On Fri, Mar 27, 2009 at 11:23 AM, David Cournapeau < > david(a)ar.media.kyoto-u.ac.jp> wrote: > >> Chris Colbert wrote: >> > Atlas 3.8.3 and Lapack 3.1.1 >> >> Hm... I am afraid I don't see what may cause this problem. Could you >> rebuild numpy from scratch and give us the log ? >> >> rm -rf build && python setup.py build &> build.log >> >> David >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >

5 30

Is it ok to include GPL scripts in the numpy *repository* ?
by David Cournapeau March 28, 2009

March 28, 2009

Hi, To build the numpy .dmg mac os x installer, I use a script from the adium project, which uses applescript and some mac os x black magic. The script seems to be GPL, as adium itself: http://trac.adiumx.com/browser/trunk/Release For now, I keep the build scripts separately from the svn repository, but it would be more practical if everything was together. As far as I understand, the GPL does not apply to the output of some build scripts, and as such, nothing would be "tainted" by the GPL - is this right ? Would it be problematic to put those in the svn repo ? cheers, David

7 6

How to tell whether I am using 32 bit or 64 bit numpy?
by John Reid March 28, 2009

March 28, 2009

I imagine I'm using 64 bit numpy as I made a vanilla install from recent source on a 64 bit box but how can I tell for sure? I have some problems creating large arrays. In [29]: a=numpy.empty((1024, 1024, 1024), dtype=int8) works just fine In [30]: a=numpy.empty((1024, 1024, 2048), dtype=int8) gives me the dimensions too large error: ValueError: dimensions too large. In [31]: a=numpy.empty((1024, 1024, 2047), dtype=int8) gives me a memory error: MemoryError: How can I create these large arrays? Do I need to make sure I have a 64 bit python? How do I do that? Thanks in advance, John.

5 17

SciPy 2009 Conference will be Aug. 18-23
by Jarrod Millman March 27, 2009

March 27, 2009

The subject says it all. Over the next few days, we will be updating the conference website with additional information. So if you are interested, please keep an eye on: http://conference.scipy.org/ Jarrod

1 0

A few more questions about build doc
by David Cournapeau March 27, 2009

March 27, 2009

Hi, I spent the whole evening on automating our whole release process on supported platforms. I am almost there, but I have a few relatively minor annoyances related to doc: - Is it ok to build the pdf doc using LANG=C ? If I run sphinx-build without setting LANG=C, I got some weird latex errors at the latex->pdf stage, which I am reluctant to track down :) - I modified doc/source/conf.py such as the reported numpy version is exactly the one used to build the doc. Am I right that building the numpy documentation requires numpy to be installed (for autodoc and co), or is this a wrong assumption ? I've realized this after the change, but I can of course revert it if that's a problem, cheers, David

3 2

Re: [Numpy-discussion] Normalization of ifft
by Joe Harrington March 27, 2009

March 27, 2009

Hi Lutz, > what you say is of course correct, but I am wondering if there is a > mistake in the user guide (p. 180 of > http://numpy.scipy.org/numpybook.pdf): according to the expressions in > the user guide, both fft and ifft are not normalized. The > implementation if ifft, on the other hand, has the additional 1/n > factor, consistent with the online documentation. You are looking at Travis Oliphant's book Guide to NumPy, last updated 2006. The routine docstrings (the "help" text) are now maintained by the community at docs.scipy.org. They get synced into the source repository relatively often. They are also the sources to the routine docs presented in the NumPy Reference Guide, also available at that site. Travis has freed his original book and large parts of it (e.g., the C API docs) are now being incorporated into the actively-maintained manuals at docs.scipy.org. Please go there for the latest docs. You'll find that the fft section gives the 1/n formula when discussing ifft. All, I can see where Lutz got the impression that Guide to Numpy was the doc to read. The descriptions of books on both numpy.scipy.org and docs.scipy.org do give that impression. But, Guide is "mature" because its scope was (necessarily) limited. At this point the Reference Guide, while not complete because of its more-ambitious scope, has every docstring in Guide to Numpy, and has substantially more complete and accurate pages for a large number of functions, and much additional text that Guide to Numpy does not have. I haven't checked in detail but much of the rest of Guide to Numpy is now included in the Reference Guide. Would it be ok to put some words on both sites to the effect that the RG is the place to go for routine, class, and module docs, or (possibly) just the place to go, period? I don't want to downplay Travis's contribution; Guide was *very* useful and it lives on in the work descended from it. But, I think the audience that should read it first is relatively limited now. --jh--

2 1

Re: [Numpy-discussion] Changeset 6729
by Charles R Harris March 27, 2009

March 27, 2009

Hi Stephan, You can actually change these functions to use fmax.reduce/fmin.reduce and get about a 50% speedup. Chuck

2 3

Behavior of numpy.random.exponential
by Yves Frederix March 27, 2009

March 27, 2009

Hi, I noticed a problem with numpy.random.exponential. Apparently, the samples generated by numpy.random.exponential(scale=scale) follow the distribution f(x)=1/scale*exp(-x/scale) (and not f(x)=scale*exp(-x*scale) as stated by the docstring). The script below illustrates this. -- import numpy as N import pylab as pl print N.__version__ pl.figure() lamda = 2. noise_modulus = N.random.exponential(scale=lamda,\ size=(100000,)) #noise_modulus = -N.log(N.random.uniform(size=(100000,)))/lamda # this works y_hist, x_hist = N.histogram(noise_modulus, bins=51,\ normed=True, new=True) x_pl = N.linspace(0, x_hist.max()) pl.semilogy(x_hist[0:-1], y_hist, label='Empirical, lambda=%s' % lamda) pl.semilogy(x_pl, lamda * N.exp(-x_pl*lamda), ':', \ label='exact, lambda=%s' % lamda) pl.semilogy(x_pl, 1./lamda * N.exp(-x_pl*1./lamda), ':', \ label='exact, lambda=1/%s' % lamda) pl.legend(loc='best') pl.show() -- Could this be a bug? I also checked with the latest svn version: In [1]: import numpy; numpy.__version__ Out[1]: '1.4.0.dev6731' Best, YVES

3 3

Built Lapack, Atlas from source.... now numpy.linalg.eig() hangs at 100% CPU
by Chris Colbert March 27, 2009

March 27, 2009

Hey Everyone, I built Lapack and Atlas from source last night on a C2D running 32-bit Linux Mint 6. I ran 'make check' and 'make time' on the lapack build, and ran the dynamic LU decomp test on atlas. Both packages checked out fine. Then, I built numpy and scipy against them using the appropriate flags in site.cfg for the parallel thread atlas libraries. This seems to have worked properly as numpy.dot() utilizes both cores at 100% on very large arrays. I have also done id(numpy.dot) and id(numpy.core.multiarray.dot) and verified that the two ids are different. So I believe the build went properly. The problem I am having now is that numpy.linalg.eig (and the eig functions in scipy) hang at 100% CPU and never returns (no matter the array size). Numpy.test() hung as well, I'm assuming for this same reason. I have included the configurations below. Any idea what would cause this? Thanks! Chris Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import scipy >>> numpy.show_config() atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] language = f77 include_dirs = ['/usr/local/atlas/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] define_macros = [('NO_ATLAS_INFO', 2)] language = c include_dirs = ['/usr/local/atlas/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] language = c include_dirs = ['/usr/local/atlas/include'] lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] define_macros = [('NO_ATLAS_INFO', 2)] language = f77 include_dirs = ['/usr/local/atlas/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE >>> scipy.show_config() umfpack_info: NOT AVAILABLE atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] language = f77 include_dirs = ['/usr/local/atlas/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] language = c include_dirs = ['/usr/local/atlas/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] language = c include_dirs = ['/usr/local/atlas/include'] lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/local/atlas/lib'] define_macros = [('NO_ATLAS_INFO', 2)] language = f77 include_dirs = ['/usr/local/atlas/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE

3 11

Re: [Numpy-discussion] Numpy v1.3.0b1 on Linux w/ Intel compilers - unknown file type
by Christian Marquardt March 27, 2009

March 27, 2009

Hi David, > I *guess* that the compiler command line does not work with your > changes, and that distutils got confused, and fails somewhere later > (or sooner, who knows). Without actually seeing the errors you got, it > is difficult to know more - but I would make sure the command line > arguments are ok instead of focusing on the .src error, > > cheers, > > David I'n not sure if I understand... The compiler options I have changed seem to work (and installation without the "build_clib --compiler=intel" option to setup.py works fine with them). To be sure I've compiled numpy from the distribution tar file without any patches. With python setup.py config --compiler=intel \ config_fc --fcompiler=intel \ build_ext --compiler=intel build everything compiles fine (and builds the internal lapack, as I haven't given the MKL paths, and have no other lapack / blas installed). With python setup.py config --compiler=intel \ config_fc --fcompiler=intel \ build_clib --compiler=intel \ build_ext --compiler=intel build the attempt to build fails (complete output is below). The python installation I use is also build with the Intel icc compiler; so it does pick up that compiler by default. Maybe something is going wrong in the implementation of build_clib in the numpy distutils? Where would I search for that in the code? Many thanks, Chris. ----<snip, snip>-------------------------------------------------------- MEDEA /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1>python setup.py config --compiler=intel config_fc --fcompiler=intel build_clib --compiler=intel build_ext --compiler=intel build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /opt/intel/mkl/10.0.2.018/lib/32 NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/apps/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /opt/apps/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1383: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /opt/apps/lib libraries blas not found in /usr/local/lib libraries blas not found in /usr/lib NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1392: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) blas_src_info: NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1395: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) NOT AVAILABLE lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /opt/intel/mkl/10.0.2.018/lib/32 NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/apps/lib libraries lapack_atlas not found in /opt/apps/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /opt/apps/lib libraries lapack_atlas not found in /opt/apps/lib libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1290: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /opt/apps/lib libraries lapack not found in /usr/local/lib libraries lapack not found in /usr/lib NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1301: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. warnings.warn(LapackNotFoundError.__doc__) lapack_src_info: NOT AVAILABLE /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1/numpy/distutils/system_info.py:1304: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. warnings.warn(LapackSrcNotFoundError.__doc__) NOT AVAILABLE running config running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_clib Found executable /opt/intel/cc/10.1.018/bin/icc Could not locate executable ecc customize IntelCCompiler customize IntelCCompiler using build_clib building 'npymath' library compiling C sources C compiler: icc error: unknown file type '.src' (from 'numpy/core/src/npy_math.c.src') MEDEA /home/marq/src/python/04_science/01_numpy/numpy-1.3.0b1>

1 0