Mailman 3 October 2012 - NumPy-Discussion

Re: [Numpy-discussion] Welcome to the "NumPy-Discussion" mailing list (Digest mode)
by Jared O'Connell 14 Oct '12

14 Oct '12

Hello! I am trying to build numpy 1.6.2 with python 2.7 in Ubuntu 12.04 and wish to link against serial ATLAS rather than the threaded version. I have modified site.cfg accordingly by uncommenting out the following lines: [blas_opt] libraries = f77blas, cblas, atlas [lapack_opt] libraries = lapack, f77blas, cblas, atlas However when I run python setup.py build it still links to the threaded versions eg. python setup.py build Running from numpy source directory.F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib Setting PTATLAS=ATLAS customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize LaheyFCompiler Could not locate executable lf95 customize PGroupFCompiler Could not locate executable pgfortran customize AbsoftFCompiler Could not locate executable f90 customize NAGFCompiler Found executable /usr/bin/f95 customize VastFCompiler customize CompaqFCompiler Could not locate executable fort customize IntelItaniumFCompiler Could not locate executable efort Could not locate executable efc customize IntelEM64TFCompiler customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib/atlas-base -lptf77blas -lptcblas -latlas -o _configtest ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011: UNAME : Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 393216 F77 : gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack not found in /usr/lib/atlas-base numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] ... Any ideas? best, Jared

1 0

ANN: WinPython v2.7.3.0
by Pierre Raybaut 12 Oct '12

12 Oct '12

Hi all, I'm pleased to introduce my new contribution to the Python community: WinPython. WinPython v2.7.3.0 has been released and is available for 32-bit and 64-bit Windows platforms: http://code.google.com/p/winpython/ WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel). Enjoy! -Pierre

2 1

Reminder: Last day of early registration for PyData NYC conference!
by Peter Wang 12 Oct '12

12 Oct '12

Hi everyone, Just a friendly reminder that today is the final day of early registration for the PyData NYC conference later this month! We have a fantastic lineup of talks and workshops on a variety of topics related to Python for data analysis, including topics that are hard to find at other conferences (e.g. practical perspectives on Python and Hadoop, using Python and R, etc.). http://nyc2012.pydata.org/ Use the discount code "numpy" for a 20% discount off of registration! We are also looking for sponsors. We are proud to feature D. E. Shaw, JP Morgan, and Appnexus as gold sponsors. If your company or organization would like some visibility in front of a few hundred Python data hackers, please visit our sponsor information page: http://nyc2012.pydata.org/sponsors/becoming/ Thanks, Peter

1 0

Fwd: [numpy] When accessing MaskedArray rows, always return an mvoid object (#483)
by Nathaniel Smith 11 Oct '12

11 Oct '12

This PR changes the behaviour of masked record arrays. The rationale makes sense to me, but I don't actually use this functionality. Any masked array users want to chime in? -n ---------- Forwarded message ---------- From: "Thomas Robitaille" <notifications(a)github.com> Date: 11 Oct 2012 19:35 Subject: [numpy] When accessing MaskedArray rows, always return an mvoid object (#483) To: "numpy/numpy" <numpy(a)noreply.github.com> The current behavior of MaskedArray when accessing a row object is that if all masks are False, the row is an np.void object, and if any are set to True, it is a np.ma.core.mvoid object. The issue with this is that users can access/modify masks for rows that already have at least one mask set to True, but not for rows that have no masks set. For example: In [1]: a = ma.array(zip([1,2,3]), mask=[0,1,0], dtype=[('x', int)]) In [2]: a[0].mask --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-7-2221dd3fce8c> in <module>() ----> 1 a[0].mask AttributeError: 'numpy.void' object has no attribute 'mask' In [3]: a[1].mask Out[3]: (True,) There is no reason why a row should behave differently whether any values are masked or not, so this PR defaults to always returning an mvoid object, otherwise mask values cannot be set for rows that start off with no mask values set. (Of course, the present implementation in Numpy also has a performance impact - for arrays with many fields, each call to a row has to flatten the mask of the whole record just to check what type to return, which is inefficient. With this PR, row access should be faster for masked arrays. But this is secondary.) ------------------------------ You can merge this Pull Request by running: git pull https://github.com/astrofrog/numpy fix-masked-getitem Or view, comment on, or merge it at: https://github.com/numpy/numpy/pull/483 Commit Summary - When accessing MaskedArray rows, always return an mvoid object File Changes - *M* numpy/ma/core.py (10) Patch Links - https://github.com/numpy/numpy/pull/483.patch - https://github.com/numpy/numpy/pull/483.diff — Reply to this email directly or view it on GitHub<https://github.com/numpy/numpy/pull/483>.

1 0

set_printoptions precision and single floats
by Dan Goodman 07 Oct '12

07 Oct '12

Hi, numpy.set_printoptions(precision=...) doesn't affect single floats, even if they are numpy floats rather than Python floats. Is this a bug or is there some reason for this behaviour? I ask because I have a class that derives from numpy.float64 and adds some extra information, and I'd like to be able to control the precision. I could fix it to use the precision set by numpy.set_printoptions, but then it would be inconsistent with how numpy itself handles precision. Thoughts? Dan

4 4

Re: [Numpy-discussion] How to copy data from a C array to a numpy array efficiently?
by Jianbao Tao 07 Oct '12

07 Oct '12

Dag, Thanks for your tip. I didn't know that I could use a numpy array directly. I tried that out, and it works like a charm. :-) Jonathan, Glad that you mentioned spacepy. I actually tried spacepy before I decided to write my own wrapper. However, on the one hand, I can't install spacepy on my mac (10.8 OS X). On the other hand, spacepy has too much stuff that I don't need, and I don't think I can rip spacepy.pycdf off the whole package. So, I eventually decided to write my own wrapper. However, as a space physics scientist, I'd like to see spacepy become better and more generic. For the moment, it seems to me that spacepy is more tailored to suit LANL's needs, especially about radiation belt research, than being a generic tool for space physics. Cheers, Jianbao

1 0

How to copy data from a C array to a numpy array efficiently?
by Jianbao Tao 07 Oct '12

07 Oct '12

Hi, I am developing a Python wrapper of the NASA CDF C library in Cython. I have got a working version of it now, but it is slower than the counterpart in IDL. For loading the same file, mine takes about 400 ms, whereas the IDL version takes about 290 ms. The main overhead in my code is caused by a for-loop of element-by-element copying. Here is the relevant code in cython: #-------------------------------- code ----------------------------------------------------- #-- double realData = numpy.zeros(lenData, np_dtype) dblEntry = <double *>malloc(lenData * sizeof(double)) status = CDFlib( SELECT_, zVAR_RECCOUNT_, numRecs, NULL_) status = CDFlib( GET_, zVAR_HYPERDATA_, dblEntry, NULL_) for ii in range(lenData): realData[ii] = dblEntry[ii] realData.shape = np_shape free(dblEntry) #--------------------------------- end of code ------------------------------------------- The time-consuming for-loop is highlighted in red. If I change range(lenData) to range(lenData/2), the cpu time will be down from 400 ms to 230 ms for the case I mentioned above. Because the element-by-element copying for-loop seems pretty naive to me, I am wondering if there is a better way to copy data from the C array, dblEntry, to the numpy array, realData. I tried the numpy C API PyArray_NewFromDescr with flag NPY_ENSURECOPY, but didn't get any luck. On the one hand, the flag above didn't seem to work as I expected, because I got memory deallocation failure error messages when I quitted ipython, where I tested my code, which I don't get if I use the naive for-loop. On the other hand, I can't figure out how to use PyArray_NewFromDescr correctly because the loaded data I got were not correct. Anyway, here is how I used PyArray_NewFromDescr: #----------------------------------------- code ------------------------------------------ cdef np.npy_intp dims[1] dims[0] = lenData realData = PyArray_NewFromDescr(numpy.ndarray, numpy.dtype(np_dtype), 1, dims, NULL, <void *>dblEntry, NPY_CARRAY|NPY_ENSURECOPY, None) free(dblEntry) #-------------------------------------- end of code -------------------------------------- BTW, it can be compiled successfully by cython, in case you are wondering if the code had all the necessary pieces, Thank you very much for reading. :-) Cheers, Jianbao

3 3

dtype '|S0' not understood
by David Warde-Farley 06 Oct '12

06 Oct '12

Howdy, It seems it's possible using e.g. In [25]: dtype([('foo', str)])Out[25]: dtype([('foo', '|S0')]) to get yourself a zero-length string. However dtype('|S0') results in a TypeError: data type not understood. I understand the stupidity of creating a 0-length string field but it's conceivable that it's accidental. For example, it could lead to a situation where you've created that field, are missing all the data you had meant to put in it, serialize with np.save, and upon np.load aren't able to get _any_ of your data back because the dtype descriptor is considered bogus (can you guess why I thought of this scenario?). It seems that either dtype(str) should do something more sensible than zero-length string, or it should be possible to create it with dtype('| S0'). Which should it be? David

3 3

specifying numpy as dependency in your project, install_requires
by Ralf Gommers 06 Oct '12

06 Oct '12

Hi, An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6'] in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list. The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason. This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py: try: import numpy except ImportError: raise ImportError("my_package requires numpy") or try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion) Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere? Ralf

7 10

numpy-1.6.2: python setup.py build bdist_rpm fails
by Ed Sexton 06 Oct '12

06 Oct '12

Hello Numpy Group- I have a working installation of Numpy-1.6.2 on a RHEL 6.3 host, now I have to create an RPM to push out to 600+ hosts. 1. I cannot get the RPM build command to work, error listed below. 2. Another issue, how do you pass --fcompiler=gnu95 to bdlist_rpm? I have everything compiled under fcompiler. # python setup.py build_bdist Tail of the error: <snip><snip><snip><snip> <snip><snip><snip><snip> RPM build errors: File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/array_from_pyobj/wrapmodule.c File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_free.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_mod.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/foo_use.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/assumed_shape/precision.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/kind/foo.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo.f File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_fixed.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/mixed/foo_free.f90 File listed twice: /usr/lib64/python2.6/site-packages/numpy/f2py/tests/src/size/foo.f90 Installed (but unpackaged) file(s) found: <snip><snip><snip><snip> <snip><snip><snip><snip> /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyc /usr/lib64/python2.6/site-packages/numpy/tests/test_matlib.pyo error: command 'rpmbuild' failed with exit status 1 Full Error on Pastebin: http://paste.ubuntu.com/1262671/ Thanks for your ideas/suggestions. Sincerely, Ed -- Sincerely, Ed Sexton e: sextonhadoop(a)gmail.com gv: (408) 475-2358

2 1