Mailman 3 January 2012 - NumPy-Discussion

advanced indexing bug with huge arrays?
by David Warde-Farley 26 Jan '12

26 Jan '12

A colleague has run into this weird behaviour with NumPy 1.6.1, EPD 7.1-2, on Linux (Fedora Core 14) 64-bit: > a = numpy.array(numpy.random.randint(256,size=(5000000,972)),dtype='uint8') > b = numpy.random.randint(5000000,size=(4993210,)) > c = a[b] It seems c is not getting filled in full, namely: > In [14]: c[1000000:].sum() > Out[14]: 0 I haven't been able to reproduce this quite yet, I'll try to find a machine with sufficient memory tomorrow. But does anyone have any insight in the mean time? It smells like some kind of integer overflow bug. Thanks, David

9 28

Re: [Numpy-discussion] Problem installing NumPy with Python 3.2.2/MacOS X 10.7.2
by Hans-Martin v. Gaudecker 26 Jan '12

26 Jan '12

I recently upgraded to Lion and just faced the same problem with both Python 2.7.2 and Python 3.2.2 installed via the python.org installers. My hunch is that the errors are related to the fact that Apple dropped gcc-4.2 from XCode 4.2. I got gcc-4.2 via [1] then, still the same error -- who knows what else got lost in that upgrade... Previous successful builds with gcc-4.2 might have been with XCode 4.1 (or 4.2 installed on top of it). In the end I decided to re-install both Python versions via homebrew, nicely described here [2] and everything seems to work fine using LLVM. Test outputs for NumPy master under 2.7.2 and 3.2.2 are below in case they are of interest. Best, Hans-Martin [1] https://github.com/kennethreitz/osx-gcc-installer [2] http://www.thisisthegreenroom.com/2011/installing-python-numpy-scipy-matplo… >>> numpy.test() Running unit tests for numpy NumPy version 2.0.0.dev-55472ca NumPy is installed in /usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy Python version 2.7.2 (default, Jan 11 2012, 15:34:30) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] nose version 1.1.2 ........................./usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py:1317: UserWarning: pytz not found, pytz compatibility tests skipped warnings.warn("pytz not found, pytz compatibility tests skipped") ......................................................................................................................................................F..................................................................................................................................................................................................................................................................S.........................................................................................................................................................../usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/core/numeric.py:2020: RuntimeWarning: invalid value encountered in absolute return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................................................................K...SK.S.......S........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. ====================================================================== FAIL: test_einsum_sums_clongdouble (test_einsum.TestEinSum) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/core/tests/test_einsum.py", line 479, in test_einsum_sums_clongdouble self.check_einsum_sums(np.clongdouble); File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/core/tests/test_einsum.py", line 231, in check_einsum_sums np.sum(a, axis=0).astype(dtype)) File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 753, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 100.0%) x: array([[ 48.0+0.0j, 50.0+0.0j, 52.0+0.0j, 54.0+0.0j, 56.0+0.0j, 58.0+0.0j, 60.0+0.0j, 62.0+0.0j, 64.0+0.0j, 66.0+0.0j, 68.0+0.0j, 70.0+0.0j, 72.0+0.0j, 74.0+0.0j, 76.0+0.0j,... y: array([[ 0.0+0.0j, 1.0+0.0j, 2.0+0.0j, 3.0+0.0j, 4.0+0.0j, 5.0+0.0j, 6.0+0.0j, 7.0+0.0j, 8.0+0.0j, 9.0+0.0j, 10.0+0.0j, 11.0+0.0j, 12.0+0.0j, 13.0+0.0j, 14.0+0.0j, 15.0+0.0j],... ====================================================================== FAIL: test_prod (test_defmatrix.TestProperties) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/matrixlib/tests/test_defmatrix.py", line 78, in test_prod assert_equal(x.prod(0), matrix([[4,10,18]])) File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 753, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 100.0%) x: matrix([[4611686018427387904, 4611686018427387904, 4449894403]]) y: matrix([[ 4, 10, 18]]) ---------------------------------------------------------------------- Ran 3553 tests in 51.254s FAILED (KNOWNFAIL=3, SKIP=5, failures=2) <nose.result.TextTestResult run=3553 errors=0 failures=2> >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 2.0.0.dev-55472ca NumPy is installed in /usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/numpy Python version 3.2.2 (default, Jan 11 2012, 15:30:18) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] nose version 1.1.2 ........................./usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/numpy/core/tests/test_datetime.py:1317: UserWarning: pytz not found, pytz compatibility tests skipped warnings.warn("pytz not found, pytz compatibility tests skipped") ...................................................................................................................................................................................................................................................................................E..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...........................................................................................................................................................................................................K....................................................................................................K...SK.S.......S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................./usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/numpy/lib/format.py:575: ResourceWarning: unclosed file <_io.BufferedReader name='/var/folders/vx/v51x110x36zcd9lppn78v0qw0000gn/T/tmpenv3j3'> mode=mode, offset=offset) ...........................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................/usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/numpy/ma/core.py:4778: RuntimeWarning: invalid value encountered in power np.power(out, 0.5, out=out, casting='unsafe') ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ====================================================================== ERROR: test_multiarray.TestFromBuffer.test_empty('', array([], dtype=float64), {}) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/nose-1.1.2-py3.2.egg/nose/case.py", line 198, in runTest self.test(*self.arg) File "/usr/local/Cellar/python3/3.2.2/lib/python3.2/site-packages/numpy/core/tests/test_multiarray.py", line 1446, in tst_basic assert_array_equal(np.frombuffer(buffer,**kwargs),expected) AttributeError: 'str' object has no attribute '__buffer__' ---------------------------------------------------------------------- Ran 3552 tests in 44.427s FAILED (KNOWNFAIL=4, SKIP=4, errors=1) <nose.result.TextTestResult run=3552 errors=1 failures=0> On 18 Dec 2011, at 23:14, numpy-discussion-request(a)scipy.org wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Problem installing NumPy with Python 3.2.2/MacOS X 10.7.2 > (McNicol, Adam) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 18 Dec 2011 22:13:48 -0000 > From: "McNicol, Adam" <amcnicol(a)longroad.ac.uk> > Subject: Re: [Numpy-discussion] Problem installing NumPy with Python > 3.2.2/MacOS X 10.7.2 > To: <numpy-discussion(a)scipy.org> > Message-ID: <2128d5dc6c318f2d07c027b5c6c4c0ef5b01bf31@localhost> > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > Definitely have the sdk installed. In the Developer/SDKs directory I have one for 10.6 and another for 10.7 - no idea where a second 10.6 would be coming from =( > > > Adam. > > > -----Original Message----- > From: numpy-discussion-request(a)scipy.org [mailto:numpy-discussion-request@scipy.org] > Sent: Sun 12/18/2011 9:52 PM > To: numpy-discussion(a)scipy.org > Subject: NumPy-Discussion Digest, Vol 63, Issue 55 > > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Problem installing NumPy with Python 3.2.2/MacOS X 10.7.2 > (McNicol, Adam) > 2. Re: Problem installing NumPy with Python 3.2.2/MacOS X 10.7.2 > (Ralf Gommers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 18 Dec 2011 18:48:47 -0000 > From: "McNicol, Adam" <amcnicol(a)longroad.ac.uk> > Subject: Re: [Numpy-discussion] Problem installing NumPy with Python > 3.2.2/MacOS X 10.7.2 > To: <numpy-discussion(a)scipy.org> > Message-ID: <fd29dc5c77c7ee4f0f0622b3459e91a5d77badc3@localhost> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Ralf, > > Thanks for the response. I tried reinstalling Xcode 4.2.1 and the GCC/Fortran installer from http://r.research.att.com/tools/ (gcc-42-5666.3-darwin11.pkg) before installing the distribute package that you suggested. > > I then reran the numpy installer being sure to enter the three export lines as suggested on the numpy installation guide for Lion. > > Still no success. I guess I'll just have to wait for more official support for my configuration. > > I have included the output from terminal just in case it is useful as there were a few lines in red that suggest something isn't quite right with something. I have placed ** before the lines that appear in red. > > I appreciate the suggestions, > > Thanks again, > > > Adam. > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options > running build_src > build_src > building py_modules sources > creating build > creating build/src.macosx-10.6-intel-3.2 > creating build/src.macosx-10.6-intel-3.2/numpy > creating build/src.macosx-10.6-intel-3.2/numpy/distutils > building library "npymath" sources > customize NAGFCompiler > **Could not locate executable f95 > customize AbsoftFCompiler > **Could not locate executable f90 > **Could not locate executable f77 > customize IBMFCompiler > **Could not locate executable xlf90 > **Could not locate executable xlf > customize IntelFCompiler > **Could not locate executable fort > **Could not locate executable ifc > customize GnuFCompiler > **Could not locate executable g77 > customize Gnu95FCompiler > **Could not locate executable gfortran > customize G95FCompiler > **Could not locate executable g95 > customize PGroupFCompiler > **Could not locate executable pgf90 > **Could not locate executable pgf77 > **don't know how to compile Fortran code on platform 'posix' > C compiler: gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c' > gcc-4.2: _configtest.c > gcc-4.2 _configtest.o -o _configtest > success! > removing: _configtest.c _configtest.o _configtest > customize NAGFCompiler > customize AbsoftFCompiler > customize IBMFCompiler > customize IntelFCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize PGroupFCompiler > **don't know how to compile Fortran code on platform 'posix' > customize NAGFCompiler > customize AbsoftFCompiler > customize IBMFCompiler > customize IntelFCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize PGroupFCompiler > **don't know how to compile Fortran code on platform 'posix' > C compiler: gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c' > gcc-4.2: _configtest.c > _configtest.c:1: warning: conflicting types for built-in function ?exp? > _configtest.c:1: warning: conflicting types for built-in function ?exp? > gcc-4.2 _configtest.o -o _configtest > success! > removing: _configtest.c _configtest.o _configtest > creating build/src.macosx-10.6-intel-3.2/numpy/core > creating build/src.macosx-10.6-intel-3.2/numpy/core/src > creating build/src.macosx-10.6-intel-3.2/numpy/core/src/npymath > conv_template:> build/src.macosx-10.6-intel-3.2/numpy/core/src/npymath/npy_math.c > conv_template:> build/src.macosx-10.6-intel-3.2/numpy/core/src/npymath/ieee754.c > conv_template:> build/src.macosx-10.6-intel-3.2/numpy/core/src/npymath/npy_math_complex.c > building extension "numpy.core._sort" sources > Generating build/src.macosx-10.6-intel-3.2/numpy/core/include/numpy/config.h > customize NAGFCompiler > customize AbsoftFCompiler > customize IBMFCompiler > customize IntelFCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize PGroupFCompiler > **don't know how to compile Fortran code on platform 'posix' > customize NAGFCompiler > customize AbsoftFCompiler > customize IBMFCompiler > customize IntelFCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize PGroupFCompiler > **don't know how to compile Fortran code on platform 'posix' > C compiler: gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c' > gcc-4.2: _configtest.c > In file included from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/bytearrayobject.h:9, > from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/Python.h:73, > from _configtest.c:1: > /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory > In file included from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/bytearrayobject.h:9, > from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/Python.h:73, > from _configtest.c:1: > /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory > lipo: can't figure out the architecture type of: /var/folders/_c/z033hf1s1cgfcxtxfnpg0lsm0000gn/T//ccKv548x.out > In file included from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/bytearrayobject.h:9, > from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/Python.h:73, > from _configtest.c:1: > /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory > In file included from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/bytearrayobject.h:9, > from /Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m/Python.h:73, > from _configtest.c:1: > /Developer/SDKs/MacOSX10.6.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory > lipo: can't figure out the architecture type of: /var/folders/_c/z033hf1s1cgfcxtxfnpg0lsm0000gn/T//ccKv548x.out > failure. > removing: _configtest.c _configtest.o > Running from numpy source directory.Traceback (most recent call last): > File "setup.py", line 196, in <module> > setup_package() > File "setup.py", line 189, in setup_package > configuration=configuration ) > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/core.py", line 186, in setup > return old_setup(**new_attr) > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/core.py", line 148, in setup > dist.run_commands() > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/dist.py", line 917, in run_commands > self.run_command(cmd) > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/dist.py", line 936, in run_command > cmd_obj.run() > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/command/build.py", line 37, in run > old_build.run(self) > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/command/build.py", line 126, in run > self.run_command(cmd_name) > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/distutils/dist.py", line 936, in run_command > cmd_obj.run() > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/command/build_src.py", line 152, in run > self.build_sources() > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/command/build_src.py", line 169, in build_sources > self.build_extension_sources(ext) > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/command/build_src.py", line 328, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/Users/adammcnicol/Desktop/numpy-1.6.1/build/py3k/numpy/distutils/command/build_src.py", line 385, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 410, in generate_config_h > moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) > File "numpy/core/setup.py", line 41, in check_types > out = check_types(*a, **kw) > File "numpy/core/setup.py", line 271, in check_types > "Cannot compile 'Python.h'. Perhaps you need to "\ > SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel. > > > > Message: 3 > Date: Sun, 18 Dec 2011 09:49:00 +0100 > From: Ralf Gommers <ralf.gommers(a)googlemail.com> > Subject: Re: [Numpy-discussion] Problem installing NumPy with Python > 3.2.2/MacOS X 10.7.2 > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CABL7CQh-+t+6p5z_S_JVycRCB-FWK0Q3wiAWjazgmDp8AJkKqw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Sat, Dec 17, 2011 at 12:59 PM, McNicol, Adam <amcnicol(a)longroad.ac.uk>wrote: > >> ** >> >> Hi There, >> >> Thanks for the responses. >> >> At this point I would settle from just being able to install matplotlib. >> Even if some of the functionality isn't present currently that is fine. >> >> I'm afraid my knowledge of Python falls down about here as well. I >> installed Python 3.2.2 via the installer from Python.org so I have no idea >> whether Python.h is present or where indeed I would find it or how I would >> add it to the search path. >> >> Do I have to install from source or something like that? >> > > No, your Python install should be fine if you just got the dmg installer > from python.org. I recommend you install the OS X SDKs and distribute ( > http://pypi.python.org/pypi/distribute), as I said before, and try again to > compile numpy. > > Unfortunately you have chosen a difficult combination of OS and Python > version, so we don't have binary installers you can use (yet). > > Ralf > > >> Thanks again, >> >> >> Adam. >> >> >> -----Original Message----- >> From: McNicol, Adam >> Sent: Fri 12/16/2011 11:07 PM >> To: numpy-discussion(a)scipy.org >> Subject: Problem installing NumPy with Python 3.2.2/MacOS X 10.7.2 >> >> Hi There, >> >> I am very new to numpy and have really only started investigating it as >> one of my students needs some functionality from matplotlib. I have managed >> to install everything under Windows for work in class but I use a Mac at >> home and have been struggling all night to get it to build and install. >> >> I should mention that I am using Python 3.2.2 both in school and at home >> and it isn't an option to use Python 2.7 as all of the rest of my class is >> taught in Python 3. I also have the most recent version of Xcode installed. >> >> I have installed the correct build of gcc-4.2 with Fortran (gcc-4.2 (Apple >> build 5666.3) with GNU Fortran 4.2.4 for Mac OS X 10.7 (Lion)) from >> http://r.research.att.com/tools/ >> >> I then followed the install instructions but the build fails with the >> following message: >> >> File "numpy/core/setup.py", line 271, in check_types >> "Cannot compile 'Python.h'. Perhaps you need to "\ >> SystemError: Cannot compile 'Python.h'. Perhaps you need to install >> python-dev|python-devel. >> >> I have got no idea what to do with this error message. Any help would be >> much appreciated. >> >> Kind Regards, >> >> >> Adam. >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >

2 1

array metadata
by Emmanuel Mayssat 26 Jan '12

26 Jan '12

Is there a way to store metadata for an array? For example, date the samples were collected, name of the operator, etc. Regards, -- Emmanuel

4 4

OT: MS C++ AMP library
by Sturla Molden 26 Jan '12

26 Jan '12

When we have nice libraries like OpenCL, OpenGL and OpenMP, I am so glad we have Microsoft to screw it up. Congratulations to Redmond: Another C++ API I cannot read, and a scientific compute library I hopefully never have to use. http://msdn.microsoft.com/en-us/library/hh265136(v=vs.110).aspx The annoying part is, with this crap there will never be a standard OpenCL DLL in Windows. Sturla Molden

3 2

view of a structured array?
by Chris Barker 25 Jan '12

25 Jan '12

HI folks, Is there a way to get a view of a subset of a structured array? I know that an arbitrary subset will not fit into the numpy "strides"offsets" model, but some will, and it would be nice to have a view: For example, here we have a stuctured array: In [56]: a Out[56]: array([(1, 2.0, 3.0, 4), (7, 8.0, 9.0, 10), (5, 123.4, 7.0, 8), (9, 10.0, 11.0, 12), (13, 14.0, 15.0, 16)], dtype=[('i', '<i4'), ('f1', '<f8'), ('f2', '<f8'), ('i2', '<i4')]) if I pull out one "field" a get a view: In [57]: b = a['f1'] In [58]: b[0] = 1000 In [59]: a Out[59]: array([(1, 1000.0, 3.0, 4), (7, 8.0, 9.0, 10), (5, 123.4, 7.0, 8), (9, 10.0, 11.0, 12), (13, 14.0, 15.0, 16)], dtype=[('i', '<i4'), ('f1', '<f8'), ('f2', '<f8'), ('i2', '<i4')]) However, if I pull out more than one field, I get a copy: In [60]: b = a[['f1','f2']] In [61]: b Out[61]: array([(1000.0, 3.0), (8.0, 9.0), (123.4, 7.0), (10.0, 11.0), (14.0, 15.0)], dtype=[('f1', '<f8'), ('f2', '<f8')]) In [62]: b[1] = (2000,3000) In [63]: b Out[63]: array([(1000.0, 3.0), (2000.0, 3000.0), (123.4, 7.0), (10.0, 11.0), (14.0, 15.0)], dtype=[('f1', '<f8'), ('f2', '<f8')]) In [64]: a Out[64]: array([(1, 1000.0, 3.0, 4), (7, 8.0, 9.0, 10), (5, 123.4, 7.0, 8), (9, 10.0, 11.0, 12), (13, 14.0, 15.0, 16)], dtype=[('i', '<i4'), ('f1', '<f8'), ('f2', '<f8'), ('i2', '<i4')]) However, in this case, the two fields are contiguous, and thus I'm pretty sure one could build a numpy array that was a view. Is there any way to do so? Ideally without manipulating the strides by hand, but I may want to do that if it's the only way. -Chris -- -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 2

Re: [Numpy-discussion] Unexpected behavior with np.min_scalar_type
by Kathleen M Tacina 25 Jan '12

25 Jan '12

Thanks! It was interesting to see why that happened. Kathy On Tue, 2012-01-24 at 18:56 -0600, Mark Wiebe wrote: > On Tue, Jan 24, 2012 at 7:29 AM, Kathleen M Tacina > <Kathleen.M.Tacina(a)nasa.gov> wrote: > > I was experimenting with np.min_scalar_type to make sure it > worked as expected, and found some unexpected results for > integers between 2**63 and 2**64-1. I would have expected > np.min_scalar_type(2**64-1) to return uint64. Instead, I get > object. Further experimenting showed that the largest integer > for which np.min_scalar_type will return uint64 is 2**63-1. > Is this expected behavior? > > > > This is a bug in how numpy detects the dtype of python objects. > > > https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/common… > > > You can see there it's only checking for a signed long long, not > accounting for the unsigned case. I created a ticket for you here: > > > http://projects.scipy.org/numpy/ticket/2028 > > > -Mark > > > On python 2.7.2 on a 64-bit linux machine: > >>> import numpy as np > >>> np.version.full_version > '2.0.0.dev-55472ca' > >>> np.min_scalar_type(2**8-1) > dtype('uint8') > >>> np.min_scalar_type(2**16-1) > dtype('uint16') > >>> np.min_scalar_type(2**32-1) > dtype('uint32') > >>> np.min_scalar_type(2**64-1) > dtype('O') > >>> np.min_scalar_type(2**63-1) > dtype('uint64') > >>> np.min_scalar_type(2**63) > dtype('O') > > I get the same results on a Windows XP machine running python > 2.7.2 and numpy 1.6.1. > > Kathy > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 0

Matplotlib and optimization tutorials at PyCon US
by Mike Müller 25 Jan '12

25 Jan '12

Hi, I will be giving a matplotlib and a optimization tutorial at PyCon in March. The first tutorial is a compact introduction to matplotlib. The optimization tutorial gives an overview over this topic. BTW, the early bird deadline is today. Mike Plotting with matplotlib ------------------------ Instructor: Mike Müller Type:Tutorial Audience level:Novice Category:Useful libraries March 8th 9 a.m. – 12:20 p.m. https://us.pycon.org/2012/schedule/presentation/238/ When it comes to plotting with Python many people think about matplotlib. It is widely used and provides a simple interface for creating a wide variety of plots from very simple diagrams to sophisticated animations. This tutorial is a hands-on introduction that teaches the basics of matplotlib. Students will learn how to create publication-ready plots with just a few lines of Python. Faster Python Programs through Optimization ------------------------------------------- Instructor: Mike Müller Type:Tutorial Audience level:Experienced Category:Best Practices/Patterns March 7th 9 a.m. – 12:20 p.m. https://us.pycon.org/2012/schedule/presentation/245/ This tutorial provides an overview of techniques to improve the performance of Python programs. The focus is on concepts such as profiling, difference of data structures and algorithms as well as a selection of tools and libraries that help to speed up Python.

1 0

Permuting sparse arrays
by Edward C. Jones 25 Jan '12

25 Jan '12

I have a vector of bits where there are many more zeros than one. I store the array as a sorted list of the indexes where the bit is one. If the bit array is (0, 1, 0, 0, 0, 1, 1), it is stored as (1, 5, 6). If the bit array, b, has length n, and p is a random permutation of arange(n), then I can permute the bit array using fancy indexing: b[p]. Is there some neat trick I can use to permute an array while leaving it in the list-of-indexes form? Currently I am doing it with a Python loop but I am looking for a faster way.

2 1

Re: [Numpy-discussion] bug in numpy.mean() ?
by Kathleen M Tacina 25 Jan '12

25 Jan '12

I found something similar, with a very simple example. On 64-bit linux, python 2.7.2, numpy development version: In [22]: a = 4000*np.ones((1024,1024),dtype=np.float32) In [23]: a.mean() Out[23]: 4034.16357421875 In [24]: np.version.full_version Out[24]: '2.0.0.dev-55472ca' But, a Windows XP machine running python 2.7.2 with numpy 1.6.1 gives: >>>a = np.ones((1024,1024),dtype=np.float32) >>>a.mean() 4000.0 >>>np.version.full_version '1.6.1' On Tue, 2012-01-24 at 17:12 -0600, eat wrote: > Hi, > > > > Oddly, but numpy 1.6 seems to behave more consistent manner: > > > In []: sys.version > Out[]: '2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit > (Intel)]' > In []: np.version.version > Out[]: '1.6.0' > > > In []: d= np.load('data.npy') > In []: d.dtype > Out[]: dtype('float32') > > > In []: d.mean() > Out[]: 3045.7471999999998 > In []: d.mean(dtype= np.float32) > Out[]: 3045.7471999999998 > In []: d.mean(dtype= np.float64) > Out[]: 3045.747251076416 > In []: (d- d.min()).mean()+ d.min() > Out[]: 3045.7472508750002 > In []: d.mean(axis= 0).mean() > Out[]: 3045.7472499999999 > In []: d.mean(axis= 1).mean() > Out[]: 3045.7472499999999 > > > Or does the results of calculations depend more on the platform? > > > > > My 2 cents, > eat -- -------------------------------------------------- Kathleen M. Tacina NASA Glenn Research Center MS 5-10 21000 Brookpark Road Cleveland, OH 44135 Telephone: (216) 433-6660 Fax: (216) 433-5802 --------------------------------------------------

4 4

numpy.percentile multiple arrays
by questions anon 25 Jan '12

25 Jan '12

I need some help understanding how to loop through many arrays to calculate the 95th percentile. I can easily do this by using numpy.concatenate to make one big array and then finding the 95th percentile using numpy.percentile but this causes a memory error when I want to run this on 100's of netcdf files (see code below). Any alternative methods will be greatly appreciated. all_TSFC=[] for (path, dirs, files) in os.walk(MainFolder): for dir in dirs: print dir path=path+'/' for ncfile in files: if ncfile[-3:]=='.nc': print "dealing with ncfiles:", ncfile ncfile=os.path.join(path,ncfile) ncfile=Dataset(ncfile, 'r+', 'NETCDF4') TSFC=ncfile.variables['T_SFC'][:] ncfile.close() all_TSFC.append(TSFC) big_array=N.ma.concatenate(all_TSFC) Percentile95th=N.percentile(big_array, 95, axis=0)

4 4