Mailman 3 May 2009 - NumPy-Discussion

Re: [Numpy-discussion] Numpy-discussion Digest, Vol 32, Issue 39
by Emerald Jasper 14 May '09

14 May '09

Hi, Actually, I am quite new in programming, so could you please send me the syntax so that I can use in my research. thank you so much. --- On Thu, 14/5/09, numpy-discussion-request(a)scipy.org <numpy-discussion-request(a)scipy.org> wrote: From: numpy-discussion-request(a)scipy.org <numpy-discussion-request(a)scipy.org> Subject: Numpy-discussion Digest, Vol 32, Issue 39 To: numpy-discussion(a)scipy.org Date: Thursday, 14 May, 2009, 10:51 AM Send Numpy-discussion mailing list submissions to numpy-discussion(a)scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/numpy-discussion or, via email, send a message with subject or body 'help' to numpy-discussion-request(a)scipy.org You can reach the person managing the list at numpy-discussion-owner(a)scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Numpy-discussion digest..." Today's Topics: 1. Re: numpy slices limited to 32 bit values? (Glenn Tarbox, PhD) 2. Re: numpy slices limited to 32 bit values? (Gael Varoquaux) 3. Re: numpy slices limited to 32 bit values? (Glenn Tarbox, PhD) 4. Re: numpy slices limited to 32 bit values? (Gael Varoquaux) 5. Re: (no subject) (Sebastian Walter) 6. Non-linear optimization in python (Emerald Jasper) 7. Re: Non-linear optimization in python (Matthieu Brucher) ---------------------------------------------------------------------- Message: 1 Date: Thu, 14 May 2009 01:31:45 -0700 From: "Glenn Tarbox, PhD" <glenn(a)tarbox.org> Subject: Re: [Numpy-discussion] numpy slices limited to 32 bit values? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <f049b35c0905140131t12efe2ddgde3285d7ade721c1(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" On Wed, May 13, 2009 at 11:22 PM, Glenn Tarbox, PhD <glenn(a)tarbox.org>wrote: > > > On Wed, May 13, 2009 at 11:04 PM, Charles R Harris < > charlesr.harris(a)gmail.com> wrote: > >> >> >> On Wed, May 13, 2009 at 10:50 PM, Glenn Tarbox, PhD <glenn(a)tarbox.org>wrote: >> >>> I'm using the latest version of Sage (3.4.2) which is python 2.5 and >>> numpy something or other (I will do more digging presently) >>> >>> I'm able to map large files and access all the elements unless I'm using >>> slices >>> >>> so, for example: >>> >>> fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64', >>> mode='r+', shape=(10000000000,)) >>> >>> which is 1e10 doubles if you don't wanna count the zeros >>> >>> gives full access to a 75 GB memory image >>> >>> But when I do: >>> >>> fp[:] = 1.0 >>> np.sum(fp) >>> >>> I get 1410065408.0 as the result >>> >> >> As doubles, that is more than 2**33 bytes, so I expect there is something >> else going on. How much physical memory/swap memory do you have? This could >> also be a python problem since python does the memmap. >> > > I've been working on some other things lately and that number seemed > related to 2^32... now that I look more closely, I don't know where that > number comes from. > > To your question, I have 32GB of RAM and virtually nothing else running... > Top tells me I'm getting between 96% and 98% for this process which seems > about right. > > Here's the thing. When I create the mmap file, I get the right number of > bytes. I can, from what I can tell, update individual values within the > array (I'm gonna bang on it a bit more with some other scripts) > > Its only when using slicing that things get strange (he says having not > really done a more thorough test) > > Of course, I was assuming this is a 32 bit thing... but you're right... > where did that result come from??? > > The other clue here is that when I create my own slice (as described above) > it returns instantly... numpy doesn't throw an error but it doesn't do > anything with the slice either. > > Since I'm IO bound anyways, maybe i'll just write a loop and see if I can't > set all the values. The machine could use a little exercise anyways. > I ran the following test: import numpy as np size=10000000000 fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64', mode='r+', shape=(size,)) for i in xrange(size): fp[i]=1.0 time np.sum(fp) 10000000000.0 Time: CPU 188.36 s, Wall: 884.33 s So, everything seems to be working and it kinda makes sense. The sum should be IO bound which it is. I didn't time the loop but it took a while (maybe 30 minutes) and it was compute bound. To make sure, I exited the program and ran everything but the initialization loop. import numpy as np size=10000000000 fp = np.memmap("/mnt/hdd/data/mmap/numpy1e10.mmap", dtype='float64', mode='r+', shape=(size,) time np.sum(fp) 10000000000.0 Time: CPU 180.02 s, Wall: 854.72 s I was a little surprised that it didn't take longer given almost half of the mmap'ed data should have been resident in the sum performed immediately after initialization, but since it needed to start at the beginning and only had the second half in memory, it makes sense So, it "appears" as though the mmap works but there's something strange with slices going on. -glenn > > >> >> Chuck >> >> >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Glenn H. Tarbox, PhD || 206-274-6919 > http://www.tarbox.org > -- Glenn H. Tarbox, PhD || 206-274-6919 http://www.tarbox.org

1 0

building inplace with numpy.distutils?
by Robert Cimrman 14 May '09

14 May '09

Hi (David)! I am evaluating numpy.distutils as a build/install system for my project - is it possible to build the extension modules in-place so that the project can be used without installing it? A pointer to documentation concerning this would be handy... Currently I use a regular Makefile for the build, which works quite well, but is not very portable and does not solve the package installation. Otherwise let me say that numpy.distutils work very well, much better than the plain old distutils. Best regards, r.

2 2

FAIL: Test bug in reduceat with structured arrays
by Nils Wagner 14 May '09

14 May '09

Hi all, Can someone reproduce the following failure ? I am using >>> numpy.__version__ '1.4.0.dev6983' ====================================================================== FAIL: Test bug in reduceat with structured arrays copied for speed. ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/home/nwagner/local/lib/python2.5/site-packages/nose-0.10.4-py2.5.egg/nose/case.py", line 182, in runTest self.test(*self.arg) File "/data/home/nwagner/local/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 664, in test_reduceat assert np.all(h1 == h2) AssertionError Nils

4 5

copy and paste arrays from matlab
by Robin 13 May '09

13 May '09

[crossposted to numpy-discussion and mlabwrap-user] Hi, I wrote a little utility class in Matlab that inherits from double and overloads the display function so you can easily print matlab arrays of arbitrary dimension in Numpy format for easy copy and pasting. I have to work a lot with other peoples code - and while mlabwrap and reading and writing is great, sometimes I find it easier and quicker just to copy and paste smaller arrays between interactive sessions. Anyway you put it in your Matlab path then you can do x = rand(2,3,4,5); a = array(x) You can specify the fprintf style format string either in the constructor or after: a = array(x,'%2.6f') a.format = '%2.2f' eg: >> x = rand(4,3,2); >> array(x) ans = array([[[2.071566461449581e-01, 3.501602151029837e-02], [1.589135260727248e-01, 3.766891927380323e-01], [8.757206127846399e-01, 7.259276565938600e-01]], [[7.570839415557700e-01, 3.974969411279816e-02], [8.109207856487061e-01, 5.043242527988604e-01], [6.351863794630047e-01, 7.013280585980169e-01]], [[8.863281096304466e-01, 9.885678912262633e-01], [4.765077527169480e-01, 7.634956792870943e-01], [9.728134909163066e-02, 4.588908258125032e-01]], [[4.722298594969571e-01, 6.861815984603373e-01], [1.162875322461844e-01, 4.887479677951201e-02], [9.084394562396312e-01, 5.822948089552498e-01]]]) It's a while since I've tried to do anything like this in Matlab and I must admit I found it pretty painful, so I hope it can be useful to someone else! I will try and do one for Python for copying and pasting to Matlab, but I'm expecting that to be a lot easier! Cheers Robin

2 2

Detecting C API mismatch (was Managing Python with NumPy and many external libraries on multiple Windows machines)
by David Cournapeau 13 May '09

13 May '09

Hi, I worked on some code to detect C API mismatches both for developers and for users: http://github.com/cournape/numpy/tree/runtime_feature It adds the following: - if a numpy or ufunc function is added in the C API without the NPY_FEATURE_VERSION to be updated, a warning is generated at built time (the warning is turned into an exception for release) - I added a function PyArray_GetNDArrayCFeatureVersion which returns the C API version, and the version is checked in import_array. If the compile-time version > import-time version, an import error is raised, so the following happens (assuming the ABI is not changed). So we keep backward compatibility (building an extension with say numpy 1.2.1 will still work after installing numpy 1.3), and forward incompatibility is detected (building an extension with numpy 1.3.0 and importing it with installed numpy 1.2.1 will fail). Ironically, adding the function means that we have to add one function to the C API, so this will not be useful for numpy < 1.4, but I don't think it is possible to do it without modifying the C API. cheers, David

4 6

Problem building 1.3.0 on x86_64 platform
by James Jackson 13 May '09

13 May '09

Hi, I am attempting (and failing...) to build numpy on a Scientific Linux 4.6 x86_64 (essentially RHEL I believe) with Python 2.4 (i386). The machine has the following Python RPM installed: python2.4-2.4-1pydotorg.i386 python2.4-tools-2.4-1pydotorg.i386 python2.4-devel-2.4-1pydotorg.i386 And also has gcc, g++ and f77 installed. Running python setup.py config appears to be successful, with lots of warnings abougt ATLAS, BLAS etc not being available (this is fine, I just want numpy for the array handling features used in matplotlib). However, the build fails (following doesn't show missing library warnings, as above): ------------------------------------------------------------------------------------------------------ running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands -- compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands -- fcompiler options running build_src building py_modules sources building library "npymath" sources building extension "numpy.core._sort" sources Generating build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h customize GnuFCompiler Found executable /usr/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using config C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall - Wstrict-prototypes -fPIC compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/ python2.4 -c' gcc: _configtest.c In file included from /usr/include/python2.4/Python.h:55, from _configtest.c:1: /usr/include/python2.4/pyport.h:612:2: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)." In file included from /usr/include/python2.4/Python.h:55, from _configtest.c:1: /usr/include/python2.4/pyport.h:612:2: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)." failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 172, in ? setup_package() File "setup.py", line 165, in setup_package configuration=configuration ) File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ core.py", line 184, in setup return old_setup(**new_attr) File "/var/tmp/python2.4-2.4-root/usr/lib/python2.4/distutils/ core.py", line 149, in setup File "/var/tmp/python2.4-2.4-root/usr/lib/python2.4/distutils/ dist.py", line 946, in run_commands File "/var/tmp/python2.4-2.4-root/usr/lib/python2.4/distutils/ dist.py", line 966, in run_command File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ command/build.py", line 37, in run old_build.run(self) File "/var/tmp/python2.4-2.4-root/usr/lib/python2.4/distutils/ command/build.py", line 112, in run File "/usr/lib/python2.4/cmd.py", line 333, in run_command del help[cmd] File "/var/tmp/python2.4-2.4-root/usr/lib/python2.4/distutils/ dist.py", line 966, in run_command File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ command/build_src.py", line 130, in run self.build_sources() File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/home/castormonitor/PythonLibs/numpy-1.3.0/numpy/distutils/ command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 286, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 30, in check_types out = check_types(*a, **kw) File "numpy/core/setup.py", line 185, in check_types raise SystemError( SystemError: Cannot compiler 'Python.h'. Perhaps you need to install python-dev|python-devel. ------------------------------------------------------------------------------------------------------ I note that the distribution directory being created is build/ src.linux-x86_64-2.4 - not i386. Can I force the architecture in the configure step, as it appears this would be the problem (hinted at by LONG_BIG wrong for platform error). Any hints gratefully received! Regards, James.

2 1

Distance Formula on an Array
by Ian Mallett 13 May '09

13 May '09

Hi, I have an array sized n*3. Each three-component is a 3D position. Given another 3D position, how is the distance between it and every three-component in the array found with NumPy? So, for example, if the array is: [[0,0,0],[0,1,0],[0,0,3]] And the position is: [0,4,0] I need this array out: [4,3,5] (Just a simple Pythagorean Distance Formula) Ideas? Thanks, Ian

6 22

Matlab/Numpy index order
by brechmos 13 May '09

13 May '09

I am very new to Numpy and relatively new to Python. I have used Matlab for 15+ years now. But, I am starting to lean toward using Numpy for all my work. One thing that I am not understanding is the order of data when read in from a file. Let's say I have a 256x256x150 uint16 dataset (MRI, 150 slices). In Matlab I would read it in as: >> fp=fopen(<binary file>); >> d = fread(fp,256*256*150, 'int16'); >> fclose(fp); >> c = reshape(d, [256 256 150]); >> imagesc(c(:,:,1)); I am very used to having it read it in and doing the reshaping such that it is 256 rows by 256 columns by 150 slices. Now, Numpy, I can read in the binary data using fromfile (or open, read, close): In [85]: a = fromfile(<binary file>, dtype='int16') In [86]: b = array(struct.unpack('<%dH'%(256*256*150), a)).reshape(150,256,256) So, in Numpy I have to reshape it so the "slices" are in the first dimension. Obviously, I can do a b.transpose( (1,2,0) ) to get it to look like Matlab, but... I don't understand why the index ordering is different between Matlab and Numpy. (It isn't a C/Fortran ordering thing, I don' think). Is the data access faster if I have b without the tranpose, or can I transpose it so it "looks" like Matlab without taking a hit when I do imshow( b[:,:,0] ). Any help for a Numpy newbie would be appreciated. -- View this message in context: http://www.nabble.com/Matlab-Numpy-index-order-tp23509178p23509178.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

5 6

Re: [Numpy-discussion] List of arrays
by Wei Su 12 May '09

12 May '09

Hi, Francesc: The codes do not work. Guess you forgot something there. Thanks. Wei Su --- On Mon, 5/11/09, Francesc Alted <faltet(a)pytables.org> wrote: From: Francesc Alted <faltet(a)pytables.org> Subject: Re: [Numpy-discussion] List of arrays To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Date: Monday, May 11, 2009, 10:40 AM A Monday 11 May 2009, Nils Wagner escrigué: > Hi all, > > How can I convert a list of arrays into one array ? > > Nils > > >>> data > > [array([ 40. , 285.6, 45. , 285.3, 50. , 285.1, > 55. , 284.8]), array([ 60. , 284.5, 65. , 282..8, > 70. , 281.1, 75. , 280. ]), array([ 80. , 278..8, > 85. , 278.1, 90. , 277.4, 95. , 276.9]), array([ > 100. , 276.3, 105. , 276.1, 110. , 275.9, 115. , > 275.7]), array([ 120. , 275.5, 125. , 275.2, 130. , > 274.8, 135. , 274.5]), array([ 140. , 274.1, 145. , > 273.7, 150. , 273.2, 155. , 272.7]), array([ 160. , > 272.2, 165. , 272.1, 170. , 272. , 175. , 271.8]), > array([ 180. , 271.6, 185. , 271. , 190. , 270.3, > 195. , 269.5]), array([ 200. , 268.5, 205. , 267.4, > 210. , 266.1, 215. , 263.5]), array([ 220. , 260.1, > 225. , 256.1, 230. , 249.9, 235. , 239.3]), array([ > 238.7, 186.2, 240., 160. , 245. , 119.7, 250. , > 111.3])] > > newdata=array([ 40. , 285.6, 45. , 285.3, 50. , > 285.1, 55. , 284.8, 60. , 284.5, 65. , 282.8, ..., > 111.3]) Try np.concatenate: In [9]: a = np.arange(10) In [10]: b = np.arange(10,20) In [11]: np.concatenate(l) Out[11]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) Hope that helps, -- Francesc Alted "One would expect people to feel threatened by the 'giant brains or machines that think'. In fact, the frightening computer becomes less frightening if it is used only to simulate a familiar noncomputer." -- Edsger W. Dykstra "On the cruelty of really teaching computer science" _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

3 2

How to include numpy headers in C across versions 1.1, 1.2, and 1.3
by Peter Wang 12 May '09

12 May '09

Hey guys, I've got a small C extension that uses isnan() and (in numpy 1.1) had been importing it from ufuncobject.h. I see that it has now moved into npy_math.h in 1.3. What is the best way to ensure that I can reliably include this function across versions 1.1, 1.2, and 1.3? (Checking NPY_FEATURE_VERSION won't work, since it did not change from 1.2 to 1.3, although the location of the function definition did.) My best idea right now is to simply do a numpy version check in my setup.py, and hard-code some macros at the top of my C extension to #include the appropriate headers for each version. Any help or suggestions would be appreciated! Thanks, Peter

3 2