Mailman 3 July 2013 - NumPy-Discussion

Re: [Numpy-discussion] unit tests / developing numpy (Pauli Virtanen)
by Graeme B. Bell July 25, 2013

July 25, 2013

> > No, runtests builds the code and sets PYTHONPATH accordingly. > > -- > Pauli Virtanen Hello Pauli, Thanks again for writing back. I agree that may be what runtests.py is intended to do, but it is unfortunately not what it actually does in its default configuration, at least on my computer. I got burned by this a few nights ago. $ pwd /home/X/github/numpy $ more numpy/version.py # THIS FILE IS GENERATED FROM NUMPY SETUP.PY short_version = '1.8.0' $ echo $PYTHONPATH $ python runtests.py Building, see build.log... Build OK Running unit tests for numpy NumPy version 1.7.1 NumPy is installed in /usr/lib64/python2.7/site-packages/numpy Python version 2.7.3 (default, Aug 9 2012, 17:23:57) [GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] nose version 1.3.0 *note the version number and directory used by runtests.py* It reported 100% tests passed (unsurprising since it was testing the release version!). In reality, the current directory at that time failed a test when I pushed it to the main repository. Can you suggest anything that I may be doing wrong here? Graeme On Jul 25, 2013, at 7:00 PM, numpy-discussion-request(a)scipy.org wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: add .H attribute? (Chris Barker - NOAA Federal) > 2. Re: unit tests / developing numpy (Graeme B. Bell) > 3. Re: unit tests / developing numpy (Pauli Virtanen) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 25 Jul 2013 07:49:44 -0700 > From: Chris Barker - NOAA Federal <chris.barker(a)noaa.gov> > Subject: Re: [Numpy-discussion] add .H attribute? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: <-29565752093052626@unknownmsgid> > Content-Type: text/plain; charset=UTF-8 > > On Jul 23, 2013, at 4:57 PM, Alan G Isaac <alan.isaac(a)gmail.com> wrote: > >> Finally, I think (?) everyone (proponents and opponents) >> would be happy if .H could provide access to an iterative >> view of the conjugate transpose. > > Except those of us that don't think numpy needs it at all. > > But I'll call it a -0 > > -Chris > > > ------------------------------ > > Message: 2 > Date: Thu, 25 Jul 2013 14:52:07 +0000 > From: "Graeme B. Bell" <grb(a)skogoglandskap.no> > Subject: Re: [Numpy-discussion] unit tests / developing numpy > To: "<numpy-discussion(a)scipy.org>" <numpy-discussion(a)scipy.org> > Message-ID: <20D22199-8C71-49A0-8292-CEBBBB70B902(a)skogoglandskap.no> > Content-Type: text/plain; charset="us-ascii" > > Nathaniel, Pauli: > > Thanks for the suggestions! > > = runtests.py is a nice solution, but unless you also set up your PYTHONPATH and install the code you've been working on, you're going to run with whichever version of numpy you have installed normally rather than the code you've just been working on (e.g. 1.7.1 rather than 1.8dev). > > Unfortunately, this caused me to submit a buggy set of commits, thinking they had passed the tests. > > > = env approach: thanks, I'll give that a try and compare it with the /tmp install approach. > > > Can we add this to the dev workflow web pages? > > Graeme. > > ------------------------------ > > Message: 3 > Date: Thu, 25 Jul 2013 18:38:11 +0300 > From: Pauli Virtanen <pav(a)iki.fi> > Subject: Re: [Numpy-discussion] unit tests / developing numpy > To: numpy-discussion(a)scipy.org > Message-ID: <ksrgos$60t$1(a)ger.gmane.org> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > 25.07.2013 17:52, Graeme B. Bell kirjoitti: > [clip] >> = runtests.py is a nice solution, but unless you also set up your >> PYTHONPATH and install the code you've been working on, you're going >> to run with whichever version of numpy you have installed normally >> rather than the code you've just been working on (e.g. 1.7.1 rather > than 1.8dev). > > No, runtests builds the code and sets PYTHONPATH accordingly. > > -- > Pauli Virtanen > > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 82, Issue 51 > ************************************************

2 1

Re: [Numpy-discussion] unit tests / developing numpy
by Graeme B. Bell July 25, 2013

July 25, 2013

Nathaniel, Pauli: Thanks for the suggestions! = runtests.py is a nice solution, but unless you also set up your PYTHONPATH and install the code you've been working on, you're going to run with whichever version of numpy you have installed normally rather than the code you've just been working on (e.g. 1.7.1 rather than 1.8dev). Unfortunately, this caused me to submit a buggy set of commits, thinking they had passed the tests. = env approach: thanks, I'll give that a try and compare it with the /tmp install approach. Can we add this to the dev workflow web pages? Graeme.

2 1

Re: [Numpy-discussion] add .H attribute?
by Alan G Isaac July 25, 2013

July 25, 2013

On 7/23/2013 6:45 PM, Dag Sverre Seljebotn wrote: > It'd be great if you could try to incorporate it to create a more balanced overview Attempt 2: I believe that propoents of adding a .H attribute have primarily emphasized - readability (and general ease of use, including in teaching) - consistency with matrix and masked array - forward looking (to a future when .H can be a view) in the following sense: it gives access now to the conjugate transpose via .H, which is likely to be implemented in the future, so as long as we document (as with ``diagonal``) that this may change, it gives a large chunk of the desired benefit now. The opponents have primarily emphasized - inconsistency with convention that for arrays instance attributes should return views - NOT forward looking (to a future when .H can be a view) in the following sense: it gives access now to the conjugate transpose via .H but NOT as a view, which is likely to be the preferred implementation in the future, and if the implementation changes in this preferred way then code that relied on behavior rather than documentation will break Finally, I think (?) everyone (proponents and opponents) would be happy if .H could provide access to an iterative view of the conjugate transpose. (Any objections?) Better? Alan

2 1

Allow == and != to raise errors
by Sebastian Berg July 25, 2013

July 25, 2013

Hey, the array comparisons == and != never raise errors but instead simply return False for invalid comparisons. The main example are arrays of non-matching dimensions, and object arrays with invalid element-wise comparisons: In [1]: np.array([1,2,3]) == np.array([1,2]) Out[1]: False In [2]: np.array([1, np.array([2, 3])], dtype=object) == [1, 2] Out[2]: False This seems wrong to me, and I am sure not just me. I doubt any large projects makes use of such comparisons and assume that most would prefer the shape mismatch to raise an error, so I would like to change it. But I am a bit unsure especially about smaller projects. So to keep the transition a bit safer could imagine implementing a FutureWarning for these cases (and that would at least notify new users that what they are doing doesn't seem like the right thing). So the question is: Is such a change safe enough, or is there some good reason for the current behavior that I am missing? Regards, Sebastian (There may be other issues with structured types that would continue returning False I think, because neither side knows how to compare)

6 17

unit tests / developing numpy
by Graeme B. Bell July 25, 2013

July 25, 2013

Does anyone know how to get the unit tests to run on a local fork, without doing a complete install of numpy? If so, please can you describe it, or better still, update: http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html It seems strange that the development workflow doesn't mention running any tests before committing/pushing/pulling. Graeme.

4 5

Treatment of the Matrix by Numpy
by Colin J. Williams July 24, 2013

July 24, 2013

1 0

Splitting numpydoc to a separate repo
by Pauli Virtanen July 24, 2013

July 24, 2013

Hi, How about splitting doc/sphinxext out from the main Numpy repository to a separate `numpydoc` repo under Numpy project? It's a separate Python package, after all. Moreover, this would make it easier to use it as a git submodule (e.g. in Scipy). Moreover, its release cycle is not in any way tied to that of Numpy. Pauli

6 6

Casting Errors in AIX
by Trevor Antczak July 24, 2013

July 24, 2013

Hello Numpy Discussion List, So I'm trying to get numpy working on an AIX 6.1 system. Initially I had a lot of problems trying to compile the package because the xlc compiler weren't installed on this machine, but apparently the Python package we installed had been built with them. Once we got xlc installed the process seemed to work pretty well until we got to compiling heapsort.c. At this point I began to get a huge number of errors in the form: compile options: '-Inumpy/core/include -Ibuild/src.aix-6.1-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/freeware/include/python2.7 -Ibuild/src.aix-6.1-2.7/numpy/core/src/multiarray -Ibuild/src.aix-6.1-2.7/numpy/core/src/umath -c' xlc_r: build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.c "/usr/include/stdio.h", line 528.12: 1506-343 (S) Redeclaration of fgetpos64 differs from previous declaration on line 323 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 528.12: 1506-377 (I) The type "long long*" of parameter 2 differs from the previous type "long* restrict". "/usr/include/stdio.h", line 531.12: 1506-343 (S) Redeclaration of fseeko64 differs from previous declaration on line 471 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 531.12: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long". "/usr/include/stdio.h", line 532.12: 1506-343 (S) Redeclaration of fsetpos64 differs from previous declaration on line 325 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 532.12: 1506-377 (I) The type "const long long*" of parameter 2 differs from the previous type "const long*". "/usr/include/stdio.h", line 533.16: 1506-343 (S) Redeclaration of ftello64 differs from previous declaration on line 472 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 533.16: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long". "/usr/include/unistd.h", line 171.17: 1506-343 (S) Redeclaration of lseek64 differs from previous declaration on line 169 of "/usr/include/unistd.h". "/usr/include/unistd.h", line 171.17: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long". "/usr/include/unistd.h", line 171.17: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long". "/usr/include/sys/lockf.h", line 64.20: 1506-343 (S) Redeclaration of lockf64 differs from previous declaration on line 62 of "/usr/include/sys/lockf.h". ................................................... "/usr/include/unistd.h", line 942.25: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long". "/usr/include/unistd.h", line 942.25: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long". "/usr/include/unistd.h", line 943.25: 1506-343 (S) Redeclaration of fsync_range64 differs from previous declaration on line 940 of "/usr/include/unistd.h". "/usr/include/unistd.h", line 943.25: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long". error: Command "/usr/vac/bin/xlc_r -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -q32 -qbitfields=signed -qmaxmem=70000 -qalloca -bmaxdata:0x80000000 -Wl,-brtl -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DNDEBUG -O2 -Inumpy/core/include -Ibuild/src.aix-6.1-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/freeware/include/python2.7 -Ibuild/src.aix-6.1-2.7/numpy/core/src/multiarray -Ibuild/src.aix-6.1-2.7/numpy/core/src/umath -c build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.c -o build/temp.aix-6.1-2.7/build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.o" failed with exit status 1 There are a lot more than this. Probably in neighborhood of 40 lines all told. I spent some time doing research and this appears to be something not terribly uncommon when compiling F/OSS on AIX. Most of the instances appeared to involve either sshd or smb. Unfortunately the most commonly cited solution (using the --disable-largefile to configure) won't work for compiling a Python module. One suggestion I found that did help was to explicitly include some of the standard libraries in the .c file. So I added: #include<unistd.h> #include<stdarg.h> To heapsort.c. That dramatically reduced the error messages. Now I get: compile options: '-Inumpy/core/include -Ibuild/src.aix-6.1-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/freeware/include/python2.7 -Ibuild/src.aix-6.1-2.7/numpy/core/src/multiarray -Ibuild/src.aix-6.1-2.7/numpy/core/src/umath -c' xlc_r: build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.c "/usr/include/stdio.h", line 528.12: 1506-343 (S) Redeclaration of fgetpos64 differs from previous declaration on line 323 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 528.12: 1506-377 (I) The type "long long*" of parameter 2 differs from the previous type "long* restrict". "/usr/include/stdio.h", line 531.12: 1506-343 (S) Redeclaration of fseeko64 differs from previous declaration on line 471 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 531.12: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long". "/usr/include/stdio.h", line 532.12: 1506-343 (S) Redeclaration of fsetpos64 differs from previous declaration on line 325 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 532.12: 1506-377 (I) The type "const long long*" of parameter 2 differs from the previous type "const long*". "/usr/include/stdio.h", line 533.16: 1506-343 (S) Redeclaration of ftello64 differs from previous declaration on line 472 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 533.16: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long". "/usr/include/stdio.h", line 528.12: 1506-343 (S) Redeclaration of fgetpos64 differs from previous declaration on line 323 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 528.12: 1506-377 (I) The type "long long*" of parameter 2 differs from the previous type "long* restrict". "/usr/include/stdio.h", line 531.12: 1506-343 (S) Redeclaration of fseeko64 differs from previous declaration on line 471 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 531.12: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long". "/usr/include/stdio.h", line 532.12: 1506-343 (S) Redeclaration of fsetpos64 differs from previous declaration on line 325 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 532.12: 1506-377 (I) The type "const long long*" of parameter 2 differs from the previous type "const long*". "/usr/include/stdio.h", line 533.16: 1506-343 (S) Redeclaration of ftello64 differs from previous declaration on line 472 of "/usr/include/stdio.h". "/usr/include/stdio.h", line 533.16: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long". error: Command "/usr/vac/bin/xlc_r -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -q32 -qbitfields=signed -qmaxmem=70000 -qalloca -bmaxdata:0x80000000 -Wl,-brtl -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DNDEBUG -O2 -Inumpy/core/include -Ibuild/src.aix-6.1-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/opt/freeware/include/python2.7 -Ibuild/src.aix-6.1-2.7/numpy/core/src/multiarray -Ibuild/src.aix-6.1-2.7/numpy/core/src/umath -c build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.c -o build/temp.aix-6.1-2.7/build/src.aix-6.1-2.7/numpy/core/src/npysort/heapsort.o" failed with exit status 1 And that's all of them, and all related to stdio.h. Unfortunately the obvious solution of explicitly including stdio.h didn't help. It also seems really odd that I would have to explicitly include standard system libraries. I'm hoping there some sort of solution to this that doesn't involve a massive amount of recoding. Thanks in advance for your help! Trevor

1 0

add .H attribute?
by Alan G Isaac July 24, 2013

July 24, 2013

With numpy arrays, I miss being able to spell a.conj().T as a.H, as one can with numpy matrices. Is adding this attribute to arrays ever under consideration? Thanks, Alan Isaac

18 58

Question regarding documentation of structured arrays
by Hanno Klemm July 24, 2013

July 24, 2013

Hi, I found the following inconsistency between the advertised and the actual behviour of structured arrays: on http://docs.scipy.org/doc/numpy/user/basics.rec.html it says in the section "Accessing multiple fields at once" Notice that the fields are always returned in the same order regardless of the sequence they are asked for. Fortunately that does not seem to be the case in my simple test (see below). Is that a change in behaviour I can rely on or am I somehow lucky in this particular example? Thanks, Hanno In [596]: test_array = np.ones((10),dtype=[('a', float), ('b',float)]) In [597]: test_array Out[597]: array([(1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0)], dtype=[('a', '<f8'), ('b', '<f8')]) In [598]: test_array['b']*=2 In [599]: test_array Out[599]: array([(1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0)], dtype=[('a', '<f8'), ('b', '<f8')]) In [600]: test_array[['b','a']] Out[600]: array([(2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0)], dtype=[('b', '<f8'), ('a', '<f8')]) In [601]: test_array[['a','b']] Out[601]: array([(1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0), (1.0, 2.0)], dtype=[('a', '<f8'), ('b', '<f8')]) In [602]: test_ab = test_array[['a','b']] In [603]: test_ba = test_array[['b','a']] In [604]: test_ba Out[604]: array([(2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0), (2.0, 1.0)], dtype=[('b', '<f8'), ('a', '<f8')]) In [605]: np.__version__ Out[605]: '1.6.1' -- Hanno Klemm klemm(a)phys.ethz.ch

2 3