Mailman 3 February 2012 - NumPy-Discussion

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer 31 Mar '16

31 Mar '16

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

SVD errors
by mtrumpis＠berkeley.edu 04 Jun '13

04 Jun '13

Hello list.. I've run into two SVD errors over the last few days. Both errors are identical in numpy/scipy. I've submitted a ticket for the 1st problem (numpy ticket #990). Summary is: some builds of the lapack_lite module linking against system LAPACK (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not converge" exception on my matrix. This error does occur using Mac's Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on said Ubuntu) Just today I've come across a negative singular value cropping up in an SVD of a different matrix. This error does occur on my ATLAS LAPACK based numpy, as well as on the Ubuntu setup. And once again, it does not happen in Octave/Matlab. I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running. Here are some npy files for the two different cases: https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy Mike

6 7

example reading binary Fortran file
by David Froger 13 Feb '13

13 Feb '13

Hy, My question is about reading Fortran binary file (oh no this question again...) Until now, I was using the unpack module like that : def lread(f,fourBeginning,fourEnd,*tuple): from struct import unpack """Reading a Fortran binary file in litte-endian""" if fourBeginning: f.seek(4,1) for array in tuple: for elt in xrange(array.size): transpose(array).flat[elt] = unpack(array.dtype.char,f.read(array.itemsize))[0] if fourEnd: f.seek(4,1) After googling, I read that fopen and npfille was deprecated, and we should use numpy.fromfile and ndarray.tofile, but despite of the documentaion, the cookbook, the mailling list and google, I don't succed in making a simple example. Considering the simple Fortran code below what is the Python script to read the four arrrays? What about if my pc is little endian and the file big endian? I think it will be a good idea to put the Fortran writting-arrays code and the Python reading-array script in the cookbook and maybe a page to help people comming from Fortran to start with Python ? Best, David Froger program makeArray implicit none integer,parameter:: nx=10,ny=20 real(4),dimension(nx,ny):: ux,uy,p integer :: i,j open(11,file='uxuyp.bin',form='unformatted') do i = 1,nx do j = 1,ny ux(i,j) = real(i*j) uy(i,j) = real(i)/real(j) p (i,j) = real(i) + real(j) enddo enddo write(11) ux,uy write(11) p close(11) end program makeArray

8 22

dtype '|S0' not understood
by David Warde-Farley 07 Oct '12

07 Oct '12

Howdy, It seems it's possible using e.g. In [25]: dtype([('foo', str)])Out[25]: dtype([('foo', '|S0')]) to get yourself a zero-length string. However dtype('|S0') results in a TypeError: data type not understood. I understand the stupidity of creating a 0-length string field but it's conceivable that it's accidental. For example, it could lead to a situation where you've created that field, are missing all the data you had meant to put in it, serialize with np.save, and upon np.load aren't able to get _any_ of your data back because the dtype descriptor is considered bogus (can you guess why I thought of this scenario?). It seems that either dtype(str) should do something more sensible than zero-length string, or it should be possible to create it with dtype('| S0'). Which should it be? David

3 3

Long-standing issue with using numpy in embedded CPython
by Yang Zhang 25 Sep '12

25 Sep '12

It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments: http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-w… http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html Is there any fix or workaround for this? Thanks. -- Yang Zhang http://yz.mit.edu/

9 15

"Symbol table not found" compiling numpy from git repository on Windows
by John Salvatier 18 Jul '12

18 Jul '12

Hello, I'm trying to compile numpy on Windows 7 using the command: "python setup.py config --compiler=mingw32 build" but I get an error about a symbol table not found. Anyone know how to work around this or what to look into? building library "npymath" sources Building msvcr library: "C:\Python26\libs\libmsvcr90.a" (from C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll) objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python26\lib\distutils\command\build.py", line 134, in run self.run_command(cmd_name) File "C:\Python26\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 646, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "C:\Python26\lib\distutils\command\config.py", line 257, in try_link self._check_compiler() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\config.py", line 45, in _check_compiler old_config._check_compiler(self) File "C:\Python26\lib\distutils\command\config.py", line 107, in _check_compiler dry_run=self.dry_run, force=1) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\ccompiler.py", line 560, in new_compiler compiler = klass(None, dry_run, force) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 94, in __init__ msvcr_success = build_msvcr_library() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 362, in build_msvcr_library generate_def(dll_file, def_file) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 282, in generate_def raise ValueError("Symbol table not found") ValueError: Symbol table not found Thank you, John

4 5

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
by Nathaniel Smith 21 Mar '12

21 Mar '12

[Re-adding the list to the To: field, after it got dropped accidentally] On Tue, Feb 28, 2012 at 12:28 AM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: > Excerpts from Nathaniel Smith's message of Mon Feb 27 17:33:52 -0500 2012: >> On Mon, Feb 27, 2012 at 6:02 PM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: >> > Excerpts from Nathaniel Smith's message of Mon Feb 27 12:07:11 -0500 2012: >> >> On Mon, Feb 27, 2012 at 2:44 PM, Erin Sheldon <erin.sheldon(a)gmail.com> wrote: >> >> > What I've got is a solution for writing and reading structured arrays to >> >> > and from files, both in text files and binary files. It is written in C >> >> > and python. It allows reading arbitrary subsets of the data efficiently >> >> > without reading in the whole file. It defines a class Recfile that >> >> > exposes an array like interface for reading, e.g. x=rf[columns][rows]. >> >> >> >> What format do you use for binary data? Something tiled? I don't >> >> understand how you can read in a single column of a standard text or >> >> mmap-style binary file any more efficiently than by reading the whole >> >> file. >> > >> > For binary, I just seek to the appropriate bytes on disk and read them, >> > no mmap. The user must have input an accurate dtype describing rows in >> > the file of course. This saves a lot of memory and time on big files if >> > you just need small subsets. >> >> Have you quantified the time savings? I'd expect this to either be the >> same speed or slower than reading the entire file. > > Nathaniel - > > Yes I've verified it, but as you point out below there are pathological > cases. See below. > >> The reason is that usually the OS cannot read just a few bytes from a >> middle of a file -- if it is reading at all, it will read at least a >> page (4K on linux). If your rows are less than 4K in size, then >> reading a little bit out of each row means that you will be loading >> the entire file from disk regardless. You avoid any parsing overhead >> for the skipped columns, but for binary files that should be zero. >> (Even if you're doing endian conversion or something it should still >> be trivial compared to disk speed.) > > I'll say up front, the speed gains for binary data are often huge over > even numpy.memmap because memmap is not column aware. My code doesn't > have that limitation. Hi Erin, I don't doubt your observations, but... there must be something more going on! In a modern VM implementation, what happens when you request to read from an arbitrary offset in the file is: 1) The OS works out which disk page (or set of pages, for a longer read) contains the given offset 2) It reads those pages from the disk, and loads them into some OS owned buffers (the "page cache") 3) It copies the relevant bytes out of the page cache into the buffer passed to read() And when you mmap() and then attempt to access some memory at an arbitrary offset within the mmap region, what happens is: 1) The processor notices that it doesn't actually know how the memory address given maps to real memory (a tlb miss), so it asks the OS 2) The OS notices that this is a memory-mapped region, and works out which disk page maps to the given memory address 3) It reads that page from disk, and loads it into some OS owned buffers (the "page cache") 4) It tells the processor That is, reading at a bunch of fixed offsets inside a large memory mapped array (which is what numpy does when you request a single column of a recarray) should end up issuing *exactly the same read commands* as writing code that explicitly seeks to those addresses and reads them. But, I realized I've never actually tested this myself, so I wrote a little test (attached). It reads a bunch of uint32's at equally-spaced offsets from a large file, using either mmap, explicit seeks, or the naive read-everything approach. I'm finding it very hard to get precise results, because I don't have a spare drive and anything that touches the disk really disrupts the timing here (and apparently Ubuntu no longer has a real single-user mode :-(), but here are some examples on a 200,000,000 byte file with different simulated row sizes: 1024 byte rows: Mode: MMAP. Checksum: bdd205e9. Time: 3.44 s Mode: SEEK. Checksum: bdd205e9. Time: 3.34 s Mode: READALL. Checksum: bdd205e9. Time: 3.53 s Mode: MMAP. Checksum: bdd205e9. Time: 3.39 s Mode: SEEK. Checksum: bdd205e9. Time: 3.30 s Mode: READALL. Checksum: bdd205e9. Time: 3.17 s Mode: MMAP. Checksum: bdd205e9. Time: 3.16 s Mode: SEEK. Checksum: bdd205e9. Time: 3.41 s Mode: READALL. Checksum: bdd205e9. Time: 3.43 s 65536 byte rows (64 KiB): Mode: MMAP. Checksum: da4f9d8d. Time: 3.25 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.27 s Mode: READALL. Checksum: da4f9d8d. Time: 3.16 s Mode: MMAP. Checksum: da4f9d8d. Time: 3.34 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.36 s Mode: READALL. Checksum: da4f9d8d. Time: 3.44 s Mode: MMAP. Checksum: da4f9d8d. Time: 3.18 s Mode: SEEK. Checksum: da4f9d8d. Time: 3.19 s Mode: READALL. Checksum: da4f9d8d. Time: 3.16 s 1048576 byte rows (1 MiB): Mode: MMAP. Checksum: 22963df9. Time: 1.57 s Mode: SEEK. Checksum: 22963df9. Time: 1.44 s Mode: READALL. Checksum: 22963df9. Time: 3.13 s Mode: MMAP. Checksum: 22963df9. Time: 1.59 s Mode: SEEK. Checksum: 22963df9. Time: 1.43 s Mode: READALL. Checksum: 22963df9. Time: 3.16 s Mode: MMAP. Checksum: 22963df9. Time: 1.55 s Mode: SEEK. Checksum: 22963df9. Time: 1.66 s Mode: READALL. Checksum: 22963df9. Time: 3.15 s And for comparison: In [32]: a = np.memmap("src/bigfile", np.uint32, "r") In [33]: time hex(np.sum(a[::1048576//4][:-1], dtype=a.dtype)) CPU times: user 0.00 s, sys: 0.01 s, total: 0.01 s Wall time: 1.54 s Out[34]: '0x22963df9L' (Ubuntu Linux 2.6.38-13, traditional spinning-disk media) So, in this test: For small rows: seeking is irrelevant, reading everything is just as fast. (And the cutoff for "small" is not very small... I tried 512KiB too and it looked like 32KiB). For large rows: seeking is faster than reading everything. But mmap, explicit seeks, and np.memmap all act the same. I guess it's possible the difference you're seeing could just mean that, like, Windows has a terrible VM subsystem, but that would be weird. > In the ascii case the gains for speed are less > for the reasons you point out; you have to read through the data even to > skip rows and fields. Then it is really about memory. > > > Even for binary, there are pathological cases, e.g. 1) reading a random > subset of nearly all rows. 2) reading a single column when rows are > small. In case 2 you will only go this route in the first place if you > need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. > I wrote this code to deal with a typical use case of mine where small > subsets of binary or ascii data are begin read from a huge file. It > solves that problem. Cool. >> If your rows are greater than 4K in size, then seeking will allow you >> to avoid loading some parts of the file from disk... but it also might >> defeat the OS's readahead heuristics, which means that instead of >> doing a streaming read, you're might be doing a disk seek for every >> row. On an SSD, this is fine, and probably a nice win. On a >> traditional spinning-disk, losing readahead will cause a huge >> slowdown; you should only win if each row is like... a megabyte >> apiece. Seeks are much much slower than continuous reads. So I'm >> surprised if this actually helps! But that's just theory, so I am >> curious to see the actual numbers. >> >> Re: memory savings -- it's definitely a win to avoid allocating the >> whole array if you just want to read one column, but you can >> accomplish that without any particular cleverness in the low-level >> file reading code. You just read the first N rows into a fixed-size >> temp buffer, copy out the relevant column into the output array, >> repeat. > > Certainly this could also be done that way, and would be equally good > for some cases. > > I've already got all the C code to do this stuff, so it not much work > for me to hack it into my numpy fork. If it turns out there are cases > that are faster using another method, we should root them out during > testing and add the appropriate logic. Cool. I'm just a little concerned that, since we seem to have like... 5 different implementations of this stuff all being worked on at the same time, we need to get some consensus on which features actually matter, so they can be melded together into the Single Best File Reader Evar. An interface where indexing and file-reading are combined is significantly more complicated than one where the core file-reading inner-loop can ignore indexing. So far I'm not sure why this complexity would be worthwhile, so that's what I'm trying to understand. Cheers, -- Nathaniel > Also, for some crazy ascii files we may want to revert to pure python > anyway, but I think these should be special cases that can be flagged > at runtime through keyword arguments to the python functions. > > BTW, did you mean to go off-list? > > cheers, > > -e > -- > Erin Scott Sheldon > Brookhaven National Laboratory

7 11

dtype comparison, hash
by Andreas Kloeckner 06 Mar '12

06 Mar '12

Hi all, Two questions: - Are dtypes supposed to be comparable (i.e. implement '==', '!=')? - Are dtypes supposed to be hashable? PyCUDA and PyOpenCL assume both in a few places, but at least hashability doesn't seem to be true. (If so, __hash__ should be implemented to throw an error. If not, we found a bug in the hash implementation.) Thanks! Andreas

3 6

test errors on deprecation/runtime warnings
by Ralf Gommers 05 Mar '12

05 Mar '12

Hi, Last week we merged https://github.com/numpy/numpy/pull/201, which causes DeprecationWarning's and RuntimeWarning's to be converted to errors if they occur when running the test suite. The purpose of that is to make sure that code that still uses other deprecated code (or code that for some reason generates warnings) to be cleaned up. In principle this is a good idea IMHO, but after merging we quickly found some problems with failing tests in scipy. Because this potentially affects any other projects or users that use the numpy NoseTester test runner, here's a proposal to deal with this issue: - make this behavior configurable by a keyword in `NoseTester.__init__()` - default to raising an error in numpy master - when making a branch for release, immediately set the default to not raise. Do this not only for the 1.7 release, but for any future release (at least until the oldest numpy version still in use has the keyword). The reason for this is that otherwise a new numpy release will trigger test failures in older releases of scipy or other packages. The pros: - We find issues that otherwise get ignored (see the issues Christoph just found when compiling with MSVC). - We're forced to clean up code when submitting PRs, instead of letting the warnings accumulate and having to deal with them just before release time. The con: - You may see test errors if you run a released version of scipy with a development version of numpy. Opinions? Concerns? Ralf

2 3

NumPy-Discussion February 2012