Mailman 3 February 2010 - NumPy-Discussion

testing binary installer for OS X
by Ralf Gommers 27 Feb '10

27 Feb '10

Hi, I built an installer for OS X and did some testing on a clean computer. All NumPy tests pass. SciPy (0.7.1 binary) gives a number of errors and failures, I copied one of each type below. For full output see http://pastebin.com/eEcwkzKr . To me it looks like the failures are harmless, and the kdtree errors are not related to changes in NumPy. Is that right? I also installed Matplotlib (0.99.1.1 binary), but I did not find a way to test just the binary install except manually. Created some plots, looked fine. Then I ran the test script examples/tests/backend_driver.py from an svn checkout, but my laptop died before the tests finished (at least 25 mins). Basic output was: <example_name.py> 1.123 0 <example_name2.py> 0.987 0 ... Can anyone tell me what the best way is to test the MPL binary? Cheers, Ralf ====================================================================== ERROR: Failure: ValueError (numpy.dtype does not appear to be the correct type object) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/cluster/__init__.py", line 9, in <module> import vq, hierarchy File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/cluster/hierarchy.py", line 199, in <module> import scipy.spatial.distance as distance File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/spatial/__init__.py", line 7, in <module> from ckdtree import * File "numpy.pxd", line 30, in scipy.spatial.ckdtree (scipy/spatial/ckdtree.c:6087) ValueError: numpy.dtype does not appear to be the correct type object ====================================================================== ERROR: test_kdtree.test_random_compiled.test_approx ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/case.py", line 364, in setUp try_run(self.inst, ('setup', 'setUp')) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-0.11.1-py2.6.egg/nose/util.py", line 487, in try_run return func() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/spatial/tests/test_kdtree.py", line 133, in setUp self.kdtree = cKDTree(self.data) File "ckdtree.pyx", line 214, in scipy.spatial.ckdtree.cKDTree.__init__ (scipy/spatial/ckdtree.c:1563) NameError: np ====================================================================== FAIL: test_asfptype (test_base.TestBSR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/tests/test_base.py", line 242, in test_asfptype assert_equal( A.dtype , 'int32' ) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/utils.py", line 284, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: dtype('int32') DESIRED: 'int32' ====================================================================== FAIL: test_nrdtrisd (test_basic.TestCephes) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/tests/test_basic.py", line 349, in test_nrdtrisd assert_equal(cephes.nrdtrisd(0.5,0.5,0.5),0.0) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/testing/utils.py", line 301, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: -0 DESIRED: 0.0 ---------------------------------------------------------------------- Ran 2585 tests in 46.196s FAILED (KNOWNFAIL=4, SKIP=31, errors=28, failures=17) Out[2]: <nose.result.TextTestResult run=2585 errors=28 failures=17>

5 15

AUTO: Jon K Peck is out of the office (returning 03/06/2010)
by Jon K Peck 27 Feb '10

27 Feb '10

I am out of the office until 03/06/2010. I will be traveling through Saturday, March 6 and will be delayed responding to your email. I will have periodic email access, but I will be many time zones away from my usual location. Note: This is an automated response to your message "NumPy-Discussion Digest, Vol 41, Issue 118" sent on 2/26/10 17:43:30. This is the only notification you will receive while this person is away.

1 0

pickling/unpickling numpy.void and numpy.record for multiprocessing
by Martin Spacek 27 Feb '10

27 Feb '10

I have a 1D structured ndarray with several different fields in the dtype. I'm using multiprocessing.Pool.map() to iterate over this structured ndarray, passing one entry (of type numpy.void) at a time to the function to be called by each process in the pool. After much confusion about why this wasn't working, I finally realized that unpickling a previously pickled numpy.void results in garbage data. Here's an example: >>> import numpy as np >>> x = np.zeros((2,), dtype=('i4,f4,a10')) >>> x[:] = [(1,2.,'Hello'), (2,3.,"World")] >>> x array([(1, 2.0, 'Hello'), (2, 3.0, 'World')], dtype=[('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')]) >>> x[0] (1, 2.0, 'Hello') >>> type(x[0]) <type 'numpy.void'> >>> import pickle >>> s = pickle.dumps(x[0]) >>> newx0 = pickle.loads(s) >>> newx0 (30917960, 1.6904535998413144e-38, '\xd0\xef\x1c\x1eZ\x03\x00d') >>> s "cnumpy.core.multiarray\nscalar\np0\n(cnumpy\ndtype\np1\n(S'V18'\np2\nI0\nI1\ntp3\nRp4\n(I4\nS'|'\np5\nN(S'f0'\np6\nS'f1'\np7\nS'f2'\np8\ntp9\n(dp10\ng6\n(g1\n(S'i4'\np11\nI0\nI1\ntp12\nRp13\n(I4\nS'<'\np14\nNNNI-1\nI-1\nI0\nNtp15\nbI0\ntp16\nsg7\n(g1\n(S'f4'\np17\nI0\nI1\ntp18\nRp19\n(I4\nS'<'\np20\nNNNI-1\nI-1\nI0\nNtp21\nbI4\ntp22\nsg8\n(g1\n(S'S10'\np23\nI0\nI1\ntp24\nRp25\n(I4\nS'|'\np26\nNNNI10\nI1\nI0\nNtp27\nbI8\ntp28\nsI18\nI1\nI0\nNtp29\nbS'\\x01\\x00\\x00\\x00\\x00\\x00\\x00@Hello\\x00\\x00\\x00\\x00\\x00'\np30\ntp31\nRp32\n." >>> type(newx0) <type 'numpy.void'> >>> newx0.dtype dtype([('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')]) >>> x[0].dtype dtype([('f0', '<i4'), ('f1', '<f4'), ('f2', '|S10')]) >>> np.version.version '1.4.0' This also seems to be the case for recarrays with their numpy.record entries. I've tried using pickle and cPickle, with both the oldest and the newest pickling protocol. This is in numpy 1.4 on win32 and win64, and numpy 1.3 on 32-bit linux. I'm using Python 2.6.4 in all cases. I also just tried it on Python 2.5.2 with numpy 1.0.4. All have the same result, although the garbage data is different each time. I suppose numpy.void is as it suggests, a pointer to a specific place in memory. I'm just surprised that this pointer isn't dereferenced before pickling Or is it? I'm not skilled in interpreting the strings returned by pickle.dumps(). I do see the word "Hello" in the string, so maybe the problem is during unpickling. I've tried doing a copy, and even a deepcopy of a structured array numpy.void entry, with no luck. Is this a known limitation? Any suggestions on how I might get around this? Pool.map() pickles each numpy.void entry as it iterates over the structured array, before sending it to the next available process. My structured array only needs to be read from by my multiple processes (one per core), so perhaps there's a better way than sending copies of entries. Multithreading (using an implementation of a ThreadPool I found somewhere) doesn't work because I'm calling scipy.optimize.leastsq, which doesn't seem to release the GIL. Thanks! Martin

3 4

Can you help me find a dumb matrix multiply mistake
by Rick Muller 27 Feb '10

27 Feb '10

I'm making a mistake here, one that I suspect is a dumb error. I'm not as familiar with the math of complex hermetian matrices as I am with real symmetry matrices. I want to diagonalize the matrix: Y = matrix([[0,-1j],[1j,0]]) # this is the Y Pauli spin matrix Ey,Uy = eigh(Y) When I try to do: print Uy.H * diag(Ey) * Uy rather than getting Y back, I get: [[ 0.+0.j -1.+0.j] [-1.+0.j 0.+0.j]] I also tried dot(Uy.H,dot(diag(Ey),Uy)) to make sure this isn't a matrix/array problem with the same result. Can someone spot what I'm doing wrong? -- Rick Muller rpmuller(a)gmail.com 505-750-7557

2 2

Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I Passive Microwave Data
by Nicolas 26 Feb '10

26 Feb '10

Hello a VERY specific question, but I thought someone in the list might have used this dataset before and could give me some advice I am trying to read the daily Antarctic Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I Passive Microwave Data, as provided by the national snow and ice data center (http://nsidc.org), more specifically, these are the files as downloaded from their ftp site (ftp://sidads.colorado.edu/pub/DATASETS/seaice/polar-stereo/nasateam/final-g…) they are provided in binary files (e.g. nt_19980101_f13_v01_s.bin for the 1st of Jan. 1998) the metadata information (http://nsidc.org/cgi-bin/get_metadata.pl?id=nsidc-0051) gives the following information (for the polar stereographic projection): """ Data are scaled and stored as one-byte integers in flat binary arrays geographical coordinates N: 90° S: 30.98° E: 180° W: -180° Latitude Resolution: 25 km Longitude Resolution: 25 km Distribution Size: 105212 bytes per southern file """ I am unfamiliar with non self-documented files (used to hdf and netcdf !) and struggle to make sense of how to read these files and plot the corresponding maps, I've tried using the array module and the fromstring function file=open(filename,'rb') a=array.array('B',file.read()) var=numpy.fromstring(a,dtype=np.int) or directly the fromfile function var = numpy.fromfile(filename,np.int) with various array types and numpy.dtypes, but I dont make sense of these values, anyone has read those files before and plot the fields using python / numpy / matplotlib ? I am using python2.6 / numpy 1.3.0 / thanks in advance Nicolas -- _/\/¯¯¯¯¯¯\/\_ 33º49'45.24"S & 18º28'45.60"E Dr. Nicolas Fauchereau Senior Researcher CSIR - NRE Research Group: Ocean systems and climate 15 Lower Hope street, Rosebank 7700 South Africa tel: 021 658 2764 _/\/¯¯¯¯¯¯\/\_ 33º49'45.24"S & 18º28'45.60"E

3 2

What are the 'p', 'P' types?
by Charles R Harris 26 Feb '10

26 Feb '10

They are now typecodes but have no entries in the typename dictionary. The 'm', 'M' types also lack dictionary entries. Chuck

2 1

Re: [Numpy-discussion] read ascii file with quote delimited strings
by Sam Tygier 26 Feb '10

26 Feb '10

On Fri, 2010-02-26 at 07:56 +0000, numpy-discussion-request(a)scipy.org wrote: > Date: Thu, 25 Feb 2010 13:56:43 -0800 > From: Chris Barker <Chris.Barker(a)noaa.gov> > Subject: Re: [Numpy-discussion] read ascii file with quote delimited > strings > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: <4B86F21B.5070507(a)noaa.gov> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Warren Weckesser wrote: > > Does each column always contain the same number of characters? That > >is, are the field widths always the same? If so, you can ... > > > if not, I'd use the std lib csv module, then convert to numpy arrays, > not as efficient, but it should be easy. > > -Chris thanks, using cvs works. do you think it would be useful for this functionality to be added to loadtxt(), or would it be good to have a loadcsv() function? sam

2 1

How to test f2py?
by Charles R Harris 26 Feb '10

26 Feb '10

Hi All, I've made PyCObject -> PyCapsule changes to f2py for python3.1. How can I check that f2py still works as advertised before making a commit? Chuck

7 24

ANN: PyTables 2.2b3 released
by Francesc Alted 26 Feb '10

26 Feb '10

=========================== Announcing PyTables 2.2b3 =========================== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. This is the third, and most probably last, beta version of 2.2 release. The main addition in this beta version is the addition of Blosc (http://blosc.pytables.org), a high-speed compressor that is meant to work at similar speeds, or higher, than the memory-cache bandwidth in modern processors. This will allow for very high performance in internal, in-memory PyTables computations while still using compression. Remember that Blosc is still in *beta* and it is not meant for production purposes yet. You have been warned! In case you want to know more in detail what has changed in this version, have a look at: http://www.pytables.org/moin/ReleaseNotes/Release_2.2b3 You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://www.pytables.org/download/preliminary For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.2b3 Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- Francesc Alted

1 0

anyone to look at #1402?
by Nathaniel Smith 26 Feb '10

26 Feb '10

So there's this patch I submitted: http://projects.scipy.org/numpy/ticket/1402 Obviously not that high a priority in the grand scheme of things (it adds a function to compute the log-determinant directly), but I don't want to release a version of scikits.sparse with this functionality while the numpy patch is hanging in needs-review status (since the API might change), so it's a bit of a blocker for me. Anyone have a minute to take a look? Thanks! -- Nathaniel

4 6