Mailman 3 June 2013 - NumPy-Discussion

ndarray: How to create and initialize with a value other than ones or zeros?
by James Adams June 6, 2013

June 6, 2013

I would like to create an array object and initialize the array's values with an arbitrary fill value, like you can do using the ones() and zeros() creation routines to create and initialize arrays with ones or zeros. Is there an easy way to do this? If this isn't possible then what is the most efficient way to initialize a numpy array with an arbitrary fill value? In order to provide such an array creation routine I can imagine that it'd be as simple as taking the code for ones() and/or zeros() and modifying that code so that it provides an additional fill value argument and then within the section which does the initialization of the array it could use that fill value instead of 1 or 0. Is this a naive assumption? Thanks in advance for your help with this issue. --James

3 2

floats coerced to string with "{:f}".format() ?
by Maccarthy, Jonathan K June 6, 2013

June 6, 2013

Hi everyone, I've looked in the mailing list archives and with the googles, but haven't yet found any hints with this question... I have a float field in a NumPy record that looks like it's being substituted as a string in the Python "{:f}".format() mini-language, thus throwing an error: In [1]: tmp = np.rec.array([('XYZZ', 2001123, -23.82396)], dtype=np.dtype([('sta', '|S6'), ('ondate', '<i8'), ('lat', '<f4')]))[0] In [2]: type(tmp) Out[3]: numpy.core.records.record In [3]: tmp Out[3]: ('XYZZ', 2001123, -23.823917388916016) In [4]: tmp.sta, tmp.ondate, tmp.lat Out[4]: ('XYZZ', 2001123, -23.823917) # strings and integers work In [5]: '{0.sta:6.6s} {0.ondate:8d}'.format(tmp) Out[5]: 'XYZZ 2001123' # "lat" is a float, but it seems to be coerced to a string first, and failing In [6]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:11.6f}'.format(tmp) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/jkmacc/<ipython-input-312-bff8066cfde8> in <module>() ----> 1 '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6f}'.format(tmp) ValueError: Unknown format code 'f' for object of type 'str' # string formatting for doesn't fail In [7]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6s}'.format(tmp) Out[7]: 'XYZZ 2001123 -23.82' This also fails: In [7]: "{:f}".format(np.array(3.2, dtype='f4')) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/jkmacc/<ipython-input-314-33119128e3e6> in <module>() ----> 1 "{:f}".format(np.array(3.2, dtype='f4')) ValueError: Unknown format code 'f' for object of type 'str' Does anyone understand what's happening? Thanks for your help. Best, Jon

3 6

Re: [Numpy-discussion] suggested change of behavior for interp
by Slavin, Jonathan June 5, 2013

June 5, 2013

The simplest monotonicity test that I've seen is: dx = np.diff(x) monotonic = np.all(dx < 0.) or np.all(dx > 0.) I expect that this is pretty fast, though I haven't tested it yet. If we want to make checking optional, then I think the default should be to check with the option to skip the check. Jon On Tue, Jun 4, 2013 at 9:03 PM, <numpy-discussion-request(a)scipy.org> wrote: > From: Eric Firing <efiring(a)hawaii.edu> > To: numpy-discussion(a)scipy.org > Cc: > Date: Tue, 04 Jun 2013 15:08:29 -1000 > Subject: Re: [Numpy-discussion] suggested change of behavior for interp > On 2013/06/04 2:05 PM, Charles R Harris wrote: > >> >> >> On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan >> <jslavin(a)cfa.harvard.edu <mailto:jslavin@cfa.harvard.**edu<jslavin(a)cfa.harvard.edu>>> >> wrote: >> >> Hi, >> >> I would like to suggest that the behavior of numpy.interp be changed >> regarding treatment of situations in which the x-coordinates are not >> monotonically increasing. Specifically, it seems to me that interp >> should work correctly when the x-coordinate is decreasing >> monotonically. Clearly it cannot work if the x-coordinate is not >> monotonic, but in that case it should raise an exception. Currently >> if x is not increasing it simply silently fails, providing incorrect >> values. This fix could be as simple as a monotonicity test and >> inversion if necessary (plus a raise statement for non-monotonic >> cases). >> >> >> Seems reasonable, although it might add a bit of execution time. >> > > The monotonicity test should be an option if it is available at all; when > interpolating a small number of points from a large pair of arrays, the > single sweep through the whole array could dominate the execution time. > Checking for increasing versus decreasing, in contrast, can be done fast, > so handling the decreasing case transparently is reasonable. > > Eric > ________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin(a)cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 fax: (617) 496-7577 USA ________________________________________________________

4 6

suggested change of behavior for interp
by Slavin, Jonathan June 5, 2013

June 5, 2013

Hi, I would like to suggest that the behavior of numpy.interp be changed regarding treatment of situations in which the x-coordinates are not monotonically increasing. Specifically, it seems to me that interp should work correctly when the x-coordinate is decreasing monotonically. Clearly it cannot work if the x-coordinate is not monotonic, but in that case it should raise an exception. Currently if x is not increasing it simply silently fails, providing incorrect values. This fix could be as simple as a monotonicity test and inversion if necessary (plus a raise statement for non-monotonic cases). Jon ________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin(a)cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 fax: (617) 496-7577 USA ________________________________________________________

5 5

numpy tests errors and failures
by Warren Weckesser June 4, 2013

June 4, 2013

I'm getting a failure and two errors with the latest master branch: $ python -c "import numpy; numpy.test('full')" Running unit tests for numpy NumPy version 1.8.0.dev-dff8c94 NumPy is installed in /home/warren/local_numpy/lib/python2.7/site-packages/numpy Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013, 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] nose version 1.3.0 .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................... ====================================================================== ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1, 1000020.0000000099], [0, nan, 1000000.0]) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 1253, in tst_isclose_allclose assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y)) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py", line 2008, in allclose return all(less_equal(abs(x-y), atol + rtol * abs(y))) RuntimeWarning: invalid value encountered in absolute ====================================================================== ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan, nan]) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 1253, in tst_isclose_allclose assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y)) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py", line 2008, in allclose return all(less_equal(abs(x-y), atol + rtol * abs(y))) RuntimeWarning: invalid value encountered in absolute ====================================================================== FAIL: Test numpy dot with different order C, F ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py", line 114, in test_dot_array_order assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 458, in assert_almost_equal return assert_array_almost_equal(actual, desired, decimal, err_msg) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 819, in assert_array_almost_equal header=('Arrays are not almost equal to %d decimals' % decimal)) File "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 652, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal to 30 decimals (mismatch 26.0%) x: array([[ 0.60970873, 1.6909554 , -1.08851945, -1.82058012, -3.95746613, 1.52435613, -0.59853059, -3.72278619, 3.82375932, 5.51367044], [-3.58154893, -2.06231236, -0.0656727 , 1.4737345 , 2.6068747 ,... y: array([[ 0.60970873, 1.6909554 , -1.08851945, -1.82058001, -3.95746613, 1.52435613, -0.59853059, -3.72278643, 3.82375932, 5.51367044], [-3.58154917, -2.06231236, -0.0656727 , 1.4737345 , 2.6068747 ,... ---------------------------------------------------------------------- Ran 5151 tests in 60.506s FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1) Warren

4 8

SVD errors
by mtrumpis＠berkeley.edu June 4, 2013

June 4, 2013

Hello list.. I've run into two SVD errors over the last few days. Both errors are identical in numpy/scipy. I've submitted a ticket for the 1st problem (numpy ticket #990). Summary is: some builds of the lapack_lite module linking against system LAPACK (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not converge" exception on my matrix. This error does occur using Mac's Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on said Ubuntu) Just today I've come across a negative singular value cropping up in an SVD of a different matrix. This error does occur on my ATLAS LAPACK based numpy, as well as on the Ubuntu setup. And once again, it does not happen in Octave/Matlab. I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running. Here are some npy files for the two different cases: https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy Mike

6 7

ANN: PyTables 3.0
by Antonio Valentino June 3, 2013

June 3, 2013

=========================== Announcing PyTables 3.0.0 =========================== We are happy to announce PyTables 3.0.0. PyTables 3.0.0 comes after about 5 years from the last major release (2.0) and 7 months since the last stable release (2.4.0). This is new major release and an important milestone for the PyTables project since it provides the long waited support for Python 3.x, which has been around for 4 years. Almost all of the core numeric/scientific packages for Python already support Python 3 so we are very happy that now also PyTables can provide this important feature. What's new ========== A short summary of main new features: - Since this release, PyTables now provides full support to Python 3 - The entire code base is now more compliant with coding style guidelines described in PEP8. - Basic support for HDF5 drivers. It now is possible to open/create an HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE drivers. - Basic support for in-memory image files. An HDF5 file can be set from or copied into a memory buffer. - Implemented methods to get/set the user block size in a HDF5 file. - All read methods now have an optional *out* argument that allows to pass a pre-allocated array to store data. - Added support for the floating point data types with extended precision (Float96, Float128, Complex192 and Complex256). - Consistent ``create_xxx()`` signatures. Now it is possible to create all data sets Array, CArray, EArray, VLArray, and Table from existing Python objects. - Complete rewrite of the `nodes.filenode` module. Now it is fully compliant with the interfaces defined in the standard `io` module. Only non-buffered binary I/O is supported currently. Please refer to the RELEASE_NOTES document for a more detailed list of changes in this release. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.0.0 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

easy way to collapse the last two dimension of an array?
by Chao YUE June 3, 2013

June 3, 2013

Dear all, I have an array with 4 dim: In [24]: dd.shape Out[24]: (12, 13, 120, 170) I would like to collapse the last two dim for applying np.sum(axis=-1) In [25]: dd.reshape(12,13,-1).shape Out[25]: (12, 13, 20400) is there a more general way to do this? something like In [21]: dd.reshape(*dd.shape[0:2],-1).shape ------------------------------------------------------------ File "<ipython console>", line 1 SyntaxError: only named arguments may follow *expression (<ipython console>, line 1) thanks a lot for any hints, cheers, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 3

About speed vs math...
by Jerome Kieffer June 3, 2013

June 3, 2013

Hello, I am giving some introduction tutorials to numpy and we notices a big difference in speed between nuumpy and math for trigonometric operations: In [3]: %timeit numpy.sin(1) 100000 loops, best of 3: 2.27 us per loop In [4]: %timeit math.sin(1) 10000000 loops, best of 3: 92.3 ns per loop While not surprising, I did not expect numpy to be so much slower (25x)... Is this normal ? Cheers, -- Jérôme Kieffer Data analysis unit - ESRF

3 3

ANN: PyTables 3.0
by Anthony Scopatz June 2, 2013

June 2, 2013

=========================== Announcing PyTables 3.0.0 =========================== We are happy to announce PyTables 3.0.0. PyTables 3.0.0 comes after about 5 years from the last major release (2.0) and 7 months since the last stable release (2.4.0). This is new major release and an important milestone for the PyTables project since it provides the long waited support for Python 3.x, which has been around for 4 years. Almost all of the core numeric/scientific packages for Python already support Python 3 so we are very happy that now also PyTables can provide this important feature. What's new ========== A short summary of main new features: - Since this release, PyTables now provides full support to Python 3 - The entire code base is now more compliant with coding style guidelines described in PEP8. - Basic support for HDF5 drivers. It now is possible to open/create an HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE drivers. - Basic support for in-memory image files. An HDF5 file can be set from or copied into a memory buffer. - Implemented methods to get/set the user block size in a HDF5 file. - All read methods now have an optional *out* argument that allows to pass a pre-allocated array to store data. - Added support for the floating point data types with extended precision (Float96, Float128, Complex192 and Complex256). - Consistent ``create_xxx()`` signatures. Now it is possible to create all data sets Array, CArray, EArray, VLArray, and Table from existing Python objects. - Complete rewrite of the `nodes.filenode` module. Now it is fully compliant with the interfaces defined in the standard `io` module. Only non-buffered binary I/O is supported currently. Please refer to the RELEASE_NOTES document for a more detailed list of changes in this release. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.0.0 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

1.8 release update
by Charles R Harris June 2, 2013

June 2, 2013

Hi All, Most of the PR's that were mentioned as desirable for the 1.8 release have been merged, or look to be merged in the next week or two. The current list of blockers<https://github.com/numpy/numpy/issues?milestone=1&page=1&state=open>doesn't look to severe to me but I suspect that it is incomplete. The major outstanding issue I see is datetime and I'd like to get a small group together to work that out. As a start I think such a group should include Christopher Barker and Wes McKinney. Suggestions for other folks, or even volunteers, to be part of such a group are welcome. A lot of stuff has piled up over the last year for inclusion in the 1.8 release, and I'm sure some bugs and regressions have crept in along with the new code. On that account a lengthy wringing out period is probably going to be needed, so the sooner we can get the 1.8 branch tagged the better. I'd like to shoot for getting that done in 2-3 weeks. If there look to be difficult issues remaining, perhaps they can be worked out at scipy2013 if there is enough interest. Thoughts? Chuck

2 1

Effect of deprecating non-integers for integer function arguments
by Sebastian Berg June 1, 2013

June 1, 2013

Hi, the current numpy master has deprecated non-integers for the use of indexing (not-fancy yet). However I think this should be moved further down in the numpy machinery which means that the conversion utils provided by numpy would generally raise warnings for non-integers. This means that for most numpy functions such as reshape, etc. the use of non-integers for arguments that are naturally integers is deprecated, which may also affect third party code in principle. Testing this against current SciPy appearently causes 56 failures (with deprecation warnings being raised). Are other project similarly affect by this change? The branch implementing this can be found in the PR https://github.com/numpy/numpy/pull/3243 Regards, Sebastian

2 2