Mailman 3 July 2008 - NumPy-Discussion

quick question about numpy deprecation warnings
by Charles Doutriaux 15 Jul '08

15 Jul '08

Hi I have a quick question and i hope somebody can answer me (I admit I should first really check the numpy doc) I have been porting old Numeric based C code to numpy/numpy.ma for the alst couple weeks. Just as I thought I was done, this morning I updatethe numpy trunk and I now get the following deprecation warnings DeprecationWarning: PyArray_FromDims: use PyArray_SimpleNew. DeprecationWarning: PyArray_FromDimsAndDataAndDescr: use PyArray_NewFromDescr. My quick question is: can I simply replace the function names or is the arg list different? Thanks, Charles.

2 1

Revised list of backport candidates for 1.1.1
by Charles R Harris 15 Jul '08

15 Jul '08

After the first round of backports the following remain. charris r5259 r5312 r5322 r5324 r5392 r5394 r5399 r5406 r5407 dhuard r5254 fperez r5298 r5301 r5303 oliphant r5245 r5255 Chuck

2 2

permissions on tests in numpy and scipy
by John Hunter 15 Jul '08

15 Jul '08

I have a rather unconventional install pipeline at work and owner only read permissions on a number of the tests are causing me minor problems. It appears the permissions on the tests are set rather inconsistently in numpy and python -- is there any reason not to make these all 644? johnh@flag:site-packages> find numpy -name "test_*.py"|xargs ls -l -rw------- 1 johnh research 2769 Jul 14 12:01 numpy/core/tests/test_defchararray.py -rw------- 1 johnh research 8283 Jul 14 12:01 numpy/core/tests/test_defmatrix.py -rw------- 1 johnh research 1769 Jun 25 10:00 numpy/core/tests/test_errstate.py -rw------- 1 johnh research 1508 Jun 25 10:00 numpy/core/tests/test_memmap.py -rw------- 1 johnh research 33334 Jul 14 12:01 numpy/core/tests/test_multiarray.py -rw------- 1 johnh research 26695 Jul 14 12:01 numpy/core/tests/test_numeric.py -rw------- 1 johnh research 13781 Jul 14 12:01 numpy/core/tests/test_numerictypes.py -rw------- 1 johnh research 1113 Jul 14 12:01 numpy/core/tests/test_print.py -rw------- 1 johnh research 4290 Jul 14 12:01 numpy/core/tests/test_records.py -rw------- 1 johnh research 39370 Jul 14 12:01 numpy/core/tests/test_regression.py -rw------- 1 johnh research 4097 Jul 14 12:01 numpy/core/tests/test_scalarmath.py -rw------- 1 johnh research 9330 Jun 25 10:00 numpy/core/tests/test_ufunc.py -rw------- 1 johnh research 7583 Jul 14 12:01 numpy/core/tests/test_umath.py -rw------- 1 johnh research 11264 Jul 14 12:01 numpy/core/tests/test_unicode.py -rw------- 1 johnh research 222 Jul 14 12:00 numpy/distutils/tests/f2py_ext/tests/test_fib2.py -rw------- 1 johnh research 220 Jul 14 12:00 numpy/distutils/tests/f2py_f90_ext/tests/test_foo.py -rw------- 1 johnh research 220 Jul 14 12:00 numpy/distutils/tests/gen_ext/tests/test_fib3.py -rw------- 1 johnh research 277 Jul 14 12:00 numpy/distutils/tests/pyrex_ext/tests/test_primes.py -rw------- 1 johnh research 387 Jul 14 12:00 numpy/distutils/tests/swig_ext/tests/test_example.py -rw------- 1 johnh research 276 Jul 14 12:00 numpy/distutils/tests/swig_ext/tests/test_example2.py -rw------- 1 johnh research 1800 Jul 14 12:00 numpy/distutils/tests/test_fcompiler_gnu.py -rw------- 1 johnh research 2421 Jun 25 09:59 numpy/distutils/tests/test_misc_util.py -rw-r--r-- 1 johnh research 64338 Jun 25 09:59 numpy/f2py/lib/parser/test_Fortran2003.py -rw-r--r-- 1 johnh research 24785 Jun 25 09:59 numpy/f2py/lib/parser/test_parser.py -rw------- 1 johnh research 574 Jul 14 12:00 numpy/fft/tests/test_fftpack.py -rw------- 1 johnh research 1256 Jul 14 12:00 numpy/fft/tests/test_helper.py -rw------- 1 johnh research 9948 Jul 14 12:01 numpy/lib/tests/test__datasource.py -rw------- 1 johnh research 4088 Jul 14 12:01 numpy/lib/tests/test_arraysetops.py -rw------- 1 johnh research 809 Jul 14 12:01 numpy/lib/tests/test_financial.py -rw------- 1 johnh research 23794 Jul 14 12:01 numpy/lib/tests/test_format.py -rw------- 1 johnh research 30772 Jul 14 12:01 numpy/lib/tests/test_function_base.py -rw------- 1 johnh research 1587 Jun 25 10:00 numpy/lib/tests/test_getlimits.py -rw------- 1 johnh research 2111 Jul 14 12:01 numpy/lib/tests/test_index_tricks.py -rw------- 1 johnh research 7691 Jul 14 12:01 numpy/lib/tests/test_io.py -rw------- 1 johnh research 999 Jun 25 10:00 numpy/lib/tests/test_machar.py -rw------- 1 johnh research 2431 Jun 25 10:00 numpy/lib/tests/test_polynomial.py -rw------- 1 johnh research 1409 Jul 14 12:01 numpy/lib/tests/test_regression.py -rw------- 1 johnh research 14353 Jul 14 12:01 numpy/lib/tests/test_shape_base.py -rw------- 1 johnh research 6958 Jul 14 12:01 numpy/lib/tests/test_stride_tricks.py -rw------- 1 johnh research 6735 Jul 14 12:01 numpy/lib/tests/test_twodim_base.py -rw------- 1 johnh research 9579 Jul 14 12:01 numpy/lib/tests/test_type_check.py -rw------- 1 johnh research 1771 Jun 25 10:00 numpy/lib/tests/test_ufunclike.py -rw------- 1 johnh research 8279 Jul 14 12:01 numpy/linalg/tests/test_linalg.py -rw------- 1 johnh research 1739 Jul 14 12:01 numpy/linalg/tests/test_regression.py -rw------- 1 johnh research 78819 Jun 25 10:00 numpy/ma/tests/test_core.py -rw------- 1 johnh research 15744 Jul 14 12:01 numpy/ma/tests/test_extras.py -rw------- 1 johnh research 17759 Jun 25 10:00 numpy/ma/tests/test_mrecords.py -rw------- 1 johnh research 33009 Jun 25 10:00 numpy/ma/tests/test_old_ma.py -rw------- 1 johnh research 5956 Jun 25 10:00 numpy/ma/tests/test_subclassing.py -rw------- 1 johnh research 3173 Jul 14 12:01 numpy/oldnumeric/tests/test_oldnumeric.py -rw------- 1 johnh research 2043 Jun 25 09:59 numpy/random/tests/test_random.py -rw------- 1 johnh research 4330 Jun 25 10:00 numpy/testing/tests/test_utils.py -rw------- 1 johnh research 3356 Jun 25 10:00 numpy/tests/test_ctypeslib.py johnh@flag:site-packages> find scipy -name "test_*.py"|xargs ls -l -rw------- 1 johnh research 7653 Jul 14 12:01 scipy/cluster/tests/test_hierarchy.py -rw------- 1 johnh research 5852 Jul 14 12:01 scipy/cluster/tests/test_vq.py -rw------- 1 johnh research 13552 Jul 14 12:01 scipy/fftpack/tests/test_basic.py -rw-r--r-- 1 johnh research 1846 Mar 10 12:57 scipy/fftpack/tests/test_helper.py -rw-r--r-- 1 johnh research 11554 Mar 10 12:57 scipy/fftpack/tests/test_pseudo_diffs.py -rw------- 1 johnh research 4192 Apr 25 09:15 scipy/integrate/tests/test_integrate.py -rw------- 1 johnh research 3728 Feb 19 17:12 scipy/integrate/tests/test_quadpack.py -rw------- 1 johnh research 1035 Feb 19 17:12 scipy/integrate/tests/test_quadrature.py -rw------- 1 johnh research 5310 Jul 14 12:01 scipy/interpolate/tests/test_fitpack.py -rw------- 1 johnh research 8014 Jul 14 12:01 scipy/interpolate/tests/test_interpolate.py -rw------- 1 johnh research 11273 May 7 12:43 scipy/interpolate/tests/test_polyint.py -rw------- 1 johnh research 929 Apr 25 09:15 scipy/interpolate/tests/test_rbf.py -rw------- 1 johnh research 8625 Jul 14 12:01 scipy/io/matlab/tests/test_mio.py -rw------- 1 johnh research 2010 Apr 25 09:15 scipy/io/tests/test_array_import.py -rw------- 1 johnh research 10246 Jul 14 12:01 scipy/io/tests/test_mmio.py -rw-r--r-- 1 johnh research 3621 Mar 10 12:57 scipy/io/tests/test_npfile.py -rw------- 1 johnh research 7276 Apr 25 09:15 scipy/io/tests/test_recaster.py -rw-r--r-- 1 johnh research 8320 Mar 10 12:57 scipy/lib/blas/tests/test_blas.py -rw-r--r-- 1 johnh research 17225 Mar 10 12:57 scipy/lib/blas/tests/test_fblas.py -rw------- 1 johnh research 4271 Apr 25 09:15 scipy/lib/lapack/tests/test_lapack.py -rw-r--r-- 1 johnh research 230 Mar 10 12:57 scipy/linalg/tests/test_atlas_version.py -rw-r--r-- 1 johnh research 13148 Mar 10 12:57 scipy/linalg/tests/test_basic.py -rw-r--r-- 1 johnh research 7597 Mar 10 12:57 scipy/linalg/tests/test_blas.py -rw------- 1 johnh research 32051 Apr 25 09:15 scipy/linalg/tests/test_decomp.py -rw-r--r-- 1 johnh research 17095 Mar 10 12:57 scipy/linalg/tests/test_fblas.py -rw-r--r-- 1 johnh research 2017 Mar 10 12:57 scipy/linalg/tests/test_lapack.py -rw-r--r-- 1 johnh research 3515 Mar 10 12:57 scipy/linalg/tests/test_matfuncs.py -rw-r--r-- 1 johnh research 961 Mar 10 12:57 scipy/maxentropy/tests/test_maxentropy.py -rw------- 1 johnh research 1399 Feb 19 17:12 scipy/misc/tests/test_pilutil.py -rw------- 1 johnh research 12120 Feb 19 17:12 scipy/odr/tests/test_odr.py -rw-r--r-- 1 johnh research 576 Mar 10 12:57 scipy/optimize/tests/test_cobyla.py -rw-r--r-- 1 johnh research 2632 Mar 10 12:57 scipy/optimize/tests/test_nonlin.py -rw------- 1 johnh research 10392 Apr 25 09:15 scipy/optimize/tests/test_optimize.py -rw-r--r-- 1 johnh research 3032 Mar 10 12:57 scipy/optimize/tests/test_slsqp.py -rw-r--r-- 1 johnh research 977 Mar 10 12:57 scipy/optimize/tests/test_zeros.py -rw-r--r-- 1 johnh research 1943 Apr 3 10:22 scipy/signal/tests/test_signaltools.py -rw-r--r-- 1 johnh research 3009 Apr 3 10:22 scipy/signal/tests/test_wavelets.py -rw------- 1 johnh research 1181 Apr 25 09:15 scipy/sparse/linalg/dsolve/tests/test_linsolve.py -rw------- 1 johnh research 5763 Apr 25 09:15 scipy/sparse/linalg/dsolve/umfpack/tests/test_umfpack.py -rw------- 1 johnh research 8458 Apr 25 09:15 scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py -rw-r--r-- 1 johnh research 2048 Mar 10 12:54 scipy/sparse/linalg/eigen/arpack/tests/test_speigs.py -rw------- 1 johnh research 1896 Jul 14 12:01 scipy/sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py -rw------- 1 johnh research 4535 Apr 25 09:15 scipy/sparse/linalg/isolve/tests/test_iterative.py -rw------- 1 johnh research 1706 Apr 25 09:15 scipy/sparse/linalg/tests/test_interface.py -rw------- 1 johnh research 47594 Jul 14 12:01 scipy/sparse/tests/test_base.py -rw------- 1 johnh research 7454 Apr 25 09:15 scipy/sparse/tests/test_construct.py -rw------- 1 johnh research 3259 Apr 25 09:15 scipy/sparse/tests/test_spfuncs.py -rw------- 1 johnh research 2411 Apr 25 09:15 scipy/sparse/tests/test_sputils.py -rw------- 1 johnh research 74547 Jul 14 12:01 scipy/special/tests/test_basic.py -rw------- 1 johnh research 1008 Feb 19 17:12 scipy/special/tests/test_spfun_stats.py -rw-r--r-- 1 johnh research 877 Mar 10 12:55 scipy/stats/models/tests/test_glm.py -rw-r--r-- 1 johnh research 1241 Mar 10 12:55 scipy/stats/models/tests/test_scale.py -rw-r--r-- 1 johnh research 1585 Mar 10 12:55 scipy/stats/models/tests/test_utils.py -rw-r--r-- 1 johnh research 237473 Mar 10 12:57 scipy/ndimage/tests/test_ndimage.py -rw------- 1 johnh research 5394 Jul 14 12:01 scipy/ndimage/tests/test_registration.py -rw------- 1 johnh research 478 Jul 14 12:01 scipy/ndimage/tests/test_regression.py -rw------- 1 johnh research 6929 Jul 14 12:01 scipy/ndimage/tests/test_segment.py -rw------- 1 johnh research 557 Jul 14 12:01 scipy/stats/models/tests/test_bspline.py -rw------- 1 johnh research 10045 Apr 25 09:15 scipy/stats/models/tests/test_formula.py -rw-r--r-- 1 johnh research 1120 Mar 10 12:55 scipy/stats/models/tests/test_regression.py -rw-r--r-- 1 johnh research 662 Mar 10 12:55 scipy/stats/models/tests/test_rlm.py -rw-r--r-- 1 johnh research 7993 Mar 10 12:55 scipy/stats/tests/test_distributions.py -rw------- 1 johnh research 4701 Apr 25 09:15 scipy/stats/tests/test_mmorestats.py -rw-r--r-- 1 johnh research 4278 Mar 10 12:55 scipy/stats/tests/test_morestats.py -rw------- 1 johnh research 20707 Apr 25 09:15 scipy/stats/tests/test_mstats.py -rw------- 1 johnh research 29167 Jul 14 12:01 scipy/stats/tests/test_stats.py -rw-r--r-- 1 johnh research 760 Mar 10 12:57 scipy/weave/tests/test_ast_tools.py -rw------- 1 johnh research 7687 Apr 25 09:15 scipy/weave/tests/test_blitz_tools.py -rw-r--r-- 1 johnh research 2178 Mar 10 12:57 scipy/weave/tests/test_build_tools.py -rw------- 1 johnh research 23557 Apr 25 09:15 scipy/weave/tests/test_c_spec.py -rw-r--r-- 1 johnh research 11938 Mar 10 12:57 scipy/weave/tests/test_catalog.py -rw-r--r-- 1 johnh research 4623 Mar 10 12:57 scipy/weave/tests/test_ext_tools.py -rw-r--r-- 1 johnh research 1240 Mar 10 12:57 scipy/weave/tests/test_inline_tools.py -rw-r--r-- 1 johnh research 5136 Mar 10 12:57 scipy/weave/tests/test_numpy_scalar_spec.py -rw-r--r-- 1 johnh research 291 Mar 10 12:57 scipy/weave/tests/test_scxx.py -rw-r--r-- 1 johnh research 8661 Mar 10 12:57 scipy/weave/tests/test_scxx_dict.py -rw-r--r-- 1 johnh research 27851 Mar 10 12:57 scipy/weave/tests/test_scxx_object.py -rw-r--r-- 1 johnh research 13442 Mar 10 12:57 scipy/weave/tests/test_scxx_sequence.py -rw-r--r-- 1 johnh research 12072 Mar 10 12:57 scipy/weave/tests/test_size_check.py -rw-r--r-- 1 johnh research 6367 Mar 10 12:57 scipy/weave/tests/test_slice_handler.py -rw-r--r-- 1 johnh research 1197 Mar 10 12:57 scipy/weave/tests/test_standard_array_spec.py -rw------- 1 johnh research 3486 Jul 14 12:01 scipy/weave/tests/test_wx_spec.py johnh@flag:site-packages>

2 2

Release 1.1.1
by Charles R Harris 15 Jul '08

15 Jul '08

All, The rc release of numpy-1.1.1 is due out next Sunday. I have gone through the commits made to the trunk since the 1.1.x branch to pull out backport candidates. If you find your name here could you make the backport or say why you think it inappropriate. David, I know that these are mostly build fixes and you have backported many of them, but I don't know the current build state of 1.1.x cdavid r5236 r5240 r5266 r5267 r5268 r5269 r5270 r5271 r5272 r5273 r5274 r5275 r5276 r5277 r5278 r5279 r5280 r5281 r5282 r5283 r5302 r5304 r5355 r5365 r5366 r5367 r5368 These are mine and I'll take care of them. charris r5259 r5312 r5322 r5324 r5392 r5394 r5399 r5406 r5407 dhuard r5254 fperez r5298 r5301 r5303 jarrod r5285 oliphant r5245 r5255 Pierre, I know you have been working diligently to get masked arrays up to speed and have made numerous fixes in the 1.1.x branch. All the tests pass for me. Is there more that needs to be done? pierregm r5242 r5244 r5248 r5249 r5251 r5253 r5256 r5260 r5263 r5264 r5284 r5292 r5314 r5329 r5332 ptvirtan r5261 rkern r5296 r5297 r5342 r5349 r5357 Stefan, these are mostly documentation related. IIRC, you planned to update the documentation in 1.1.1, which probably also needs ptvirtan's commit above. What is the current status of this project? stefan r5290 r5293 r5294 r5299 r5360 r5371 r5372 Chuck

8 14

RFC: A proposal for implementing some date/time types in NumPy
by Francesc Alted 15 Jul '08

15 Jul '08

Hi, We are planning to implement some date/time types for NumPy, and I'm sending a document that explains our approach. We would love to hear the feedback of the NumPy community in order to cover their needs as much as possible. Cheers, Francesc =========================================================== A proposal for implementing some date/time types in NumPy =========================================================== :Author: Francesc Alted i Abad :Contact: faltet(a)pytables.com :Author: Ivan Vilata i Balaguer :Contact: ivan(a)selidor.net Executive summary ================= A date/time mark is something very handy to have in many fields where one has to deal with data sets. While Python have several modules that define a date/time type, like ``mx.DateTime`` or the integrated ``datetime`` [1]_, NumPy has a lack of them. In this document, we are proposing the addition of a series of date/time types to fill this gap. The requirements for the proposed types are two-folded: 1) they have to be fast to operate with and 2) they have to be as compatible as possible with the existing ``datetime`` module that comes with Python. Types proposed ============== To start with, it is virtually impossible to come up with a single date/time type that fills the needs of every case of use. So, after pondering about different possibilities, we have stick with three different types, namely ``datetime64``, ``timestamp64`` and ``timefloat64`` -- these names are preliminary and can be changed indeed; they are mostly useful for the sake of the discussion -- that cover different needs. Here it goes a detailed description of the different types: ``datetime64`` - Implemented internally as an ``int64`` type. - Expressed in microseconds since POSIX epoch (January 1, 1970). - Resolution: nanoseconds. - Time span: 278922 years in each direction since the POSIX epoch. Observations:: This will be compatible with the Python ``datetime`` module not only in terms of precision (it also have a resolution of microseconds) and time-span (its range is year 1 to year 9999), but also in that we will provide getters and setters for it. ``timestamp64`` - Implemented internally as an ``int64`` type. - Expressed in nanoseconds since POSIX epoch (January 1, 1970). - Resolution: nanoseconds. - Time span: 272 years in each direction since the POSIX epoch. Observations:: This will be not be fully compatible with the Python ``datetime`` module neither in terms of precision nor time-span. However, getters and setters will be provided for it (loosing precision or overflowing as needed). * ``timefloat64`` - Implemented internally as a float64. - Expressed in microseconds since POSIX epoch. - Resolution: 1 microsecond (for +-32 years from epoch) or 14 digits (for distant years from epoch). So the precision is *variable*. - Time span: 1e+308 years in each direction since the POSIX epoch. Observations:: In general, this will be not be fully compatible with the Python datetime neither in terms of precision nor time-span. However, getters and setters will be provided for it (loosing precision or overflowing as needed). Example of use ============== Here it is an example of usage of one of the types described above (``datetime64``):: In [10]: t = numpy.zeros(5, dtype="datetime64") In [11]: t[0] = datetime.datetime.now() # setter in action In [12]: t[0] Out[12]: 733234384724 # representation as an int64 (scalar) In [13]: t Out[13]: array([12155899511985929, 0, 0, 0, 0], dtype=datetime64) In [14]: t[0].item() # getter in action Out[14]: datetime.datetime(2008, 7, 11, 14, 27, 3, 384724) Final considerations ==================== About the ``mx.DateTime`` module -------------------------------- In this document, the emphasis has been put in comparing the compatibility of future NumPy date/time types against the ``datetime`` module that comes with Python. Should we consider the compatibility with mx.DateTime as well? Are there many people using ``mx.DateTime`` [2]_ out there? If so, which are their advantages over ``datetime`` module?. A final note on time scales --------------------------- [Only for people with high precision time requirements or just for those that love the feel of their brain exploding] POSIX (or UTC, in which POSIX is based) time scale [3]_ is based on the time that the Earth takes to revolve around the Sun. However, after the adoption of more precise time patterns (read: atomic clocks), it became clear that this time is pretty imprecise compared with the latter. As a result, the UTC standard is adding ``leap seconds`` from time to time (at a rate of 1 second per year, approximately) in order to compensate these differences. Because of this, when computing time deltas (using the UTC standard) between two instants that differ in more than one year, it is extremely probable that an error of a second or several (depending of the time span) would be introduced. While this in general is harmless for common use cases, there are situations that this can bite people quite a lot. For example, realize that IERS (International Earth Rotation and Reference Systems Service) decided to add a leap second the past June 30th 00:00:00 UTC and that you were doing an experiment precisely at that time (this is not so rare, and has probably happened already to someone, somewhere). With this, in the analysis phase of your experiment, you could have the surprise that the time deltas computed during this leap second are actually 1 second shorter (and then, why the heck do we want to use types that supposedly support micro- or nano-seconds of precision?). Because of this, we were initially tempted to use the TAI (Temps Atomique International) [4]_ standard because it is strictly *continuous* (contrarily to UTC or POSIX), so avoiding the sort of problems exposed above. However, the omnipresence of the UTC clocks in the computing world would force us to be continuously converting UTC timestamps to TAI ones and vice-versa. Unfortunately, this is not easy to do because there is not a simple mathematical relationship between UTC and TAI. Instead, you have to use a table in order to check when the IERS added the leap seconds, and take them into account. The problem is that, as the revolution of the Earth is imprecise, one cannot determine ahead of time when the leap seconds will be added. This can lead to problems with code that performs the TAI <-> UTC conversion in the sense that the internal conversion table has to be continuously updated if we don't want to have problems of precision. And this is a too much added complication to be worth the effort, in our opinion. These are the difficulties that driven us to prefer the POSIX time scale over TAI for this implementation. However, more input on this issue is very welcome. .. [1] http://docs.python.org/lib/module-datetime.html .. [2] http://www.egenix.com/products/python/mxBase/mxDateTime .. [3] http://en.wikipedia.org/wiki/Unix_time .. [4] http://en.wikipedia.org/wiki/International_Atomic_Time

10 31

Run np.test() twice, get message.
by Charles R Harris 15 Jul '08

15 Jul '08

Alan, Any idea what this is: *** DocTestRunner.merge: '__main__' in both testers; summing outcomes. Chuck

3 3

Doc on dtypes
by Pierre GM 15 Jul '08

15 Jul '08

All, Anybody could point me to some docs on dtypes ? Michael Droettboom's recent question made me realize that things were far more complex than I thought. For example, how can I find the shape and type of a field (without using dtype.descr). Thanks a lot in advance P.

2 8

mrecarray
by Michael Droettboom 14 Jul '08

14 Jul '08

I'm running into a couple of small problems with mrecarray. I'm not sure if they're bugs or a usage error. First, the constructor throws an exception when the format string contains nested arrays (if that is the proper term) such as "(2,2)f8". This creates a three-element tuple in the dtype.descr list, whereas the mrecord.py code seems to assume each descr will always have 2 elements. ======== In [1]: from numpy.ma import mrecords In [2]: x = mrecords.mrecarray(1, formats="(2,2)f8") --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /wonkabar/data1/scraps/<ipython console> in <module>() /home/mdroe/usr/lib/python2.5/site-packages/numpy/ma/mrecords.py in __new__(cls, shape, dtype, buf, offset, strides, formats, names, titles, byteorder, aligned, mask, hard_mask, fill_value, keep_mask, copy, **options) 121 self = recarray.__new__(cls, shape, dtype=dtype, buf=buf, offset=offset, 122 strides=strides, formats=formats, --> 123 byteorder=byteorder, aligned=aligned,) 124 # 125 mdtype = [(k,'|b1') for (k,_) in self.dtype.descr] /home/mdroe/usr/lib/python2.5/site-packages/numpy/core/records.py in __new__(subtype, shape, dtype, buf, offset, strides, formats, names, titles, byteorder, aligned) 248 249 if buf is None: --> 250 self = ndarray.__new__(subtype, shape, (record, descr)) 251 else: 252 self = ndarray.__new__(subtype, shape, (record, descr), /home/mdroe/usr/lib/python2.5/site-packages/numpy/ma/mrecords.py in __array_finalize__(self, obj) 157 _fieldmask = getattr(obj, '_fieldmask', None) 158 if _fieldmask is None: --> 159 mdescr = [(n,'|b1') for (n,_) in self.dtype.descr] 160 objmask = getattr(obj, '_mask', nomask) 161 if objmask is nomask: <type 'exceptions.ValueError'>: too many values to unpack ======== In my own use case, I don't care if the individual elements of the nested array are maskable -- either the whole array being masked or not is good enough -- but perhaps that shortcoming is why this wasn't designed to work? I have attached a patch to mrecords.py that gets me past this and does seem to allow the nested array to be masked as a whole. Secondly, the "names" and "titles" kwargs seem to be ignored by the mrecarray constructor. ======== In [3]: x = mrecords.mrecarray(1, formats="f8", names="foo") In [4]: x[0]['foo'] = 42.0 --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /wonkabar/data1/scraps/<ipython console> in <module>() /home/mdroe/usr/lib/python2.5/site-packages/numpy/ma/mrecords.py in __setitem__(self, indx, value) 309 def __setitem__(self, indx, value): 310 "Sets the given record to value." --> 311 MaskedArray.__setitem__(self, indx, value) 312 if isinstance(indx, basestring): 313 self._mask[indx] = ma.getmaskarray(value) /home/mdroe/usr/lib/python2.5/site-packages/numpy/ma/core.py in __setitem__(self, indx, value) 1437 # raise IndexError, msg 1438 if isinstance(indx, basestring): -> 1439 ndarray.__setitem__(self._data, indx, value) 1440 ndarray.__setitem__(self._mask, indx, getmask(value)) 1441 return <type 'exceptions.ValueError'>: field named foo not found. ======== The included patch delegates these kwargs onto the underlying recarray. Cheers, Mike -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA Index: numpy/ma/mrecords.py =================================================================== --- numpy/ma/mrecords.py (revision 5378) +++ numpy/ma/mrecords.py (working copy) @@ -119,10 +119,10 @@ **options): # self = recarray.__new__(cls, shape, dtype=dtype, buf=buf, offset=offset, - strides=strides, formats=formats, - byteorder=byteorder, aligned=aligned,) + strides=strides, formats=formats, names=names, + titles=titles, byteorder=byteorder, aligned=aligned,) # - mdtype = [(k,'|b1') for (k,_) in self.dtype.descr] + mdtype = [(k[0],'|b1') for k in self.dtype.descr] if mask is nomask or not np.size(mask): if not keep_mask: self._mask = tuple([False]*len(mdtype)) @@ -156,7 +156,7 @@ # Make sure we have a _fieldmask by default .. _fieldmask = getattr(obj, '_fieldmask', None) if _fieldmask is None: - mdescr = [(n,'|b1') for (n,_) in self.dtype.descr] + mdescr = [(n[0],'|b1') for n in self.dtype.descr] objmask = getattr(obj, '_mask', nomask) if objmask is nomask: _mask = np.empty(self.shape, dtype=mdescr).view(recarray)

2 1

loadtxt and usecols
by Bryan Fodness 14 Jul '08

14 Jul '08

i would like to load my data without knowing the length, i have explicitly stated the rows data = loadtxt('18B180.dat', skiprows = 1, usecols = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45)) and would like to use something like, data = loadtxt('18B180.dat', skiprows = 1, usecols = (1,:)) the first column is the only that i do not need. -- "The game of science can accurately be described as a never-ending insult to human intelligence." - João Magueijo "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius - and a lot of courage - to move in the opposite direction. " -Albert Einstein

3 2

Unused private functions
by Alan McIntyre 14 Jul '08

14 Jul '08

Does anybody know whether there's any reason to keep the following functions? They don't appear to be used anywhere. linalg/linalg.py:95 _castCopyAndTranspose lib/function_base.py:659 _asarray1d

2 2