From cgohlke at uci.edu Sun May 1 00:11:48 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 30 Apr 2011 21:11:48 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: References: <4DBC9761.9070908@uci.edu> <4DBCAE45.8080402@uci.edu> Message-ID: <4DBCDD84.9020709@uci.edu> On 4/30/2011 6:37 PM, Charles R Harris wrote: > > > On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke > wrote: > > > > On 4/30/2011 4:58 PM, Charles R Harris wrote: > > > > > > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris > > > >> wrote: > > > > > > > > > > I get a null pointer access violation during numpy.test() > with all > > msvc9/MKL builds for win32 (not win-amd64). The crash > occurs during > > test_result_type() in test_numeric.py and can be reduced > to the > > following code: > > > > > >> import numpy as np > > > >> np.result_type(np.array([np.float32(0)]), np.complex128(0)) > > > > np.float64(0) and np.float16(0) also crash. Unfortunately > the debug > > builds do not crash. > > > > This is new, right? > > > > > > Does it depend on the optimization level? > > > > Chuck > > > > Yes it's new. The pure msvc9 builds without MKL also crash. The crash > disapperars When compiling with /Od (disable optimization) instead of > /Ox (maximum optimization; the default for distutils). > > > So all of np.float16(0), np.float32(0), np.float64(0), etc crash? Does > it depend at all on 0 as the argument, or is it the same for 1, 0.0, > etc. What about string arguments like np.float64("0"). I want to pin the > location down a bit more. Too bad it doesn't crash in the debugger. > > Chuck > Sorry, I should have been more precise. The crash occurs in `numpy.result_type(a, b)` with the following inputs: b = numpy.complex128(0) a = numpy.array([numpy.float16(0)]) a = numpy.array([numpy.float32(0)]) a = numpy.array([numpy.float64(0)]) Christoph From mwwiebe at gmail.com Sun May 1 00:27:56 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 30 Apr 2011 21:27:56 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: <4DBCDD84.9020709@uci.edu> References: <4DBC9761.9070908@uci.edu> <4DBCAE45.8080402@uci.edu> <4DBCDD84.9020709@uci.edu> Message-ID: On Sat, Apr 30, 2011 at 9:11 PM, Christoph Gohlke wrote: > > > On 4/30/2011 6:37 PM, Charles R Harris wrote: > > > > > > On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke > > wrote: > > > > > > > > On 4/30/2011 4:58 PM, Charles R Harris wrote: > > > > > > > > > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris > > > > > > >> wrote: > > > > > > > > > > > > > > > I get a null pointer access violation during numpy.test() > > with all > > > msvc9/MKL builds for win32 (not win-amd64). The crash > > occurs during > > > test_result_type() in test_numeric.py and can be reduced > > to the > > > following code: > > > > > > > >> import numpy as np > > > > >> np.result_type(np.array([np.float32(0)]), np.complex128(0)) > > > > > > np.float64(0) and np.float16(0) also crash. Unfortunately > > the debug > > > builds do not crash. > > > > > > This is new, right? > > > > > > > > > Does it depend on the optimization level? > > > > > > Chuck > > > > > > > Yes it's new. The pure msvc9 builds without MKL also crash. The crash > > disapperars When compiling with /Od (disable optimization) instead of > > /Ox (maximum optimization; the default for distutils). > > > > > > So all of np.float16(0), np.float32(0), np.float64(0), etc crash? Does > > it depend at all on 0 as the argument, or is it the same for 1, 0.0, > > etc. What about string arguments like np.float64("0"). I want to pin the > > location down a bit more. Too bad it doesn't crash in the debugger. > > > > Chuck > > > > Sorry, I should have been more precise. The crash occurs in > `numpy.result_type(a, b)` with the following inputs: > > b = numpy.complex128(0) > a = numpy.array([numpy.float16(0)]) > a = numpy.array([numpy.float32(0)]) > a = numpy.array([numpy.float64(0)]) > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Can you try it with this commit backed out? https://github.com/numpy/numpy/commit/fc0148bef18b8fd34d124d5edd90887f63e3bfce#L0R1147 What does the stack trace of the crash look like? Compiling with debug information still works with optimizations, just the line numbers and stack traces don't match the source as well. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Sun May 1 03:19:33 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 01 May 2011 00:19:33 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: References: <4DBC9761.9070908@uci.edu> <4DBCAE45.8080402@uci.edu> <4DBCDD84.9020709@uci.edu> Message-ID: <4DBD0985.1080900@uci.edu> On 4/30/2011 9:27 PM, Mark Wiebe wrote: > On Sat, Apr 30, 2011 at 9:11 PM, Christoph Gohlke > wrote: > > > > On 4/30/2011 6:37 PM, Charles R Harris wrote: > > > > > > On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke > > >> wrote: > > > > > > > > On 4/30/2011 4:58 PM, Charles R Harris wrote: > > > > > > > > > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris > > > > > > > > > >>> wrote: > > > > > > > > > > > > > > > I get a null pointer access violation during numpy.test() > > with all > > > msvc9/MKL builds for win32 (not win-amd64). The crash > > occurs during > > > test_result_type() in test_numeric.py and can be reduced > > to the > > > following code: > > > > > > > >> import numpy as np > > > > >> np.result_type(np.array([np.float32(0)]), np.complex128(0)) > > > > > > np.float64(0) and np.float16(0) also crash. Unfortunately > > the debug > > > builds do not crash. > > > > > > This is new, right? > > > > > > > > > Does it depend on the optimization level? > > > > > > Chuck > > > > > > > Yes it's new. The pure msvc9 builds without MKL also crash. > The crash > > disapperars When compiling with /Od (disable optimization) > instead of > > /Ox (maximum optimization; the default for distutils). > > > > > > So all of np.float16(0), np.float32(0), np.float64(0), etc crash? Does > > it depend at all on 0 as the argument, or is it the same for 1, 0.0, > > etc. What about string arguments like np.float64("0"). I want to > pin the > > location down a bit more. Too bad it doesn't crash in the debugger. > > > > Chuck > > > > Sorry, I should have been more precise. The crash occurs in > `numpy.result_type(a, b)` with the following inputs: > > b = numpy.complex128(0) > a = numpy.array([numpy.float16(0)]) > a = numpy.array([numpy.float32(0)]) > a = numpy.array([numpy.float64(0)]) > > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > Can you try it with this commit backed out? > > https://github.com/numpy/numpy/commit/fc0148bef18b8fd34d124d5edd90887f63e3bfce#L0R1147 > > What does the stack trace of the crash look like? Compiling with debug > information still works with optimizations, just the line numbers and > stack traces don't match the source as well. > > -Mark > > Sorry, I get no stack trace; apparently the stack is corrupted. I printf-debugged it down to the following preprocessor directive, which already caused trouble in gcc before : #if NPY_SIZEOF_LONGLONG >= NPY_SIZEOF_CLONGDOUBLE npy_longlong value; #else npy_clongdouble value; #endif Replacing the two occasions of this code in multiarray/convert_datatype.c with `npy_longlong value[4];` solved the crash and all tests pass. Btw, where is NPY_SIZEOF_CLONGDOUBLE defined? I searched the source code and the build directory but could not find it. Christoph -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: convert_datatype.diff URL: From mwwiebe at gmail.com Sun May 1 22:58:25 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sun, 1 May 2011 19:58:25 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: <4DBD0985.1080900@uci.edu> References: <4DBC9761.9070908@uci.edu> <4DBCAE45.8080402@uci.edu> <4DBCDD84.9020709@uci.edu> <4DBD0985.1080900@uci.edu> Message-ID: On Sun, May 1, 2011 at 12:19 AM, Christoph Gohlke wrote: > > > On 4/30/2011 9:27 PM, Mark Wiebe wrote: > >> On Sat, Apr 30, 2011 at 9:11 PM, Christoph Gohlke > > wrote: >> >> >> >> On 4/30/2011 6:37 PM, Charles R Harris wrote: >> > >> > >> > On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke > >> > >> wrote: >> > >> > >> > >> > On 4/30/2011 4:58 PM, Charles R Harris wrote: >> > > >> > > >> > > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris >> > > >> > >> > > > >> > > >>> wrote: >> > > >> > > >> > > >> > > >> > > I get a null pointer access violation during numpy.test() >> > with all >> > > msvc9/MKL builds for win32 (not win-amd64). The crash >> > occurs during >> > > test_result_type() in test_numeric.py and can be reduced >> > to the >> > > following code: >> > > >> > > > >> import numpy as np >> > > > >> np.result_type(np.array([np.float32(0)]), np.complex128(0)) >> > > >> > > np.float64(0) and np.float16(0) also crash. Unfortunately >> > the debug >> > > builds do not crash. >> > > >> > > This is new, right? >> > > >> > > >> > > Does it depend on the optimization level? >> > > >> > > Chuck >> > > >> > >> > Yes it's new. The pure msvc9 builds without MKL also crash. >> The crash >> > disapperars When compiling with /Od (disable optimization) >> instead of >> > /Ox (maximum optimization; the default for distutils). >> > >> > >> > So all of np.float16(0), np.float32(0), np.float64(0), etc crash? >> Does >> > it depend at all on 0 as the argument, or is it the same for 1, 0.0, >> > etc. What about string arguments like np.float64("0"). I want to >> pin the >> > location down a bit more. Too bad it doesn't crash in the debugger. >> > >> > Chuck >> > >> >> Sorry, I should have been more precise. The crash occurs in >> `numpy.result_type(a, b)` with the following inputs: >> >> b = numpy.complex128(0) >> a = numpy.array([numpy.float16(0)]) >> a = numpy.array([numpy.float32(0)]) >> a = numpy.array([numpy.float64(0)]) >> >> Christoph >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> Can you try it with this commit backed out? >> >> >> https://github.com/numpy/numpy/commit/fc0148bef18b8fd34d124d5edd90887f63e3bfce#L0R1147 >> >> What does the stack trace of the crash look like? Compiling with debug >> information still works with optimizations, just the line numbers and >> stack traces don't match the source as well. >> >> < >> https://github.com/numpy/numpy/commit/fc0148bef18b8fd34d124d5edd90887f63e3bfce#L0R1147 >> >-Mark >> >> >> > Sorry, I get no stack trace; apparently the stack is corrupted. I > printf-debugged it down to the following preprocessor directive, which > already caused trouble in gcc before < > http://projects.scipy.org/numpy/ticket/1737>: > > #if NPY_SIZEOF_LONGLONG >= NPY_SIZEOF_CLONGDOUBLE > npy_longlong value; > #else > npy_clongdouble value; > #endif > > Replacing the two occasions of this code in multiarray/convert_datatype.c > with `npy_longlong value[4];` solved the crash and all tests pass. > > Btw, where is NPY_SIZEOF_CLONGDOUBLE defined? I searched the source code > and the build directory but could not find it. > > Christoph > > I committed that fix. I think NPY_SIZEOF_CLONGDOUBLE isn't actually defined, but it appears the C preprocessor doesn't consider that an error or even a warning, rather an opportunity to provide a surprise for the user of the code. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Mon May 2 11:18:07 2011 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 2 May 2011 11:18:07 -0400 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: References: Message-ID: <4DBECB2F.4040500@stsci.edu> I've found a few reference counting bugs running the regression tests under Valgrind. Pull request here: https://github.com/numpy/numpy/pull/79 Mike On 04/30/2011 04:19 PM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first release > candidate of NumPy 1.6.0. If no new problems are reported, the final > release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > This release includes several new features as well as numerous bug fixes and > improved documentation. It is backward compatible with the 1.5.0 release, and > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > Highlights > ========== > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > * A new 16-bit floating point type. > > * A new iterator, which improves performance of many functions. > > > New features > ============ > > New 16-bit floating point type > ------------------------------ > > This release adds support for the IEEE 754-2008 binary16 format, available as > the data type ``numpy.half``. Within Python, the type behaves similarly to > `float` or `double`, and C extensions can add support for it with the exposed > half-float API. > > > New iterator > ------------ > > A new iterator has been added, replacing the functionality of the > existing iterator and multi-iterator with a single object and API. > This iterator works well with general memory layouts different from > C or Fortran contiguous, and handles both standard NumPy and > customized broadcasting. The buffering, automatic data type > conversion, and optional output parameters, offered by > ufuncs but difficult to replicate elsewhere, are now exposed by this > iterator. > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > ------------------------------------------------------------------------- > > Extend the number of polynomials available in the polynomial package. In > addition, a new ``window`` attribute has been added to the classes in > order to specify the range the ``domain`` maps to. This is mostly useful > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > are infinite and provides a more intuitive way to get the correct mapping > of values without playing unnatural tricks with the domain. > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > as ``shape(array, dim-1)`` which makes it possible to automatically > wrap Fortran routines that use two argument ``size`` function in > dimension specifications. Before users were forced to apply this > mapping manually. > > > Other new functions > ------------------- > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > an array of flat indices, applying boundary modes to the indices. > > ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the > Einstein summation convention, many common multi-dimensional array operations > can be represented in a simple fashion. This function provides a way compute > such summations. > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose > the underlying type promotion used by the ufuncs and other operations to > determine the types of outputs. These improve upon the ``numpy.common_type`` > and ``numpy.mintypecode`` which provide similar functionality but do > not match the ufunc implementation. > > > Changes > ======= > > Changes and improvements in the numpy core > ------------------------------------------ > > ``default error handling`` > -------------------------- > > The default error handling has been change from ``print`` to ``warn`` for > all except for ``underflow``, which remains as ``ignore``. > > > ``numpy.distutils`` > ------------------- > > Several new compilers are supported for building Numpy: the Portland Group > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C > compiler on Linux. > > > ``numpy.testing`` > ----------------- > > The testing framework gained ``numpy.testing.assert_allclose``, which provides > a more convenient way to compare floating point arrays than > `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. > > > ``C API`` > --------- > > In addition to the APIs for the new iterator and half data type, a number > of other additions have been made to the C API. The type promotion > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > ``NPY_CASTING`` has been added which controls what types of casts are > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > and ``PyArray_CanCastTypeTo``. A more flexible way to handle > conversion of arbitrary python objects into arrays is exposed by > ``PyArray_GetArrayParamsFromObject``. > > > Deprecated features > =================== > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality > will be replaced by the new "density" keyword. > > > Removed features > ================ > > ``numpy.fft`` > ------------- > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, > which were aliases for the same functions without the 'e' in the name, were > removed. > > > ``numpy.memmap`` > ---------------- > > The `sync()` and `close()` methods of memmap were removed. Use `flush()` and > "del memmap" instead. > > > ``numpy.lib`` > ------------- > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > ``numpy.ma`` > ------------ > > Several deprecated items were removed from the ``numpy.ma`` module:: > > * ``numpy.ma.MaskedArray`` "raw_data" method > * ``numpy.ma.MaskedArray`` constructor "flag" keyword > * ``numpy.ma.make_mask`` "flag" keyword > * ``numpy.ma.allclose`` "fill_value" keyword > > > ``numpy.distutils`` > ------------------- > > The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` > instead. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From Chris.Barker at noaa.gov Mon May 2 12:17:15 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 02 May 2011 09:17:15 -0700 Subject: [Numpy-discussion] Less dimensions than expected with record array In-Reply-To: References: Message-ID: <4DBED90B.50604@noaa.gov> >> ndarray has 2 dimensions but record array has 1 dimensions >> >> This makes seemingly reasonable things, like using apply_along_axis() >> over a table of data with named columns, impossible: > each row (record) is treated as one array element, so the structured > array is only 1d. > > If you have rows/records with content that is not homogenous, then > working along axis=1 (across elements of a record) doesn't make sense. > for example I just struggle with 2 datetime columns and the rest are > integers. > > If you want an array with homogenous elements (all floats or all ints) > with operations along axis, then larry (la) is, I think, still the > best bet. another option is to use views. There are time when I want the same array visible as both a structured array, and a regular old array, depending on what I'm doing with it. and you can do that: In [77]: data Out[77]: [(1, 2, 3), (4, 5, 6), (7, 8, 9)] In [80]: dt = [('a',int),('b',int),('c',int)] In [81]: record_array = np.array(data, dtype=dt) In [84]: array = record_array.view(dtype=np.int).reshape(-1,3) In [85]: array Out[85]: array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) # array and record_array share the same data: In [88]: array[:,1] *= 2 In [89]: array Out[89]: array([[ 1, 4, 3], [ 4, 10, 6], [ 7, 16, 9]]) In [90]: record_array Out[90]: array([(1, 4, 3), (4, 10, 6), (7, 16, 9)], dtype=[('a', ' References: <4DBECB2F.4040500@stsci.edu> Message-ID: I applied everything, since they're all obviously bugs and the fixes look straightforward. -Mark On Mon, May 2, 2011 at 8:18 AM, Michael Droettboom wrote: > I've found a few reference counting bugs running the regression tests > under Valgrind. Pull request here: > > https://github.com/numpy/numpy/pull/79 > > Mike > > On 04/30/2011 04:19 PM, Ralf Gommers wrote: > > Hi, > > > > I am pleased to announce the availability of the first release > > candidate of NumPy 1.6.0. If no new problems are reported, the final > > release will be in one week. > > > > Sources and binaries can be found at > > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/ > > For (preliminary) release notes see below. > > > > Enjoy, > > Ralf > > > > > > ========================= > > NumPy 1.6.0 Release Notes > > ========================= > > > > This release includes several new features as well as numerous bug fixes > and > > improved documentation. It is backward compatible with the 1.5.0 > release, and > > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > > > > Highlights > > ========== > > > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > > > * A new 16-bit floating point type. > > > > * A new iterator, which improves performance of many functions. > > > > > > New features > > ============ > > > > New 16-bit floating point type > > ------------------------------ > > > > This release adds support for the IEEE 754-2008 binary16 format, > available as > > the data type ``numpy.half``. Within Python, the type behaves similarly > to > > `float` or `double`, and C extensions can add support for it with the > exposed > > half-float API. > > > > > > New iterator > > ------------ > > > > A new iterator has been added, replacing the functionality of the > > existing iterator and multi-iterator with a single object and API. > > This iterator works well with general memory layouts different from > > C or Fortran contiguous, and handles both standard NumPy and > > customized broadcasting. The buffering, automatic data type > > conversion, and optional output parameters, offered by > > ufuncs but difficult to replicate elsewhere, are now exposed by this > > iterator. > > > > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > > ------------------------------------------------------------------------- > > > > Extend the number of polynomials available in the polynomial package. In > > addition, a new ``window`` attribute has been added to the classes in > > order to specify the range the ``domain`` maps to. This is mostly useful > > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > > are infinite and provides a more intuitive way to get the correct mapping > > of values without playing unnatural tricks with the domain. > > > > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > > ----------------------------------------------------------------------- > > > > F2py now supports wrapping Fortran 90 routines that use assumed shape > > arrays. Before such routines could be called from Python but the > > corresponding Fortran routines received assumed shape arrays as zero > > length arrays which caused unpredicted results. Thanks to Lorenz > > H?depohl for pointing out the correct way to interface routines with > > assumed shape arrays. > > > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > > as ``shape(array, dim-1)`` which makes it possible to automatically > > wrap Fortran routines that use two argument ``size`` function in > > dimension specifications. Before users were forced to apply this > > mapping manually. > > > > > > Other new functions > > ------------------- > > > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > > an array of flat indices, applying boundary modes to the indices. > > > > ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the > > Einstein summation convention, many common multi-dimensional array > operations > > can be represented in a simple fashion. This function provides a way > compute > > such summations. > > > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an > array. > > > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions > expose > > the underlying type promotion used by the ufuncs and other operations to > > determine the types of outputs. These improve upon the > ``numpy.common_type`` > > and ``numpy.mintypecode`` which provide similar functionality but do > > not match the ufunc implementation. > > > > > > Changes > > ======= > > > > Changes and improvements in the numpy core > > ------------------------------------------ > > > > ``default error handling`` > > -------------------------- > > > > The default error handling has been change from ``print`` to ``warn`` for > > all except for ``underflow``, which remains as ``ignore``. > > > > > > ``numpy.distutils`` > > ------------------- > > > > Several new compilers are supported for building Numpy: the Portland > Group > > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit > Intel C > > compiler on Linux. > > > > > > ``numpy.testing`` > > ----------------- > > > > The testing framework gained ``numpy.testing.assert_allclose``, which > provides > > a more convenient way to compare floating point arrays than > > `assert_almost_equal`, `assert_approx_equal` and > `assert_array_almost_equal`. > > > > > > ``C API`` > > --------- > > > > In addition to the APIs for the new iterator and half data type, a number > > of other additions have been made to the C API. The type promotion > > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > > ``NPY_CASTING`` has been added which controls what types of casts are > > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > > and ``PyArray_CanCastTypeTo``. A more flexible way to handle > > conversion of arbitrary python objects into arrays is exposed by > > ``PyArray_GetArrayParamsFromObject``. > > > > > > Deprecated features > > =================== > > > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its > functionality > > will be replaced by the new "density" keyword. > > > > > > Removed features > > ================ > > > > ``numpy.fft`` > > ------------- > > > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, > `irefftn`, > > which were aliases for the same functions without the 'e' in the name, > were > > removed. > > > > > > ``numpy.memmap`` > > ---------------- > > > > The `sync()` and `close()` methods of memmap were removed. Use `flush()` > and > > "del memmap" instead. > > > > > > ``numpy.lib`` > > ------------- > > > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > > > > ``numpy.ma`` > > ------------ > > > > Several deprecated items were removed from the ``numpy.ma`` module:: > > > > * ``numpy.ma.MaskedArray`` "raw_data" method > > * ``numpy.ma.MaskedArray`` constructor "flag" keyword > > * ``numpy.ma.make_mask`` "flag" keyword > > * ``numpy.ma.allclose`` "fill_value" keyword > > > > > > ``numpy.distutils`` > > ------------------- > > > > The ``numpy.get_numpy_include`` function was removed, use > ``numpy.get_include`` > > instead. > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Michael Droettboom > Science Software Branch > Space Telescope Science Institute > Baltimore, Maryland, USA > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue May 3 12:20:27 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 3 May 2011 18:20:27 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: References: <4DBECB2F.4040500@stsci.edu> Message-ID: On Mon, May 2, 2011 at 10:55 PM, Mark Wiebe wrote: > I applied everything, since they're all obviously bugs and the fixes look > straightforward. I tested those commits on a few python/OS combinations, all looks good. So I'll try to tag an RC2 tonight. Ralf From matthew.brett at gmail.com Tue May 3 13:07:26 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 3 May 2011 13:07:26 -0400 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Hi, On Sat, Apr 30, 2011 at 5:21 AM, Ralf Gommers wrote: > On Wed, Apr 27, 2011 at 8:52 PM, Matthew Brett wrote: >> Hi, >> >> This is just to follow up on a dead thread of mine a little while back. >> >> I was asking about letters for Clint Whaley's tenure case, from numpy, >> but I realized that I don't know who 'numpy' is :) >> >> Is there in fact a numpy steering group? ? ?Who is best to write >> letters representing the 'numpy community'? > > At http://scipy.org/Developer_Zone there's a list of people under a > big header "steering committee". It seems to me that writing such a > letter representing the community is one of the purposes that > committee could serve. Ah - yes - thanks for the reply. In the interests of general transparency - and given that no-one from that group has replied to this email - how should the group best be addressed? By personal email? That seems to break the open-source matra of everything on-list: http://producingoss.com/en/setting-tone.html#avoid-private-discussions Thanks again, Matthew From ralf.gommers at googlemail.com Tue May 3 14:18:31 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 3 May 2011 20:18:31 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 Message-ID: Hi, I am pleased to announce the availability of the second release candidate of NumPy 1.6.0. Compared to the first release candidate, one segfault on (32-bit Windows + MSVC) and several memory leaks were fixed. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ For (preliminary) release notes see below. Enjoy, Ralf ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py interprets Fortran expression ``size(array, dim)`` as ``shape(array, dim-1)`` which makes it possible to automatically wrap Fortran routines that use two argument ``size`` function in dimension specifications. Before users were forced to apply this mapping manually. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= Changes and improvements in the numpy core ------------------------------------------ ``default error handling`` -------------------------- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Deprecated features =================== The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality will be replaced by the new "density" keyword. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. From rowen at uw.edu Tue May 3 15:37:28 2011 From: rowen at uw.edu (Russell E. Owen) Date: Tue, 03 May 2011 12:37:28 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 References: Message-ID: In article , Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. >... > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > For (preliminary) release notes see below. Thanks! It tests fine built from source on my Mac OS X 10.5.8 with python.org Python 2.6.6: Ran 3537 tests in 24.833s OK (KNOWNFAIL=3, SKIP=1) -- Russell From cgohlke at uci.edu Tue May 3 16:35:37 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Tue, 03 May 2011 13:35:37 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: <4DC06719.7030909@uci.edu> On 5/3/2011 11:18 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. > > Compared to the first release candidate, one segfault on (32-bit > Windows + MSVC) and several memory leaks were fixed. If no new > problems are reported, the final release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > Looks good. The msvc9/MKL builds now pass all tests on win32 and win-amd64, python 2.6, 2.7, 3.1, and 3.2. One scipy test failure reported earlier remains, but that is probably no release blocker. Christoph From ralf.gommers at googlemail.com Tue May 3 17:06:49 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 3 May 2011 23:06:49 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: <4DC06719.7030909@uci.edu> References: <4DC06719.7030909@uci.edu> Message-ID: On Tue, May 3, 2011 at 10:35 PM, Christoph Gohlke wrote: > > > On 5/3/2011 11:18 AM, Ralf Gommers wrote: >> Hi, >> >> I am pleased to announce the availability of the second release >> candidate of NumPy 1.6.0. >> >> Compared to the first release candidate, one segfault on (32-bit >> Windows + MSVC) and several memory leaks were fixed. If no new >> problems are reported, the final release will be in one week. >> >> Sources and binaries can be found at >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ >> For (preliminary) release notes see below. >> >> Enjoy, >> Ralf >> > > Looks good. The msvc9/MKL builds now pass all tests on win32 and > win-amd64, python 2.6, 2.7, 3.1, and 3.2. Good, thanks for testing. > > One scipy test failure reported earlier remains, but that is probably no > release blocker. > That's a problem in scipy.stats, that only showed up recently because of a bug fix in numpy.testing. Cheers, Ralf From ischnell at enthought.com Tue May 3 17:22:07 2011 From: ischnell at enthought.com (Ilan Schnell) Date: Tue, 3 May 2011 16:22:07 -0500 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: <4DC06719.7030909@uci.edu> Message-ID: I'm seeing these three failures on Solaris 5.10 (x86_64, using Python 2.7.1): ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/demo/master/lib/python2.7/site-packages/numpy/testing/decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 321, in test_floating_exceptions lambda a,b:a/b, ft_tiny, ft_max) File "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 271, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "/home/demo/master/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error 'underflow'. ====================================================================== FAIL: test_zero_nzero (test_umath.TestArctan2SpecialValues) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", line 322, in test_zero_nzero assert_almost_equal(ncu.arctan2(np.PZERO, np.NZERO), np.pi) File "/home/demo/master/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 3.141592653589793 ====================================================================== FAIL: test_zero_pzero (test_umath.TestArctan2SpecialValues) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", line 328, in test_zero_pzero assert_arctan2_isnzero(np.NZERO, np.PZERO) File "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", line 310, in assert_arctan2_isnzero assert (ncu.arctan2(x, y) == 0 and np.signbit(ncu.arctan2(x, y))), "arctan(%s, %s) is %s, not -0" % (x, y, ncu.arctan2(x, y)) AssertionError: arctan(-0.0, 0.0) is 0.0, not -0 ---------------------------------------------------------------------- Ran 3152 tests in 16.990s I'm not sure what the state of Solaris support is these days, but I remember being able to run all tests without any failures. - Ilan On Tue, May 3, 2011 at 4:06 PM, Ralf Gommers wrote: > On Tue, May 3, 2011 at 10:35 PM, Christoph Gohlke wrote: >> >> >> On 5/3/2011 11:18 AM, Ralf Gommers wrote: >>> Hi, >>> >>> I am pleased to announce the availability of the second release >>> candidate of NumPy 1.6.0. >>> >>> Compared to the first release candidate, one segfault on (32-bit >>> Windows + MSVC) and several memory leaks were fixed. If no new >>> problems are reported, the final release will be in one week. >>> >>> Sources and binaries can be found at >>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ >>> For (preliminary) release notes see below. >>> >>> Enjoy, >>> Ralf >>> >> >> Looks good. The msvc9/MKL builds now pass all tests on win32 and >> win-amd64, python 2.6, 2.7, 3.1, and 3.2. > > Good, thanks for testing. >> >> One scipy test failure reported earlier remains, but that is probably no >> release blocker. >> > > That's a problem in scipy.stats, that only showed up recently because > of a bug fix in numpy.testing. > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Tue May 3 17:24:30 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 3 May 2011 17:24:30 -0400 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: <4DC06719.7030909@uci.edu> Message-ID: On Tue, May 3, 2011 at 5:06 PM, Ralf Gommers wrote: > On Tue, May 3, 2011 at 10:35 PM, Christoph Gohlke wrote: >> >> >> On 5/3/2011 11:18 AM, Ralf Gommers wrote: >>> Hi, >>> >>> I am pleased to announce the availability of the second release >>> candidate of NumPy 1.6.0. >>> >>> Compared to the first release candidate, one segfault on (32-bit >>> Windows + MSVC) and several memory leaks were fixed. If no new >>> problems are reported, the final release will be in one week. >>> >>> Sources and binaries can be found at >>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ >>> For (preliminary) release notes see below. >>> >>> Enjoy, >>> Ralf >>> >> >> Looks good. The msvc9/MKL builds now pass all tests on win32 and >> win-amd64, python 2.6, 2.7, 3.1, and 3.2. > > Good, thanks for testing. >> >> One scipy test failure reported earlier remains, but that is probably no >> release blocker. >> > > That's a problem in scipy.stats, that only showed up recently because > of a bug fix in numpy.testing. Sorry, I don't have 1.6 to test, but what are assertions like np.testing.assert_array_less(2, np.inf) np.testing.assert_array_less(np.array([ 0.911, 1.065, 1.325, 1.587]), np.inf) supposed to be with numpy 1.6 ? Josef > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Tue May 3 18:58:39 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Wed, 4 May 2011 00:58:39 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: Hi Ralf, > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. > > Compared to the first release candidate, one segfault on (32-bit > Windows + MSVC) and several memory leaks were fixed. If no new > problems are reported, the final release will be in one week. I found a problem apparently related to string handling on MacOS X 10.5/ppc with Python3 - not a new one though, at least it seemed to be present with 1.6.0b2: >>> numpy.test('full') Running unit tests for numpy NumPy version 1.6.0rc2 NumPy is installed in /sw/lib/python3.2/site-packages/numpy Python version 3.2 (r32:88445, Mar 1 2011, 18:28:16) [GCC 4.0.1 (Apple Inc. build 5493)] nose version 1.0.0 ... ====================================================================== FAIL: test_return_character.TestF77ReturnCharacter.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest self.test(*self.arg) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 78, in test_all self.check_function(getattr(self.module, name)) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 12, in check_function r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: b' ' ====================================================================== FAIL: test_return_character.TestF90ReturnCharacter.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest self.test(*self.arg) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 136, in test_all self.check_function(getattr(self.module.f90_return_char, name)) File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 12, in check_function r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: b' ' All the previous failures on ppc are gone now, and with Python2.[4567] all tests are passing! On Intel MacOS X (10.5/i386 and 10.6/x86_64) everything is OK for Python 2.5-3.2 as well. Cheers, Derek From kwgoodman at gmail.com Tue May 3 19:45:52 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 3 May 2011 16:45:52 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 11:18 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. I get one failure when I run from ipython but not python. In ipython I import a few packages at startup. One of those packages must be changing the numpy error settings (would that explain it?). Should that be a failure? The failing test: class TestSeterr(TestCase): def test_default(self): err = geterr() self.assertEqual(err, dict( divide='warn', invalid='warn', over='warn', under='ignore', )) ====================================================================== FAIL: test_default (test_numeric.TestSeterr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/numpy/core/tests/test_numeric.py", line 232, in test_default under='ignore', AssertionError: {'over': 'warn', 'divide': 'ignore', 'invalid': 'ignore', 'under': 'ignore'} != {'over': 'warn', 'divide': 'warn', 'invalid': 'warn', 'under': 'ignore'} ---------------------------------------------------------------------- From gaston.fiore at gmail.com Tue May 3 19:52:36 2011 From: gaston.fiore at gmail.com (Gaston Fiore) Date: Tue, 3 May 2011 19:52:36 -0400 Subject: [Numpy-discussion] Vector/matrix with 3200975422129 elements Message-ID: Hello, I'm trying to create a vector (or a matrix, could be either) with 3200975422129 elements but I'm not being successful, get the following error: >>> zeros(width * height, dtype=float32) Killed or >>> zeros((1789127,1789127), dtype=float32) Killed I'm assuming that I'm running out of memory. How could I go around this? This is some extra information: -bash-3.2$ python Python 2.7.1 (r271:86832, Apr 25 2011, 11:02:55) [GCC 4.3.3] on linux2 -bash-3.2$ python -c 'import numpy as np; print np.__version__' 1.6.0b2 Thanks a lot, -Gaston From robert.kern at gmail.com Tue May 3 19:52:16 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 18:52:16 -0500 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 18:45, Keith Goodman wrote: > On Tue, May 3, 2011 at 11:18 AM, Ralf Gommers > wrote: > >> I am pleased to announce the availability of the second release >> candidate of NumPy 1.6.0. > > I get one failure when I run from ipython but not python. In ipython I > import a few packages at startup. One of those packages must be > changing the numpy error settings (would that explain it?). Yes. > Should > that be a failure? Yes, I believe it's important to test numpy's default settings (and make sure that nothing else in numpy fudges with them). We have had regressions in the past that went unnoticed for years. Rather, you should not be running the test suite from an environment with a bunch of other packages imported (I know, I know, the existence of np.test() kind of implicitly encourages this, but still...). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From michaeladamkatz at yahoo.com Tue May 3 19:58:25 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Tue, 3 May 2011 16:58:25 -0700 (PDT) Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array Message-ID: <388444.51076.qm@web161307.mail.bf1.yahoo.com> I have a basic question about applying functions to array elements. There is a rambling introduction here and then I get to the ACTUAL QUESTION about 2/3 of the way down. RAMBLING INTRODUCTION I was trying to do a simple mapping of array values to other values. So I had: unmapped_colors = np.array((0,0,1,1,2,0,0,1)) BNA_LAND_FEATURE_CODE = 0 BNA_WATER_FEATURE_CODE = 1 # color_to_int() produces some unsigned int value green = color_to_int( 0.25, 0.5, 0, 0.75 ) blue = color_to_int( 0.0, 0.0, 0.5, 0.75 ) gray = color_to_int( 0.5, 0.5, 0.5, 0.75 ) def map_type_to_color( t ): if ( t == BNA_LAND_FEATURE_CODE ): return green elif ( t == BNA_WATER_FEATURE_CODE ): return blue else: return gray mapped_colors = np.vectorize( map_type_to_color )( unmapped_colors ) I found that by using vectorize I got about a 3x speedup compared to using a python for loop to loop through the array. One small question is I wonder where that speedup comes from; i.e., what vectorize is actually doing for me. Is it just the time for the actual looping logic? That seems like too much speedup. Does it also include a faster access of the array data items? So I was trying to find a "pure numpy" solution for this. I then learned about fancy indexing and boolean indexing, and saw that I could do boolean array version: mapped_colors = np.zeros(unmapped_colors.shape, dtype=np.uint32) + gray mapped_colors[unmapped_colors==BNA_LAND_FEATURE_CODE] = green mapped_colors[unmapped_colors==BNA_WATER_FEATURE_CODE] = blue and in fact I could do "fancy indexing" version: color_array = np.array((green, blue, gray), dtype=np.uint32) colors = color_array[unmapped_colors] The boolean array version is pretty simple, but makes three "passes" over the data. The fancy indexing version is simple and one pass, but it relies on the fact that my constant values happen to be nice to use as array indexes. And in fact to use it in actual code I'd need to do one or more other passes to check unmapped_colors for any indexes < 0 or > 2. ACTUAL QUESTION So, the actual question here is, even though I was able to find a couple solutions for this particular problem, I am wondering in general about applying arbitrary functions to each array element in a "pure numpy" way, ideally using just a single pass. It seems to me like there should be a some way to build up a "numpy compiled" function that you could then apply to all elements in an array. Is that what "universal functions" are about? That's what it sounded like it was about to me at first, but then after looking closer it didn't seem like that. I guess I was hoping vectorize was some magical thing that could compile certain python expressions down into "numpy primitives". It seems like as long as you stuck to a limited set of math and logical operations on arrays, even if those expressions included calls to subroutines that were similarly restricted, there should be an automated way to convert such expressions into the equivalent of a numpy built-in. I guess it's similar to cython, but it wouldn't require a separate file or special compilation. I could just say def capped_sqrt( n ): if ( x > 100 ) return 10 else: return sqrt( n ) f = numpy.compile( lambda(x): 0 if ( x < 10 ) else capped_sqrt( x ) ) numpy.map( f, a ) or something like that, and it would all happen in a single pass within numpy, with no "python code" being run. Is that something that exists? I realize that for any function like this I could build it out of a sequence of individual numpy operations on the whole array, but it seems that in general that requires doing several passes, creating multiple temp arrays, and computing functions for elements where the results will later be thrown out. This might be just because I'm a numpy newbie, but it's not clear to me how to do the example above on an array of size N without computing N square roots, even if the number of elements in my array between 10 and 100 is much smaller than N . Granted, I could set things up so that by the time sqrt was applied it would only be computing, say, sqrt( 0 ) for all those values that were originally < 10 or > 100. But still, it would be nice not to compute unnecessarily at all. And efficiency concerns aside, it just seems unfortunate, language-wise, to always have to think cleverly about some sequence of whole-array transformations instead of just being able to think about the "single pass computation" that is in your head (at least is in my head) in the first place and is after all the most direct and non-redundant way to get the result. Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue May 3 19:58:15 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 18:58:15 -0500 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 12:07, Matthew Brett wrote: > Hi, > > On Sat, Apr 30, 2011 at 5:21 AM, Ralf Gommers > wrote: >> On Wed, Apr 27, 2011 at 8:52 PM, Matthew Brett wrote: >>> Hi, >>> >>> This is just to follow up on a dead thread of mine a little while back. >>> >>> I was asking about letters for Clint Whaley's tenure case, from numpy, >>> but I realized that I don't know who 'numpy' is :) >>> >>> Is there in fact a numpy steering group? ? ?Who is best to write >>> letters representing the 'numpy community'? >> >> At http://scipy.org/Developer_Zone there's a list of people under a >> big header "steering committee". It seems to me that writing such a >> letter representing the community is one of the purposes that >> committee could serve. > > Ah - yes - thanks for the reply. > > In the interests of general transparency - and given that no-one from > that group has replied to this email - how should the group best be > addressed? ? By personal email? ?That seems to break the open-source > matra of everything on-list: > > http://producingoss.com/en/setting-tone.html#avoid-private-discussions Having project-relevant *discussions* on-list doesn't preclude getting someone's *attention* off-list. I can't speak for the rest of the group, but as for myself, if you would like to draft such a letter, I'm sure I will agree with its contents. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ben.root at ou.edu Tue May 3 20:03:32 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 3 May 2011 19:03:32 -0500 Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: <388444.51076.qm@web161307.mail.bf1.yahoo.com> References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> Message-ID: On Tue, May 3, 2011 at 6:58 PM, Michael Katz wrote: > I have a basic question about applying functions to array elements. There > is a rambling introduction here and then I get to the ACTUAL QUESTION about > 2/3 of the way down. > > RAMBLING INTRODUCTION > > I was trying to do a simple mapping of array values to other values. So I > had: > > unmapped_colors = np.array((0,0,1,1,2,0,0,1)) > BNA_LAND_FEATURE_CODE = 0 > BNA_WATER_FEATURE_CODE = 1 > # color_to_int() produces some unsigned int value > green = color_to_int( 0.25, 0.5, 0, 0.75 ) > blue = color_to_int( 0.0, 0.0, 0.5, 0.75 ) > gray = color_to_int( 0.5, 0.5, 0.5, 0.75 ) > def map_type_to_color( t ): > if ( t == BNA_LAND_FEATURE_CODE ): > return green > elif ( t == BNA_WATER_FEATURE_CODE ): > return blue > else: > return gray > mapped_colors = np.vectorize( map_type_to_color )( > unmapped_colors ) > > I found that by using vectorize I got about a 3x speedup compared to using > a python for loop to loop through the array. One small question is I wonder > where that speedup comes from; i.e., what vectorize is actually doing for > me. Is it just the time for the actual looping logic? That seems like too > much speedup. Does it also include a faster access of the array data items? > > So I was trying to find a "pure numpy" solution for this. I then learned > about fancy indexing and boolean indexing, and saw that I could do boolean > array version: > > mapped_colors = np.zeros(unmapped_colors.shape, dtype=np.uint32) + gray > > mapped_colors[unmapped_colors==BNA_LAND_FEATURE_CODE] = green > mapped_colors[unmapped_colors==BNA_WATER_FEATURE_CODE] = blue > > and in fact I could do "fancy indexing" version: > > color_array = np.array((green, blue, gray), dtype=np.uint32) > colors = color_array[unmapped_colors] > > The boolean array version is pretty simple, but makes three "passes" over > the data. > > The fancy indexing version is simple and one pass, but it relies on the > fact that my constant values happen to be nice to use as array indexes. And > in fact to use it in actual code I'd need to do one or more other passes to > check unmapped_colors for any indexes < 0 or > 2. > > ACTUAL QUESTION > > So, the actual question here is, even though I was able to find a couple > solutions for this particular problem, I am wondering in general about > applying arbitrary functions to each array element in a "pure numpy" way, > ideally using just a single pass. > > It seems to me like there should be a some way to build up a "numpy > compiled" function that you could then apply to all elements in an array. Is > that what "universal functions" are about? That's what it sounded like it > was about to me at first, but then after looking closer it didn't seem like > that. > > I guess I was hoping vectorize was some magical thing that could compile > certain python expressions down into "numpy primitives". It seems like as > long as you stuck to a limited set of math and logical operations on arrays, > even if those expressions included calls to subroutines that were similarly > restricted, there should be an automated way to convert such expressions > into the equivalent of a numpy built-in. I guess it's similar to cython, but > it wouldn't require a separate file or special compilation. I could just say > > def capped_sqrt( n ): > if ( x > 100 ) > return 10 > else: > return sqrt( n ) > f = numpy.compile( lambda(x): 0 if ( x < 10 ) else capped_sqrt( x ) ) > numpy.map( f, a ) > > or something like that, and it would all happen in a single pass within > numpy, with no "python code" being run. > > Is that something that exists? > > I realize that for any function like this I could build it out of a > sequence of individual numpy operations on the whole array, but it seems > that in general that requires doing several passes, creating multiple temp > arrays, and computing functions for elements where the results will later be > thrown out. This might be just because I'm a numpy newbie, but it's not > clear to me how to do the example above on an array of size N without > computing N square roots, even if the number of elements in my array between > 10 and 100 is much smaller than N . Granted, I could set things up so that > by the time sqrt was applied it would only be computing, say, sqrt( 0 ) for > all those values that were originally < 10 or > 100. But still, it would be > nice not to compute unnecessarily at all. And efficiency concerns aside, it > just seems unfortunate, language-wise, to always have to think cleverly > about some sequence of whole-array transformations instead of just being > able to think about the "single pass computation" that is in your head (at > least is in my head) in the first place and is after all the most direct > and non-redundant way to get the result. > > Michael > > I can't comment much on the questions you raise, but I can certainly suggest that np.where() function might be a much better choice than some of the if-statements. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue May 3 20:07:44 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 19:07:44 -0500 Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: <388444.51076.qm@web161307.mail.bf1.yahoo.com> References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> Message-ID: On Tue, May 3, 2011 at 18:58, Michael Katz wrote: > I have a basic question about applying functions to array elements. There is > a rambling introduction here and then I get to the ACTUAL QUESTION about 2/3 > of the way down. > > RAMBLING INTRODUCTION > > I was trying to do a simple mapping of array values to other values. So I > had: > > ??? ??? ??? ??? ??? unmapped_colors = np.array((0,0,1,1,2,0,0,1)) > ??? ??? ??? ??? ??? BNA_LAND_FEATURE_CODE = 0 > ??? ??? ??? ??? ??? BNA_WATER_FEATURE_CODE = 1 > ??? ??? ??? ??? ??? # color_to_int() produces some unsigned int value > ??????????????????? green = color_to_int( 0.25, 0.5, 0, 0.75 ) > ??????????????????? blue = color_to_int( 0.0, 0.0, 0.5, 0.75 ) > ??????????????????? gray = color_to_int( 0.5, 0.5, 0.5, 0.75 ) > ??????????????????? def map_type_to_color( t ): > ??????????????????????? if ( t == BNA_LAND_FEATURE_CODE ): > ??????????????????????????? return green > ??????????????????????? elif ( t == BNA_WATER_FEATURE_CODE ): > ??????????????????????????? return blue > ??????????????????????? else: > ??????????????????????????? return gray > ??? ??? ??? ??? ??? mapped_colors = np.vectorize( map_type_to_color )( > unmapped_colors ) > > I found that by using vectorize I got about a 3x speedup compared to using a > python for loop to loop through the array. One small question is I wonder > where that speedup comes from; i.e., what vectorize is actually doing for > me. Is it just the time for the actual looping logic? Pretty much. > That seems like too > much speedup. Does it also include a faster access of the array data items? > > So I was trying to find a "pure numpy" solution for this. I then learned > about fancy indexing and boolean indexing, and saw that I could do boolean > array version: > > ????mapped_colors = np.zeros(unmapped_colors.shape, dtype=np.uint32) + gray > ????mapped_colors[unmapped_colors==BNA_LAND_FEATURE_CODE] = green > ????mapped_colors[unmapped_colors==BNA_WATER_FEATURE_CODE] = blue > > and in fact I could do "fancy indexing" version: > > ??? color_array = np.array((green, blue, gray), dtype=np.uint32) > ??? colors = color_array[unmapped_colors] > > The boolean array version is pretty simple, but makes three "passes" over > the data. > > The fancy indexing version is simple and one pass, but it relies on the fact > that my constant values happen to be nice to use as array indexes. And in > fact to use it in actual code I'd need to do one or more other passes to > check unmapped_colors for any indexes < 0 or > 2. > > ACTUAL QUESTION > > So, the actual question here is, even though I was able to find a couple > solutions for this particular problem, I am wondering in general about > applying arbitrary functions to each array element in a "pure numpy" way, > ideally using just a single pass. > > It seems to me like there should be a some way to build up a "numpy > compiled" function that you could then apply to all elements in an array. Is > that what "universal functions" are about? That's what it sounded like it > was about to me at first, but then after looking closer it didn't seem like > that. Universal functions, or ufuncs, are called universal because they will accept any sequence that can be treated like an array. Roughly, if np.array(x) and np.array(y) work, then np.add(x, y) will work (assuming compatible dimensions, etc.). You do have to write C code in order to make really fast ones, though. np.vectorize() makes a ufunc from a Python function, but it still has a lot of Python overhead for each function call. > I guess I was hoping vectorize was some magical thing that could compile > certain python expressions down into "numpy primitives". It seems like as > long as you stuck to a limited set of math and logical operations on arrays, > even if those expressions included calls to subroutines that were similarly > restricted, there should be an automated way to convert such expressions > into the equivalent of a numpy built-in. I guess it's similar to cython, but > it wouldn't require a separate file or special compilation. I could just say > > ??? def capped_sqrt( n ): > ??? ??? if ( x > 100 ) > ??? ??? ??? return 10 > ??? ??? else: > ??? ??? ??? return sqrt( n ) > ??? f = numpy.compile( lambda(x): 0 if ( x < 10 ) else capped_sqrt( x ) ) > ??? numpy.map( f, a ) > > or something like that, and it would all happen in a single pass within > numpy, with no "python code" being run. > > Is that something that exists? Look at numexpr: http://code.google.com/p/numexpr/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Tue May 3 20:09:00 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 19:09:00 -0500 Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> Message-ID: On Tue, May 3, 2011 at 19:03, Benjamin Root wrote: > I can't comment much on the questions you raise, but I can certainly suggest > that np.where() function might be a much better choice than some of the > if-statements. But it won't be any better than the two fancy-indexing approaches he's used. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From robert.kern at gmail.com Tue May 3 20:14:06 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 19:14:06 -0500 Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: <388444.51076.qm@web161307.mail.bf1.yahoo.com> References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> Message-ID: On Tue, May 3, 2011 at 18:58, Michael Katz wrote: > So I was trying to find a "pure numpy" solution for this. I then learned > about fancy indexing and boolean indexing, and saw that I could do boolean > array version: > > ????mapped_colors = np.zeros(unmapped_colors.shape, dtype=np.uint32) + gray > ????mapped_colors[unmapped_colors==BNA_LAND_FEATURE_CODE] = green > ????mapped_colors[unmapped_colors==BNA_WATER_FEATURE_CODE] = blue > > and in fact I could do "fancy indexing" version: > > ??? color_array = np.array((green, blue, gray), dtype=np.uint32) > ??? colors = color_array[unmapped_colors] > > The boolean array version is pretty simple, but makes three "passes" over > the data. > > The fancy indexing version is simple and one pass, but it relies on the fact > that my constant values happen to be nice to use as array indexes. And in > fact to use it in actual code I'd need to do one or more other passes to > check unmapped_colors for any indexes < 0 or > 2. Also, still not *quite* as general as you might like, but sufficient for the problem as stated: colors = color_array[np.clip(unmapped_colors, 0, 2)] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Tue May 3 20:29:32 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 May 2011 18:29:32 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 4:58 PM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > Hi Ralf, > > > I am pleased to announce the availability of the second release > > candidate of NumPy 1.6.0. > > > > Compared to the first release candidate, one segfault on (32-bit > > Windows + MSVC) and several memory leaks were fixed. If no new > > problems are reported, the final release will be in one week. > > I found a problem apparently related to string handling on MacOS X > 10.5/ppc with Python3 - not a new one though, at least it seemed to > be present with 1.6.0b2: > > >>> numpy.test('full') > Running unit tests for numpy > NumPy version 1.6.0rc2 > NumPy is installed in /sw/lib/python3.2/site-packages/numpy > Python version 3.2 (r32:88445, Mar 1 2011, 18:28:16) [GCC 4.0.1 (Apple > Inc. build 5493)] > nose version 1.0.0 > ... > ====================================================================== > FAIL: test_return_character.TestF77ReturnCharacter.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest > self.test(*self.arg) > File > "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", > line 78, in test_all > self.check_function(getattr(self.module, name)) > File > "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", > line 12, in check_function > r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) > File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError: b' ' > > ====================================================================== > FAIL: test_return_character.TestF90ReturnCharacter.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest > self.test(*self.arg) > File > "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", > line 136, in test_all > self.check_function(getattr(self.module.f90_return_char, name)) > File > "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", > line 12, in check_function > r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) > File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError: b' ' > > All the previous failures on ppc are gone now, and with Python2.[4567] all > tests are passing! > On Intel MacOS X (10.5/i386 and 10.6/x86_64) everything is OK for Python > 2.5-3.2 as well. > > Looks like Fortran. Maybe we should lst the Fortran compiler when the full test suite is run. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michaeladamkatz at yahoo.com Tue May 3 20:34:48 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Tue, 3 May 2011 17:34:48 -0700 (PDT) Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> Message-ID: <280538.34249.qm@web161309.mail.bf1.yahoo.com> Robert, thanks for these quick replies. So, it sounds like the main thing is for me to get better at thinking "declaratively" in terms of these array operations, instead of the "imperative" way I normally think about processing arrays in C++. I did look at Numexpr, and that's in the direction I had in mind. It's not quite what I was thinking, in terms of being able to use arbitrary if/else statements and function calls, but it certainly seems to let you at least express your computations in a natural way without having to worry so much about the number of temp arrays being created. Michael ________________________________ From: Robert Kern To: Discussion of Numerical Python Sent: Tue, May 3, 2011 5:14:06 PM Subject: Re: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array On Tue, May 3, 2011 at 18:58, Michael Katz wrote: > So I was trying to find a "pure numpy" solution for this. I then learned > about fancy indexing and boolean indexing, and saw that I could do boolean > array version: > > mapped_colors = np.zeros(unmapped_colors.shape, dtype=np.uint32) + gray > mapped_colors[unmapped_colors==BNA_LAND_FEATURE_CODE] = green > mapped_colors[unmapped_colors==BNA_WATER_FEATURE_CODE] = blue > > and in fact I could do "fancy indexing" version: > > color_array = np.array((green, blue, gray), dtype=np.uint32) > colors = color_array[unmapped_colors] > > The boolean array version is pretty simple, but makes three "passes" over > the data. > > The fancy indexing version is simple and one pass, but it relies on the fact > that my constant values happen to be nice to use as array indexes. And in > fact to use it in actual code I'd need to do one or more other passes to > check unmapped_colors for any indexes < 0 or > 2. Also, still not *quite* as general as you might like, but sufficient for the problem as stated: colors = color_array[np.clip(unmapped_colors, 0, 2)] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Tue May 3 21:23:20 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 3 May 2011 18:23:20 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 11:18 AM, Ralf Gommers wrote: > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. nanmin and nanmax are much faster in Numpy 1.6. Plus they now return an object that has dtype, etc attributes when the input is all NaN. Broke Bottleneck unit tests and makes the benchmarks less impressive but: Nice. From halbert at halwitz.org Tue May 3 21:34:31 2011 From: halbert at halwitz.org (Dan Halbert) Date: Tue, 03 May 2011 21:34:31 -0400 Subject: [Numpy-discussion] Vector/matrix with 3200975422129 elements In-Reply-To: References: Message-ID: <4DC0AD27.4050106@halwitz.org> On 5/3/2011 7:52 PM, Gaston Fiore wrote: > I'm trying to create a vector (or a matrix, could be either) with > 3200975422129 elements but I'm not being successful In words, that's about 3.2 trillion float32's, so about 24 trillion bytes, > 2^41. So, yes, you are running out of memory, by a factor of several thousand, and I don't think you have 24T of swap space on your disk. So perhaps you need to rethink the problem. Maybe this matrix is sparse? (I hope so.) From halbert at halwitz.org Tue May 3 21:39:41 2011 From: halbert at halwitz.org (Dan Halbert) Date: Tue, 03 May 2011 21:39:41 -0400 Subject: [Numpy-discussion] Vector/matrix with 3200975422129 elements In-Reply-To: <4DC0AD27.4050106@halwitz.org> References: <4DC0AD27.4050106@halwitz.org> Message-ID: <4DC0AE5D.4040003@halwitz.org> On 5/3/2011 9:34 PM, Dan Halbert wrote: > On 5/3/2011 7:52 PM, Gaston Fiore wrote: >> I'm trying to create a vector (or a matrix, could be either) with >> 3200975422129 elements but I'm not being successful > In words, that's about 3.2 trillion float32's, so about 24 trillion > bytes,> 2^41. So, yes, you are running out of memory, by a factor of > several thousand, and I don't think you have 24T of swap space on your > disk. So perhaps you need to rethink the problem. Maybe this matrix is > sparse? (I hope so.) My mistake, only about 12 trillion. From robert.kern at gmail.com Tue May 3 22:33:41 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 3 May 2011 21:33:41 -0500 Subject: [Numpy-discussion] general newbie question about applying an arbitrary function to all elements in a numpy array In-Reply-To: <280538.34249.qm@web161309.mail.bf1.yahoo.com> References: <388444.51076.qm@web161307.mail.bf1.yahoo.com> <280538.34249.qm@web161309.mail.bf1.yahoo.com> Message-ID: On Tue, May 3, 2011 at 19:34, Michael Katz wrote: > Robert, thanks for these quick replies. > > So, it sounds like the main thing is for me to get better at thinking > "declaratively" in terms of these array operations, instead of the > "imperative" way I normally think about processing arrays in C++. > > I did look at Numexpr, and that's in the direction I had in mind. It's not > quite what I was thinking, in terms of being able to use arbitrary if/else > statements and function calls, but it certainly seems to let you at least > express your computations in a natural way without having to worry so much > about the number of temp arrays being created. Theano does something along those lines. http://deeplearning.net/software/theano/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From paul.anton.letnes at gmail.com Tue May 3 22:49:18 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Tue, 3 May 2011 19:49:18 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Mac OS X 10.6.7 with macports python 2.7 I get as follows: OK (KNOWNFAIL=3, SKIP=1) Python version: Python 2.7.1 (r271:86832, Jan 12 2011, 10:44:27) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin % gfortran --version GNU Fortran (GCC) 4.5.3 "Well done" to everyone working on numpy 1.6! Paul. On 3. mai 2011, at 11.18, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. > > Compared to the first release candidate, one segfault on (32-bit > Windows + MSVC) and several memory leaks were fixed. If no new > problems are reported, the final release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > This release includes several new features as well as numerous bug fixes and > improved documentation. It is backward compatible with the 1.5.0 release, and > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > Highlights > ========== > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > * A new 16-bit floating point type. > > * A new iterator, which improves performance of many functions. > > > New features > ============ > > New 16-bit floating point type > ------------------------------ > > This release adds support for the IEEE 754-2008 binary16 format, available as > the data type ``numpy.half``. Within Python, the type behaves similarly to > `float` or `double`, and C extensions can add support for it with the exposed > half-float API. > > > New iterator > ------------ > > A new iterator has been added, replacing the functionality of the > existing iterator and multi-iterator with a single object and API. > This iterator works well with general memory layouts different from > C or Fortran contiguous, and handles both standard NumPy and > customized broadcasting. The buffering, automatic data type > conversion, and optional output parameters, offered by > ufuncs but difficult to replicate elsewhere, are now exposed by this > iterator. > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > ------------------------------------------------------------------------- > > Extend the number of polynomials available in the polynomial package. In > addition, a new ``window`` attribute has been added to the classes in > order to specify the range the ``domain`` maps to. This is mostly useful > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > are infinite and provides a more intuitive way to get the correct mapping > of values without playing unnatural tricks with the domain. > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > as ``shape(array, dim-1)`` which makes it possible to automatically > wrap Fortran routines that use two argument ``size`` function in > dimension specifications. Before users were forced to apply this > mapping manually. > > > Other new functions > ------------------- > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > an array of flat indices, applying boundary modes to the indices. > > ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the > Einstein summation convention, many common multi-dimensional array operations > can be represented in a simple fashion. This function provides a way compute > such summations. > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose > the underlying type promotion used by the ufuncs and other operations to > determine the types of outputs. These improve upon the ``numpy.common_type`` > and ``numpy.mintypecode`` which provide similar functionality but do > not match the ufunc implementation. > > > Changes > ======= > > Changes and improvements in the numpy core > ------------------------------------------ > > ``default error handling`` > -------------------------- > > The default error handling has been change from ``print`` to ``warn`` for > all except for ``underflow``, which remains as ``ignore``. > > > ``numpy.distutils`` > ------------------- > > Several new compilers are supported for building Numpy: the Portland Group > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C > compiler on Linux. > > > ``numpy.testing`` > ----------------- > > The testing framework gained ``numpy.testing.assert_allclose``, which provides > a more convenient way to compare floating point arrays than > `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. > > > ``C API`` > --------- > > In addition to the APIs for the new iterator and half data type, a number > of other additions have been made to the C API. The type promotion > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > ``NPY_CASTING`` has been added which controls what types of casts are > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > and ``PyArray_CanCastTypeTo``. A more flexible way to handle > conversion of arbitrary python objects into arrays is exposed by > ``PyArray_GetArrayParamsFromObject``. > > > Deprecated features > =================== > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality > will be replaced by the new "density" keyword. > > > Removed features > ================ > > ``numpy.fft`` > ------------- > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, > which were aliases for the same functions without the 'e' in the name, were > removed. > > > ``numpy.memmap`` > ---------------- > > The `sync()` and `close()` methods of memmap were removed. Use `flush()` and > "del memmap" instead. > > > ``numpy.lib`` > ------------- > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > ``numpy.ma`` > ------------ > > Several deprecated items were removed from the ``numpy.ma`` module:: > > * ``numpy.ma.MaskedArray`` "raw_data" method > * ``numpy.ma.MaskedArray`` constructor "flag" keyword > * ``numpy.ma.make_mask`` "flag" keyword > * ``numpy.ma.allclose`` "fill_value" keyword > > > ``numpy.distutils`` > ------------------- > > The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` > instead. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From opossumnano at gmail.com Wed May 4 06:28:03 2011 From: opossumnano at gmail.com (Tiziano Zito) Date: Wed, 4 May 2011 12:28:03 +0200 Subject: [Numpy-discussion] [ANN] EuroScipy 2011 - deadline approaching Message-ID: <20110504102803.GE25988@tulpenbaum.cognition.tu-berlin.de> ===================================== EuroScipy 2011 - Deadline Approaching ===================================== Beware: talk submission deadline is approaching. You can submit your contribution until Sunday May 8. --------------------------------------------- The 4th European meeting on Python in Science --------------------------------------------- **Paris, Ecole Normale Sup?rieure, August 25-28 2011** We are happy to announce the 4th EuroScipy meeting, in Paris, August 2011. The EuroSciPy meeting is a cross-disciplinary gathering focused on the use and development of the Python language in scientific research. This event strives to bring together both users and developers of scientific tools, as well as academic research and state of the art industry. Main topics =========== - Presentations of scientific tools and libraries using the Python language, including but not limited to: - vector and array manipulation - parallel computing - scientific visualization - scientific data flow and persistence - algorithms implemented or exposed in Python - web applications and portals for science and engineering. - Reports on the use of Python in scientific achievements or ongoing projects. - General-purpose Python tools that can be of special interest to the scientific community. Tutorials ========= There will be two tutorial tracks at the conference, an introductory one, to bring up to speed with the Python language as a scientific tool, and an advanced track, during which experts of the field will lecture on specific advanced topics such as advanced use of numpy, scientific visualization, software engineering... Keynote Speaker: Fernando Perez =============================== We are excited to welcome Fernando Perez (UC Berkeley, Helen Wills Neuroscience Institute, USA) as our keynote speaker. Fernando Perez is the original author of the enhanced interactive python shell IPython and a very active contributor to the Python for Science ecosystem. Important dates =============== Talk submission deadline: Sunday May 8 Program announced: Sunday May 29 Tutorials tracks: Thursday August 25 - Friday August 26 Conference track: Saturday August 27 - Sunday August 28 Call for papers =============== We are soliciting talks that discuss topics related to scientific computing using Python. These include applications, teaching, future development directions, and research. We welcome contributions from the industry as well as the academic world. Indeed, industrial research and development as well academic research face the challenge of mastering IT tools for exploration, modeling and analysis. We look forward to hearing your recent breakthroughs using Python! Submission guidelines ===================== - We solicit talk proposals in the form of a one-page long abstract. - Submissions whose main purpose is to promote a commercial product or service will be refused. - All accepted proposals must be presented at the EuroSciPy conference by at least one author. The one-page long abstracts are for conference planing and selection purposes only. We will later select papers for publication of post-proceedings in a peer-reviewed journal. How to submit an abstract ========================= To submit a talk to the EuroScipy conference follow the instructions here: http://www.euroscipy.org/card/euroscipy2011_call_for_papers Organizers ========== Chairs: - Ga?l Varoquaux (INSERM, Unicog team, and INRIA, Parietal team) - Nicolas Chauvat (Logilab) Local organization committee: - Emmanuelle Gouillart (Saint-Gobain Recherche) - Jean-Philippe Chauvat (Logilab) Tutorial chair: - Valentin Haenel (MKP, Technische Universit?t Berlin) Program committee: - Chair: Tiziano Zito (MKP, Technische Universit?t Berlin) - Romain Brette (ENS Paris, DEC) - Emmanuelle Gouillart (Saint-Gobain Recherche) - Eric Lebigot (Laboratoire Kastler Brossel, Universit? Pierre et Marie Curie) - Konrad Hinsen (Soleil Synchrotron, CNRS) - Hans Petter Langtangen (Simula laboratories) - Jarrod Millman (UC Berkeley, Helen Wills NeuroScience institute) - Mike M?ller (Python Academy) - Didrik Pinte (Enthought Inc) - Marc Poinot (ONERA) - Christophe Pradal (CIRAD/INRIA, Virtual Plantes team) - Andreas Schreiber (DLR) - St?fan van der Walt (University of Stellenbosch) Website ======= http://www.euroscipy.org/conference/euroscipy_2011 From cwg at falma.de Wed May 4 09:19:20 2011 From: cwg at falma.de (Christoph Groth) Date: Wed, 04 May 2011 15:19:20 +0200 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ Message-ID: <87aaf27gdj.fsf@falma.de> Dear numpy experts, I have noticed that with Numpy 1.5.1 the operation m[::2] += 1.0 takes twice as long as t = m[::2] t += 1.0 where "m" is some large matrix. This is of course because the first snippet is equivalent to t = m[::2] t += 1.0 m[::2] = t I wonder whether it would not be a good idea to optimize ndarray.__setitem__ to not execute an assignment of a slice onto itself. Is there any good reason why this is not being done already? best, Christoph From robert.kern at gmail.com Wed May 4 11:45:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 May 2011 10:45:38 -0500 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ In-Reply-To: <87aaf27gdj.fsf@falma.de> References: <87aaf27gdj.fsf@falma.de> Message-ID: On Wed, May 4, 2011 at 08:19, Christoph Groth wrote: > Dear numpy experts, > > I have noticed that with Numpy 1.5.1 the operation > > m[::2] += 1.0 > > takes twice as long as > > t = m[::2] > t += 1.0 > > where "m" is some large matrix. ?This is of course because the first > snippet is equivalent to > > t = m[::2] > t += 1.0 > m[::2] = t > > I wonder whether it would not be a good idea to optimize > ndarray.__setitem__ to not execute an assignment of a slice onto itself. > Is there any good reason why this is not being done already? We didn't think of it. If you can write up a patch that works safely and shows a performance improvement, it's probably worth putting in. It's probably not *that* common of a bottleneck, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From mwwiebe at gmail.com Wed May 4 11:57:42 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 4 May 2011 08:57:42 -0700 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ In-Reply-To: <87aaf27gdj.fsf@falma.de> References: <87aaf27gdj.fsf@falma.de> Message-ID: On Wed, May 4, 2011 at 6:19 AM, Christoph Groth wrote: > Dear numpy experts, > > I have noticed that with Numpy 1.5.1 the operation > > m[::2] += 1.0 > > takes twice as long as > > t = m[::2] > t += 1.0 > > where "m" is some large matrix. This is of course because the first > snippet is equivalent to > > t = m[::2] > t += 1.0 > m[::2] = t > > I wonder whether it would not be a good idea to optimize > ndarray.__setitem__ to not execute an assignment of a slice onto itself. > Is there any good reason why this is not being done already? > > best, Christoph > You'd better time this in 1.6 too. ;) https://github.com/numpy/numpy/commit/f60797ba64ccf33597225d23b893b6eb11149860 The case of boolean mask indexing can't benefit so easily from this optimization, but I think could see a big performance benefit if combined __index__ + __i__ operators were added to Python. Something to consider, anyway. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed May 4 12:14:26 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 4 May 2011 12:14:26 -0400 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Hi, On Tue, May 3, 2011 at 7:58 PM, Robert Kern wrote: > On Tue, May 3, 2011 at 12:07, Matthew Brett wrote: >> Hi, >> >> On Sat, Apr 30, 2011 at 5:21 AM, Ralf Gommers >> wrote: >>> On Wed, Apr 27, 2011 at 8:52 PM, Matthew Brett wrote: >>>> Hi, >>>> >>>> This is just to follow up on a dead thread of mine a little while back. >>>> >>>> I was asking about letters for Clint Whaley's tenure case, from numpy, >>>> but I realized that I don't know who 'numpy' is :) >>>> >>>> Is there in fact a numpy steering group? ? ?Who is best to write >>>> letters representing the 'numpy community'? >>> >>> At http://scipy.org/Developer_Zone there's a list of people under a >>> big header "steering committee". It seems to me that writing such a >>> letter representing the community is one of the purposes that >>> committee could serve. >> >> Ah - yes - thanks for the reply. >> >> In the interests of general transparency - and given that no-one from >> that group has replied to this email - how should the group best be >> addressed? ? By personal email? ?That seems to break the open-source >> matra of everything on-list: >> >> http://producingoss.com/en/setting-tone.html#avoid-private-discussions > > Having project-relevant *discussions* on-list doesn't preclude getting > someone's *attention* off-list. Yes, that's true. My worry was that, having put the question on the list, and not had an answer, it might send a bad signal if it was obvious that I had only got a reply because I'd asked for one off-list. > I can't speak for the rest of the group, but as for myself, if you > would like to draft such a letter, I'm sure I will agree with its > contents. Thank you - sadly I am not confident in deserving your confidence, but I will do my best to say something sensible. Any objections to a public google doc? See you, Matthew From robert.kern at gmail.com Wed May 4 12:24:38 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 4 May 2011 11:24:38 -0500 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: On Wed, May 4, 2011 at 11:14, Matthew Brett wrote: > Hi, > > On Tue, May 3, 2011 at 7:58 PM, Robert Kern wrote: >> I can't speak for the rest of the group, but as for myself, if you >> would like to draft such a letter, I'm sure I will agree with its >> contents. > > Thank you - sadly I am not confident in deserving your confidence, but > I will do my best to say something sensible. ? Any objections to a > public google doc? Even better! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From matthew.brett at gmail.com Wed May 4 12:53:13 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 4 May 2011 12:53:13 -0400 Subject: [Numpy-discussion] numpy easy_install fails for python 3.2 Message-ID: Hi, I can imagine that this is low-priority, but I have just been enjoying pytox for automated virtualenv testing: http://codespeak.net/tox/index.html which revealed that numpy download-build-install via easy_install (distribute) fails with the appended traceback ending in "ValueError: 'build/py3k/numpy' is not a directory". easy_install for pythons 2.5 - 2.7 work fine. Best, Matthew RefactoringTool: /tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/compat/py3k.py Running from numpy source directory.Converting to Python3 via 2to3... Traceback (most recent call last): File "../bin/easy_install", line 9, in load_entry_point('distribute==0.6.14', 'console_scripts', 'easy_install')() File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 1855, in main with_ei_usage(lambda: File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage return f() File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 1859, in distclass=DistributionWithoutHelpCommands, **kw File "/usr/lib/python3.2/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python3.2/distutils/dist.py", line 919, in run_commands self.run_command(cmd) File "/usr/lib/python3.2/distutils/dist.py", line 938, in run_command cmd_obj.run() File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 342, in run self.easy_install(spec, not self.no_deps) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 582, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 612, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 802, in install_eggs return self.build_and_install(setup_script, setup_base) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 1079, in build_and_install self.run_setup(setup_script, setup_base, args) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/command/easy_install.py", line 1068, in run_setup run_setup(setup_script, args) File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/sandbox.py", line 30, in run_setup lambda: exec(compile(open( File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/sandbox.py", line 71, in run return func() File "/home/mb312/dev_trees/nibabel/.tox/py32/lib/python3.2/site-packages/distribute-0.6.14-py3.2.egg/setuptools/sandbox.py", line 33, in {'__file__':setup_script, '__name__':'__main__'}) File "setup.py", line 211, in File "setup.py", line 204, in setup_package File "/tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/distutils/core.py", line 152, in setup File "setup.py", line 151, in configuration File "/tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 972, in add_subpackage File "/tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 941, in get_subpackage File "/tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py File "numpy/setup.py", line 5, in configuration File "/tmp/easy_install-xr2px3/numpy-1.5.1/build/py3k/numpy/distutils/misc_util.py", line 713, in __init__ ValueError: 'build/py3k/numpy' is not a directory From ralf.gommers at googlemail.com Wed May 4 13:23:42 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 May 2011 19:23:42 +0200 Subject: [Numpy-discussion] numpy easy_install fails for python 3.2 In-Reply-To: References: Message-ID: On Wed, May 4, 2011 at 6:53 PM, Matthew Brett wrote: > Hi, > > I can imagine that this is low-priority, but I have just been enjoying > pytox for automated virtualenv testing: > > http://codespeak.net/tox/index.html > > which revealed that numpy download-build-install via easy_install > (distribute) fails with the appended traceback ending in "ValueError: > 'build/py3k/numpy' is not a directory". > I think it would be good to just say "wontfix" immediately, rather than just leaving a ticket open and not do anything (like we did with http://projects.scipy.org/numpy/ticket/860). It seems tox can also use pip (which works with py3k now), does that work for you? Ralf > > easy_install for pythons 2.5 - 2.7 work fine. > > Best, > > Matthew > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed May 4 14:37:38 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 May 2011 20:37:38 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: <4DC06719.7030909@uci.edu> Message-ID: On Tue, May 3, 2011 at 11:24 PM, wrote: > On Tue, May 3, 2011 at 5:06 PM, Ralf Gommers > wrote: > > On Tue, May 3, 2011 at 10:35 PM, Christoph Gohlke > wrote: > >> > >> > >> On 5/3/2011 11:18 AM, Ralf Gommers wrote: > >>> Hi, > >>> > >>> I am pleased to announce the availability of the second release > >>> candidate of NumPy 1.6.0. > >>> > >>> Compared to the first release candidate, one segfault on (32-bit > >>> Windows + MSVC) and several memory leaks were fixed. If no new > >>> problems are reported, the final release will be in one week. > >>> > >>> Sources and binaries can be found at > >>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > >>> For (preliminary) release notes see below. > >>> > >>> Enjoy, > >>> Ralf > >>> > >> > >> Looks good. The msvc9/MKL builds now pass all tests on win32 and > >> win-amd64, python 2.6, 2.7, 3.1, and 3.2. > > > > Good, thanks for testing. > >> > >> One scipy test failure reported earlier remains, but that is probably no > >> release blocker. > >> < > http://mail.scipy.org/pipermail/numpy-discussion/2011-April/055877.html> > > > > That's a problem in scipy.stats, that only showed up recently because > > of a bug fix in numpy.testing. > > Sorry, I don't have 1.6 to test, but what are assertions like > > np.testing.assert_array_less(2, np.inf) > np.testing.assert_array_less(np.array([ 0.911, 1.065, 1.325, 1.587]), > np.inf) > > supposed to be with numpy 1.6 ? > > The same as what they were before ideally, but I get an AssertionError. No tests for assert_array_less, so I missed this. The easiest way to fix this I can think of is to add a switch to assert_array_compare that determines whether or not to special-case nan/inf. What do you think about these ones: >>> assert_array_less([1, np.inf], [2, np.inf]) >>> assert_array_less([1, np.nan], [2, np.nan]) They both pass now. The first one did not with 1.5, the second one did (I think). It seems to me that there's no obvious answer, it's just not very well-defined. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed May 4 14:42:44 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 May 2011 20:42:44 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Wed, May 4, 2011 at 2:29 AM, Charles R Harris wrote: > > > On Tue, May 3, 2011 at 4:58 PM, Derek Homeier < > derek at astro.physik.uni-goettingen.de> wrote: > >> Hi Ralf, >> >> > I am pleased to announce the availability of the second release >> > candidate of NumPy 1.6.0. >> > >> > Compared to the first release candidate, one segfault on (32-bit >> > Windows + MSVC) and several memory leaks were fixed. If no new >> > problems are reported, the final release will be in one week. >> >> I found a problem apparently related to string handling on MacOS X >> 10.5/ppc with Python3 - not a new one though, at least it seemed to >> be present with 1.6.0b2: >> >> >>> numpy.test('full') >> Running unit tests for numpy >> NumPy version 1.6.0rc2 >> NumPy is installed in /sw/lib/python3.2/site-packages/numpy >> Python version 3.2 (r32:88445, Mar 1 2011, 18:28:16) [GCC 4.0.1 (Apple >> Inc. build 5493)] >> nose version 1.0.0 >> ... >> ====================================================================== >> FAIL: test_return_character.TestF77ReturnCharacter.test_all >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in >> runTest >> self.test(*self.arg) >> File >> "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", >> line 78, in test_all >> self.check_function(getattr(self.module, name)) >> File >> "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", >> line 12, in check_function >> r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) >> File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, >> in assert_ >> raise AssertionError(msg) >> AssertionError: b' ' >> >> ====================================================================== >> FAIL: test_return_character.TestF90ReturnCharacter.test_all >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in >> runTest >> self.test(*self.arg) >> File >> "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", >> line 136, in test_all >> self.check_function(getattr(self.module.f90_return_char, name)) >> File >> "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", >> line 12, in check_function >> r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) >> File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, >> in assert_ >> raise AssertionError(msg) >> AssertionError: b' ' >> >> All the previous failures on ppc are gone now, and with Python2.[4567] all >> tests are passing! >> On Intel MacOS X (10.5/i386 and 10.6/x86_64) everything is OK for Python >> 2.5-3.2 as well. >> >> > Looks like Fortran. Maybe we should lst the Fortran compiler when the full > test suite is run. > > lst = list? That would be useful. Is this failure a problem or can it wait? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed May 4 14:51:33 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 4 May 2011 14:51:33 -0400 Subject: [Numpy-discussion] numpy easy_install fails for python 3.2 In-Reply-To: References: Message-ID: Hi, On Wed, May 4, 2011 at 1:23 PM, Ralf Gommers wrote: > > > On Wed, May 4, 2011 at 6:53 PM, Matthew Brett > wrote: >> >> Hi, >> >> I can imagine that this is low-priority, but I have just been enjoying >> pytox for automated virtualenv testing: >> >> http://codespeak.net/tox/index.html >> >> which revealed that numpy download-build-install via easy_install >> (distribute) fails with the appended traceback ending in "ValueError: >> 'build/py3k/numpy' is not a directory". > > I think it would be good to just say "wontfix" immediately, rather than just > leaving a ticket open and not do anything (like we did with > http://projects.scipy.org/numpy/ticket/860). Ouch - yes - I see what you mean. > It seems tox can also use pip (which works with py3k now), does that work > for you? I think current tox 0.9 uses virtualenv5 for python3.2 and has to use distribute, I believe. Current tip of pytox appears to use virtualenv 1.6.1 for python 3.2, and does use pip, but generates the same error in the end. I've appended the result of a fresh python3.2 virtualenv and a "pip install numpy". Sorry - I know these are not fun problems, See you, Matthew RefactoringTool: /home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/core/defchararray.py Running from numpy source directory.Traceback (most recent call last): File "", line 14, in File "/home/mb312/.virtualenvs/bare-32/build/numpy/setup.py", line 211, in setup_package() File "/home/mb312/.virtualenvs/bare-32/build/numpy/setup.py", line 204, in setup_package configuration=configuration ) File "/home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/distutils/core.py", line 152, in setup config = configuration() File "/home/mb312/.virtualenvs/bare-32/build/numpy/setup.py", line 151, in configuration config.add_subpackage('numpy') File "/home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/setup.py", line 5, in configuration config = Configuration('numpy',parent_package,top_path) File "/home/mb312/.virtualenvs/bare-32/build/numpy/build/py3k/numpy/distutils/misc_util.py", line 713, in __init__ raise ValueError("%r is not a directory" % (package_path,)) ValueError: 'build/py3k/numpy' is not a directory From ralf.gommers at googlemail.com Wed May 4 14:51:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 4 May 2011 20:51:50 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: <4DC06719.7030909@uci.edu> Message-ID: On Tue, May 3, 2011 at 11:22 PM, Ilan Schnell wrote: > I'm seeing these three failures on Solaris 5.10 (x86_64, using Python > 2.7.1): > > ====================================================================== > FAIL: Test basic arithmetic function errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/demo/master/lib/python2.7/site-packages/numpy/testing/decorators.py", > line 215, in knownfailer > return f(*args, **kwargs) > File > "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > line 321, in test_floating_exceptions > lambda a,b:a/b, ft_tiny, ft_max) > File > "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > line 271, in assert_raises_fpe > "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) > File > "/home/demo/master/lib/python2.7/site-packages/numpy/testing/utils.py", > line 34, in assert_ > raise AssertionError(msg) > AssertionError: Type did not raise fpe error > 'underflow'. > This is #1755, it's just not marked as knownfail on Solaris. I'll do that. > > ====================================================================== > FAIL: test_zero_nzero (test_umath.TestArctan2SpecialValues) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", > line 322, in test_zero_nzero > assert_almost_equal(ncu.arctan2(np.PZERO, np.NZERO), np.pi) > File > "/home/demo/master/lib/python2.7/site-packages/numpy/testing/utils.py", > line 468, in assert_almost_equal > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 7 decimals > ACTUAL: 0.0 > DESIRED: 3.141592653589793 > > ====================================================================== > FAIL: test_zero_pzero (test_umath.TestArctan2SpecialValues) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", > line 328, in test_zero_pzero > assert_arctan2_isnzero(np.NZERO, np.PZERO) > File > "/home/demo/master/lib/python2.7/site-packages/numpy/core/tests/test_umath.py", > line 310, in assert_arctan2_isnzero > assert (ncu.arctan2(x, y) == 0 and np.signbit(ncu.arctan2(x, y))), > "arctan(%s, %s) is %s, not -0" % (x, y, ncu.arctan2(x, y)) > AssertionError: arctan(-0.0, 0.0) is 0.0, not -0 > > These are really corner cases. Can you open a ticket? > > I'm not sure what the state of Solaris support is these days, but > I remember being able to run all tests without any failures. > > Pretty good apparently. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Wed May 4 18:47:26 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 5 May 2011 00:47:26 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: <8645AF5A-34BA-4A90-A213-597F7FF6ADC0@astro.physik.uni-goettingen.de> On 04.05.2011, at 8:42PM, Ralf Gommers wrote: > ====================================================================== > FAIL: test_return_character.TestF90ReturnCharacter.test_all > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python3.2/site-packages/nose/case.py", line 188, in runTest > self.test(*self.arg) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 136, in test_all > self.check_function(getattr(self.module.f90_return_char, name)) > File "/sw/lib/python3.2/site-packages/numpy/f2py/tests/test_return_character.py", line 12, in check_function > r = t(array('ab'));assert_( r==asbytes('a'),repr(r)) > File "/sw/lib/python3.2/site-packages/numpy/testing/utils.py", line 34, in assert_ > raise AssertionError(msg) > AssertionError: b' ' > > All the previous failures on ppc are gone now, and with Python2.[4567] all tests are passing! > On Intel MacOS X (10.5/i386 and 10.6/x86_64) everything is OK for Python 2.5-3.2 as well. > > > Looks like Fortran. Maybe we should lst the Fortran compiler when the full test suite is run. > > lst = list? That would be useful. > It seems to be listed in case of an f2py error, so maybe this could be done for failures as well... The compiler versions used to compile numpy would also be useful, I suppose. Anyway, the failure above turns up with numpy built with either of the system gcc (4.0.1 or 4.2.1) and gfortran from the fink-installed gcc 4.2.4 or 4.4.4. Unfortunately I cannot build numpy itself with the fink gcc versions, since they don't support the '-faltivec' flag. > Is this failure a problem or can it wait? Definitely not a show-stopper for 1.6.0, afaiac. I noticed another strange behaviour of f2py under OS X 10.5 - ppc and i386: When I have g77 installed, it seems to no longer find the (still present) gfortran, causing other test errors (maybe not surprising, since it is g77 3.4.3, but I'd still think gfortran should be given preference, if both are installed). Cheers, Derek ERROR: test_return_real.TestF77ReturnReal.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 371, in setUp try_run(self.inst, ('setup', 'setUp')) File "/sw/lib/python2.7/site-packages/nose/util.py", line 478, in try_run return func() File "/sw/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 347, in setUp module_name=self.module_name) File "/sw/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper memo[key] = func(*a, **kw) File "/sw/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 162, in build_code module_name=module_name) File "/sw/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 73, in wrapper memo[key] = func(*a, **kw) File "/sw/lib/python2.7/site-packages/numpy/f2py/tests/util.py", line 134, in build_module % (cmd[4:], asstr(out))) RuntimeError: Running f2py failed: ['-m', '_test_ext_module_5403', '/scratch/derek/tmp/tmp6cjHzz/tmpZt9fgZ.f'] running build [...] build_src: building npy-pkg config files running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize NAGFCompiler Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Found executable /sw/bin/f77 absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Found executable /sw/bin/g77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize GnuFCompiler using build_ext building '_test_ext_module_5403' extension compiling C sources C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes creating /scratch/derek/tmp/tmpeRWXKV/scratch creating /scratch/derek/tmp/tmpeRWXKV/scratch/derek creating /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp creating /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV creating /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7 compile options: '-I/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7 -I/sw/lib/python2.7/site-packages/numpy/core/include -I/sw/include/python2.7 -c' gcc: /scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403module.c gcc: /scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/fortranobject.c compiling Fortran sources Fortran f77 compiler: /sw/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -funroll-loops creating /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmp6cjHzz compile options: '-I/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7 -I/sw/lib/python2.7/site-packages/numpy/core/include -I/sw/include/python2.7 -c' g77:f77: /scratch/derek/tmp/tmp6cjHzz/tmpZt9fgZ.f g77:f77: /scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403-f2pywrappers.f /sw/bin/g77 -g -Wall -g -Wall -undefined dynamic_lookup -bundle /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403module.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/fortranobject.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmp6cjHzz/tmpZt9fgZ.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403-f2pywrappers.o -L/sw/lib/gcc/powerpc-apple-darwin9.8.0/3.4.3 -lg2c -lcc_dynamic -o ./_test_ext_module_5403.so ld: library not found for -lcc_dynamic collect2: ld returned 1 exit status ld: library not found for -lcc_dynamic collect2: ld returned 1 exit status error: Command "/sw/bin/g77 -g -Wall -g -Wall -undefined dynamic_lookup -bundle /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403module.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/fortranobject.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmp6cjHzz/tmpZt9fgZ.o /scratch/derek/tmp/tmpeRWXKV/scratch/derek/tmp/tmpeRWXKV/src.macosx-10.5-ppc-2.7/_test_ext_module_5403-f2pywrappers.o -L/sw/lib/gcc/powerpc-apple-darwin9.8.0/3.4.3 -lg2c -lcc_dynamic -o ./_test_ext_module_5403.so" failed with exit status 1 From ben.root at ou.edu Wed May 4 19:10:54 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 4 May 2011 18:10:54 -0500 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 1:18 PM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. > > Compared to the first release candidate, one segfault on (32-bit > Windows + MSVC) and several memory leaks were fixed. If no new > problems are reported, the final release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > Minor issue I just noticed on my recently installed Ubuntu 11.04 machine. The setup script is making a call to 'svnversion'. Doesn't impact the build or anything, but I only noticed it because svn hasn't been installed yet on that machine. Don't know if it is something that ought to be cleaned up or not. Otherwise, all tests on my machines (Fedora 13 x86 python 2.6, and Ubuntu 11.04 x86, python 2.7) work just fine. I will be testing some of the changes to file-loading later tonight. Good release! Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Wed May 4 20:34:25 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 5 May 2011 02:34:25 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> Message-ID: <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> Hi Paul, I've got back to your suggestion re. the ndmin flag for loadtxt from a few weeks ago... On 27.03.2011, at 12:09PM, Paul Anton Letnes wrote: >>> 1562: >>> I attach a possible patch. This could also be the default >>> behavior to my mind, since the function caller can simply call >>> numpy.squeeze if needed. Changing default behavior would probably >>> break old code, however. >> >> See comments on Trac as well. > > Your patch is better, but there is one thing I disagree with. > 808 if X.ndim < ndmin: > 809 if ndmin == 1: > 810 X.shape = (X.size, ) > 811 elif ndmin == 2: > 812 X.shape = (X.size, 1) > The last line should be: > 812 X.shape = (1, X.size) > If someone wants a 2D array out, they would most likely expect a one-row file to come out as a one-row array, not the other way around. IMHO. I think you are completely right for the test case with one row. More generally though, since a file of N rows and M columns is read into an array of shape (N, M), ndmin=2 should enforce X.shape = (1, X.size) for single-row input, and X.shape = (X.size, 1) for single-column input. I thought this would be handled automatically by preserving the original 2 dimensions, but apparently with single-row/multi-column input an extra dimension 1 is prepended when the array is returned from the parser. I've put up a fix for this at https://github.com/dhomeier/numpy/compare/master...ndmin-cols and also tested the patch against 1.6.0.rc2. Cheers, Derek From paul.anton.letnes at gmail.com Wed May 4 20:40:59 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 4 May 2011 17:40:59 -0700 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> Message-ID: <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> On 4. mai 2011, at 17.34, Derek Homeier wrote: > Hi Paul, > > I've got back to your suggestion re. the ndmin flag for loadtxt from a few weeks ago... > > On 27.03.2011, at 12:09PM, Paul Anton Letnes wrote: > >>>> 1562: >>>> I attach a possible patch. This could also be the default >>>> behavior to my mind, since the function caller can simply call >>>> numpy.squeeze if needed. Changing default behavior would probably >>>> break old code, however. >>> >>> See comments on Trac as well. >> >> Your patch is better, but there is one thing I disagree with. >> 808 if X.ndim < ndmin: >> 809 if ndmin == 1: >> 810 X.shape = (X.size, ) >> 811 elif ndmin == 2: >> 812 X.shape = (X.size, 1) >> The last line should be: >> 812 X.shape = (1, X.size) >> If someone wants a 2D array out, they would most likely expect a one-row file to come out as a one-row array, not the other way around. IMHO. > > I think you are completely right for the test case with one row. More generally though, > since a file of N rows and M columns is read into an array of shape (N, M), ndmin=2 > should enforce X.shape = (1, X.size) for single-row input, and X.shape = (X.size, 1) > for single-column input. > I thought this would be handled automatically by preserving the original 2 dimensions, > but apparently with single-row/multi-column input an extra dimension 1 is prepended > when the array is returned from the parser. I've put up a fix for this at > > https://github.com/dhomeier/numpy/compare/master...ndmin-cols > > and also tested the patch against 1.6.0.rc2. > > Cheers, > Derek Looks sensible to me at least! But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? Paul From derek at astro.physik.uni-goettingen.de Wed May 4 20:54:36 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 5 May 2011 02:54:36 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? Yes, good point, one could replace the X.shape = (X.size, ) with X = np.atleast_1d(X), but for the ndmin=2 case, we'd need to replace X.shape = (X.size, 1) with X = np.atleast_2d(X).T - not sure which solution is more efficient in terms of memory access etc... Cheers, Derek From ben.root at ou.edu Wed May 4 23:33:28 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 4 May 2011 22:33:28 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written > for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it > will reintroduce the 'transposed' problem? > > Yes, good point, one could replace the > X.shape = (X.size, ) with X = np.atleast_1d(X), > but for the ndmin=2 case, we'd need to replace > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > not sure which solution is more efficient in terms of memory access etc... > > Cheers, > Derek > > I can confirm that the current behavior is not sufficient for all of the original corner cases that ndmin was supposed to address. Keep in mind that np.loadtxt takes a one-column data file and a one-row data file down to the same shape. I don't see how the current code is able to produce the correct array shape when ndmin=2. Do we have some sort of counter in loadtxt for counting the number of rows and columns read? Could we use those to help guide the ndmin=2 case? I think that using atleast_1d(X) might be a bit overkill, but it would be very clear as to the code's intent. I don't think we have to worry about memory usage if we limit its use to only situations where ndmin is greater than the number of dimensions of the array. In those cases, the array is either an empty result, a scalar value (in which memory access is trivial), or 1-d (in which a transpose is cheap). Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Thu May 5 00:08:23 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Wed, 4 May 2011 21:08:23 -0700 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On 4. mai 2011, at 20.33, Benjamin Root wrote: > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier wrote: > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? > > Yes, good point, one could replace the > X.shape = (X.size, ) with X = np.atleast_1d(X), > but for the ndmin=2 case, we'd need to replace > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > not sure which solution is more efficient in terms of memory access etc... > > Cheers, > Derek > > > I can confirm that the current behavior is not sufficient for all of the original corner cases that ndmin was supposed to address. Keep in mind that np.loadtxt takes a one-column data file and a one-row data file down to the same shape. I don't see how the current code is able to produce the correct array shape when ndmin=2. Do we have some sort of counter in loadtxt for counting the number of rows and columns read? Could we use those to help guide the ndmin=2 case? > > I think that using atleast_1d(X) might be a bit overkill, but it would be very clear as to the code's intent. I don't think we have to worry about memory usage if we limit its use to only situations where ndmin is greater than the number of dimensions of the array. In those cases, the array is either an empty result, a scalar value (in which memory access is trivial), or 1-d (in which a transpose is cheap). What if one does things the other way around - avoid calling squeeze until _after_ doing the atleast_Nd() magic? That way the row/column information should be conserved, right? Also, we avoid transposing, memory use, ... Oh, and someone could conceivably have a _looong_ 1D file, but would want it read as a 2D array. Paul From cwg at falma.de Thu May 5 03:29:23 2011 From: cwg at falma.de (Christoph Groth) Date: Thu, 05 May 2011 09:29:23 +0200 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ References: <87aaf27gdj.fsf@falma.de> Message-ID: <87wri561ws.fsf@falma.de> > On Wed, May 4, 2011 at 6:19 AM, Christoph Groth wrote: >> >> Dear numpy experts, >> >> I have noticed that with Numpy 1.5.1 the operation >> >> m[::2] += 1.0 >> >> takes twice as long as >> >> t = m[::2] >> t += 1.0 Mark Wiebe writes: > You'd better time this in 1.6 too. ;) > > https://github.com/numpy/numpy/commit/f60797ba64ccf33597225d23b893b6eb11149860 This seems to be exactly what I had in mind. Thanks for finding this. > The case of boolean mask indexing can't benefit so easily from this > optimization, but I think could see a big performance benefit if > combined __index__ + __i__ operators were added to > Python. Something to consider, anyway. Has something like __index_iadd__ ever been considered seriously? Not to my (limited) knowledge. Indeed, the second loop executes twice as fast than the first in the following example (again with Numpy 1.5.1). import numpy m = numpy.zeros((1000, 1000)) mask = numpy.arange(0, 1000, 2, dtype=int) for i in xrange(40): m[mask] += 1.0 for i in xrange(40): t = m[mask] t += 1.0 But wouldn't it be easy to optimize this as well, by not executing assignments where the source and the destination is indexed by the same mask object? This would be a bit weaker, as it would work only for same ("a is b"), and not for equal masks, but it should still cover the most common case. Christoph From gael.varoquaux at normalesup.org Thu May 5 04:02:42 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 5 May 2011 10:02:42 +0200 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ In-Reply-To: References: <87aaf27gdj.fsf@falma.de> Message-ID: <20110505080242.GA15656@phare.normalesup.org> On Wed, May 04, 2011 at 08:57:42AM -0700, Mark Wiebe wrote: > You'd better time this in 1.6 too. ;) > ?[2]https://github.com/numpy/numpy/commit/f60797ba64ccf33597225d23b893b6eb11149860 Awesome work! That's all I have to say. I find very enjoyable the generalized use of the time machine in the SciPy community to add features before they are requested: http://permalink.gmane.org/gmane.comp.python.ipython.devel/478 http://mail.scipy.org/pipermail/ipython-dev/2010-November/006954.html G From ben.root at ou.edu Thu May 5 11:49:19 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 May 2011 10:49:19 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 4. mai 2011, at 20.33, Benjamin Root wrote: > > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < > derek at astro.physik.uni-goettingen.de> wrote: > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written > for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it > will reintroduce the 'transposed' problem? > > > > Yes, good point, one could replace the > > X.shape = (X.size, ) with X = np.atleast_1d(X), > > but for the ndmin=2 case, we'd need to replace > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > > not sure which solution is more efficient in terms of memory access > etc... > > > > Cheers, > > Derek > > > > > > I can confirm that the current behavior is not sufficient for all of the > original corner cases that ndmin was supposed to address. Keep in mind that > np.loadtxt takes a one-column data file and a one-row data file down to the > same shape. I don't see how the current code is able to produce the correct > array shape when ndmin=2. Do we have some sort of counter in loadtxt for > counting the number of rows and columns read? Could we use those to help > guide the ndmin=2 case? > > > > I think that using atleast_1d(X) might be a bit overkill, but it would be > very clear as to the code's intent. I don't think we have to worry about > memory usage if we limit its use to only situations where ndmin is greater > than the number of dimensions of the array. In those cases, the array is > either an empty result, a scalar value (in which memory access is trivial), > or 1-d (in which a transpose is cheap). > > What if one does things the other way around - avoid calling squeeze until > _after_ doing the atleast_Nd() magic? That way the row/column information > should be conserved, right? Also, we avoid transposing, memory use, ... > > Oh, and someone could conceivably have a _looong_ 1D file, but would want > it read as a 2D array. > > Paul > > > @Derek, good catch with noticing the error in the tests. We do still need to handle the case I mentioned, however. I have attached an example script to demonstrate the issue. In this script, I would expect the second-to-last array to be a shape of (1, 5). I believe that the single-row, multi-column case would actually be the more common type of edge-case encountered by users than the others. Therefore, I believe that this ndmin fix is not adequate until this is addressed. @Paul, we can't call squeeze after doing the atleast_Nd() magic. That would just undo whatever we had just done. Also, wrt the transpose, a (1, 100000) array looks the same in memory as a (100000, 1) array, right? Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: loadtest.py Type: application/octet-stream Size: 734 bytes Desc: not available URL: From ben.root at ou.edu Thu May 5 11:52:11 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 May 2011 10:52:11 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Thu, May 5, 2011 at 10:49 AM, Benjamin Root wrote: > > > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > >> >> On 4. mai 2011, at 20.33, Benjamin Root wrote: >> >> > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < >> derek at astro.physik.uni-goettingen.de> wrote: >> > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: >> > >> > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written >> for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it >> will reintroduce the 'transposed' problem? >> > >> > Yes, good point, one could replace the >> > X.shape = (X.size, ) with X = np.atleast_1d(X), >> > but for the ndmin=2 case, we'd need to replace >> > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - >> > not sure which solution is more efficient in terms of memory access >> etc... >> > >> > Cheers, >> > Derek >> > >> > >> > I can confirm that the current behavior is not sufficient for all of the >> original corner cases that ndmin was supposed to address. Keep in mind that >> np.loadtxt takes a one-column data file and a one-row data file down to the >> same shape. I don't see how the current code is able to produce the correct >> array shape when ndmin=2. Do we have some sort of counter in loadtxt for >> counting the number of rows and columns read? Could we use those to help >> guide the ndmin=2 case? >> > >> > I think that using atleast_1d(X) might be a bit overkill, but it would >> be very clear as to the code's intent. I don't think we have to worry about >> memory usage if we limit its use to only situations where ndmin is greater >> than the number of dimensions of the array. In those cases, the array is >> either an empty result, a scalar value (in which memory access is trivial), >> or 1-d (in which a transpose is cheap). >> >> What if one does things the other way around - avoid calling squeeze until >> _after_ doing the atleast_Nd() magic? That way the row/column information >> should be conserved, right? Also, we avoid transposing, memory use, ... >> >> Oh, and someone could conceivably have a _looong_ 1D file, but would want >> it read as a 2D array. >> >> Paul >> >> >> > @Derek, good catch with noticing the error in the tests. We do still need > to handle the case I mentioned, however. I have attached an example script > to demonstrate the issue. In this script, I would expect the second-to-last > array to be a shape of (1, 5). I believe that the single-row, multi-column > case would actually be the more common type of edge-case encountered by > users than the others. Therefore, I believe that this ndmin fix is not > adequate until this is addressed. > > Apologies Derek, your patch does address the issue I raised. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 5 12:54:40 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 5 May 2011 11:54:40 -0500 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ In-Reply-To: <87wri561ws.fsf@falma.de> References: <87aaf27gdj.fsf@falma.de> <87wri561ws.fsf@falma.de> Message-ID: On Thu, May 5, 2011 at 02:29, Christoph Groth wrote: >> On Wed, May 4, 2011 at 6:19 AM, Christoph Groth wrote: >>> >>> ? ? Dear numpy experts, >>> >>> ? ? I have noticed that with Numpy 1.5.1 the operation >>> >>> ? ? m[::2] += 1.0 >>> >>> ? ? takes twice as long as >>> >>> ? ? t = m[::2] >>> ? ? t += 1.0 > > Mark Wiebe writes: > >> You'd better time this in 1.6 too. ;) >> >> https://github.com/numpy/numpy/commit/f60797ba64ccf33597225d23b893b6eb11149860 > > This seems to be exactly what I had in mind. ?Thanks for finding this. > >> The case of boolean mask indexing can't benefit so easily from this >> optimization, but I think could see a big performance benefit if >> combined __index__ + __i__ operators were added to >> Python. Something to consider, anyway. > > Has something like __index_iadd__ ever been considered seriously? ?Not > to my (limited) knowledge. Only on this list, I think. :-) I don't think it will ever happen. Only numpy really cares about it, and adding another __special__ method for each __iop__ is a lot of additional methods that need to be supported. > Indeed, the second loop executes twice as fast than the first in the > following example (again with Numpy 1.5.1). > > import numpy > m = numpy.zeros((1000, 1000)) > mask = numpy.arange(0, 1000, 2, dtype=int) > > for i in xrange(40): > ? ?m[mask] += 1.0 > > for i in xrange(40): > ? ?t = m[mask] > ? ?t += 1.0 > > But wouldn't it be easy to optimize this as well, by not executing > assignments where the source and the destination is indexed by the same > mask object? No. These two are not semantically equivalent. Your second example does not actually modify m. For integer and bool mask arrays, m[mask] necessarily makes a copy, so when you modify t via inplace addition, you have only modified t and not m. The assignment back to m[mask] is necessary. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From njs at pobox.com Thu May 5 13:53:10 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 May 2011 10:53:10 -0700 Subject: [Numpy-discussion] optimizing ndarray.__setitem__ In-Reply-To: References: <87aaf27gdj.fsf@falma.de> <87wri561ws.fsf@falma.de> Message-ID: On Thu, May 5, 2011 at 9:54 AM, Robert Kern wrote: > On Thu, May 5, 2011 at 02:29, Christoph Groth wrote: >> Has something like __index_iadd__ ever been considered seriously? ?Not >> to my (limited) knowledge. > > Only on this list, I think. :-) > > I don't think it will ever happen. Only numpy really cares about it, > and adding another __special__ method for each __iop__ is a lot of > additional methods that need to be supported. Maybe in the context of PyPy someone will come up with a clever way to implement template-expression style operator fusion for numpy. That'd be kinda neat. -- Nathaniel From paul.anton.letnes at gmail.com Thu May 5 14:08:30 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Thu, 5 May 2011 11:08:30 -0700 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On 5. mai 2011, at 08.49, Benjamin Root wrote: > > > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes wrote: > > On 4. mai 2011, at 20.33, Benjamin Root wrote: > > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier wrote: > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions written for this? Shouldn't we reuse them? Perhaps it's overkill, and perhaps it will reintroduce the 'transposed' problem? > > > > Yes, good point, one could replace the > > X.shape = (X.size, ) with X = np.atleast_1d(X), > > but for the ndmin=2 case, we'd need to replace > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > > not sure which solution is more efficient in terms of memory access etc... > > > > Cheers, > > Derek > > > > > > I can confirm that the current behavior is not sufficient for all of the original corner cases that ndmin was supposed to address. Keep in mind that np.loadtxt takes a one-column data file and a one-row data file down to the same shape. I don't see how the current code is able to produce the correct array shape when ndmin=2. Do we have some sort of counter in loadtxt for counting the number of rows and columns read? Could we use those to help guide the ndmin=2 case? > > > > I think that using atleast_1d(X) might be a bit overkill, but it would be very clear as to the code's intent. I don't think we have to worry about memory usage if we limit its use to only situations where ndmin is greater than the number of dimensions of the array. In those cases, the array is either an empty result, a scalar value (in which memory access is trivial), or 1-d (in which a transpose is cheap). > > What if one does things the other way around - avoid calling squeeze until _after_ doing the atleast_Nd() magic? That way the row/column information should be conserved, right? Also, we avoid transposing, memory use, ... > > Oh, and someone could conceivably have a _looong_ 1D file, but would want it read as a 2D array. > > Paul > > > > @Derek, good catch with noticing the error in the tests. We do still need to handle the case I mentioned, however. I have attached an example script to demonstrate the issue. In this script, I would expect the second-to-last array to be a shape of (1, 5). I believe that the single-row, multi-column case would actually be the more common type of edge-case encountered by users than the others. Therefore, I believe that this ndmin fix is not adequate until this is addressed. > > @Paul, we can't call squeeze after doing the atleast_Nd() magic. That would just undo whatever we had just done. Also, wrt the transpose, a (1, 100000) array looks the same in memory as a (100000, 1) array, right? Agree. I thought more along the lines of (pseudocode-ish) if ndmin == 0: squeeze() if ndmin == 1: atleast_1D() elif ndmin == 2: atleast_2D() else: I don't rightly know what would go here, maybe raise ValueError? That would avoid the squeeze call before the atleast_Nd magic. But the code was changed, so I think my comment doesn't make sense anymore. It's probably fine the way it is! Paul From ralf.gommers at googlemail.com Thu May 5 14:38:42 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 May 2011 20:38:42 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Thu, May 5, 2011 at 1:10 AM, Benjamin Root wrote: > > > On Tue, May 3, 2011 at 1:18 PM, Ralf Gommers wrote: > >> Hi, >> >> I am pleased to announce the availability of the second release >> candidate of NumPy 1.6.0. >> >> Compared to the first release candidate, one segfault on (32-bit >> Windows + MSVC) and several memory leaks were fixed. If no new >> problems are reported, the final release will be in one week. >> >> Sources and binaries can be found at >> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ >> For (preliminary) release notes see below. >> >> Enjoy, >> Ralf >> >> >> > Minor issue I just noticed on my recently installed Ubuntu 11.04 machine. > The setup script is making a call to 'svnversion'. Doesn't impact the build > or anything, but I only noticed it because svn hasn't been installed yet on > that machine. Don't know if it is something that ought to be cleaned up or > not. > > That's harmless and better left alone for backwards compatibility I think. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu May 5 15:18:34 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 May 2011 14:18:34 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Thu, May 5, 2011 at 1:08 PM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 5. mai 2011, at 08.49, Benjamin Root wrote: > > > > > > > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > > > > On 4. mai 2011, at 20.33, Benjamin Root wrote: > > > > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < > derek at astro.physik.uni-goettingen.de> wrote: > > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > > > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions > written for this? Shouldn't we reuse them? Perhaps it's overkill, and > perhaps it will reintroduce the 'transposed' problem? > > > > > > Yes, good point, one could replace the > > > X.shape = (X.size, ) with X = np.atleast_1d(X), > > > but for the ndmin=2 case, we'd need to replace > > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > > > not sure which solution is more efficient in terms of memory access > etc... > > > > > > Cheers, > > > Derek > > > > > > > > > I can confirm that the current behavior is not sufficient for all of > the original corner cases that ndmin was supposed to address. Keep in mind > that np.loadtxt takes a one-column data file and a one-row data file down to > the same shape. I don't see how the current code is able to produce the > correct array shape when ndmin=2. Do we have some sort of counter in > loadtxt for counting the number of rows and columns read? Could we use > those to help guide the ndmin=2 case? > > > > > > I think that using atleast_1d(X) might be a bit overkill, but it would > be very clear as to the code's intent. I don't think we have to worry about > memory usage if we limit its use to only situations where ndmin is greater > than the number of dimensions of the array. In those cases, the array is > either an empty result, a scalar value (in which memory access is trivial), > or 1-d (in which a transpose is cheap). > > > > What if one does things the other way around - avoid calling squeeze > until _after_ doing the atleast_Nd() magic? That way the row/column > information should be conserved, right? Also, we avoid transposing, memory > use, ... > > > > Oh, and someone could conceivably have a _looong_ 1D file, but would want > it read as a 2D array. > > > > Paul > > > > > > > > @Derek, good catch with noticing the error in the tests. We do still need > to handle the case I mentioned, however. I have attached an example script > to demonstrate the issue. In this script, I would expect the second-to-last > array to be a shape of (1, 5). I believe that the single-row, multi-column > case would actually be the more common type of edge-case encountered by > users than the others. Therefore, I believe that this ndmin fix is not > adequate until this is addressed. > > > > @Paul, we can't call squeeze after doing the atleast_Nd() magic. That > would just undo whatever we had just done. Also, wrt the transpose, a (1, > 100000) array looks the same in memory as a (100000, 1) array, right? > Agree. I thought more along the lines of (pseudocode-ish) > if ndmin == 0: > squeeze() > if ndmin == 1: > atleast_1D() > elif ndmin == 2: > atleast_2D() > else: > I don't rightly know what would go here, maybe raise ValueError? > > That would avoid the squeeze call before the atleast_Nd magic. But the code > was changed, so I think my comment doesn't make sense anymore. It's probably > fine the way it is! > > Paul > > I have thought of that too, but the problem with that approach is that after reading the file, X will have 2 or 3 dimensions, regardless of how many singleton dims were in the file. A squeeze will always be needed. Also, the purpose of squeeze is opposite that of the atleast_*d() functions: squeeze reduces dimensions, while atleast_*d will add dimensions. Therefore, I re-iterate... the patch by Derek gets the job done. I have tested it for a wide variety of inputs for both regular arrays and record arrays. Is there room for improvements? Yes, but I think that can wait for later. Derek's patch however fixes an important bug in the ndmin implementation and should be included for the release. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu May 5 15:33:04 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 May 2011 21:33:04 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Thu, May 5, 2011 at 9:18 PM, Benjamin Root wrote: > > > On Thu, May 5, 2011 at 1:08 PM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > >> >> On 5. mai 2011, at 08.49, Benjamin Root wrote: >> >> > >> > >> > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < >> paul.anton.letnes at gmail.com> wrote: >> > >> > On 4. mai 2011, at 20.33, Benjamin Root wrote: >> > >> > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < >> derek at astro.physik.uni-goettingen.de> wrote: >> > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: >> > > >> > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions >> written for this? Shouldn't we reuse them? Perhaps it's overkill, and >> perhaps it will reintroduce the 'transposed' problem? >> > > >> > > Yes, good point, one could replace the >> > > X.shape = (X.size, ) with X = np.atleast_1d(X), >> > > but for the ndmin=2 case, we'd need to replace >> > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - >> > > not sure which solution is more efficient in terms of memory access >> etc... >> > > >> > > Cheers, >> > > Derek >> > > >> > > >> > > I can confirm that the current behavior is not sufficient for all of >> the original corner cases that ndmin was supposed to address. Keep in mind >> that np.loadtxt takes a one-column data file and a one-row data file down to >> the same shape. I don't see how the current code is able to produce the >> correct array shape when ndmin=2. Do we have some sort of counter in >> loadtxt for counting the number of rows and columns read? Could we use >> those to help guide the ndmin=2 case? >> > > >> > > I think that using atleast_1d(X) might be a bit overkill, but it would >> be very clear as to the code's intent. I don't think we have to worry about >> memory usage if we limit its use to only situations where ndmin is greater >> than the number of dimensions of the array. In those cases, the array is >> either an empty result, a scalar value (in which memory access is trivial), >> or 1-d (in which a transpose is cheap). >> > >> > What if one does things the other way around - avoid calling squeeze >> until _after_ doing the atleast_Nd() magic? That way the row/column >> information should be conserved, right? Also, we avoid transposing, memory >> use, ... >> > >> > Oh, and someone could conceivably have a _looong_ 1D file, but would >> want it read as a 2D array. >> > >> > Paul >> > >> > >> > >> > @Derek, good catch with noticing the error in the tests. We do still >> need to handle the case I mentioned, however. I have attached an example >> script to demonstrate the issue. In this script, I would expect the >> second-to-last array to be a shape of (1, 5). I believe that the >> single-row, multi-column case would actually be the more common type of >> edge-case encountered by users than the others. Therefore, I believe that >> this ndmin fix is not adequate until this is addressed. >> > >> > @Paul, we can't call squeeze after doing the atleast_Nd() magic. That >> would just undo whatever we had just done. Also, wrt the transpose, a (1, >> 100000) array looks the same in memory as a (100000, 1) array, right? >> Agree. I thought more along the lines of (pseudocode-ish) >> if ndmin == 0: >> squeeze() >> if ndmin == 1: >> atleast_1D() >> elif ndmin == 2: >> atleast_2D() >> else: >> I don't rightly know what would go here, maybe raise ValueError? >> >> That would avoid the squeeze call before the atleast_Nd magic. But the >> code was changed, so I think my comment doesn't make sense anymore. It's >> probably fine the way it is! >> >> Paul >> >> > I have thought of that too, but the problem with that approach is that > after reading the file, X will have 2 or 3 dimensions, regardless of how > many singleton dims were in the file. A squeeze will always be needed. > Also, the purpose of squeeze is opposite that of the atleast_*d() > functions: squeeze reduces dimensions, while atleast_*d will add > dimensions. > > Therefore, I re-iterate... the patch by Derek gets the job done. I have > tested it for a wide variety of inputs for both regular arrays and record > arrays. Is there room for improvements? Yes, but I think that can wait for > later. Derek's patch however fixes an important bug in the ndmin > implementation and should be included for the release. > > Two questions: can you point me to the patch/ticket, and is this a regression? Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Thu May 5 15:46:17 2011 From: ben.root at ou.edu (Benjamin Root) Date: Thu, 5 May 2011 14:46:17 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Thu, May 5, 2011 at 2:33 PM, Ralf Gommers wrote: > > > On Thu, May 5, 2011 at 9:18 PM, Benjamin Root wrote: > >> >> >> On Thu, May 5, 2011 at 1:08 PM, Paul Anton Letnes < >> paul.anton.letnes at gmail.com> wrote: >> >>> >>> On 5. mai 2011, at 08.49, Benjamin Root wrote: >>> >>> > >>> > >>> > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < >>> paul.anton.letnes at gmail.com> wrote: >>> > >>> > On 4. mai 2011, at 20.33, Benjamin Root wrote: >>> > >>> > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < >>> derek at astro.physik.uni-goettingen.de> wrote: >>> > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: >>> > > >>> > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions >>> written for this? Shouldn't we reuse them? Perhaps it's overkill, and >>> perhaps it will reintroduce the 'transposed' problem? >>> > > >>> > > Yes, good point, one could replace the >>> > > X.shape = (X.size, ) with X = np.atleast_1d(X), >>> > > but for the ndmin=2 case, we'd need to replace >>> > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - >>> > > not sure which solution is more efficient in terms of memory access >>> etc... >>> > > >>> > > Cheers, >>> > > Derek >>> > > >>> > > >>> > > I can confirm that the current behavior is not sufficient for all of >>> the original corner cases that ndmin was supposed to address. Keep in mind >>> that np.loadtxt takes a one-column data file and a one-row data file down to >>> the same shape. I don't see how the current code is able to produce the >>> correct array shape when ndmin=2. Do we have some sort of counter in >>> loadtxt for counting the number of rows and columns read? Could we use >>> those to help guide the ndmin=2 case? >>> > > >>> > > I think that using atleast_1d(X) might be a bit overkill, but it >>> would be very clear as to the code's intent. I don't think we have to worry >>> about memory usage if we limit its use to only situations where ndmin is >>> greater than the number of dimensions of the array. In those cases, the >>> array is either an empty result, a scalar value (in which memory access is >>> trivial), or 1-d (in which a transpose is cheap). >>> > >>> > What if one does things the other way around - avoid calling squeeze >>> until _after_ doing the atleast_Nd() magic? That way the row/column >>> information should be conserved, right? Also, we avoid transposing, memory >>> use, ... >>> > >>> > Oh, and someone could conceivably have a _looong_ 1D file, but would >>> want it read as a 2D array. >>> > >>> > Paul >>> > >>> > >>> > >>> > @Derek, good catch with noticing the error in the tests. We do still >>> need to handle the case I mentioned, however. I have attached an example >>> script to demonstrate the issue. In this script, I would expect the >>> second-to-last array to be a shape of (1, 5). I believe that the >>> single-row, multi-column case would actually be the more common type of >>> edge-case encountered by users than the others. Therefore, I believe that >>> this ndmin fix is not adequate until this is addressed. >>> > >>> > @Paul, we can't call squeeze after doing the atleast_Nd() magic. That >>> would just undo whatever we had just done. Also, wrt the transpose, a (1, >>> 100000) array looks the same in memory as a (100000, 1) array, right? >>> Agree. I thought more along the lines of (pseudocode-ish) >>> if ndmin == 0: >>> squeeze() >>> if ndmin == 1: >>> atleast_1D() >>> elif ndmin == 2: >>> atleast_2D() >>> else: >>> I don't rightly know what would go here, maybe raise ValueError? >>> >>> That would avoid the squeeze call before the atleast_Nd magic. But the >>> code was changed, so I think my comment doesn't make sense anymore. It's >>> probably fine the way it is! >>> >>> Paul >>> >>> >> I have thought of that too, but the problem with that approach is that >> after reading the file, X will have 2 or 3 dimensions, regardless of how >> many singleton dims were in the file. A squeeze will always be needed. >> Also, the purpose of squeeze is opposite that of the atleast_*d() >> functions: squeeze reduces dimensions, while atleast_*d will add >> dimensions. >> >> Therefore, I re-iterate... the patch by Derek gets the job done. I have >> tested it for a wide variety of inputs for both regular arrays and record >> arrays. Is there room for improvements? Yes, but I think that can wait for >> later. Derek's patch however fixes an important bug in the ndmin >> implementation and should be included for the release. >> >> Two questions: can you point me to the patch/ticket, and is this a > regression? > > Thanks, > Ralf > > > I don't know if he did a pull-request or not, but here is the link he provided earlier in the thread. https://github.com/dhomeier/numpy/compare/master...ndmin-cols Technically, this is not a "regression" as the ndmin feature is new in this release. However, the problem that ndmin is supposed to address is not fixed by the current implementation for the rc. Essentially, a single-row, multi-column file with ndmin=2 comes out as a Nx1 array which is the same result for a multi-row, single-column file. My feeling is that if we let the current implementation stand as is, and developers use it in their code, then fixing it in a later release would introduce more problems (maybe the devels would transpose the result themselves or something). Better to fix it now in rc with the two lines of code (and the correction to the tests), then to introduce a buggy feature that will be hard to fix in future releases, IMHO. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu May 5 16:31:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 5 May 2011 22:31:25 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Thu, May 5, 2011 at 9:46 PM, Benjamin Root wrote: > > > On Thu, May 5, 2011 at 2:33 PM, Ralf Gommers wrote: > >> >> >> On Thu, May 5, 2011 at 9:18 PM, Benjamin Root wrote: >> >>> >>> >>> On Thu, May 5, 2011 at 1:08 PM, Paul Anton Letnes < >>> paul.anton.letnes at gmail.com> wrote: >>> >>>> >>>> On 5. mai 2011, at 08.49, Benjamin Root wrote: >>>> >>>> > >>>> > >>>> > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes < >>>> paul.anton.letnes at gmail.com> wrote: >>>> > >>>> > On 4. mai 2011, at 20.33, Benjamin Root wrote: >>>> > >>>> > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier < >>>> derek at astro.physik.uni-goettingen.de> wrote: >>>> > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: >>>> > > >>>> > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions >>>> written for this? Shouldn't we reuse them? Perhaps it's overkill, and >>>> perhaps it will reintroduce the 'transposed' problem? >>>> > > >>>> > > Yes, good point, one could replace the >>>> > > X.shape = (X.size, ) with X = np.atleast_1d(X), >>>> > > but for the ndmin=2 case, we'd need to replace >>>> > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - >>>> > > not sure which solution is more efficient in terms of memory access >>>> etc... >>>> > > >>>> > > Cheers, >>>> > > Derek >>>> > > >>>> > > >>>> > > I can confirm that the current behavior is not sufficient for all of >>>> the original corner cases that ndmin was supposed to address. Keep in mind >>>> that np.loadtxt takes a one-column data file and a one-row data file down to >>>> the same shape. I don't see how the current code is able to produce the >>>> correct array shape when ndmin=2. Do we have some sort of counter in >>>> loadtxt for counting the number of rows and columns read? Could we use >>>> those to help guide the ndmin=2 case? >>>> > > >>>> > > I think that using atleast_1d(X) might be a bit overkill, but it >>>> would be very clear as to the code's intent. I don't think we have to worry >>>> about memory usage if we limit its use to only situations where ndmin is >>>> greater than the number of dimensions of the array. In those cases, the >>>> array is either an empty result, a scalar value (in which memory access is >>>> trivial), or 1-d (in which a transpose is cheap). >>>> > >>>> > What if one does things the other way around - avoid calling squeeze >>>> until _after_ doing the atleast_Nd() magic? That way the row/column >>>> information should be conserved, right? Also, we avoid transposing, memory >>>> use, ... >>>> > >>>> > Oh, and someone could conceivably have a _looong_ 1D file, but would >>>> want it read as a 2D array. >>>> > >>>> > Paul >>>> > >>>> > >>>> > >>>> > @Derek, good catch with noticing the error in the tests. We do still >>>> need to handle the case I mentioned, however. I have attached an example >>>> script to demonstrate the issue. In this script, I would expect the >>>> second-to-last array to be a shape of (1, 5). I believe that the >>>> single-row, multi-column case would actually be the more common type of >>>> edge-case encountered by users than the others. Therefore, I believe that >>>> this ndmin fix is not adequate until this is addressed. >>>> > >>>> > @Paul, we can't call squeeze after doing the atleast_Nd() magic. That >>>> would just undo whatever we had just done. Also, wrt the transpose, a (1, >>>> 100000) array looks the same in memory as a (100000, 1) array, right? >>>> Agree. I thought more along the lines of (pseudocode-ish) >>>> if ndmin == 0: >>>> squeeze() >>>> if ndmin == 1: >>>> atleast_1D() >>>> elif ndmin == 2: >>>> atleast_2D() >>>> else: >>>> I don't rightly know what would go here, maybe raise ValueError? >>>> >>>> That would avoid the squeeze call before the atleast_Nd magic. But the >>>> code was changed, so I think my comment doesn't make sense anymore. It's >>>> probably fine the way it is! >>>> >>>> Paul >>>> >>>> >>> I have thought of that too, but the problem with that approach is that >>> after reading the file, X will have 2 or 3 dimensions, regardless of how >>> many singleton dims were in the file. A squeeze will always be needed. >>> Also, the purpose of squeeze is opposite that of the atleast_*d() >>> functions: squeeze reduces dimensions, while atleast_*d will add >>> dimensions. >>> >>> Therefore, I re-iterate... the patch by Derek gets the job done. I have >>> tested it for a wide variety of inputs for both regular arrays and record >>> arrays. Is there room for improvements? Yes, but I think that can wait for >>> later. Derek's patch however fixes an important bug in the ndmin >>> implementation and should be included for the release. >>> >>> Two questions: can you point me to the patch/ticket, and is this a >> regression? >> >> Thanks, >> Ralf >> >> >> > I don't know if he did a pull-request or not, but here is the link he > provided earlier in the thread. > > https://github.com/dhomeier/numpy/compare/master...ndmin-cols > > Technically, this is not a "regression" as the ndmin feature is new in this > release. > Yes right, I forgot this was a recent change. > However, the problem that ndmin is supposed to address is not fixed by the > current implementation for the rc. Essentially, a single-row, multi-column > file with ndmin=2 comes out as a Nx1 array which is the same result for a > multi-row, single-column file. My feeling is that if we let the current > implementation stand as is, and developers use it in their code, then fixing > it in a later release would introduce more problems (maybe the devels would > transpose the result themselves or something). Better to fix it now in rc > with the two lines of code (and the correction to the tests), then to > introduce a buggy feature that will be hard to fix in future releases, IMHO. > > Looks okay, and I agree that it's better to fix it now. The timing is a bit unfortunate though, just after RC2. I'll have closer look tomorrow and if it can go in, probably tag RC3. If in the meantime a few more people could test this, that would be helpful. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Thu May 5 16:53:04 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 5 May 2011 22:53:04 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On 5 May 2011, at 22:31, Ralf Gommers wrote: > > On Thu, May 5, 2011 at 9:46 PM, Benjamin Root wrote: > > > On Thu, May 5, 2011 at 2:33 PM, Ralf Gommers > wrote: > > > On Thu, May 5, 2011 at 9:18 PM, Benjamin Root wrote: > > > On Thu, May 5, 2011 at 1:08 PM, Paul Anton Letnes > wrote: > > On 5. mai 2011, at 08.49, Benjamin Root wrote: > > > > > > > On Wed, May 4, 2011 at 11:08 PM, Paul Anton Letnes > wrote: > > > > On 4. mai 2011, at 20.33, Benjamin Root wrote: > > > > > On Wed, May 4, 2011 at 7:54 PM, Derek Homeier > wrote: > > > On 05.05.2011, at 2:40AM, Paul Anton Letnes wrote: > > > > > > > But: Isn't the numpy.atleast_2d and numpy.atleast_1d functions > written for this? Shouldn't we reuse them? Perhaps it's overkill, > and perhaps it will reintroduce the 'transposed' problem? > > > > > > Yes, good point, one could replace the > > > X.shape = (X.size, ) with X = np.atleast_1d(X), > > > but for the ndmin=2 case, we'd need to replace > > > X.shape = (X.size, 1) with X = np.atleast_2d(X).T - > > > not sure which solution is more efficient in terms of memory > access etc... > > > > > > Cheers, > > > Derek > > > > > > > > > I can confirm that the current behavior is not sufficient for > all of the original corner cases that ndmin was supposed to > address. Keep in mind that np.loadtxt takes a one-column data file > and a one-row data file down to the same shape. I don't see how the > current code is able to produce the correct array shape when > ndmin=2. Do we have some sort of counter in loadtxt for counting > the number of rows and columns read? Could we use those to help > guide the ndmin=2 case? > > > > > > I think that using atleast_1d(X) might be a bit overkill, but it > would be very clear as to the code's intent. I don't think we have > to worry about memory usage if we limit its use to only situations > where ndmin is greater than the number of dimensions of the array. > In those cases, the array is either an empty result, a scalar value > (in which memory access is trivial), or 1-d (in which a transpose is > cheap). > > > > What if one does things the other way around - avoid calling > squeeze until _after_ doing the atleast_Nd() magic? That way the row/ > column information should be conserved, right? Also, we avoid > transposing, memory use, ... > > > > Oh, and someone could conceivably have a _looong_ 1D file, but > would want it read as a 2D array. > > > > Paul > > > > > > > > @Derek, good catch with noticing the error in the tests. We do > still need to handle the case I mentioned, however. I have attached > an example script to demonstrate the issue. In this script, I would > expect the second-to-last array to be a shape of (1, 5). I believe > that the single-row, multi-column case would actually be the more > common type of edge-case encountered by users than the others. > Therefore, I believe that this ndmin fix is not adequate until this > is addressed. > > > > @Paul, we can't call squeeze after doing the atleast_Nd() magic. > That would just undo whatever we had just done. Also, wrt the > transpose, a (1, 100000) array looks the same in memory as a > (100000, 1) array, right? > Agree. I thought more along the lines of (pseudocode-ish) > if ndmin == 0: > squeeze() > if ndmin == 1: > atleast_1D() > elif ndmin == 2: > atleast_2D() > else: > I don't rightly know what would go here, maybe raise > ValueError? > > That would avoid the squeeze call before the atleast_Nd magic. But > the code was changed, so I think my comment doesn't make sense > anymore. It's probably fine the way it is! > > Paul > > > I have thought of that too, but the problem with that approach is > that after reading the file, X will have 2 or 3 dimensions, > regardless of how many singleton dims were in the file. A squeeze > will always be needed. Also, the purpose of squeeze is opposite > that of the atleast_*d() functions: squeeze reduces dimensions, > while atleast_*d will add dimensions. > > Therefore, I re-iterate... the patch by Derek gets the job done. I > have tested it for a wide variety of inputs for both regular arrays > and record arrays. Is there room for improvements? Yes, but I > think that can wait for later. Derek's patch however fixes an > important bug in the ndmin implementation and should be included for > the release. > Thanks for the additional tests and discussion, Paul and Ben! Yes, I think the main issue is that multi-column input first comes back as 3- dimensional, with a "singleton" dimension in front, so I was getting rid of that first - maybe could add a comment on that. Single-column, multi-row apparently already come back as 1-dimensional, so we only need the expansion there. > Two questions: can you point me to the patch/ticket, and is this a > regression? > > Thanks, > Ralf > This was ticket #1562, had been closed as "resolved" in the meantime, though. > > I don't know if he did a pull-request or not, but here is the link > he provided earlier in the thread. > > https://github.com/dhomeier/numpy/compare/master...ndmin-cols > > Technically, this is not a "regression" as the ndmin feature is new > in this release. > > Yes right, I forgot this was a recent change. > > However, the problem that ndmin is supposed to address is not fixed > by the current implementation for the rc. Essentially, a single- > row, multi-column file with ndmin=2 comes out as a Nx1 array which > is the same result for a multi-row, single-column file. My feeling > is that if we let the current implementation stand as is, and > developers use it in their code, then fixing it in a later release > would introduce more problems (maybe the devels would transpose the > result themselves or something). Better to fix it now in rc with > the two lines of code (and the correction to the tests), then to > introduce a buggy feature that will be hard to fix in future > releases, IMHO. > > Looks okay, and I agree that it's better to fix it now. The timing > is a bit unfortunate though, just after RC2. I'll have closer look > tomorrow and if it can go in, probably tag RC3. > > If in the meantime a few more people could test this, that would be > helpful. > > Ralf I agree, wish I had time to push this before rc2. I could add the explanatory comments mentioned above and switch to use the atleast_[12]d() solution, test that and push it in a couple of minutes, or should I better leave it as is now for testing? Cheers, Derek From djl at lanl.gov Thu May 5 16:51:06 2011 From: djl at lanl.gov (DJ Luscher) Date: Thu, 5 May 2011 20:51:06 +0000 (UTC) Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 References: Message-ID: Ralf Gommers googlemail.com> writes: > > Hi, > > I am pleased to announce the availability of the second release > candidate of NumPy 1.6.0. > > Compared to the first release candidate, one segfault on (32-bit > Windows + MSVC) and several memory leaks were fixed. If no new > problems are reported, the final release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > as ``shape(array, dim-1)`` which makes it possible to automatically > wrap Fortran routines that use two argument ``size`` function in > dimension specifications. Before users were forced to apply this > mapping manually. > Regarding the f2py support for assumed shape arrays: I'm just struggling along trying to learn how to use f2py to interface with fortran source, so please be patient if I am missing something obvious. That said, in test cases I've run the new f2py assumed-shape-array support in Numpy 1.6.0.rc2 seems to conflict with the support for f90-style modules. For example: ! -*- fix -*- module easy real, parameter :: anx(4) = (/1.,2.,3.,4./) contains subroutine sum(x, res) implicit none real, intent(in) :: x(:) real, intent(out) :: res integer :: i !print *, "sum: size(x) = ", size(x) res = 0.0 do i = 1, size(x) res = res + x(i) enddo end subroutine sum end module easy when compiled with: f2py -c --fcompiler=intelem foo_mod.f90 -m e then: python import e print e.easy.sum(e.easy.anx) returns: 0.0 Also (and I believe related) f2py can no longer compile source with assumed shape array valued functions within a module. Even though the python wrapped code did not function properly when called from python, it did work when called from other fortran code. It seems that the interface has been broken. The previous version of Numpy I was using was 1.3.0 all on Ubuntu 10.04, Python 2.6, and using Intel fortran compiler. thanks for your consideration and feedback. DJ From derek at astro.physik.uni-goettingen.de Thu May 5 18:12:41 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 6 May 2011 00:12:41 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On 5 May 2011, at 22:53, Derek Homeier wrote: >> >> However, the problem that ndmin is supposed to address is not fixed >> by the current implementation for the rc. Essentially, a single- >> row, multi-column file with ndmin=2 comes out as a Nx1 array which >> is the same result for a multi-row, single-column file. My feeling >> is that if we let the current implementation stand as is, and >> developers use it in their code, then fixing it in a later release >> would introduce more problems (maybe the devels would transpose the >> result themselves or something). Better to fix it now in rc with >> the two lines of code (and the correction to the tests), then to >> introduce a buggy feature that will be hard to fix in future >> releases, IMHO. >> >> Looks okay, and I agree that it's better to fix it now. The timing >> is a bit unfortunate though, just after RC2. I'll have closer look >> tomorrow and if it can go in, probably tag RC3. >> >> If in the meantime a few more people could test this, that would be >> helpful. >> >> Ralf > > I agree, wish I had time to push this before rc2. I could add the > explanatory comments > mentioned above and switch to use the atleast_[12]d() solution, test > that and push it > in a couple of minutes, or should I better leave it as is now for > testing? Quick follow-up: I just applied the above changes, added some tests to cover Ben's test cases and tested this with 1.6.0rc2 on OS X 10.5 i386+ppc + 10.6 x86_64 (Python2.7+3.2). So I'd be ready to push it to my repo and do my (first) pull request... Cheers, Derek From pearu.peterson at gmail.com Thu May 5 18:40:25 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 6 May 2011 01:40:25 +0300 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Thu, May 5, 2011 at 11:51 PM, DJ Luscher wrote: > > Ralf Gommers googlemail.com> writes: > > > > > Hi, > > > > I am pleased to announce the availability of the second release > > candidate of NumPy 1.6.0. > > > > Compared to the first release candidate, one segfault on (32-bit > > Windows + MSVC) and several memory leaks were fixed. If no new > > problems are reported, the final release will be in one week. > > > > Sources and binaries can be found at > > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc2/ > > For (preliminary) release notes see below. > > > > Enjoy, > > Ralf > > > > ========================= > > NumPy 1.6.0 Release Notes > > ========================= > > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > > ----------------------------------------------------------------------- > > > > F2py now supports wrapping Fortran 90 routines that use assumed shape > > arrays. Before such routines could be called from Python but the > > corresponding Fortran routines received assumed shape arrays as zero > > length arrays which caused unpredicted results. Thanks to Lorenz > > H?depohl for pointing out the correct way to interface routines with > > assumed shape arrays. > > > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > > as ``shape(array, dim-1)`` which makes it possible to automatically > > wrap Fortran routines that use two argument ``size`` function in > > dimension specifications. Before users were forced to apply this > > mapping manually. > > > > > Regarding the f2py support for assumed shape arrays: > > I'm just struggling along trying to learn how to use f2py to interface with > fortran source, so please be patient if I am missing something obvious. > That > said, in test cases I've run the new f2py assumed-shape-array support in > Numpy > 1.6.0.rc2 seems to conflict with the support for f90-style modules. For > example: > > > > ! -*- fix -*- > > module easy > > real, parameter :: anx(4) = (/1.,2.,3.,4./) > > contains > > subroutine sum(x, res) > implicit none > real, intent(in) :: x(:) > real, intent(out) :: res > > integer :: i > > !print *, "sum: size(x) = ", size(x) > > res = 0.0 > > do i = 1, size(x) > res = res + x(i) > enddo > > end subroutine sum > > end module easy > > > when compiled with: > f2py -c --fcompiler=intelem foo_mod.f90 -m e > > then: > > python > import e > print e.easy.sum(e.easy.anx) > > returns: 0.0 > > Also (and I believe related) f2py can no longer compile source with assumed > shape array valued functions within a module. Even though the python > wrapped > code did not function properly when called from python, it did work when > called > from other fortran code. It seems that the interface has been broken. The > previous version of Numpy I was using was 1.3.0 all on Ubuntu 10.04, Python > 2.6, > and using Intel fortran compiler. > > thanks for your consideration and feedback. > > Thanks for the bug report! These issues are now fixed in: https://github.com/numpy/numpy/commit/f393b604 Ralf, feel free to apply this changeset to 1.6.x branch if appropriate. Regards, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu May 5 20:41:35 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 May 2011 17:41:35 -0700 Subject: [Numpy-discussion] FYI -- numpy.linalg.lstsq as distributed by Ubuntu is crashing on some inputs Message-ID: Probably just another standard "your BLAS is compiled wrong!" bug, but in this case I'm seeing it with the stock versions of ATLAS, numpy, etc. included in the latest Ubuntu release (11.04 Natty Narwhal): https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/778217 So I thought people might like a heads up in case they run into it as well. -- Nathaniel From ralf.gommers at googlemail.com Fri May 6 01:53:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 6 May 2011 07:53:25 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: On Fri, May 6, 2011 at 12:12 AM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > > On 5 May 2011, at 22:53, Derek Homeier wrote: > > >> > >> However, the problem that ndmin is supposed to address is not fixed > >> by the current implementation for the rc. Essentially, a single- > >> row, multi-column file with ndmin=2 comes out as a Nx1 array which > >> is the same result for a multi-row, single-column file. My feeling > >> is that if we let the current implementation stand as is, and > >> developers use it in their code, then fixing it in a later release > >> would introduce more problems (maybe the devels would transpose the > >> result themselves or something). Better to fix it now in rc with > >> the two lines of code (and the correction to the tests), then to > >> introduce a buggy feature that will be hard to fix in future > >> releases, IMHO. > >> > >> Looks okay, and I agree that it's better to fix it now. The timing > >> is a bit unfortunate though, just after RC2. I'll have closer look > >> tomorrow and if it can go in, probably tag RC3. > >> > >> If in the meantime a few more people could test this, that would be > >> helpful. > >> > >> Ralf > > > > I agree, wish I had time to push this before rc2. I could add the > > explanatory comments > > mentioned above and switch to use the atleast_[12]d() solution, test > > that and push it > > in a couple of minutes, or should I better leave it as is now for > > testing? > > Quick follow-up: I just applied the above changes, added some tests to > cover Ben's test cases and tested this with 1.6.0rc2 on OS X 10.5 > i386+ppc > + 10.6 x86_64 (Python2.7+3.2). So I'd be ready to push it to my repo > and do > my (first) pull request... > Go ahead, I'll have a look at it tonight. Thanks for testing on several Pythons, that definitely helps. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri May 6 06:04:05 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 6 May 2011 10:04:05 +0000 (UTC) Subject: [Numpy-discussion] FYI -- numpy.linalg.lstsq as distributed by Ubuntu is crashing on some inputs References: Message-ID: Thu, 05 May 2011 17:41:35 -0700, Nathaniel Smith wrote: > Probably just another standard "your BLAS is compiled wrong!" bug, but > in this case I'm seeing it with the stock versions of ATLAS, numpy, etc. > included in the latest Ubuntu release (11.04 Natty Narwhal): > https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/778217 > > So I thought people might like a heads up in case they run into it as > well. It's terminating with SIGILL, which means that Ubuntu no longer ships a precompiled version of Atlas usable on machines without SSE3. -- Pauli Virtanen From derek at astro.physik.uni-goettingen.de Fri May 6 06:57:39 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 6 May 2011 12:57:39 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> Message-ID: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> On 6 May 2011, at 07:53, Ralf Gommers wrote: > > >> Looks okay, and I agree that it's better to fix it now. The timing > >> is a bit unfortunate though, just after RC2. I'll have closer look > >> tomorrow and if it can go in, probably tag RC3. > >> > >> If in the meantime a few more people could test this, that would be > >> helpful. > >> > >> Ralf > > > > I agree, wish I had time to push this before rc2. I could add the > > explanatory comments > > mentioned above and switch to use the atleast_[12]d() solution, test > > that and push it > > in a couple of minutes, or should I better leave it as is now for > > testing? > > Quick follow-up: I just applied the above changes, added some tests to > cover Ben's test cases and tested this with 1.6.0rc2 on OS X 10.5 > i386+ppc > + 10.6 x86_64 (Python2.7+3.2). So I'd be ready to push it to my repo > and do > my (first) pull request... > > Go ahead, I'll have a look at it tonight. Thanks for testing on > several Pythons, that definitely helps. Done, the request only appears on my repo https://github.com/dhomeier/numpy/ is that correct? If someone could test it on Linux and Windows as well... Cheers, Derek From djl at lanl.gov Fri May 6 10:55:21 2011 From: djl at lanl.gov (DJ Luscher) Date: Fri, 6 May 2011 14:55:21 +0000 (UTC) Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 References: Message-ID: Pearu Peterson gmail.com> writes: > > > Thanks for the bug report!These issues are now fixed in:? https://github.com/numpy/numpy/commit/f393b604? Ralf, feel free to apply this changeset to 1.6.x branch if appropriate.Regards,Pearu > Excellent! Thank you. I'll cautiously add another concern because I believe it is related. Using f2py to compile subroutines where dimensions for result variable are derived from two argument usage of size of an assumed shape input variable does not compile for me. example: subroutine trans(x,y) implicit none real, intent(in), dimension(:,:) :: x real, intent(out), dimension( size(x,2), size(x,1) ) :: y integer :: N, M, i, j N = size(x,1) M = size(x,2) DO i=1,N do j=1,M y(j,i) = x(i,j) END DO END DO end subroutine trans For this example the autogenerated fortran wrapper is: subroutine f2pywraptrans (x, y, f2py_x_d0, f2py_x_d1) integer f2py_x_d0 integer f2py_x_d1 real x(f2py_x_d0,f2py_x_d1) real y(shape(x,-1+2),shape(x,-1+1)) interface subroutine trans(x,y) real, intent(in),dimension(:,:) :: x real, intent(out),dimension( size(x,2), size(x,1) ) :: y end subroutine trans end interface call trans(x, y) end which (inappropriately?) translates the fortran SIZE(var,dim) into fortran SHAPE(var, kind). Please let me know if it is poor form for me to follow on with this secondary issue, but it seems like it is related and a straight-forward fix. Thanks again for your previous fix. DJ From mail.till at gmx.de Fri May 6 11:30:32 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Fri, 6 May 2011 15:30:32 +0000 (UTC) Subject: [Numpy-discussion] Bug: np.linalg.qr crash the python interpreter if getting a shape (0, x) array. Message-ID: Hi, discovered another small bug. Windows 7 32 bit, Python 2.6. In [1]: np.__version__ Out[1]: '1.5.1' In [2]: a=np.zeros((0,2)) In [3]: np.linalg.qr(a) ** On entry to DGEQRF parameter number 4 had an illegal value --Here python is crashed. While np.linalg.lstsq doesn't crashs, the error message is not very helpfull. greetings Till From njs at pobox.com Fri May 6 13:06:20 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 6 May 2011 10:06:20 -0700 Subject: [Numpy-discussion] FYI -- numpy.linalg.lstsq as distributed by Ubuntu is crashing on some inputs In-Reply-To: References: Message-ID: On Fri, May 6, 2011 at 3:04 AM, Pauli Virtanen wrote: > It's terminating with SIGILL, which means that Ubuntu no longer ships > a precompiled version of Atlas usable on machines without SSE3. Yes, that's right. BTW, where'd you get that neat detect_cpu_extensions program? -- Nathaniel From pearu.peterson at gmail.com Fri May 6 14:16:40 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 6 May 2011 21:16:40 +0300 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Fri, May 6, 2011 at 5:55 PM, DJ Luscher wrote: > Pearu Peterson gmail.com> writes: > > > > > > Thanks for the bug report!These issues are now fixed in: > https://github.com/numpy/numpy/commit/f393b604 Ralf, feel free to apply > this > changeset to 1.6.x branch if appropriate.Regards,Pearu > > > > Excellent! Thank you. > > I'll cautiously add another concern because I believe it is related. Using > f2py to compile subroutines where dimensions for result variable are > derived > from two argument usage of size of an assumed shape input variable does not > compile for me. > > example: > > > subroutine trans(x,y) > > implicit none > > real, intent(in), dimension(:,:) :: x > real, intent(out), dimension( size(x,2), size(x,1) ) :: y > > integer :: N, M, i, j > > N = size(x,1) > M = size(x,2) > > DO i=1,N > do j=1,M > y(j,i) = x(i,j) > END DO > END DO > > end subroutine trans > > For this example the autogenerated fortran wrapper is: > > subroutine f2pywraptrans (x, y, f2py_x_d0, f2py_x_d1) > integer f2py_x_d0 > integer f2py_x_d1 > real x(f2py_x_d0,f2py_x_d1) > real y(shape(x,-1+2),shape(x,-1+1)) > interface > subroutine trans(x,y) > real, intent(in),dimension(:,:) :: x > real, intent(out),dimension( size(x,2), size(x,1) ) :: y > end subroutine trans > end interface > call trans(x, y) > end > > which (inappropriately?) translates the fortran SIZE(var,dim) into fortran > SHAPE(var, kind). Please let me know if it is poor form for me to follow > on > with this secondary issue, but it seems like it is related and a > straight-forward fix. > This issue is now fixed in https://github.com/numpy/numpy/commit/a859492c I had to implement size function in C that can be called both as size(var) and size(var, dim). The size-to-shape mapping feature is now removed. I have updated the corresponding release notes in https://github.com/numpy/numpy/commit/1f2e751b Thanks for testing these new f2py features, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri May 6 14:24:22 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 6 May 2011 18:24:22 +0000 (UTC) Subject: [Numpy-discussion] FYI -- numpy.linalg.lstsq as distributed by Ubuntu is crashing on some inputs References: Message-ID: On Fri, 06 May 2011 10:06:20 -0700, Nathaniel Smith wrote: > On Fri, May 6, 2011 at 3:04 AM, Pauli Virtanen wrote: >> It's terminating with SIGILL, which means that Ubuntu no longer ships a >> precompiled version of Atlas usable on machines without SSE3. > > Yes, that's right. > > BTW, where'd you get that neat detect_cpu_extensions program? Here: http://projects.scipy.org/scipy/attachment/ticket/1170/detect_cpu_extensions From djl at lanl.gov Fri May 6 15:18:16 2011 From: djl at lanl.gov (DJ Luscher) Date: Fri, 6 May 2011 19:18:16 +0000 (UTC) Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 References: Message-ID: Pearu Peterson gmail.com> writes: > > On Fri, May 6, 2011 at 5:55 PM, DJ Luscher lanl.gov> wrote: > Pearu Peterson gmail.com> writes: > > > > > > Thanks for the bug report!These issues are now fixed in:?https://github.com/numpy/numpy/commit/f393b604? Ralf, feel free to apply this > changeset to 1.6.x branch if appropriate.Regards,Pearu > > > Excellent! Thank you. > I'll cautiously add another concern because I believe it is related. ?Using > f2py to compile subroutines where dimensions for result variable are derived > from two argument usage of size of an assumed shape input variable does not > compile for me. > > This issue is now fixed in? https://github.com/numpy/numpy/commit/a859492cI had to implement size function in C that can be called both as size(var) and size(var, dim). > The size-to-shape mapping feature is now removed. I have updated the correspondingrelease notes inhttps://github.com/numpy/numpy/commit/1f2e751bThanks for testing these new f2py features,Pearu > Pearu, I greatly appreciate how promptly you provide support for f2py. Thank you. I have encountered another minor hangup. For assumed-shape array-valued functions defined within a fortran module there seems to be some trouble in the autogenerated subroutine wrapper interface. I think it has to do with the order in which variables are declared in the interface specification. for example: ! -*- fix -*- module foo contains function outer(a,b) implicit none real, dimension(:), intent(in) :: a, b real, dimension(size(a),size(b)) :: outer outer = spread(a,dim=2,ncopies=size(b) ) * & spread(b,dim=1,ncopies=size(a) ) end function outer end module when compiled by f2py creates the file: ! -*- f90 -*- ! This file is autogenerated with f2py (version:2) ! It contains Fortran 90 wrappers to fortran functions. subroutine f2pywrap_foo_outer (outerf2pywrap, a, b, f2py_a_d0, f2p& &y_b_d0) use foo, only : outer integer f2py_a_d0 integer f2py_b_d0 real a(f2py_a_d0) real b(f2py_b_d0) real outerf2pywrap(size(a),size(b)) outerf2pywrap = outer(a, b) end subroutine f2pywrap_foo_outer subroutine f2pyinitfoo(f2pysetupfunc) interface subroutine f2pywrap_foo_outer (outerf2pywrap, outer, a, b, f2py_a_& &d0, f2py_b_d0) integer f2py_a_d0 integer f2py_b_d0 real outer(size(a),size(b)) real a(f2py_a_d0) real b(f2py_b_d0) real outerf2pywrap(size(a),size(b)) end subroutine f2pywrap_foo_outer end interface external f2pysetupfunc call f2pysetupfunc(f2pywrap_foo_outer) end subroutine f2pyinitfoo in the subroutine interface specification the size(a) and size(b) are used to dimension outer above (before) the declaration of a and b themselves. This halts my compiler. The wrapper seems to compile OK if a and b are declared above outer in the interface. thanks again for your help, DJ From pearu.peterson at gmail.com Fri May 6 16:04:56 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 6 May 2011 23:04:56 +0300 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Fri, May 6, 2011 at 10:18 PM, DJ Luscher wrote: > > I have encountered another minor hangup. For assumed-shape array-valued > functions defined within a fortran module there seems to be some trouble in > the > autogenerated subroutine wrapper interface. I think it has to do with the > order > in which variables are declared in the interface specification. > > for example: > > ! -*- fix -*- > module foo > contains > function outer(a,b) > implicit none > real, dimension(:), intent(in) :: a, b > real, dimension(size(a),size(b)) :: outer > > outer = spread(a,dim=2,ncopies=size(b) ) * > & spread(b,dim=1,ncopies=size(a) ) > > end function outer > end module > > when compiled by f2py creates the file: > > ! -*- f90 -*- > ! This file is autogenerated with f2py (version:2) > ! It contains Fortran 90 wrappers to fortran functions. > > subroutine f2pywrap_foo_outer (outerf2pywrap, a, b, f2py_a_d0, f2p& > &y_b_d0) > use foo, only : outer > integer f2py_a_d0 > integer f2py_b_d0 > real a(f2py_a_d0) > real b(f2py_b_d0) > real outerf2pywrap(size(a),size(b)) > outerf2pywrap = outer(a, b) > end subroutine f2pywrap_foo_outer > > subroutine f2pyinitfoo(f2pysetupfunc) > interface > subroutine f2pywrap_foo_outer (outerf2pywrap, outer, a, b, f2py_a_& > &d0, f2py_b_d0) > integer f2py_a_d0 > integer f2py_b_d0 > real outer(size(a),size(b)) > real a(f2py_a_d0) > real b(f2py_b_d0) > real outerf2pywrap(size(a),size(b)) > end subroutine f2pywrap_foo_outer > end interface > external f2pysetupfunc > call f2pysetupfunc(f2pywrap_foo_outer) > end subroutine f2pyinitfoo > > in the subroutine interface specification the size(a) and size(b) are used > to > dimension outer above (before) the declaration of a and b themselves. This > halts my compiler. The wrapper seems to compile OK if a and b are declared > above outer in the interface. > thanks again for your help, > DJ > > Your example works fine here: $ f2py -m foo foo_out.f90 -c $ python -c 'import foo; print foo.foo.outer([1,2],[3,4])' [[ 3. 4.] [ 6. 8.]] with outer defined before a and b. I would presume that compiler would give a warning, at least, when this would be a problem. Anyway, try to apply the following patch: -------------------------------------------------------------------- diff --git a/numpy/f2py/func2subr.py b/numpy/f2py/func2subr.py index 0f76920..f746108 100644 --- a/numpy/f2py/func2subr.py +++ b/numpy/f2py/func2subr.py @@ -90,7 +90,6 @@ def createfuncwrapper(rout,signature=0): v['dimension'][i] = dn rout['args'].extend(extra_args) need_interface = bool(extra_args) - ret = [''] def add(line,ret=ret): @@ -143,8 +142,13 @@ def createfuncwrapper(rout,signature=0): dumped_args.append(a) for a in args: if a in dumped_args: continue + if isintent_in(vars[a]): + add(var2fixfortran(vars,a,f90mode=f90mode)) + dumped_args.append(a) + for a in args: + if a in dumped_args: continue add(var2fixfortran(vars,a,f90mode=f90mode)) - + add(l) if need_interface: -------------------------------------------------------------------- to see if changing the order will fix the hang. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From djl at lanl.gov Fri May 6 17:00:43 2011 From: djl at lanl.gov (DJ Luscher) Date: Fri, 6 May 2011 21:00:43 +0000 (UTC) Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 References: Message-ID: Pearu Peterson gmail.com> writes: > > On Fri, May 6, 2011 at 10:18 PM, DJ Luscher lanl.gov> wrote: > > I have encountered another minor hangup. ?For assumed-shape array-valued > functions defined within a fortran module there seems to be some trouble in > the autogenerated subroutine wrapper interface. ?I think it has to do with > the order in which variables are declared in the interface specification. > > in the subroutine interface specification the size(a) and size(b) are used to > dimension outer above (before) the declaration of a and b themselves. ?This > halts my compiler. ?The wrapper seems to compile OK if a and b are declared > above outer in the interface. > thanks again for your help, > > DJ > > Your example works fine here:$ f2py? -m foo foo_out.f90 -c$ python -c 'import > foo; print foo.foo.outer([1,2],[3,4])'[[ 3.? 4.]?[ 6.? 8.]]with outer defined > before a and b. I would presume that compiler would > give a warning, at least, when this would be a problem. Anyway, try to apply > the following patch: > to see if changing the order will fix the hang.Pearu > > indeed - it works fine as is when I compile with gfortran, but not ifort. I suppose there may be some compiler option for ifort to overcome that, but I couldn't tell from a brief scan of the doc. the patch works when I add in two separate loops over args: (~line 138 in func2subr.py): for a in args: if a in dumped_args: continue if isscalar(vars[a]): add(var2fixfortran(vars,a,f90mode=f90mode)) dumped_args.append(a) for a in args: if a in dumped_args: continue if isintent_in(vars[a]): add(var2fixfortran(vars,a,f90mode=f90mode)) dumped_args.append(a) not sure if that was your intention, but when I tried to use just "isintent_in" or to include both conditions in same loop, the input arrays (a and b) were declared ahead of the derived shape-array (outer), but also ahead of the integers used to define a and b (e.g. f2py_a_d0). this works. many thanks, DJ From pearu.peterson at gmail.com Fri May 6 17:25:28 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Sat, 7 May 2011 00:25:28 +0300 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 2 In-Reply-To: References: Message-ID: On Sat, May 7, 2011 at 12:00 AM, DJ Luscher wrote: > Pearu Peterson gmail.com> writes: > > > > On Fri, May 6, 2011 at 10:18 PM, DJ Luscher lanl.gov> wrote: > > > > I have encountered another minor hangup. For assumed-shape array-valued > > functions defined within a fortran module there seems to be some trouble > in > > the autogenerated subroutine wrapper interface. I think it has to do > with > > the order in which variables are declared in the interface specification. > > > > in the subroutine interface specification the size(a) and size(b) are > used to > > dimension outer above (before) the declaration of a and b themselves. > This > > halts my compiler. The wrapper seems to compile OK if a and b are > declared > > above outer in the interface. > > thanks again for your help, > > > > DJ > > > > Your example works fine here:$ f2py -m foo foo_out.f90 -c$ python -c > 'import > > foo; print foo.foo.outer([1,2],[3,4])'[[ 3. 4.] [ 6. 8.]]with outer > defined > > before a and b. I would presume that compiler would > > give a warning, at least, when this would be a problem. Anyway, try to > apply > > the following patch: > > to see if changing the order will fix the hang.Pearu > > > > > indeed - it works fine as is when I compile with gfortran, but not ifort. > I > suppose there may be some compiler option for ifort to overcome that, but I > couldn't tell from a brief scan of the doc. > > the patch works when I add in two separate loops over args: (~line 138 in > func2subr.py): > > for a in args: > if a in dumped_args: continue > if isscalar(vars[a]): > add(var2fixfortran(vars,a,f90mode=f90mode)) > dumped_args.append(a) > for a in args: > if a in dumped_args: continue > if isintent_in(vars[a]): > add(var2fixfortran(vars,a,f90mode=f90mode)) > dumped_args.append(a) > > not sure if that was your intention, yes, that is what the patch was generated from. > but when I tried to use just "isintent_in" > or to include both conditions in same loop, that would not work as you noticed.. > the input arrays (a and b) were > declared ahead of the derived shape-array (outer), but also ahead of the > integers used to define a and b (e.g. f2py_a_d0). > > I have committed the patch: https://github.com/numpy/numpy/commit/6df2ac21 Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From uniccornio at hotmail.com Sat May 7 11:40:55 2011 From: uniccornio at hotmail.com (Pedro G) Date: Sat, 7 May 2011 08:40:55 -0700 (PDT) Subject: [Numpy-discussion] re turning a tuple of arrays with apply_along_axis Message-ID: <31566316.post@talk.nabble.com> Hello, I'm trying to use apply_along_axis with a function that returns 2 arrays, but I always get different error messages. For example, here is a test: >>> from numpy import * >>> def f(x): ... a = array([1, 2, 3], dtype=float) ... b = array([4, 5, 6], dtype=float) ... return (a, b) ... >>> x = arange(60).reshape(5, 4, 3) >>> x array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]], [[12, 13, 14], [15, 16, 17], [18, 19, 20], [21, 22, 23]], [[24, 25, 26], [27, 28, 29], [30, 31, 32], [33, 34, 35]], [[36, 37, 38], [39, 40, 41], [42, 43, 44], [45, 46, 47]], [[48, 49, 50], [51, 52, 53], [54, 55, 56], [57, 58, 59]]]) >>> f(1) (array([ 1., 2., 3.]), array([ 4., 5., 6.])) >>> f(array([1, 5, 9])) (array([ 1., 2., 3.]), array([ 4., 5., 6.])) >>> a, b = apply_along_axis(f, 0, x) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.2/dist-packages/numpy/lib/shape_base.py", line 106, in apply_along_axis outarr[tuple(i.tolist())] = res ValueError: operands could not be broadcast together with shapes (2) (2,3) I'm using python 3.2 and numpy 1.6.0 RC2. Does anyone know an efficient way to do this? Thanks Pedro -- View this message in context: http://old.nabble.com/returning-a-tuple-of-arrays-with-apply_along_axis-tp31566316p31566316.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From ralf.gommers at googlemail.com Sat May 7 16:17:28 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 7 May 2011 22:17:28 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> Message-ID: On Fri, May 6, 2011 at 12:57 PM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: > > On 6 May 2011, at 07:53, Ralf Gommers wrote: > > > > > >> Looks okay, and I agree that it's better to fix it now. The timing > > >> is a bit unfortunate though, just after RC2. I'll have closer look > > >> tomorrow and if it can go in, probably tag RC3. > > >> > > >> If in the meantime a few more people could test this, that would be > > >> helpful. > > >> > > >> Ralf > > > > > > I agree, wish I had time to push this before rc2. I could add the > > > explanatory comments > > > mentioned above and switch to use the atleast_[12]d() solution, test > > > that and push it > > > in a couple of minutes, or should I better leave it as is now for > > > testing? > > > > Quick follow-up: I just applied the above changes, added some tests to > > cover Ben's test cases and tested this with 1.6.0rc2 on OS X 10.5 > > i386+ppc > > + 10.6 x86_64 (Python2.7+3.2). So I'd be ready to push it to my repo > > and do > > my (first) pull request... > > > > Go ahead, I'll have a look at it tonight. Thanks for testing on > > several Pythons, that definitely helps. > > > Done, the request only appears on my repo > https://github.com/dhomeier/numpy/ > > is that correct? If someone could test it on Linux and Windows as > well... > Committed, thanks for all the work. The pull request was in the wrong place, that's a minor flaw in the github UI. After you press "Pull Request" you need to read the small print to see where it's going. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat May 7 16:30:43 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 7 May 2011 22:30:43 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 3 Message-ID: Hi, I am pleased to announce the availability of the third release candidate of NumPy 1.6.0. Compared to the second release candidate, two issues in f2py and one in loadtxt were fixed. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc3/ For (preliminary) release notes see below. Enjoy, Ralf ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py supports now automatic wrapping of Fortran routines that use two argument ``size`` function in dimension specifications. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= ``default error handling`` -------------------------- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Deprecated features =================== The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality will be replaced by the new "density" keyword. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. Checksums ========= 295c456e5008873a14a56c11bce7cb60 release/installers/numpy-1.6.0rc3-py2.7-python.org-macosx10.6.dmg faaa5e8ec8992263ad444d809a511f07 release/installers/numpy-1.6.0rc3-win32-superpack-python2.5.exe 3a27307c4a5dcd56f8c8a1ae1d858f93 release/installers/numpy-1.6.0rc3-win32-superpack-python2.6.exe 6bf42735d54f7043a0d0b5a9652d3dbb release/installers/numpy-1.6.0rc3-win32-superpack-python2.7.exe 450e39f4666805e53fd750d1d0d07243 release/installers/numpy-1.6.0rc3-win32-superpack-python3.1.exe 500214e83b4823f90569ccf2f947ff6b release/installers/numpy-1.6.0rc3-win32-superpack-python3.2.exe 09efcfdb987899d0ff6b79d79e707ac9 release/installers/numpy-1.6.0rc3.tar.gz 466fe879964de5e9ecb562f48ba18131 release/installers/numpy-1.6.0rc3.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesmckinn at gmail.com Sat May 7 20:04:37 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 7 May 2011 20:04:37 -0400 Subject: [Numpy-discussion] Bug? NumPy types circumvent some overloaded methods in ndarray subclasses Message-ID: This strikes me as a bug-- haven't checked NumPy 1.6 yet but this happens in 1.5.1. Here's a toy example: class MyNdarray(np.ndarray): def __new__(cls, data): subarr = np.array(data, dtype=np.float64).view(cls) return subarr def __radd__(self, other): print 'hello' return (self.view(np.ndarray) + other).view(MyNdarray) def __add__(self, other): print 'hello' return (self.view(np.ndarray) + other).view(MyNdarray) In [25]: foo = MyNdarray(np.arange(10.)) In [26]: foo + 3 hello Out[26]: MyNdarray([ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.]) In [27]: foo + np.float64(3) hello Out[27]: MyNdarray([ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.]) In [28]: 3 + foo hello Out[28]: MyNdarray([ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.]) In [29]: np.float64(3) + foo Out[29]: MyNdarray([ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12.]) I'm not sure if there's an easy workaround here. I thought maybe it was an array priority issue but after some tinkering it doesn't look like it. Any ideas? - Wes From opossumnano at gmail.com Sun May 8 07:17:59 2011 From: opossumnano at gmail.com (Tiziano Zito) Date: Sun, 8 May 2011 13:17:59 +0200 Subject: [Numpy-discussion] [ANN] EuroScipy 2011 - deadline extended Message-ID: <20110508111759.GA19748@multivac.zonafranca> =================================== EuroScipy 2011 - Deadline Extended! =================================== Deadline extended! You can submit your contribution until Friday May 13. --------------------------------------------- The 4th European meeting on Python in Science --------------------------------------------- **Paris, Ecole Normale Sup?rieure, August 25-28 2011** We are happy to announce the 4th EuroScipy meeting, in Paris, August 2011. The EuroSciPy meeting is a cross-disciplinary gathering focused on the use and development of the Python language in scientific research. This event strives to bring together both users and developers of scientific tools, as well as academic research and state of the art industry. Main topics =========== - Presentations of scientific tools and libraries using the Python language, including but not limited to: - vector and array manipulation - parallel computing - scientific visualization - scientific data flow and persistence - algorithms implemented or exposed in Python - web applications and portals for science and engineering. - Reports on the use of Python in scientific achievements or ongoing projects. - General-purpose Python tools that can be of special interest to the scientific community. Tutorials ========= There will be two tutorial tracks at the conference, an introductory one, to bring up to speed with the Python language as a scientific tool, and an advanced track, during which experts of the field will lecture on specific advanced topics such as advanced use of numpy, scientific visualization, software engineering... Keynote Speaker: Fernando Perez =============================== We are excited to welcome Fernando Perez (UC Berkeley, Helen Wills Neuroscience Institute, USA) as our keynote speaker. Fernando Perez is the original author of the enhanced interactive python shell IPython and a very active contributor to the Python for Science ecosystem. Important dates =============== Talk submission deadline: Sunday May 8 Program announced: Sunday May 29 Tutorials tracks: Thursday August 25 - Friday August 26 Conference track: Saturday August 27 - Sunday August 28 Call for papers =============== We are soliciting talks that discuss topics related to scientific computing using Python. These include applications, teaching, future development directions, and research. We welcome contributions from the industry as well as the academic world. Indeed, industrial research and development as well academic research face the challenge of mastering IT tools for exploration, modeling and analysis. We look forward to hearing your recent breakthroughs using Python! Submission guidelines ===================== - We solicit talk proposals in the form of a one-page long abstract. - Submissions whose main purpose is to promote a commercial product or service will be refused. - All accepted proposals must be presented at the EuroSciPy conference by at least one author. The one-page long abstracts are for conference planing and selection purposes only. We will later select papers for publication of post-proceedings in a peer-reviewed journal. How to submit an abstract ========================= To submit a talk to the EuroScipy conference follow the instructions here: http://www.euroscipy.org/card/euroscipy2011_call_for_papers Organizers ========== Chairs: - Ga?l Varoquaux (INSERM, Unicog team, and INRIA, Parietal team) - Nicolas Chauvat (Logilab) Local organization committee: - Emmanuelle Gouillart (Saint-Gobain Recherche) - Jean-Philippe Chauvat (Logilab) Tutorial chair: - Valentin Haenel (MKP, Technische Universit?t Berlin) Program committee: - Chair: Tiziano Zito (MKP, Technische Universit?t Berlin) - Romain Brette (ENS Paris, DEC) - Emmanuelle Gouillart (Saint-Gobain Recherche) - Eric Lebigot (Laboratoire Kastler Brossel, Universit? Pierre et Marie Curie) - Konrad Hinsen (Soleil Synchrotron, CNRS) - Hans Petter Langtangen (Simula laboratories) - Jarrod Millman (UC Berkeley, Helen Wills NeuroScience institute) - Mike M?ller (Python Academy) - Didrik Pinte (Enthought Inc) - Marc Poinot (ONERA) - Christophe Pradal (CIRAD/INRIA, Virtual Plantes team) - Andreas Schreiber (DLR) - St?fan van der Walt (University of Stellenbosch) Website ======= http://www.euroscipy.org/conference/euroscipy_2011 _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From jsseabold at gmail.com Sun May 8 11:42:32 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sun, 8 May 2011 11:42:32 -0400 Subject: [Numpy-discussion] Bug: np.linalg.qr crash the python interpreter if getting a shape (0, x) array. In-Reply-To: References: Message-ID: On Fri, May 6, 2011 at 11:30 AM, Till Stensitzki wrote: > Hi, > discovered another small bug. Windows 7 32 bit, Python 2.6. > > In [1]: np.__version__ > Out[1]: '1.5.1' > > In [2]: a=np.zeros((0,2)) > > In [3]: np.linalg.qr(a) > ?** On entry to DGEQRF parameter number ?4 had an illegal value > In [3]: a Out[3]: array([], shape=(0, 2), dtype=float64) It doesn't crash for me on 1.5.1, but returns error code -4 from dgeqrf. Maybe np.linalg.linalg._assertRank2 should also check that a.size > 0, or there should an _assertNotEmpty? Maybe you should file a ticket so it doesn't get lost. Skipper > > --Here python is crashed. > While np.linalg.lstsq doesn't crashs, the error message is not very helpfull. > > greetings Till > From jsseabold at gmail.com Sun May 8 14:24:01 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sun, 8 May 2011 14:24:01 -0400 Subject: [Numpy-discussion] Bug: np.linalg.qr crash the python interpreter if getting a shape (0, x) array. In-Reply-To: References: Message-ID: On Sun, May 8, 2011 at 11:42 AM, Skipper Seabold wrote: > On Fri, May 6, 2011 at 11:30 AM, Till Stensitzki wrote: >> Hi, >> discovered another small bug. Windows 7 32 bit, Python 2.6. >> >> In [1]: np.__version__ >> Out[1]: '1.5.1' >> >> In [2]: a=np.zeros((0,2)) >> >> In [3]: np.linalg.qr(a) >> ?** On entry to DGEQRF parameter number ?4 had an illegal value >> > > In [3]: a > Out[3]: array([], shape=(0, 2), dtype=float64) > > It doesn't crash for me on 1.5.1, but returns error code -4 from > dgeqrf. Maybe np.linalg.linalg._assertRank2 should also check that > a.size > 0, or there should an _assertNotEmpty? > > Maybe you should file a ticket so it doesn't get lost. > Added an _assertNonEmpty to qr and filed a pull request. https://github.com/jseabold/numpy/compare/master...qr-nonempty-check Skipper From paul.anton.letnes at gmail.com Sun May 8 17:15:31 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 8 May 2011 14:15:31 -0700 Subject: [Numpy-discussion] bug in numpy.ndarray? Message-ID: Hi, it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the problem. % python testcrash.py [14:13:27 on 11-05-08] [ 12.+0.1j] [ 1.+0.1j] complex128 Traceback (most recent call last): File "testcrash.py", line 11, in A[0] = A[0] + (eps1 - eps2) TypeError: can't convert complex to float % cat testcrash.py #!/usr/bin/env python import numpy A = numpy.zeros(10, dtype=numpy.complex128) eps1 = numpy.complex128([12.0 + 0.1j]) eps2 = numpy.complex128([1.0 + 0.1j]) print type(eps1), eps1 print type(eps2), eps2 print A.dtype A[0] = A[0] + (eps1 - eps2) From kwgoodman at gmail.com Sun May 8 17:45:45 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 8 May 2011 14:45:45 -0700 Subject: [Numpy-discussion] Mapping of dtype to C types Message-ID: I'm writing a function that accepts four possible dtypes: int32, int64, float32, float64. The function will call a C extension (wrapped in Cython). What are the equivalent C types? int, long, float, double, respectively? Will that work on all systems? From jsalvati at u.washington.edu Sun May 8 17:51:51 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Sun, 8 May 2011 14:51:51 -0700 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: References: Message-ID: I want to add a question: is there a function that returns the name of the corresponding C type given the numpy type? On Sun, May 8, 2011 at 2:45 PM, Keith Goodman wrote: > I'm writing a function that accepts four possible dtypes: int32, > int64, float32, float64. The function will call a C extension (wrapped > in Cython). What are the equivalent C types? int, long, float, double, > respectively? Will that work on all systems? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun May 8 17:57:49 2011 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 8 May 2011 14:57:49 -0700 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: References: Message-ID: On Sun, May 8, 2011 at 2:45 PM, Keith Goodman wrote: > I'm writing a function that accepts four possible dtypes: int32, > int64, float32, float64. The function will call a C extension (wrapped > in Cython). What are the equivalent C types? int, long, float, double, > respectively? Will that work on all systems? I think you're fine on float and double, but on many systems 'long' is 32 bits. #include int32_t this_is_32_bits; int64_t this_is_64_bits; (There may still be compilers out there that don't come with stdint.h by default, but if you need to support them then you'll be able to find appropriate autoconf goo to create it, since it's a standard interface.) -- Nathaniel From charlesr.harris at gmail.com Sun May 8 20:23:10 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 May 2011 18:23:10 -0600 Subject: [Numpy-discussion] bug in numpy.ndarray? In-Reply-To: References: Message-ID: On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > Hi, > > it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python > 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 > with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the > problem. > > % python testcrash.py > > [14:13:27 on 11-05-08] > [ 12.+0.1j] > [ 1.+0.1j] > complex128 > Traceback (most recent call last): > File "testcrash.py", line 11, in > A[0] = A[0] + (eps1 - eps2) > TypeError: can't convert complex to float > > % cat testcrash.py > #!/usr/bin/env python > > import numpy > > A = numpy.zeros(10, dtype=numpy.complex128) > eps1 = numpy.complex128([12.0 + 0.1j]) > eps2 = numpy.complex128([1.0 + 0.1j]) > It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a scalar. The error message is less than helpful though. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun May 8 20:31:47 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 8 May 2011 19:31:47 -0500 Subject: [Numpy-discussion] bug in numpy.ndarray? In-Reply-To: References: Message-ID: On Sun, May 8, 2011 at 19:23, Charles R Harris wrote: > > On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes > wrote: >> >> Hi, >> >> it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python >> 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 >> with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the >> problem. >> >> % python testcrash.py >> >> ? ? ? ? ?[14:13:27 on 11-05-08] >> [ 12.+0.1j] >> [ 1.+0.1j] >> complex128 >> Traceback (most recent call last): >> ?File "testcrash.py", line 11, in >> ? ?A[0] = A[0] + (eps1 - eps2) >> TypeError: can't convert complex to float >> >> ?% cat testcrash.py >> #!/usr/bin/env python >> >> import numpy >> >> A = numpy.zeros(10, dtype=numpy.complex128) >> eps1 = numpy.complex128([12.0 + 0.1j]) >> eps2 = numpy.complex128([1.0 + 0.1j]) > > It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a > scalar. The error message is less than helpful though. Well, not quite. It looks like we do handle (1,) arrays on the RHS just fine: [~/scratch] |46> A = np.arange(10) [~/scratch] |47> A[0] = np.array([1]) [~/scratch] |48> A[0] 1 It works for a variety of dtypes except for the complexes: [~/scratch] |53> for dt in [bool, np.uint8, np.int8, int, float, np.float32, np.complex64, np.complex128, object]: ...> A = np.zeros(10, dtype=dt) ...> try: ...> A[0] = np.array([1], dtype=dt) ...> except Exception, e: ...> print '%s failed: %s: %s' % (dt.__name__, type(e).__name__, e) ...> else: ...> print '%s succeeded' % dt.__name__ ...> bool succeeded uint8 succeeded int8 succeeded int succeeded float succeeded float32 succeeded complex64 failed: TypeError: can't convert complex to float complex128 failed: TypeError: can't convert complex to float object succeeded -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From warren.weckesser at enthought.com Sun May 8 20:32:24 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sun, 8 May 2011 19:32:24 -0500 Subject: [Numpy-discussion] bug in numpy.ndarray? In-Reply-To: References: Message-ID: On Sun, May 8, 2011 at 7:23 PM, Charles R Harris wrote: > > > On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > >> Hi, >> >> it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python >> 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 >> with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the >> problem. >> >> % python testcrash.py >> >> [14:13:27 on 11-05-08] >> [ 12.+0.1j] >> [ 1.+0.1j] >> complex128 >> Traceback (most recent call last): >> File "testcrash.py", line 11, in >> A[0] = A[0] + (eps1 - eps2) >> TypeError: can't convert complex to float >> >> % cat testcrash.py >> #!/usr/bin/env python >> >> import numpy >> >> A = numpy.zeros(10, dtype=numpy.complex128) >> eps1 = numpy.complex128([12.0 + 0.1j]) >> eps2 = numpy.complex128([1.0 + 0.1j]) >> > > It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a > scalar. The error message is less than helpful though. > > But the same pattern works fine with float64: In [2]: x = array([1.0, 2.0]) In [3]: y = array([10.0]) In [4]: x[0] = y # Works In [5]: a = array([1.0, 2.0], dtype=complex128) In [6]: b = array([10.0 + 1j]) In [7]: a[0] = b # Error --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /Users/warren/ in () TypeError: can't convert complex to float Something is fishy about that. Warren > > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.anton.letnes at gmail.com Sun May 8 21:22:40 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sun, 8 May 2011 18:22:40 -0700 Subject: [Numpy-discussion] bug in numpy.ndarray? In-Reply-To: References: Message-ID: <39D74CDA-1748-4DE5-9461-1DC6FF7BC6FA@gmail.com> On 8. mai 2011, at 17.32, Warren Weckesser wrote: > > > On Sun, May 8, 2011 at 7:23 PM, Charles R Harris wrote: > > > On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes wrote: > Hi, > > it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the problem. > > % python testcrash.py [14:13:27 on 11-05-08] > [ 12.+0.1j] > [ 1.+0.1j] > complex128 > Traceback (most recent call last): > File "testcrash.py", line 11, in > A[0] = A[0] + (eps1 - eps2) > TypeError: can't convert complex to float > > % cat testcrash.py > #!/usr/bin/env python > > import numpy > > A = numpy.zeros(10, dtype=numpy.complex128) > eps1 = numpy.complex128([12.0 + 0.1j]) > eps2 = numpy.complex128([1.0 + 0.1j]) > > It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a scalar. The error message is less than helpful though. > > > > But the same pattern works fine with float64: > > In [2]: x = array([1.0, 2.0]) > > In [3]: y = array([10.0]) > > In [4]: x[0] = y # Works > > In [5]: a = array([1.0, 2.0], dtype=complex128) > > In [6]: b = array([10.0 + 1j]) > > In [7]: a[0] = b # Error > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > > /Users/warren/ in () > > TypeError: can't convert complex to float > > > Something is fishy about that. > I agree. One thing is if arrays don't support this kind of assignment, but behavior should be the same for complex and float. IMHO. Paul From pav at iki.fi Mon May 9 04:46:10 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 9 May 2011 08:46:10 +0000 (UTC) Subject: [Numpy-discussion] Mapping of dtype to C types References: Message-ID: Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: > I'm writing a function that accepts four possible dtypes: int32, int64, > float32, float64. The function will call a C extension (wrapped in > Cython). What are the equivalent C types? int, long, float, double, > respectively? Will that work on all systems? Long can be 32-bit or 64-bit, depending on the platform. The types available in Numpy are listed here, including the information which of them are compatible with which C types: http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html The C long seems not to be listed -- but it's the same as "Python int", i.e., np.int_ will work. IIRC, the Numpy type codes, dtype.kind, map directly to C types. Pauli From kwgoodman at gmail.com Mon May 9 09:58:55 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 9 May 2011 06:58:55 -0700 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: References: Message-ID: On Mon, May 9, 2011 at 1:46 AM, Pauli Virtanen wrote: > Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: >> I'm writing a function that accepts four possible dtypes: int32, int64, >> float32, float64. The function will call a C extension (wrapped in >> Cython). What are the equivalent C types? int, long, float, double, >> respectively? Will that work on all systems? > > Long can be 32-bit or 64-bit, depending on the platform. > > The types available in Numpy are listed here, including > the information which of them are compatible with which C types: > > http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html > > The C long seems not to be listed -- but it's the same as > "Python int", i.e., np.int_ will work. > > IIRC, the Numpy type codes, dtype.kind, map directly to C types. Does this mapping look right? np.float32 --> float np.float64 --> double if np.int_ == np.int32: # np.int32 --> long # np.int64 --> long long elif np.int_ == np.int64: # np.int32 --> int # np.int64 --> long From charlesr.harris at gmail.com Mon May 9 15:26:05 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 May 2011 13:26:05 -0600 Subject: [Numpy-discussion] bug in numpy.ndarray? In-Reply-To: <39D74CDA-1748-4DE5-9461-1DC6FF7BC6FA@gmail.com> References: <39D74CDA-1748-4DE5-9461-1DC6FF7BC6FA@gmail.com> Message-ID: On Sun, May 8, 2011 at 7:22 PM, Paul Anton Letnes < paul.anton.letnes at gmail.com> wrote: > > On 8. mai 2011, at 17.32, Warren Weckesser wrote: > > > > > > > On Sun, May 8, 2011 at 7:23 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > > > > > > On Sun, May 8, 2011 at 3:15 PM, Paul Anton Letnes < > paul.anton.letnes at gmail.com> wrote: > > Hi, > > > > it seems that I have found a bug in numpy.ndarray. numpy 1.5.1, python > 2.7.1 from macports on mac os x 10.6.7. I got the same error on Fedora 14 > with numpy 1.4.1 and python 2.7. Appending a [0] to the last line solves the > problem. > > > > % python testcrash.py > > [14:13:27 on 11-05-08] > > [ 12.+0.1j] > > [ 1.+0.1j] > > complex128 > > Traceback (most recent call last): > > File "testcrash.py", line 11, in > > A[0] = A[0] + (eps1 - eps2) > > TypeError: can't convert complex to float > > > > % cat testcrash.py > > #!/usr/bin/env python > > > > import numpy > > > > A = numpy.zeros(10, dtype=numpy.complex128) > > eps1 = numpy.complex128([12.0 + 0.1j]) > > eps2 = numpy.complex128([1.0 + 0.1j]) > > > > It's the brackets, numpy.complex128([1.0 + 0.1j]) is a 1d array, not a > scalar. The error message is less than helpful though. > > > > > > > > But the same pattern works fine with float64: > > > > In [2]: x = array([1.0, 2.0]) > > > > In [3]: y = array([10.0]) > > > > In [4]: x[0] = y # Works > > > > In [5]: a = array([1.0, 2.0], dtype=complex128) > > > > In [6]: b = array([10.0 + 1j]) > > > > In [7]: a[0] = b # Error > > > --------------------------------------------------------------------------- > > TypeError Traceback (most recent call > last) > > > > /Users/warren/ in () > > > > TypeError: can't convert complex to float > > > > > > Something is fishy about that. > > > I agree. One thing is if arrays don't support this kind of assignment, but > behavior should be the same for complex and float. IMHO. > > Seems that it works for most types as long as there is just one element: In [1]: a = ones(10) In [2]: b = ones((1,1,1,1)) In [3]: a[0] = b So that seems to be the standard. IMHO, most uses of this sort are likely to be programming errors, but there it is. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue May 10 13:16:13 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 10 May 2011 13:16:13 -0400 Subject: [Numpy-discussion] Inconsistent output dtype in mixed scalar-array operations Message-ID: Hi, I opened a ticket about this (http://projects.scipy.org/numpy/ticket/1827), but thought I'd also ask on the mailing list whether this is working as intended. It looks like the resulting dtype of a "+" operation between a scalar and an array can depend on the order of the arguments. This seems wrong. Tested with numpy 1.5.1 under Ubuntu 11.04 (64-bit). In [4]: (numpy.array(0, dtype='uint16') + numpy.array([1], dtype='int8')).dtype Out[4]: dtype('int16') In [6]: (numpy.array([1], dtype='int8') + numpy.array(0, dtype='uint16')).dtype Out[6]: dtype('int8') Thanks for any feedback, -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 10 13:27:19 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 May 2011 11:27:19 -0600 Subject: [Numpy-discussion] Inconsistent output dtype in mixed scalar-array operations In-Reply-To: References: Message-ID: On Tue, May 10, 2011 at 11:16 AM, Olivier Delalleau wrote: > Hi, > > I opened a ticket about this (http://projects.scipy.org/numpy/ticket/1827 > ), > but thought I'd also ask on the mailing list whether this is working as > intended. > > It looks like the resulting dtype of a "+" operation between a scalar and > an > array can depend on the order of the arguments. This seems wrong. > > Tested with numpy 1.5.1 under Ubuntu 11.04 (64-bit). > > In [4]: (numpy.array(0, dtype='uint16') + numpy.array([1], > dtype='int8')).dtype > Out[4]: dtype('int16') > > In [6]: (numpy.array([1], dtype='int8') + numpy.array(0, > dtype='uint16')).dtype > Out[6]: dtype('int8') > > Thanks for any feedback, > > This has been fixed in the coming 1.6.0 release. In [2]: (numpy.array(0, dtype='uint16') + numpy.array([1],dtype='int8')).dtype Out[2]: dtype('int8') In [3]: (numpy.array([1], dtype='int8') + numpy.array(0,dtype='uint16')).dtype Out[3]: dtype('int8') Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From shish at keba.be Tue May 10 13:41:09 2011 From: shish at keba.be (Olivier Delalleau) Date: Tue, 10 May 2011 13:41:09 -0400 Subject: [Numpy-discussion] Inconsistent output dtype in mixed scalar-array operations In-Reply-To: References: Message-ID: 2011/5/10 Charles R Harris > > > On Tue, May 10, 2011 at 11:16 AM, Olivier Delalleau wrote: > >> Hi, >> >> I opened a ticket about this (http://projects.scipy.org/numpy/ticket/1827 >> ), >> but thought I'd also ask on the mailing list whether this is working as >> intended. >> >> It looks like the resulting dtype of a "+" operation between a scalar and >> an >> array can depend on the order of the arguments. This seems wrong. >> >> Tested with numpy 1.5.1 under Ubuntu 11.04 (64-bit). >> >> In [4]: (numpy.array(0, dtype='uint16') + numpy.array([1], >> dtype='int8')).dtype >> Out[4]: dtype('int16') >> >> In [6]: (numpy.array([1], dtype='int8') + numpy.array(0, >> dtype='uint16')).dtype >> Out[6]: dtype('int8') >> >> Thanks for any feedback, >> >> > This has been fixed in the coming 1.6.0 release. > > In [2]: (numpy.array(0, dtype='uint16') + > numpy.array([1],dtype='int8')).dtype > Out[2]: dtype('int8') > > In [3]: (numpy.array([1], dtype='int8') + > numpy.array(0,dtype='uint16')).dtype > Out[3]: dtype('int8') > > Chuck > Cool, thanks, I'll close the ticket then :) -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 10 13:44:07 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 10 May 2011 11:44:07 -0600 Subject: [Numpy-discussion] Inconsistent output dtype in mixed scalar-array operations In-Reply-To: References: Message-ID: On Tue, May 10, 2011 at 11:41 AM, Olivier Delalleau wrote: > > 2011/5/10 Charles R Harris > >> >> >> On Tue, May 10, 2011 at 11:16 AM, Olivier Delalleau wrote: >> >>> Hi, >>> >>> I opened a ticket about this ( >>> http://projects.scipy.org/numpy/ticket/1827), >>> but thought I'd also ask on the mailing list whether this is working as >>> intended. >>> >>> It looks like the resulting dtype of a "+" operation between a scalar and >>> an >>> array can depend on the order of the arguments. This seems wrong. >>> >>> Tested with numpy 1.5.1 under Ubuntu 11.04 (64-bit). >>> >>> In [4]: (numpy.array(0, dtype='uint16') + numpy.array([1], >>> dtype='int8')).dtype >>> Out[4]: dtype('int16') >>> >>> In [6]: (numpy.array([1], dtype='int8') + numpy.array(0, >>> dtype='uint16')).dtype >>> Out[6]: dtype('int8') >>> >>> Thanks for any feedback, >>> >>> >> This has been fixed in the coming 1.6.0 release. >> >> In [2]: (numpy.array(0, dtype='uint16') + >> numpy.array([1],dtype='int8')).dtype >> Out[2]: dtype('int8') >> >> In [3]: (numpy.array([1], dtype='int8') + >> numpy.array(0,dtype='uint16')).dtype >> Out[3]: dtype('int8') >> >> Chuck >> > > Cool, thanks, I'll close the ticket then :) > > Already done. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From elfnor at gmail.com Wed May 11 01:28:22 2011 From: elfnor at gmail.com (Elfnor) Date: Tue, 10 May 2011 22:28:22 -0700 (PDT) Subject: [Numpy-discussion] Need to eliminate a nested loop Message-ID: <31591457.post@talk.nabble.com> Hi The following code produces the desired result but has a slow triple loop iterating over the matrix multiplication. I'm sure it can be eliminated with a neat indexing trick but I can't figure out how. Any suggestions please? ----------------------------- import numpy #define domain of function x = numpy.linspace(-5,5,64) y = numpy.linspace(-5,5,64) z = numpy.linspace(-5,5,64) #calculate f at each point in domain a = 5.0 b = 3.0 c = 2.0 #ellipsoid E = numpy.array([[1/a**2, 0 , 0 , 0 ], [ 0 ,1/b**2 , 0 , 0 ], [ 0 , 0 ,1/c**2, 0 ], [ 0 , 0 , 0 , -1 ]]) f = numpy.zeros((x.size,y.size,z.size)) for i,xi in enumerate(x): for j,yj in enumerate(y): for k,zk in enumerate(z): X = numpy.array([xi,yj,zk,1]) f[i,j,k] = numpy.dot(numpy.dot(X.T,E),X) ----------------------------------- Thanks Eleanor -- View this message in context: http://old.nabble.com/Need-to-eliminate-a-nested-loop-tp31591457p31591457.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From palla74 at gmail.com Wed May 11 05:42:56 2011 From: palla74 at gmail.com (Palla) Date: Wed, 11 May 2011 11:42:56 +0200 Subject: [Numpy-discussion] EuroPython: Early Bird will end in 2 days! Message-ID: Hi all, If you plan to attend, you could save quite a bit on registration fees! The end of Early bird is on May 12th, Friday, 23:59:59 CEST. We'd like to ask to you to forward this post to anyone that you feel may be interested. We have an amazing lineup of tutorials, events and talks. We have some excellent keynote speakers and a very complete partner program... but early bird registration ends in 2 days! Right now, you still get discounts on talks and tutorials so if you plan to attend Register Now: http://ep2011.europython.eu/registration/ While you are booking, remember to have a look at the partner program and our offer for a prepaid, data+voice+tethering SIM. We'd like to ask to you to forward this post to anyone that you feel may be interested. All the best, -- ->PALLA From e.antero.tammi at gmail.com Wed May 11 08:00:07 2011 From: e.antero.tammi at gmail.com (eat) Date: Wed, 11 May 2011 15:00:07 +0300 Subject: [Numpy-discussion] Need to eliminate a nested loop In-Reply-To: <31591457.post@talk.nabble.com> References: <31591457.post@talk.nabble.com> Message-ID: Hi, On Wed, May 11, 2011 at 8:28 AM, Elfnor wrote: > > Hi > > The following code produces the desired result but has a slow triple loop > iterating over the matrix multiplication. > > I'm sure it can be eliminated with a neat indexing trick but I can't figure > out how. > > Any suggestions please? > ----------------------------- > import numpy > #define domain of function > x = numpy.linspace(-5,5,64) > y = numpy.linspace(-5,5,64) > z = numpy.linspace(-5,5,64) > > #calculate f at each point in domain > a = 5.0 > b = 3.0 > c = 2.0 > #ellipsoid > E = numpy.array([[1/a**2, 0 , 0 , 0 ], > [ 0 ,1/b**2 , 0 , 0 ], > [ 0 , 0 ,1/c**2, 0 ], > [ 0 , 0 , 0 , -1 ]]) > > f = numpy.zeros((x.size,y.size,z.size)) > > for i,xi in enumerate(x): > for j,yj in enumerate(y): > for k,zk in enumerate(z): > X = numpy.array([xi,yj,zk,1]) > f[i,j,k] = numpy.dot(numpy.dot(X.T,E),X) > ----------------------------------- > Something like this: n= 64 u= np.linspace(-5, 5, n)[None, :] u0= u.repeat(n** 2)[None, :] u1= u.repeat(n)[None, :].repeat(n, axis= 0).reshape(1, -1) u2= u.repeat(n** 2, axis= 0).reshape(1, -1) U= np.r_[u0, u1, u2, np.ones((1, n** 3))] f= (np.dot(E, U)* U).sum(0).reshape(n, n, n) Regards,eat > > Thanks Eleanor > -- > View this message in context: > http://old.nabble.com/Need-to-eliminate-a-nested-loop-tp31591457p31591457.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed May 11 14:52:07 2011 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 May 2011 20:52:07 +0200 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: References: Message-ID: <4DCADAD7.4070607@molden.no> Den 09.05.2011 15:58, skrev Keith Goodman: > On Mon, May 9, 2011 at 1:46 AM, Pauli Virtanen wrote: >> Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: >>> I'm writing a function that accepts four possible dtypes: int32, int64, >>> float32, float64. The function will call a C extension (wrapped in >>> Cython). What are the equivalent C types? int, long, float, double, >>> respectively? Will that work on all systems? >> Long can be 32-bit or 64-bit, depending on the platform. >> >> The types available in Numpy are listed here, including >> the information which of them are compatible with which C types: >> >> http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html >> >> The C long seems not to be listed -- but it's the same as >> "Python int", i.e., np.int_ will work. >> >> IIRC, the Numpy type codes, dtype.kind, map directly to C types. > Does this mapping look right? > No. C int is at least 16 bits, C long is at least 32 bits. The size of long and size of int depend on compiler and platform. I'd write a small Cython function and ask the C compiler for an authorative answer. Alternatively you could use ctypes. This is 64-bit Python on Windows: >>> import ctypes >>> ctypes.sizeof(ctypes.c_long) 4 >>> ctypes.sizeof(ctypes.c_int) 4 Sturla From shish at keba.be Wed May 11 15:07:19 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 11 May 2011 15:07:19 -0400 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: <4DCADAD7.4070607@molden.no> References: <4DCADAD7.4070607@molden.no> Message-ID: 2011/5/11 Sturla Molden > Den 09.05.2011 15:58, skrev Keith Goodman: > > On Mon, May 9, 2011 at 1:46 AM, Pauli Virtanen wrote: > >> Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: > >>> I'm writing a function that accepts four possible dtypes: int32, int64, > >>> float32, float64. The function will call a C extension (wrapped in > >>> Cython). What are the equivalent C types? int, long, float, double, > >>> respectively? Will that work on all systems? > >> Long can be 32-bit or 64-bit, depending on the platform. > >> > >> The types available in Numpy are listed here, including > >> the information which of them are compatible with which C types: > >> > >> http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html > >> > >> The C long seems not to be listed -- but it's the same as > >> "Python int", i.e., np.int_ will work. > >> > >> IIRC, the Numpy type codes, dtype.kind, map directly to C types. > > Does this mapping look right? > > > > No. > > C int is at least 16 bits, C long is at least 32 bits. > > The size of long and size of int depend on compiler and platform. > > I'd write a small Cython function and ask the C compiler for an > authorative answer. > > Alternatively you could use ctypes. This is 64-bit Python on Windows: > > >>> import ctypes > >>> ctypes.sizeof(ctypes.c_long) > 4 > >>> ctypes.sizeof(ctypes.c_int) > 4 > > I think Keith's approach should work, as long as there is one C type listed on the url you mention that corresponds to your four dtypes. Something like (not tested at all): map = {} for dtype in ('int32', 'int64', 'float32', 'float64'): for ctype (N.byte, N.short, N.intc, N.long.long, ...): # List all compatible C types here if N.dtype(ctype) == dtype: map{dtype} = ctype break assert len(map) == 4 -=- Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed May 11 15:42:37 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 11 May 2011 14:42:37 -0500 Subject: [Numpy-discussion] Mapping of dtype to C types In-Reply-To: References: <4DCADAD7.4070607@molden.no> Message-ID: On Wed, May 11, 2011 at 2:07 PM, Olivier Delalleau wrote: > 2011/5/11 Sturla Molden >> >> Den 09.05.2011 15:58, skrev Keith Goodman: >> > On Mon, May 9, 2011 at 1:46 AM, Pauli Virtanen ?wrote: >> >> Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: >> >>> I'm writing a function that accepts four possible dtypes: int32, >> >>> int64, >> >>> float32, float64. The function will call a C extension (wrapped in >> >>> Cython). What are the equivalent C types? int, long, float, double, >> >>> respectively? Will that work on all systems? >> >> Long can be 32-bit or 64-bit, depending on the platform. >> >> >> >> The types available in Numpy are listed here, including >> >> the information which of them are compatible with which C types: >> >> >> >> http://docs.scipy.org/doc/numpy/reference/arrays.scalars.html >> >> >> >> The C long seems not to be listed -- but it's the same as >> >> "Python int", i.e., np.int_ will work. >> >> >> >> IIRC, the Numpy type codes, dtype.kind, map directly to C types. >> > Does this mapping look right? >> > >> >> No. >> >> C int is at least 16 bits, C long is at least 32 bits. >> >> The size of long and size of int depend on compiler and platform. >> >> I'd write a small Cython function and ask the C compiler for an >> authorative answer. >> >> Alternatively you could use ctypes. This is 64-bit Python on Windows: >> >> ?>>> import ctypes >> ?>>> ctypes.sizeof(ctypes.c_long) >> 4 >> ?>>> ctypes.sizeof(ctypes.c_int) >> 4 >> > > I think Keith's approach should work, as long as there is one C type listed > on the url you mention that corresponds to your four dtypes. > Something like (not tested at all): > > map = {} > for dtype in ('int32', 'int64', 'float32', 'float64'): > ? for ctype (N.byte, N.short, N.intc, N.long.long, ...): # List all > compatible C types here > ??? if N.dtype(ctype) == dtype: > ?????? map{dtype} = ctype > ?????? break > assert len(map) == 4 > > -=- Olivier > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > But that result is just due to how Microsoft implemented it's 64-bit support so it will not work for 64-bit Linux and other similar OSes. I thought that you would just use things like npy_intp (or intp) and npy_float as used by mtrandom: numpy/random/mtrand/numpy.pxi These are defined in numpy/core/include/numpy/ndarraytypes.h Bruce From chanyone.claudia at gmail.com Thu May 12 06:21:46 2011 From: chanyone.claudia at gmail.com (Claudia Chan Yone) Date: Thu, 12 May 2011 12:21:46 +0200 Subject: [Numpy-discussion] Problem with Numpy and array Message-ID: Hi, I have a problem with the Numpy module, but I think it is a very basic issue for many of you... I have a file with numerical data (2 columns, and 5 lignes) as : 1 2 3 4 ... ... And I woulid like to convert it in a matrix as : [[1,2] [3,4] ...] My python script is : fic=open('file.txt','r') ligne=fic.readlines() fic.close() m=numpy.array(ligne) and I get : ['1,2\n' '3,4'] So I cannot call m[0][0]... Even if I modify my text file with : [[1,2], [3,4] ...] I get : ['[[1,2],[3,4]]'] and I cannot call m[0][0]. Thank you very much for your help, Clo -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel.deval at gmail.com Thu May 12 06:30:03 2011 From: miguel.deval at gmail.com (Miguel de Val-Borro) Date: Thu, 12 May 2011 12:30:03 +0200 Subject: [Numpy-discussion] Problem with Numpy and array In-Reply-To: References: Message-ID: <20110512103001.GD7477@poincare.pc.linmpi.mpg.de> On Thu, May 12, 2011 at 12:21:46PM +0200, Claudia Chan Yone wrote: > Hi, > > I have a problem with the Numpy module, but I think it is a very basic issue > for many of you... > I have a file with numerical data (2 columns, and 5 lignes) as : > 1 2 > 3 4 > ... ... > > And I woulid like to convert it in a matrix as : > [[1,2] > [3,4] > ...] > > My python script is : > > fic=open('file.txt','r') > ligne=fic.readlines() > fic.close() Try using the loadtxt function in numpy >>> m = numpy.loadtxt('file.txt') array([[ 1., 2.], [ 3., 4.]]) Miguel From youngung.jeong at gmail.com Thu May 12 06:30:20 2011 From: youngung.jeong at gmail.com (Youngung Jeong) Date: Thu, 12 May 2011 19:30:20 +0900 Subject: [Numpy-discussion] Problem with Numpy and array In-Reply-To: References: Message-ID: Hello, Use np.loadtxt >>> a = np.loadtxt(filename) ##in case no rows to skip. >>> a = a.transpose() ##in your case, I guess you should transpose it.. Youngung Jeong On Thu, May 12, 2011 at 7:21 PM, Claudia Chan Yone < chanyone.claudia at gmail.com> wrote: > Hi, > > I have a problem with the Numpy module, but I think it is a very basic > issue > for many of you... > I have a file with numerical data (2 columns, and 5 lignes) as : > 1 2 > 3 4 > ... ... > > And I woulid like to convert it in a matrix as : > [[1,2] > [3,4] > ...] > > My python script is : > > fic=open('file.txt','r') > ligne=fic.readlines() > fic.close() > > m=numpy.array(ligne) > > and I get : > ['1,2\n' '3,4'] > > So I cannot call m[0][0]... > > Even if I modify my text file with : > [[1,2], > [3,4] > ...] > > I get : > > ['[[1,2],[3,4]]'] and I cannot call m[0][0]. > > Thank you very much for your help, > > > Clo > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From youngung.jeong at gmail.com Thu May 12 06:32:15 2011 From: youngung.jeong at gmail.com (Youngung Jeong) Date: Thu, 12 May 2011 19:32:15 +0900 Subject: [Numpy-discussion] Problem with Numpy and array In-Reply-To: References: Message-ID: Sorry you don't have to transpose the matrix... Youngung Jeong On Thu, May 12, 2011 at 7:30 PM, Youngung Jeong wrote: > Hello, > > Use np.loadtxt > > >>> a = np.loadtxt(filename) ##in case no rows to skip. > >>> a = a.transpose() ##in your case, I guess you should > transpose it.. > > > > Youngung Jeong > > > > > On Thu, May 12, 2011 at 7:21 PM, Claudia Chan Yone < > chanyone.claudia at gmail.com> wrote: > >> Hi, >> >> I have a problem with the Numpy module, but I think it is a very basic >> issue >> for many of you... >> I have a file with numerical data (2 columns, and 5 lignes) as : >> 1 2 >> 3 4 >> ... ... >> >> And I woulid like to convert it in a matrix as : >> [[1,2] >> [3,4] >> ...] >> >> My python script is : >> >> fic=open('file.txt','r') >> ligne=fic.readlines() >> fic.close() >> >> m=numpy.array(ligne) >> >> and I get : >> ['1,2\n' '3,4'] >> >> So I cannot call m[0][0]... >> >> Even if I modify my text file with : >> [[1,2], >> [3,4] >> ...] >> >> I get : >> >> ['[[1,2],[3,4]]'] and I cannot call m[0][0]. >> >> Thank you very much for your help, >> >> >> Clo >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at depagne.org Thu May 12 06:32:41 2011 From: eric at depagne.org (=?iso-8859-15?q?=C9ric_Depagne?=) Date: Thu, 12 May 2011 12:32:41 +0200 Subject: [Numpy-discussion] Problem with Numpy and array In-Reply-To: References: Message-ID: <201105121232.41970.eric@depagne.org> Le jeudi 12 mai 2011 12:21:46, Claudia Chan Yone a ?crit : > Hi, > > I have a problem with the Numpy module, but I think it is a very basic > issue for many of you... > I have a file with numerical data (2 columns, and 5 lignes) as : > 1 2 > 3 4 > ... ... > > And I woulid like to convert it in a matrix as : > [[1,2] > [3,4] > ...] > > My python script is : > > fic=open('file.txt','r') > ligne=fic.readlines() > fic.close() > > m=numpy.array(ligne) > > and I get : > ['1,2\n' '3,4'] > > So I cannot call m[0][0]... > > Even if I modify my text file with : > [[1,2], > [3,4] > ...] > > I get : > > ['[[1,2],[3,4]]'] and I cannot call m[0][0]. > > Thank you very much for your help, > > > Clo numpy.loadtxt() is probably what you need here. hth. ?ric. -- Un clavier azerty en vaut deux ---------------------------------------------------------- ?ric Depagne eric at depagne.org From chanyone.claudia at gmail.com Thu May 12 07:40:40 2011 From: chanyone.claudia at gmail.com (Claudia Chan Yone) Date: Thu, 12 May 2011 13:40:40 +0200 Subject: [Numpy-discussion] Problem with Numpy and array In-Reply-To: <201105121232.41970.eric@depagne.org> References: <201105121232.41970.eric@depagne.org> Message-ID: Merci beaucoup beaucoup!! 2011/5/12 ?ric Depagne > Le jeudi 12 mai 2011 12:21:46, Claudia Chan Yone a ?crit : > > Hi, > > > > I have a problem with the Numpy module, but I think it is a very basic > > issue for many of you... > > I have a file with numerical data (2 columns, and 5 lignes) as : > > 1 2 > > 3 4 > > ... ... > > > > And I woulid like to convert it in a matrix as : > > [[1,2] > > [3,4] > > ...] > > > > My python script is : > > > > fic=open('file.txt','r') > > ligne=fic.readlines() > > fic.close() > > > > m=numpy.array(ligne) > > > > and I get : > > ['1,2\n' '3,4'] > > > > So I cannot call m[0][0]... > > > > Even if I modify my text file with : > > [[1,2], > > [3,4] > > ...] > > > > I get : > > > > ['[[1,2],[3,4]]'] and I cannot call m[0][0]. > > > > Thank you very much for your help, > > > > > > Clo > numpy.loadtxt() is probably what you need here. > > hth. > > ?ric. > -- > Un clavier azerty en vaut deux > ---------------------------------------------------------- > ?ric Depagne eric at depagne.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgomezdans at gmail.com Thu May 12 09:14:49 2011 From: jgomezdans at gmail.com (Jose Gomez-Dans) Date: Thu, 12 May 2011 14:14:49 +0100 Subject: [Numpy-discussion] f2py complications Message-ID: Hi, We have some fortran code that we'd like to wrap using f2py. The code consists of a library that we compile into an .so file, and a file that wraps that library, which is then wrapped by f2py to provide a clean interface to the library. The process thus includes two steps: 1.- Use a makefile to compile the library 2.- Use f2py to create the python bindings from a fortran file, and link dynamically to the library created in the previous step. Now, I don't see why just a call to f2py shouldn't suffice (we don't really need to have an .so lying around, and it implies that we have to set eg $LD_LIBRARY_PATH to search for it). I thought about using a pyf file for this, and use the only :: option: $ f2py -h my_model.pyf -m my_model my_wrapper.f90 only: func1 func2 func3 : all_my_other_files.f even_more_files.f90 $ f2py -c -m my_model --f90flags="-fdefault-real-8 -O3 -march=native -m32" --f90exec=/usr/bin/gfortran --f77exec=/usr/bin/gfortran --opt=-O3 my_model.pyf This however, doesn't seem to work, with python complaining about missing things. If I put all my *.f and *f90 files after the my_model.pyf (which doesn't seem to have them in the file), I get undefined symbol errors when importing the .so in python. Additionally, it would be great to have this compilation in a distutils-friendly package, but how do you specify all these compiler flags? Thanks! Jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Thu May 12 12:37:19 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Thu, 12 May 2011 19:37:19 +0300 Subject: [Numpy-discussion] f2py complications In-Reply-To: References: Message-ID: On Thu, May 12, 2011 at 4:14 PM, Jose Gomez-Dans wrote: > Hi, > > We have some fortran code that we'd like to wrap using f2py. The code > consists of a library that we compile into an .so file, and a file that > wraps that library, which is then wrapped by f2py to provide a clean > interface to the library. The process thus includes two steps: > 1.- Use a makefile to compile the library > 2.- Use f2py to create the python bindings from a fortran file, and link > dynamically to the library created in the previous step. > > Now, I don't see why just a call to f2py shouldn't suffice (we don't really > need to have an .so lying around, and it implies that we have to set eg > $LD_LIBRARY_PATH to search for it). > It would be sufficient to just call f2py once. > I thought about using a pyf file for this, and use the only :: option: > $ f2py -h my_model.pyf -m my_model my_wrapper.f90 only: func1 func2 func3 > : all_my_other_files.f even_more_files.f90 > The above command (with using -h option) will just create the my_model.pyf file, no extra magic here, > $ f2py -c -m my_model --f90flags="-fdefault-real-8 -O3 -march=native > -m32" --f90exec=/usr/bin/gfortran --f77exec=/usr/bin/gfortran --opt=-O3 > my_model.pyf > > You need to include all .f and .f90 files to the f2py command and -m has no effect when .pyf is specified: f2py -c --f90flags="-fdefault-real-8 -O3 -march=native -m32" --f90exec=/usr/bin/gfortran --f77exec=/usr/bin/gfortran --opt=-O3 my_model.pyf all_my_other_files.f even_more_files.f90 This command (with .pyf file in command line) reads only the my_model.pyf file and creates wrapper code. It does not scan any Fortran files but only compiles them (with -c in command line) and links to the extension module. In fact, IIRC, the two above command lines can be joined to one: f2py -m my_model my_wrapper.f90 only: func1 func2 func3 : all_my_other_files.f even_more_files.f90 --f90flags="-fdefault-real-8 -O3 -march=native -m32" --f90exec=/usr/bin/gfortran --f77exec=/usr/bin/gfortran --opt=-O3 > This however, doesn't seem to work, with python complaining about missing > things. If I put all my *.f and *f90 files after the my_model.pyf (which > doesn't seem to have them in the file), I get undefined symbol errors when > importing the .so in python. > > Are you sure that you specified all needed Fortran files in the f2py command line? Where are these symbols defined that are reported to be undefined? Additionally, it would be great to have this compilation in a > distutils-friendly package, but how do you specify all these compiler flags? > > It is possible. See numpy/distutils/tests for examples. To use gfortran, run python setup.py build --fcompiler=gnu95 HTH, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu May 12 17:17:44 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 12 May 2011 16:17:44 -0500 Subject: [Numpy-discussion] RHE 4 and test failure with complex numbers (tickets: 1323, 1324 and 1325) Message-ID: <4DCC4E78.9090208@gmail.com> Hi, I think tickets 1323, 1324 and 1325 have a common source of error and, thus, are duplicates. http://projects.scipy.org/numpy/ticket/1323 http://projects.scipy.org/numpy/ticket/1324 http://projects.scipy.org/numpy/ticket/1325 These tickets could be closed just because Red Hat Enterprise Linux (RHE) 4 has been replaced by RHE 5 (first released 2007-03-14), which in turn is being replaced by RHE 6 (2010-11-10). I could not replicate this with numpy version 1.5.0 and 1.5.1 with Python 2.5.5 build with GCC 4.5.1 20100924 (Red Hat 4.5.1-4). As indicated by ticket 1325, this might be the gcc version used since RHE 4 used GCC 3.4 (RHE 5 is GCC 4.1 and RHE 6 is GCC 4.4 - http://www.redhat.com/rhel/compare/). The compiler versions used to build Python and numpy are needed unless someone can replicate these failures using either RHE 4 or older gcc version with the current numpy release. Bruce From srean.list at gmail.com Thu May 12 17:21:02 2011 From: srean.list at gmail.com (srean) Date: Thu, 12 May 2011 16:21:02 -0500 Subject: [Numpy-discussion] ufunc 's order of execution [relevant when output overlaps with input] Message-ID: Hi, is there a guarantee that ufuncs will execute left to right and in sequential order ? For instance is the following code standards compliant ? >>> import numpy as n >>> a=n.arange(0,5) array([0, 1, 2, 3, 4]) >>> n.add(a[0:-1], a[1:], a[0:-1]) array([1, 3, 5, 7]) The idea was to reuse and hence save space. The place where I write to is not accessed again. I am quite surprised that the following works correctly. >>>n.add.accumulate(a,out=a) I guess it uses a buffer and rebinds `a` to that buffer at the end. It is also faster than >>> n.add(a[0:-1], a[1:], a[1:]) --sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 12 17:36:01 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 12 May 2011 16:36:01 -0500 Subject: [Numpy-discussion] ufunc 's order of execution [relevant when output overlaps with input] In-Reply-To: References: Message-ID: On Thu, May 12, 2011 at 16:21, srean wrote: > Hi, > > ? is there a guarantee that ufuncs will execute left to right and in > sequential order ? By which you mean that it will traverse the elements in a 1D array left-to-right? No, this is not guaranteed. numpy reserves the right to reorder the computation for efficiency, to make the inner loop go down the axis with the smallest stride and in the direction of increasing memory addresses. We may or may not actually *do* that right now, but we reserve the right to. :-) It is possible that we can make an exception for inputs and outputs that overlap each other and pick a standard traversal. In those cases, the order of traversal can affect the semantics, so it might be nice to standardize on the most expected order and guarantee that it won't change. > For instance is the following code standards compliant ? > >>>> import numpy as n >>>> a=n.arange(0,5) > array([0, 1, 2, 3, 4]) >>>> n.add(a[0:-1], a[1:], a[0:-1]) > array([1, 3, 5, 7]) > > The idea was to reuse and hence save space. The place where I write to is > not accessed again. > > I am quite surprised that the following works correctly. > >>>>n.add.accumulate(a,out=a) > > I guess it uses a buffer and rebinds `a` to that buffer at the end. No, it's using `a` as the accumulator array. It's essentially doing the following: out[0] = a[0] + 0 out[1] = a[1] + out[0] out[2] = a[2] + out[1] ... It always reads from a[i] before it writes to out[i], so it's always consistent. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From srean.list at gmail.com Thu May 12 17:52:12 2011 From: srean.list at gmail.com (srean) Date: Thu, 12 May 2011 16:52:12 -0500 Subject: [Numpy-discussion] ufunc 's order of execution [relevant when output overlaps with input] In-Reply-To: References: Message-ID: > It is possible that we can make an exception for inputs and outputs > that overlap each other and pick a standard traversal. In those cases, > the order of traversal can affect the semantics, Exactly. If there is no overlap then it does not matter and can potentially be done in parallel. On the other hand if there is some standardized traversal that might allow one to write nested loops compactly. I dont really need it, but found the possibility quite intriguing. It always reads from a[i] before it writes to out[i], so it's always > consistent. > Ah I see, thanks. Should have seen through it. --sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From peck at us.ibm.com Thu May 12 18:03:19 2011 From: peck at us.ibm.com (Jon K Peck) Date: Thu, 12 May 2011 16:03:19 -0600 Subject: [Numpy-discussion] AUTO: Jon K Peck is out of the office (returning 05/18/2011) Message-ID: I am out of the office until 05/18/2011. I am out of the office traveling Wed - Thursday, May 11-12 and Saturday-Tuesday, May 14-17. I will have limited access to email during this time, so I will be delayed in responding. Note: This is an automated response to your message "NumPy-Discussion Digest, Vol 56, Issue 25" sent on 5/12/11 11:00:02. This is the only notification you will receive while this person is away. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri May 13 10:58:05 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 13 May 2011 09:58:05 -0500 Subject: [Numpy-discussion] Creating a 1-d structured array Message-ID: <4DCD46FD.2020101@gmail.com> Hi, How do you create a 'single' structured array using np.array()? Basically I am attempting to do something like this that does not work: a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) I realize that this is essentially redundant as if A is an 1-d array then a structured array with a named field 'foo' is the same thing - A would be A['foo'], just shorter. So if that is valid then a clearer error message is required to indicate this and provide the suitable error message to address ticket 1264 (http://projects.scipy.org/numpy/ticket/1264). $ python Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) Traceback (most recent call last): File "", line 1, in TypeError: expected a readable buffer object >>> np.array([(1,2), (3,4), (5,6)], dtype=np.dtype([('foo', int), ('foo2', int)])) array([(1, 2), (3, 4), (5, 6)], dtype=[('foo', '>> a=np.array([(1), (3), (5)], dtype=np.dtype([('foo', int)])) Traceback (most recent call last): File "", line 1, in TypeError: expected a readable buffer object >>> np.array([(1), (3), (5)], dtype=np.dtype(int)) array([1, 3, 5]) >>> np.array((1,2, 3,4, 5,6)) array([1, 2, 3, 4, 5, 6]) Thanks Bruce From josef.pktd at gmail.com Fri May 13 11:20:29 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 May 2011 11:20:29 -0400 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: <4DCD46FD.2020101@gmail.com> References: <4DCD46FD.2020101@gmail.com> Message-ID: On Fri, May 13, 2011 at 10:58 AM, Bruce Southey wrote: > Hi, > How do you create a 'single' structured array using np.array()? > Basically I am attempting to do something like this that does not work: > a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) > > I realize that this is essentially redundant as if A is an 1-d array > then a structured array with a named field 'foo' is the same thing - A > would be A['foo'], just shorter. > > So if that is valid then a clearer error message is required to indicate > this and provide the suitable error message to address ticket 1264 > (http://projects.scipy.org/numpy/ticket/1264). > > $ python > Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) > [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ?>>> import numpy as np > ?>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: expected a readable buffer object > ?>>> np.array([(1,2), (3,4), (5,6)], dtype=np.dtype([('foo', int), > ('foo2', int)])) > array([(1, 2), (3, 4), (5, 6)], > ? ? ? dtype=[('foo', ' ?>>> a=np.array([(1), (3), (5)], dtype=np.dtype([('foo', int)])) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: expected a readable buffer object > ?>>> np.array([(1), (3), (5)], dtype=np.dtype(int)) > array([1, 3, 5]) > ?>>> np.array((1,2, 3,4, 5,6)) > array([1, 2, 3, 4, 5, 6]) Using a view works, (and direct assignment of dtype) >>> a=np.array([1,2, 3,4, 5,6]).view(([('foo', int)])) >>> a array([(1,), (2,), (3,), (4,), (5,), (6,)], dtype=[('foo', '>> b = a.copy() >>> b array([(1,), (2,), (3,), (4,), (5,), (6,)], dtype=[('foo', '>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) Traceback (most recent call last): File "", line 1, in a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) TypeError: expected a readable buffer object >>> a1 = np.array([1,2, 3,4, 5,6]) >>> a1.dtype dtype('int32') >>> a1.dtype = np.dtype([('foo', int)]) >>> a1 array([(1,), (2,), (3,), (4,), (5,), (6,)], dtype=[('foo', ' > Thanks > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ondrej.marsalek at gmail.com Fri May 13 11:39:26 2011 From: ondrej.marsalek at gmail.com (Ondrej Marsalek) Date: Fri, 13 May 2011 17:39:26 +0200 Subject: [Numpy-discussion] Trying to turn warning to errors Message-ID: Dear all, I am trying to turn Python warnings to errors, but it does not work as expected with numpy. The following two piece of code work: $ python -W error -c 'import warnings; import numpy; warnings.warn("test", numpy.core.numeric.ComplexWarning)' Traceback (most recent call last): File "", line 1, in numpy.core.numeric.ComplexWarning: test $ python -W error -c 'import warnings; import numpy; raise numpy.core.numeric.ComplexWarning' Traceback (most recent call last): File "", line 1, in numpy.core.numeric.ComplexWarning while this does not (i.e. still produces just a warning): $ python -W error -c 'import numpy; x=numpy.ones(2); x+=1j' numpy.core.numeric.ComplexWarning: Casting complex values to real discards the imaginary part This is with Python 2.6.6 and numpy 1.5.1, but also confirmed with Python 2.7. Thanks for any suggested solutions, Ondrej Marsalek From pav at iki.fi Fri May 13 12:54:08 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 13 May 2011 16:54:08 +0000 (UTC) Subject: [Numpy-discussion] Trying to turn warning to errors References: Message-ID: Fri, 13 May 2011 17:39:26 +0200, Ondrej Marsalek wrote: [clip] > while this does not (i.e. still produces just a warning): > > $ python -W error -c 'import numpy; x=numpy.ones(2); x+=1j' > numpy.core.numeric.ComplexWarning: Casting complex values to real > discards the imaginary part > > This is with Python 2.6.6 and numpy 1.5.1, but also confirmed with > Python 2.7. Hmm, I suspect that there is some code in the type casting path that swallows exceptions... Seems to also occur on 1.6.0rc3 Can you file a bug ticket on this on the Numpy Trac, so it won't get lost? From derek at astro.physik.uni-goettingen.de Fri May 13 15:11:44 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 13 May 2011 21:11:44 +0200 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: Hi, just a comment since I first thought the solution below might not be what Bruce was looking for, but having realised it's probably what he's been asking for... On 13 May 2011, at 17:20, josef.pktd at gmail.com wrote: > On Fri, May 13, 2011 at 10:58 AM, Bruce Southey > wrote: >> Hi, >> How do you create a 'single' structured array using np.array()? >> Basically I am attempting to do something like this that does not >> work: >> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >> >> I realize that this is essentially redundant as if A is an 1-d array >> then a structured array with a named field 'foo' is the same thing >> - A >> would be A['foo'], just shorter. ... > > Using a view works, (and direct assignment of dtype) > >>>> a=np.array([1,2, 3,4, 5,6]).view(([('foo', int)])) >>>> a > array([(1,), (2,), (3,), (4,), (5,), (6,)], > dtype=[('foo', '>>> b = a.copy() >>>> b > array([(1,), (2,), (3,), (4,), (5,), (6,)], > dtype=[('foo', ' > >>>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) > > Traceback (most recent call last): > File "", line 1, in > a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) > TypeError: expected a readable buffer object > >>>> a1 = np.array([1,2, 3,4, 5,6]) >>>> a1.dtype > dtype('int32') >>>> a1.dtype = np.dtype([('foo', int)]) >>>> a1 > array([(1,), (2,), (3,), (4,), (5,), (6,)], > dtype=[('foo', '>> a['foo'] array([ 1, 2, 3, 4, 5, 6]) >>> a['foo'].shape (6,) >>> a.shape (6,) >>> a['foo']+np.arange(6) array([ 1, 3, 5, 7, 9, 11]) >>> a+np.arange(6) Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' >>> a[0]+1 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'numpy.void' and 'int' Thus I am wondering why broadcasting should not be possible in this case, and if it really isn't, the first error message certainly is not very helpful... (maybe inevitably though, since type() of a structured array is the same as for a "plain" array (unlike for a record array). Cheers, Derek From ondrej.marsalek at gmail.com Fri May 13 15:15:30 2011 From: ondrej.marsalek at gmail.com (Ondrej Marsalek) Date: Fri, 13 May 2011 21:15:30 +0200 Subject: [Numpy-discussion] Trying to turn warning to errors In-Reply-To: References: Message-ID: On Fri, May 13, 2011 at 18:54, Pauli Virtanen wrote: > Fri, 13 May 2011 17:39:26 +0200, Ondrej Marsalek wrote: > [clip] >> while this does not (i.e. still produces just a warning): >> >> $ python -W error -c 'import numpy; x=numpy.ones(2); x+=1j' >> numpy.core.numeric.ComplexWarning: Casting complex values to real >> discards the imaginary part >> >> This is with Python 2.6.6 and numpy 1.5.1, but also confirmed with >> Python 2.7. > > Hmm, I suspect that there is some code in the type casting path > that swallows exceptions... Seems to also occur on 1.6.0rc3 > > Can you file a bug ticket on this on the Numpy Trac, so it won't get lost? > If someone with an account could do this, that would be great. Thanks, Ondrej From josef.pktd at gmail.com Fri May 13 16:04:01 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 May 2011 16:04:01 -0400 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: On Fri, May 13, 2011 at 3:11 PM, Derek Homeier wrote: > Hi, > > just a comment since I first thought the solution below might not be > what Bruce > was looking for, but having realised it's probably what he's been > asking for... > > On 13 May 2011, at 17:20, josef.pktd at gmail.com wrote: > >> On Fri, May 13, 2011 at 10:58 AM, Bruce Southey >> wrote: >>> Hi, >>> How do you create a 'single' structured array using np.array()? >>> Basically I am attempting to do something like this that does not >>> work: >>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >>> >>> I realize that this is essentially redundant as if A is an 1-d array >>> then a structured array with a named field 'foo' is the same thing >>> - A >>> would be A['foo'], just shorter. > ... >> >> Using a view works, (and direct assignment of dtype) >> >>>>> a=np.array([1,2, 3,4, 5,6]).view(([('foo', int)])) >>>>> a >> array([(1,), (2,), (3,), (4,), (5,), (6,)], >> ? ? ?dtype=[('foo', '>>>> b = a.copy() >>>>> b >> array([(1,), (2,), (3,), (4,), (5,), (6,)], >> ? ? ?dtype=[('foo', '> >> >>>>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >> >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >> TypeError: expected a readable buffer object >> >>>>> a1 = np.array([1,2, 3,4, 5,6]) >>>>> a1.dtype >> dtype('int32') >>>>> a1.dtype = np.dtype([('foo', int)]) >>>>> a1 >> array([(1,), (2,), (3,), (4,), (5,), (6,)], >> ? ? ?dtype=[('foo', ' > This is a 1-d structured array, yet a is in fact not the same as > a['foo']: > > ?>>> a['foo'] > array([ 1, 2, 3, 4, 5, 6]) > ?>>> a['foo'].shape > (6,) > ?>>> a.shape > (6,) > ?>>> a['foo']+np.arange(6) > array([ 1, ?3, ?5, ?7, ?9, 11]) > ?>>> a+np.arange(6) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and > 'numpy.ndarray' funny things with subclasses, I saw this also in an error message in another package with a subclass of ndarray, should read: 'this ndarray-(sub)class' and 'that ndarray-(sub)class' for some messages it's just getting used to the "coding" "TypeError: expected a readable buffer object" translates into "I cannot interpret the data as the desired array" e.g. list of list or list of tuples > ?>>> a[0]+1 > Traceback (most recent call last): > ? File "", line 1, in > TypeError: unsupported operand type(s) for +: 'numpy.void' and 'int' > > Thus I am wondering why broadcasting should not be possible in this > case, Even a 1 column table is still a table (or a list of records), and a 1 item row is still a row. Josef > and if it really isn't, the first error message certainly is not very > helpful... (maybe > inevitably though, since type() of a structured array is the same as > for a "plain" > array (unlike for a record array). > > Cheers, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From derek at astro.physik.uni-goettingen.de Fri May 13 16:29:10 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Fri, 13 May 2011 22:29:10 +0200 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: <578A7004-636D-4A80-81CC-03AB7AA81D8C@astro.physik.uni-goettingen.de> On 13 May 2011, at 22:04, josef.pktd at gmail.com wrote: >> Thus I am wondering why broadcasting should not be possible in this >> case, > > Even a 1 column table is still a table (or a list of records), and a 1 > item row is still a row. > True, but even multiplying the shape (6, ) array e.g. with a shape (2, 6) array works just fine. I assume it does not work for a structured array because the records _could_ in principle have incompatible data types (like str), even though in this case they haven't... Cheers, Derek From bsouthey at gmail.com Fri May 13 17:03:59 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 13 May 2011 16:03:59 -0500 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: <4DCD9CBF.9040801@gmail.com> On 05/13/2011 03:04 PM, josef.pktd at gmail.com wrote: > On Fri, May 13, 2011 at 3:11 PM, Derek Homeier > wrote: >> Hi, >> >> just a comment since I first thought the solution below might not be >> what Bruce >> was looking for, but having realised it's probably what he's been >> asking for... >> >> On 13 May 2011, at 17:20, josef.pktd at gmail.com wrote: >> >>> On Fri, May 13, 2011 at 10:58 AM, Bruce Southey >>> wrote: >>>> Hi, >>>> How do you create a 'single' structured array using np.array()? >>>> Basically I am attempting to do something like this that does not >>>> work: >>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >>>> >>>> I realize that this is essentially redundant as if A is an 1-d array >>>> then a structured array with a named field 'foo' is the same thing >>>> - A >>>> would be A['foo'], just shorter. >> ... >>> Using a view works, (and direct assignment of dtype) >>> >>>>>> a=np.array([1,2, 3,4, 5,6]).view(([('foo', int)])) >>>>>> a >>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>> dtype=[('foo', '>>>>> b = a.copy() >>>>>> b >>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>> dtype=[('foo', '>> >>> >>>>>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >>> Traceback (most recent call last): >>> File "", line 1, in >>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >>> TypeError: expected a readable buffer object >>> >>>>>> a1 = np.array([1,2, 3,4, 5,6]) >>>>>> a1.dtype >>> dtype('int32') >>>>>> a1.dtype = np.dtype([('foo', int)]) >>>>>> a1 >>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>> dtype=[('foo', '> This is a 1-d structured array, yet a is in fact not the same as >> a['foo']: >> >> >>> a['foo'] >> array([ 1, 2, 3, 4, 5, 6]) >> >>> a['foo'].shape >> (6,) >> >>> a.shape >> (6,) >> >>> a['foo']+np.arange(6) >> array([ 1, 3, 5, 7, 9, 11]) >> >>> a+np.arange(6) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and >> 'numpy.ndarray' > funny things with subclasses, I saw this also in an error message in > another package with a subclass of ndarray, > should read: 'this ndarray-(sub)class' and 'that ndarray-(sub)class' > > for some messages it's just getting used to the "coding" > > "TypeError: expected a readable buffer object" translates into "I > cannot interpret the data as the desired array" > e.g. list of list or list of tuples > > >> >>> a[0]+1 >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unsupported operand type(s) for +: 'numpy.void' and 'int' >> >> Thus I am wondering why broadcasting should not be possible in this >> case, > Even a 1 column table is still a table (or a list of records), and a 1 > item row is still a row. > > Josef > >> and if it really isn't, the first error message certainly is not very >> helpful... (maybe >> inevitably though, since type() of a structured array is the same as >> for a "plain" >> array (unlike for a record array). >> >> Cheers, >> Derek >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Thanks for the replies, especially the dtype example Josef! So this should be reported as a bug? Bruce From josef.pktd at gmail.com Fri May 13 17:33:47 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 May 2011 17:33:47 -0400 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: <4DCD9CBF.9040801@gmail.com> References: <4DCD46FD.2020101@gmail.com> <4DCD9CBF.9040801@gmail.com> Message-ID: On Fri, May 13, 2011 at 5:03 PM, Bruce Southey wrote: > On 05/13/2011 03:04 PM, josef.pktd at gmail.com wrote: >> On Fri, May 13, 2011 at 3:11 PM, Derek Homeier >> ?wrote: >>> Hi, >>> >>> just a comment since I first thought the solution below might not be >>> what Bruce >>> was looking for, but having realised it's probably what he's been >>> asking for... >>> >>> On 13 May 2011, at 17:20, josef.pktd at gmail.com wrote: >>> >>>> On Fri, May 13, 2011 at 10:58 AM, Bruce Southey >>>> wrote: >>>>> Hi, >>>>> How do you create a 'single' structured array using np.array()? >>>>> Basically I am attempting to do something like this that does not >>>>> work: >>>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >>>>> >>>>> I realize that this is essentially redundant as if A is an 1-d array >>>>> then a structured array with a named field 'foo' is the same thing >>>>> - A >>>>> would be A['foo'], just shorter. >>> ... >>>> Using a view works, (and direct assignment of dtype) >>>> >>>>>>> a=np.array([1,2, 3,4, 5,6]).view(([('foo', int)])) >>>>>>> a >>>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>>> ? ? ? dtype=[('foo', '>>>>>> b = a.copy() >>>>>>> b >>>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>>> ? ? ? dtype=[('foo', '>>> >>>> >>>>>>> a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >>>> Traceback (most recent call last): >>>> ? File "", line 1, in >>>> ? ? a1 = np.array([1,2, 3,4, 5,6]).astype([('foo', int)]) >>>> TypeError: expected a readable buffer object >>>> >>>>>>> a1 = np.array([1,2, 3,4, 5,6]) >>>>>>> a1.dtype >>>> dtype('int32') >>>>>>> a1.dtype = np.dtype([('foo', int)]) >>>>>>> a1 >>>> array([(1,), (2,), (3,), (4,), (5,), (6,)], >>>> ? ? ? dtype=[('foo', '>> This is a 1-d structured array, yet a is in fact not the same as >>> a['foo']: >>> >>> ? >>> ?a['foo'] >>> array([ 1, 2, 3, 4, 5, 6]) >>> ? >>> ?a['foo'].shape >>> (6,) >>> ? >>> ?a.shape >>> (6,) >>> ? >>> ?a['foo']+np.arange(6) >>> array([ 1, ?3, ?5, ?7, ?9, 11]) >>> ? >>> ?a+np.arange(6) >>> Traceback (most recent call last): >>> ? ?File "", line 1, in >>> TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and >>> 'numpy.ndarray' >> funny things with subclasses, I saw this also in an error message in >> another package with a subclass of ndarray, >> should read: ?'this ndarray-(sub)class' and 'that ndarray-(sub)class' >> >> for some messages it's just getting used to the "coding" >> >> "TypeError: expected a readable buffer object" ?translates into "I >> cannot interpret the data as the desired array" >> e.g. list of list or list of tuples >> >> >>> ? >>> ?a[0]+1 >>> Traceback (most recent call last): >>> ? ?File "", line 1, in >>> TypeError: unsupported operand type(s) for +: 'numpy.void' and 'int' >>> >>> Thus I am wondering why broadcasting should not be possible in this >>> case, >> Even a 1 column table is still a table (or a list of records), and a 1 >> item row is still a row. >> >> Josef >> >>> and if it really isn't, the first error message certainly is not very >>> helpful... (maybe >>> inevitably though, since type() of a structured array is the same as >>> for a "plain" >>> array (unlike for a record array). >>> >>> Cheers, >>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Derek >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > Thanks for the replies, especially the dtype example Josef! > > So this should be reported as a bug? It's by design, a structured array takes a list of tuples and is an array of records, -- and tuples and records are not lists or arrays and don't behave that way. I have no opinion about whether it should be considered a bug, I just learned to live with it. Josef > > Bruce > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Fri May 13 17:38:55 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 13 May 2011 16:38:55 -0500 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: <4DCD46FD.2020101@gmail.com> References: <4DCD46FD.2020101@gmail.com> Message-ID: On Fri, May 13, 2011 at 09:58, Bruce Southey wrote: > Hi, > How do you create a 'single' structured array using np.array()? > Basically I am attempting to do something like this that does not work: > a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) > > I realize that this is essentially redundant as if A is an 1-d array > then a structured array with a named field 'foo' is the same thing - A > would be A['foo'], just shorter. > > So if that is valid then a clearer error message is required to indicate > this and provide the suitable error message to address ticket 1264 > (http://projects.scipy.org/numpy/ticket/1264). > > $ python > Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) > [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ?>>> import numpy as np > ?>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: expected a readable buffer object > ?>>> np.array([(1,2), (3,4), (5,6)], dtype=np.dtype([('foo', int), > ('foo2', int)])) > array([(1, 2), (3, 4), (5, 6)], > ? ? ? dtype=[('foo', ' ?>>> a=np.array([(1), (3), (5)], dtype=np.dtype([('foo', int)])) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: expected a readable buffer object You are missing the commas: [~] |12> a=np.array([(1,), (3,), (5,)], dtype=np.dtype([('foo', int)])) [~] |13> a array([(1,), (3,), (5,)], dtype=[('foo', ' Hi, I am pleased to announce the release of NumPy 1.6.0. This release is the result of 9 months of work, and includes many new features, performance improvements and bug fixes. Some highlights are: - Re-introduction of datetime dtype support to deal with dates in arrays. - A new 16-bit floating point type. - A new iterator, which improves performance of many functions. Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ For release notes see below. Thank you to everyone who contributed to this release. Enjoy, The NumPy developers ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py supports now automatic wrapping of Fortran routines that use two argument ``size`` function in dimension specifications. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= ``default error handling`` -------------------------- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Deprecated features =================== The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality will be replaced by the new "density" keyword. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat May 14 09:09:30 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 May 2011 07:09:30 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 In-Reply-To: References: Message-ID: On Sat, May 14, 2011 at 3:54 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the release of NumPy 1.6.0. This release is the > result of 9 months of work, and includes many new features, performance > improvements and bug fixes. Some highlights are: > > - Re-introduction of datetime dtype support to deal with dates in arrays. > - A new 16-bit floating point type. > - A new iterator, which improves performance of many functions. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ > For release notes see below. > > Thank you to everyone who contributed to this release. > > Enjoy, > The NumPy developers > > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > This release includes several new features as well as numerous bug fixes > and > improved documentation. It is backward compatible with the 1.5.0 release, > and > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > Highlights > ========== > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > * A new 16-bit floating point type. > > * A new iterator, which improves performance of many functions. > > > New features > ============ > > New 16-bit floating point type > ------------------------------ > > This release adds support for the IEEE 754-2008 binary16 format, available > as > the data type ``numpy.half``. Within Python, the type behaves similarly to > `float` or `double`, and C extensions can add support for it with the > exposed > half-float API. > > > New iterator > ------------ > > A new iterator has been added, replacing the functionality of the > existing iterator and multi-iterator with a single object and API. > This iterator works well with general memory layouts different from > C or Fortran contiguous, and handles both standard NumPy and > customized broadcasting. The buffering, automatic data type > conversion, and optional output parameters, offered by > ufuncs but difficult to replicate elsewhere, are now exposed by this > iterator. > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > ------------------------------------------------------------------------- > > Extend the number of polynomials available in the polynomial package. In > addition, a new ``window`` attribute has been added to the classes in > order to specify the range the ``domain`` maps to. This is mostly useful > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > are infinite and provides a more intuitive way to get the correct mapping > of values without playing unnatural tricks with the domain. > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py supports now automatic wrapping of Fortran routines > that use two argument ``size`` function in dimension specifications. > > > Other new functions > ------------------- > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > an array of flat indices, applying boundary modes to the indices. > > ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the > Einstein summation convention, many common multi-dimensional array > operations > can be represented in a simple fashion. This function provides a way > compute > such summations. > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an > array. > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions > expose > the underlying type promotion used by the ufuncs and other operations to > determine the types of outputs. These improve upon the > ``numpy.common_type`` > and ``numpy.mintypecode`` which provide similar functionality but do > not match the ufunc implementation. > > > Changes > ======= > > ``default error handling`` > -------------------------- > > The default error handling has been change from ``print`` to ``warn`` for > all except for ``underflow``, which remains as ``ignore``. > > > ``numpy.distutils`` > ------------------- > > Several new compilers are supported for building Numpy: the Portland Group > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel > C > compiler on Linux. > > > ``numpy.testing`` > ----------------- > > The testing framework gained ``numpy.testing.assert_allclose``, which > provides > a more convenient way to compare floating point arrays than > `assert_almost_equal`, `assert_approx_equal` and > `assert_array_almost_equal`. > > > ``C API`` > --------- > > In addition to the APIs for the new iterator and half data type, a number > of other additions have been made to the C API. The type promotion > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > ``NPY_CASTING`` has been added which controls what types of casts are > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > and ``PyArray_CanCastTypeTo``. A more flexible way to handle > conversion of arbitrary python objects into arrays is exposed by > ``PyArray_GetArrayParamsFromObject``. > > > Deprecated features > =================== > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its > functionality > will be replaced by the new "density" keyword. > > > Removed features > ================ > > ``numpy.fft`` > ------------- > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, > which were aliases for the same functions without the 'e' in the name, were > removed. > > > ``numpy.memmap`` > ---------------- > > The `sync()` and `close()` methods of memmap were removed. Use `flush()` > and > "del memmap" instead. > > > ``numpy.lib`` > ------------- > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > ``numpy.ma`` > ------------ > > Several deprecated items were removed from the ``numpy.ma`` module:: > > * ``numpy.ma.MaskedArray`` "raw_data" method > * ``numpy.ma.MaskedArray`` constructor "flag" keyword > * ``numpy.ma.make_mask`` "flag" keyword > * ``numpy.ma.allclose`` "fill_value" keyword > > > ``numpy.distutils`` > ------------------- > > The ``numpy.get_numpy_include`` function was removed, use > ``numpy.get_include`` > instead. > > The link is http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat May 14 11:45:12 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 May 2011 09:45:12 -0600 Subject: [Numpy-discussion] Documentation errors Message-ID: A number of the ufuncs in the reference doc are missing arguments, for example remainder(x1) Return element-wise remainder of division. mod(x1) Return element-wise remainder of division. fmod(x1) Return the element-wise remainder of division. absolute() Calculate the absolute value element-wise. rint() Round elements of the array to the nearest integer. sign() Returns an element-wise indication of the sign of a number. conj() Return the And this is true of binary ufuncs as well greater(x1) Return the truth value of (x1 > x2) element-wise. greater_equal(x1) Return the truth value of (x1 >= x2) element-wise. less(x1) Return the truth value of (x1 < x2) element-wise. less_equal(x1) Return the truth value of (x1 =< x2) element-wise. not_equal(x1) Return (x1 != x2) element-wise. equal(x1) Return (x1 == x2) element-wise. These are generated by the .. autosummary:: command, so the error probably lies there. Also, what is the easiest way to generate the reference guide for testing purposes? Can the autosummary command deal with classes that are only generated when modules are imported? I want to document the new polynomial classes but they are generated on the fly from templates. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat May 14 12:48:16 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 14 May 2011 16:48:16 +0000 (UTC) Subject: [Numpy-discussion] Documentation errors References: Message-ID: On Sat, 14 May 2011 09:45:12 -0600, Charles R Harris wrote: [clip] > These are generated by the .. autosummary:: command, so the error > probably lies there. The problem is that the signature of these routines is written as remainder(x1, x2[, out]) and not as remainder(x1, x2, out=None) on the first line of the docstrings. The autosummary part tries to truncate the argument list if it is too long, but its parser chokes on the nonstandard syntax. > Also, what is the easiest way to generate the reference guide for > testing purposes? cd doc export PYTHONPATH=wherever-you-installed-numpy make html or just make dist > Can the autosummary command deal with classes that are > only generated when modules are imported? I want to document the new > polynomial classes but they are generated on the fly from templates. Yes. Pauli From charlesr.harris at gmail.com Sat May 14 13:13:18 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 May 2011 11:13:18 -0600 Subject: [Numpy-discussion] Documentation errors In-Reply-To: References: Message-ID: On Sat, May 14, 2011 at 10:48 AM, Pauli Virtanen wrote: > On Sat, 14 May 2011 09:45:12 -0600, Charles R Harris wrote: > [clip] > > These are generated by the .. autosummary:: command, so the error > > probably lies there. > > The problem is that the signature of these routines is written as > > remainder(x1, x2[, out]) > > and not as > > remainder(x1, x2, out=None) > > on the first line of the docstrings. The autosummary part tries to > truncate the argument list if it is too long, but its parser chokes on > the nonstandard syntax. > > Looks like we also need to add the new keywords. There are a lot of them. > > Also, what is the easiest way to generate the reference guide for > > testing purposes? > > cd doc > export PYTHONPATH=wherever-you-installed-numpy > make html > > or just > > make dist > > Thanks. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun May 15 00:22:37 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 14 May 2011 22:22:37 -0600 Subject: [Numpy-discussion] Documentation errors In-Reply-To: References: Message-ID: On Sat, May 14, 2011 at 10:48 AM, Pauli Virtanen wrote: > On Sat, 14 May 2011 09:45:12 -0600, Charles R Harris wrote: > [clip] > > These are generated by the .. autosummary:: command, so the error > > probably lies there. > > The problem is that the signature of these routines is written as > > remainder(x1, x2[, out]) > > and not as > > remainder(x1, x2, out=None) > > OK, the format for that part of the signature is in line 4910 in ufunc_object.c. The question is, which should we fix, the format or the autosummary? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun May 15 04:32:17 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 15 May 2011 10:32:17 +0200 Subject: [Numpy-discussion] Documentation errors In-Reply-To: References: Message-ID: On Sun, May 15, 2011 at 6:22 AM, Charles R Harris wrote: > > > On Sat, May 14, 2011 at 10:48 AM, Pauli Virtanen wrote: > >> On Sat, 14 May 2011 09:45:12 -0600, Charles R Harris wrote: >> [clip] >> > These are generated by the .. autosummary:: command, so the error >> > probably lies there. >> >> The problem is that the signature of these routines is written as >> >> remainder(x1, x2[, out]) >> >> and not as >> >> remainder(x1, x2, out=None) >> >> > OK, the format for that part of the signature is in line 4910 in > ufunc_object.c. The question is, which should we fix, the format or the > autosummary? > > The format please. That [, out] never made sense to me. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun May 15 05:35:26 2011 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 15 May 2011 09:35:26 +0000 (UTC) Subject: [Numpy-discussion] Documentation errors References: Message-ID: On Sun, 15 May 2011 10:32:17 +0200, Ralf Gommers wrote: >> OK, the format for that part of the signature is in line 4910 in >> ufunc_object.c. The question is, which should we fix, the format or the >> autosummary? > > The format please. That [, out] never made sense to me. The problem here is that there is no standard format for this: ceil(a, out=None) does not work, since `out` is not a keyword parameter. Anyway, I have a fix for the autosummary parser, so I think both fixing it, and making the above arguments real keyword arguments makes sense. Pauli From ralf.gommers at googlemail.com Sun May 15 06:23:27 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 15 May 2011 12:23:27 +0200 Subject: [Numpy-discussion] RHE 4 and test failure with complex numbers (tickets: 1323, 1324 and 1325) In-Reply-To: <4DCC4E78.9090208@gmail.com> References: <4DCC4E78.9090208@gmail.com> Message-ID: On Thu, May 12, 2011 at 11:17 PM, Bruce Southey wrote: > Hi, > I think tickets 1323, 1324 and 1325 have a common source of error and, > thus, are duplicates. > http://projects.scipy.org/numpy/ticket/1323 > http://projects.scipy.org/numpy/ticket/1324 > http://projects.scipy.org/numpy/ticket/1325 > > These tickets could be closed just because Red Hat Enterprise Linux > (RHE) 4 has been replaced by RHE 5 (first released 2007-03-14), which in > turn is being replaced by RHE 6 (2010-11-10). > > I could not replicate this with numpy version 1.5.0 and 1.5.1 with > Python 2.5.5 build with GCC 4.5.1 20100924 (Red Hat 4.5.1-4). As > indicated by ticket 1325, this might be the gcc version used since RHE > 4 used GCC 3.4 (RHE 5 is GCC 4.1 and RHE 6 is GCC 4.4 - > http://www.redhat.com/rhel/compare/). The compiler versions used to > build Python and numpy are needed unless someone can replicate these > failures using either RHE 4 or older gcc version with the current numpy > release. > > Do we have a policy on supported GCC versions? Is closing as wontfix because GCC 3.4 is too old fine? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun May 15 12:39:56 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 15 May 2011 18:39:56 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 In-Reply-To: References: Message-ID: On Sat, May 14, 2011 at 3:09 PM, Charles R Harris wrote: > > > On Sat, May 14, 2011 at 3:54 AM, Ralf Gommers > wrote: > >> Hi, >> >> I am pleased to announce the release of NumPy 1.6.0. This release is the >> result of 9 months of work, and includes many new features, performance >> improvements and bug fixes. Some highlights are: >> >> - Re-introduction of datetime dtype support to deal with dates in >> arrays. >> - A new 16-bit floating point type. >> - A new iterator, which improves performance of many functions. >> >> > The link is http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ > > The OS X binaries are also up now. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun May 15 16:50:56 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 May 2011 14:50:56 -0600 Subject: [Numpy-discussion] Documentation errors In-Reply-To: References: Message-ID: On Sun, May 15, 2011 at 3:35 AM, Pauli Virtanen wrote: > On Sun, 15 May 2011 10:32:17 +0200, Ralf Gommers wrote: > >> OK, the format for that part of the signature is in line 4910 in > >> ufunc_object.c. The question is, which should we fix, the format or the > >> autosummary? > > > > The format please. That [, out] never made sense to me. > > The problem here is that there is no standard format for this: > > ceil(a, out=None) > > does not work, since `out` is not a keyword parameter. > > Anyway, I have a fix for the autosummary parser, so I think both fixing > it, and making the above arguments real keyword arguments makes sense. > > I think the formatting could be moved up into the Ufunc class in generate_umath.py or even into the ufunc docs themselves. That would make it more accessible than leaving it in the c code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Sun May 15 21:49:15 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 15 May 2011 20:49:15 -0500 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: On Fri, May 13, 2011 at 4:38 PM, Robert Kern wrote: > On Fri, May 13, 2011 at 09:58, Bruce Southey wrote: >> Hi, >> How do you create a 'single' structured array using np.array()? >> Basically I am attempting to do something like this that does not work: >> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >> >> I realize that this is essentially redundant as if A is an 1-d array >> then a structured array with a named field 'foo' is the same thing - A >> would be A['foo'], just shorter. >> >> So if that is valid then a clearer error message is required to indicate >> this and provide the suitable error message to address ticket 1264 >> (http://projects.scipy.org/numpy/ticket/1264). >> >> $ python >> Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) >> [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> ?>>> import numpy as np >> ?>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >> Traceback (most recent call last): >> ? File "", line 1, in >> TypeError: expected a readable buffer object >> ?>>> np.array([(1,2), (3,4), (5,6)], dtype=np.dtype([('foo', int), >> ('foo2', int)])) >> array([(1, 2), (3, 4), (5, 6)], >> ? ? ? dtype=[('foo', '> ?>>> a=np.array([(1), (3), (5)], dtype=np.dtype([('foo', int)])) >> Traceback (most recent call last): >> ? File "", line 1, in >> TypeError: expected a readable buffer object > > You are missing the commas: > > [~] > |12> a=np.array([(1,), (3,), (5,)], dtype=np.dtype([('foo', int)])) > > [~] > |13> a > array([(1,), (3,), (5,)], > ? ? ?dtype=[('foo', ' > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > I am still 'digesting' this and what it means. In any case, many thanks! Bruce From robert.kern at gmail.com Sun May 15 23:51:52 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 15 May 2011 22:51:52 -0500 Subject: [Numpy-discussion] Creating a 1-d structured array In-Reply-To: References: <4DCD46FD.2020101@gmail.com> Message-ID: On Sun, May 15, 2011 at 20:49, Bruce Southey wrote: > On Fri, May 13, 2011 at 4:38 PM, Robert Kern wrote: >> On Fri, May 13, 2011 at 09:58, Bruce Southey wrote: >>> Hi, >>> How do you create a 'single' structured array using np.array()? >>> Basically I am attempting to do something like this that does not work: >>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >>> >>> I realize that this is essentially redundant as if A is an 1-d array >>> then a structured array with a named field 'foo' is the same thing - A >>> would be A['foo'], just shorter. >>> >>> So if that is valid then a clearer error message is required to indicate >>> this and provide the suitable error message to address ticket 1264 >>> (http://projects.scipy.org/numpy/ticket/1264). >>> >>> $ python >>> Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) >>> [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> ?>>> import numpy as np >>> ?>>> a=np.array([1,2, 3,4, 5,6], dtype=np.dtype([('foo', int)])) >>> Traceback (most recent call last): >>> ? File "", line 1, in >>> TypeError: expected a readable buffer object >>> ?>>> np.array([(1,2), (3,4), (5,6)], dtype=np.dtype([('foo', int), >>> ('foo2', int)])) >>> array([(1, 2), (3, 4), (5, 6)], >>> ? ? ? dtype=[('foo', '>> ?>>> a=np.array([(1), (3), (5)], dtype=np.dtype([('foo', int)])) >>> Traceback (most recent call last): >>> ? File "", line 1, in >>> TypeError: expected a readable buffer object >> >> You are missing the commas: >> >> [~] >> |12> a=np.array([(1,), (3,), (5,)], dtype=np.dtype([('foo', int)])) >> >> [~] >> |13> a >> array([(1,), (3,), (5,)], >> ? ? ?dtype=[('foo', ' > I am still 'digesting' this and what it means. > In any case, many thanks! You make record arrays from lists of tuples (or lists of lists ... of tuples). If you want one-field records, you need one-element tuples. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From gael.varoquaux at normalesup.org Mon May 16 03:18:36 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 16 May 2011 09:18:36 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm Message-ID: <20110516071836.GB14868@phare.normalesup.org> Following a suggestion by Joseph, I am trying to implement the Jonker-Volgenant algorithm for best linear assignment in Python, using numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend too much time on this, as it not to solve a problem of mine, but for the scikits.learn. Thus I was wondering if someone had a BSD-licensed Python version of the algorithm that he would be willing to share. Cheers, Gael From a.h.jaffe at gmail.com Mon May 16 06:41:55 2011 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Mon, 16 May 2011 11:41:55 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 [Problems on OS X w/ Python 2.6 and XCode 4] In-Reply-To: References: Message-ID: <4DD0FF73.6080701@gmail.com> Dear all, I have OSX 10.6.7, XCode 4, and Python.org python 2.6.6 and 2.7.1, where 2.7 is 64-bit. With 2.7, easy_install successfully compiles and installs the package, both over the web and with an explicit download. With 2.6, there seems to be a problem with attempting to compile the PPC architecture, seen in this error. /usr/libexec/gcc/powerpc-apple-darwin10/4.0.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed If I explicitly set CFLAGS and LDFLAGS to have "-arch i386 -arch x86_64" in them as appropriate, it seems to work... Is this a correctable bug in the packaging, or just a quirk of my setup? Andrew On 15/05/2011 17:39, Ralf Gommers wrote: > > > On Sat, May 14, 2011 at 3:09 PM, Charles R Harris > > wrote: > > > > On Sat, May 14, 2011 at 3:54 AM, Ralf Gommers > > > wrote: > > Hi, > > I am pleased to announce the release of NumPy 1.6.0. This > release is the result of 9 months of work, and includes many new > features, performance improvements and bug fixes. Some > highlights are: > > - Re-introduction of datetime dtype support to deal with > dates in arrays. > - A new 16-bit floating point type. > - A new iterator, which improves performance of many functions. > > > The link is http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ > > The OS X binaries are also up now. > > Ralf > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From a.h.jaffe at gmail.com Mon May 16 06:41:55 2011 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Mon, 16 May 2011 11:41:55 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 [Problems on OS X w/ Python 2.6 and XCode 4] In-Reply-To: References: Message-ID: <4DD0FF73.6080701@gmail.com> Dear all, I have OSX 10.6.7, XCode 4, and Python.org python 2.6.6 and 2.7.1, where 2.7 is 64-bit. With 2.7, easy_install successfully compiles and installs the package, both over the web and with an explicit download. With 2.6, there seems to be a problem with attempting to compile the PPC architecture, seen in this error. /usr/libexec/gcc/powerpc-apple-darwin10/4.0.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed If I explicitly set CFLAGS and LDFLAGS to have "-arch i386 -arch x86_64" in them as appropriate, it seems to work... Is this a correctable bug in the packaging, or just a quirk of my setup? Andrew On 15/05/2011 17:39, Ralf Gommers wrote: > > > On Sat, May 14, 2011 at 3:09 PM, Charles R Harris > > wrote: > > > > On Sat, May 14, 2011 at 3:54 AM, Ralf Gommers > > > wrote: > > Hi, > > I am pleased to announce the release of NumPy 1.6.0. This > release is the result of 9 months of work, and includes many new > features, performance improvements and bug fixes. Some > highlights are: > > - Re-introduction of datetime dtype support to deal with > dates in arrays. > - A new 16-bit floating point type. > - A new iterator, which improves performance of many functions. > > > The link is http://sourceforge.net/projects/numpy/files/NumPy/1.6.0/ > > The OS X binaries are also up now. > > Ralf > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Mon May 16 08:53:40 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 May 2011 06:53:40 -0600 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110516071836.GB14868@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> Message-ID: On Mon, May 16, 2011 at 1:18 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > Following a suggestion by Joseph, I am trying to implement the > Jonker-Volgenant algorithm for best linear assignment in Python, using > numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend > too much time on this, as it not to solve a problem of mine, but for the > scikits.learn. Thus I was wondering if someone had a BSD-licensed Python > version of the algorithm that he would be willing to share. > > I was at a presentation two weeks ago where open source software for linear assignment was referenced, so I think some is available, although that part went by so quickly I might have missed something. I still don't know what linear assignment does, but that is another problem... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon May 16 09:07:08 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 May 2011 09:07:08 -0400 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> Message-ID: On Mon, May 16, 2011 at 8:53 AM, Charles R Harris wrote: > > > On Mon, May 16, 2011 at 1:18 AM, Gael Varoquaux > wrote: >> >> Following a suggestion by Joseph, I am trying to implement the >> Jonker-Volgenant algorithm for best linear assignment in Python, using >> numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend >> too much time on this, as it not to solve a problem of mine, but for the >> scikits.learn. Thus I was wondering if someone had a BSD-licensed Python >> version of the algorithm that he would be willing to share. >> > > I was at a presentation two weeks ago where open source software for linear > assignment was referenced, so I think some is available, although that part > went by so quickly I might have missed something. I still don't know what > linear assignment does, but that is another problem... OT for the implementation question: linear assignment in the simplest case is just optimal matching of pairs, male-female, seller-buyer, where each side has full ranking preferences about being matched with anyone on the other side. more general case, college and student matching, or interns and hospitals, where one side has a capacity to be matched with more than one from the other side. (more fun if you allow for side-payments, i.e. prices) Josef > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Mon May 16 10:04:29 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 16 May 2011 08:04:29 -0600 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> Message-ID: On Mon, May 16, 2011 at 7:07 AM, wrote: > On Mon, May 16, 2011 at 8:53 AM, Charles R Harris > wrote: > > > > > > On Mon, May 16, 2011 at 1:18 AM, Gael Varoquaux > > wrote: > >> > >> Following a suggestion by Joseph, I am trying to implement the > >> Jonker-Volgenant algorithm for best linear assignment in Python, using > >> numpy. Unsuprisingly, it is proving time-costly. I cannot afford to > spend > >> too much time on this, as it not to solve a problem of mine, but for the > >> scikits.learn. Thus I was wondering if someone had a BSD-licensed Python > >> version of the algorithm that he would be willing to share. > >> > > > > I was at a presentation two weeks ago where open source software for > linear > > assignment was referenced, so I think some is available, although that > part > > went by so quickly I might have missed something. I still don't know what > > linear assignment does, but that is another problem... > > OT for the implementation question: > linear assignment in the simplest case is just optimal matching of > pairs, male-female, seller-buyer, where each side has full ranking > preferences about being matched with anyone on the other side. > > more general case, college and student matching, or interns and > hospitals, where one side has a capacity to be matched with more than > one from the other side. > > (more fun if you allow for side-payments, i.e. prices) > > Thanks Josef, that makes sense in the context of the presentation I attended. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Mon May 16 11:49:58 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Mon, 16 May 2011 08:49:58 -0700 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110516071836.GB14868@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> Message-ID: On Mon, May 16, 2011 at 12:18 AM, Gael Varoquaux wrote: > Following a suggestion by Joseph, I am trying to implement the > Jonker-Volgenant algorithm for best linear assignment in Python, using > numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend > too much time on this, as it not to solve a problem of mine, but for the > scikits.learn. Thus I was wondering if someone had a BSD-licensed Python > version of the algorithm that he would be willing to share. There are a number available. The lemon graph library (http://lemon.cs.elte.hu/trac/lemon) has solvers for this problem and has python bindings. It's under the boost license (is that ok?). It might be a bit heavyweight for this, though, but it's great software. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From gael.varoquaux at normalesup.org Mon May 16 11:55:17 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 16 May 2011 17:55:17 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> Message-ID: <20110516155517.GN20930@phare.normalesup.org> Hey On Mon, May 16, 2011 at 08:49:58AM -0700, Hoyt Koepke wrote: > On Mon, May 16, 2011 at 12:18 AM, Gael Varoquaux > wrote: > > Jonker-Volgenant algorithm for best linear assignment in Python, using > > numpy. Unsuprisingly, it is proving time-costly. I cannot afford to spend > > too much time on this, as it not to solve a problem of mine, but for the > > scikits.learn. Thus I was wondering if someone had a BSD-licensed Python > > version of the algorithm that he would be willing to share. > There are a number available. The lemon graph library > (http://lemon.cs.elte.hu/trac/lemon) has solvers for this problem and > has python bindings. It's under the boost license (is that ok?). It > might be a bit heavyweight for this, though, but it's great software. Thanks for the suggestion. Unfortunately, I do not want to add compiled code (or an external dependency) to the scikit for this solver, as it is a fairly minor step for us. I particular chances are that it will never be a computational bottleneck on our poblems. Cheers, Gael From hoytak at stat.washington.edu Mon May 16 12:15:21 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Mon, 16 May 2011 09:15:21 -0700 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110516155517.GN20930@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> Message-ID: In this case, lemon exists only as C++ header files, with no compiled code. A few cython functions should make it pretty simple. However, yeah, it is a bit heavyweight. > Thanks for the suggestion. Unfortunately, I do not want to add compiled > code (or an external dependency) to the scikit for this solver, as it is > a fairly minor step for us. I particular chances are that it will never > be a computational bottleneck on our poblems. If that's the case, I suggest just using the Hungarian algorithm. It's a little slower on large graphs, but it still is pretty quick. And there's tons of implementations out there, as it's often a standard coding project in undergrad algorithms courses. http://en.wikipedia.org/wiki/Hungarian_algorithm has links to a few. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From gael.varoquaux at normalesup.org Mon May 16 12:28:51 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 16 May 2011 18:28:51 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> Message-ID: <20110516162851.GL11221@phare.normalesup.org> On Mon, May 16, 2011 at 09:15:21AM -0700, Hoyt Koepke wrote: > > Thanks for the suggestion. Unfortunately, I do not want to add compiled > > code (or an external dependency) to the scikit for this solver, as it is > > a fairly minor step for us. I particular chances are that it will never > > be a computational bottleneck on our poblems. > If that's the case, I suggest just using the Hungarian algorithm. I might go that way: I already have pure-Python code that implements it and that I have been using for a year or so. > It's a little slower on large graphs, but it still is pretty quick. Can you put any numbers on the difference, or rule of thumbs? Is there an algorithmic complexity difference? If so, what kind of dependency are we talking about? Is it only in the prefactors? Sorry for drowning you with questions, but linear programming has always seem a dark craft to me, due to my lack of training on these issues. Cheers, Gael From hoytak at stat.washington.edu Mon May 16 13:03:09 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Mon, 16 May 2011 10:03:09 -0700 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110516162851.GL11221@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> Message-ID: > I might go that way: I already have pure-Python code that implements it > and that I have been using for a year or so. Fair enough -- though you'd probably get a big speed up moving to cython. >> It's a little slower on large graphs, but it still is pretty quick. > > Can you put any numbers on the difference, or rule of thumbs? Is there an > algorithmic complexity difference? If so, what kind of dependency are we > talking about? Is it only in the prefactors? Well, the hungarian algorithm has a theoretical upper bound of O(n^3), with n being the number of nodes, which is pretty much the best you can do if you have a dense graph and make no assumptions on capacities. If you have a sparse graph, other algorithms can exploit that structure a bit more efficiently (e.g. lemon's implementation is O(nm logn), with m being the number of edges). Other algorithms can use capacities and include the capacity bounds as part of the theoretic performance of the algorithms, e.g. http://arxiv.org/abs/1105.1569. However, all these bounds are generally very much upper bounds and actual practical performance varies greatly. The hungarian algorithm is popular mainly because it's the best one that can be coded up efficiently in a few hundred lines of code. Hope that answers your questions :-). --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From ralf.gommers at googlemail.com Mon May 16 13:45:12 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 16 May 2011 19:45:12 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 [Problems on OS X w/ Python 2.6 and XCode 4] In-Reply-To: <4DD0FF73.6080701@gmail.com> References: <4DD0FF73.6080701@gmail.com> Message-ID: On Mon, May 16, 2011 at 12:41 PM, Andrew Jaffe wrote: > Dear all, > > I have OSX 10.6.7, XCode 4, and Python.org python 2.6.6 and 2.7.1, where > 2.7 is 64-bit. > > With 2.7, easy_install successfully compiles and installs the package, > both over the web and with an explicit download. > > With 2.6, there seems to be a problem with attempting to compile the PPC > architecture, seen in this error. > Please just use "python setup.py install", easy_install is very unreliable. > > /usr/libexec/gcc/powerpc-apple-darwin10/4.0.1/as: assembler > (/usr/bin/../libexec/gcc/darwin/ppc/as or > /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not > installed > > XCode 4 does not support PPC, use version 3.2. Then it should work as advertized. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Mon May 16 14:30:23 2011 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Mon, 16 May 2011 19:30:23 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 [Problems on OS X w/ Python 2.6 and XCode 4] In-Reply-To: References: <4DD0FF73.6080701@gmail.com> Message-ID: On 16/05/2011 18:45, Ralf Gommers wrote: > > On Mon, May 16, 2011 at 12:41 PM, Andrew Jaffe > wrote: > > Dear all, > > I have OSX 10.6.7, XCode 4, and Python.org python 2.6.6 and 2.7.1, where > 2.7 is 64-bit. > > With 2.7, easy_install successfully compiles and installs the package, > both over the web and with an explicit download. > > With 2.6, there seems to be a problem with attempting to compile the PPC > architecture, seen in this error. > > Please just use "python setup.py install", easy_install is very unreliable. > > > /usr/libexec/gcc/powerpc-apple-darwin10/4.0.1/as: assembler > (/usr/bin/../libexec/gcc/darwin/ppc/as or > /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not > installed > > XCode 4 does not support PPC, use version 3.2. Then it should work as > advertized. Aha, thanks! But they're both installed, but how do force this? (Does 64-bit 2.7 not care about PPC?) Yours, Andrew From virtual at gmx.de Mon May 16 14:55:30 2011 From: virtual at gmx.de (Nikolas Tautenhahn) Date: Mon, 16 May 2011 20:55:30 +0200 Subject: [Numpy-discussion] convert integer into bit array Message-ID: <4DD17322.5080708@gmx.de> Hi, for some research, I need to convert lots of integers into their bit representation - but as a bit array, not a string like numpy.binary_repr() returns it. So instead of In [22]: numpy.binary_repr(23) Out[22]: '10111 I'd need: numpy.binary_magic(23) Out: array([ True, False, True, True, True], dtype=bool) is there any way to do this efficiently? best regards, Nik From ralf.gommers at googlemail.com Mon May 16 14:58:11 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 16 May 2011 20:58:11 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.6.0 [Problems on OS X w/ Python 2.6 and XCode 4] In-Reply-To: References: <4DD0FF73.6080701@gmail.com> Message-ID: On Mon, May 16, 2011 at 8:30 PM, Andrew Jaffe wrote: > On 16/05/2011 18:45, Ralf Gommers wrote: > > > > On Mon, May 16, 2011 at 12:41 PM, Andrew Jaffe > > wrote: > > > > Dear all, > > > > I have OSX 10.6.7, XCode 4, and Python.org python 2.6.6 and 2.7.1, > where > > 2.7 is 64-bit. > > > > With 2.7, easy_install successfully compiles and installs the > package, > > both over the web and with an explicit download. > > > > With 2.6, there seems to be a problem with attempting to compile the > PPC > > architecture, seen in this error. > > > > Please just use "python setup.py install", easy_install is very > unreliable. > > > > > > /usr/libexec/gcc/powerpc-apple-darwin10/4.0.1/as: assembler > > (/usr/bin/../libexec/gcc/darwin/ppc/as or > > /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not > > installed > > > > XCode 4 does not support PPC, use version 3.2. Then it should work as > > advertized. > > Aha, thanks! > > But they're both installed, but how do force this? You have to figure out where the right compiler is located. Then try "export CC=/path/to/compiler". Or if you have MacPorts installed you can use its gcc_select. Or google for another solution. > (Does 64-bit 2.7 not care about PPC?) > No, it's an Intel 32 + 64-bit bundle. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Mon May 16 15:01:51 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Mon, 16 May 2011 22:01:51 +0300 Subject: [Numpy-discussion] convert integer into bit array In-Reply-To: <4DD17322.5080708@gmx.de> References: <4DD17322.5080708@gmx.de> Message-ID: Hi, I have used bitarray for that http://pypi.python.org/pypi/bitarray/ Here http://code.google.com/p/pylibtiff/source/browse/#svn%2Ftrunk%2Flibtiff%2Fbitarray-0.3.5-numpy you can find bitarray with numpy support. HTH, Pearu On Mon, May 16, 2011 at 9:55 PM, Nikolas Tautenhahn wrote: > Hi, > > for some research, I need to convert lots of integers into their bit > representation - but as a bit array, not a string like > numpy.binary_repr() returns it. > > So instead of > In [22]: numpy.binary_repr(23) > Out[22]: '10111 > > > I'd need: > numpy.binary_magic(23) > Out: array([ True, False, True, True, True], dtype=bool) > > is there any way to do this efficiently? > > best regards, > Nik > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jthomas1 at decisionanalyst.com Mon May 16 15:09:20 2011 From: jthomas1 at decisionanalyst.com (Justin Thomas) Date: Mon, 16 May 2011 14:09:20 -0500 Subject: [Numpy-discussion] Performance issues with numpy cross tabulation Message-ID: Hello, I am having trouble with performance when trying to create a cross tabulation using numpy. Ideally, I would calculate each cell in the cross tabulation separately because this gives me the greatest amount of flexibility. I have included some sample code as a reference and am really looking for better approaches to the simpleLoop method. So far the histogram2d and histogramdd methods seem to outperform any code I write by a factor of about 100, at least. I chalk this up to I just don't understand enough about numpy, yet. Any help would be appreciated. Here is the test code: import numpy as np import time import random # Create a simple loop and count up the number of matching cases # Basic cross tabulation or histogram of the data # This approach is prefered because of the need to customize the calculation potentially for each cell. def simpleLoop(c): #number of items per inner loop a_cnt = len(np.unique(c[:,0])) b_cnt = len(np.unique(c[:,1])) idx = 0 result = np.zeros(b_cnt * a_cnt) for i in np.unique(c[:,0]): for j in np.unique(c[:,1]): result[idx] = np.sum(1*(c[:,0] == i) & (c[:,1] == j)) idx += 1 result.resize(len(result)/b_cnt,b_cnt) return result # Use numpys histogram method to calculate the matrix of combinations and the number of cases in each one. def simpleHistogram(c): #number of items per inner loop return np.histogramdd((c[:,0],c[:,1]), bins=[np.unique(c[:, 0]),range(1,11)]) # Variation1 of simple histogram def simpleHistogram1(c): #number of items per inner loop results = [] for i in np.unique(c[:,1]): results.append(np.histogramdd((c[:,0][c[:,1]==i]), bins=[np.unique(c[:,0])]) or 0) return np.column_stack([result[0] for result in results]) if __name__ == '__main__': a = np.random.randint(1,900,200000) b = np.random.randint(1,10,200000) c = np.column_stack((a,b)) print '---- Simple Loop ----' start = time.time() results = simpleLoop(c) print results[0] print time.time() - start print '---- Histogram dd no looping ----' start = time.time() results = simpleHistogram(c) print results[0][0] print time.time() - start print '---- Histogram run 1 time for each item in column 1 (10 times) ----' start = time.time() results = simpleHistogram1(c) print results[0] print time.time() - start -------------- next part -------------- An HTML attachment was scrubbed... URL: From virtual at gmx.de Mon May 16 17:04:17 2011 From: virtual at gmx.de (Nikolas Tautenhahn) Date: Mon, 16 May 2011 23:04:17 +0200 Subject: [Numpy-discussion] convert integer into bit array In-Reply-To: References: <4DD17322.5080708@gmx.de> Message-ID: <4DD19151.7090601@gmx.de> Hi, > Here > > http://code.google.com/p/pylibtiff/source/browse/#svn%2Ftrunk%2Flibtiff%2Fbitarray-0.3.5-numpy > > you can find bitarray with numpy support. > Thanks, that looks promising - to get a numpy array, I need to do numpy.array(bitarray.bitarray(numpy.binary_repr(i, l))) for an integer i and l with i < 2**l, right? best regards, Nik From elfnor at gmail.com Mon May 16 18:33:53 2011 From: elfnor at gmail.com (Elfnor) Date: Mon, 16 May 2011 15:33:53 -0700 (PDT) Subject: [Numpy-discussion] Need to eliminate a nested loop In-Reply-To: <31591457.post@talk.nabble.com> References: <31591457.post@talk.nabble.com> Message-ID: <31633558.post@talk.nabble.com> Thanks that works well Execution time is down from ~3 seconds to 0.06 seconds Eleanor -- View this message in context: http://old.nabble.com/Need-to-eliminate-a-nested-loop-tp31591457p31633558.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From thomas.p.krauss at gmail.com Mon May 16 19:06:25 2011 From: thomas.p.krauss at gmail.com (Tom Krauss) Date: Mon, 16 May 2011 18:06:25 -0500 Subject: [Numpy-discussion] convert integer into bit array In-Reply-To: <4DD19151.7090601@gmx.de> References: <4DD17322.5080708@gmx.de> <4DD19151.7090601@gmx.de> Message-ID: Here's a one-liner: In [31]: x=np.arange(-8,8) In [32]: x Out[32]: array([-8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7]) In [33]: 1&(x[:,np.newaxis]/2**np.arange(3,-1,-1)) Out[33]: array([[1, 0, 0, 0], [1, 0, 0, 1], [1, 0, 1, 0], [1, 0, 1, 1], [1, 1, 0, 0], [1, 1, 0, 1], [1, 1, 1, 0], [1, 1, 1, 1], [0, 0, 0, 0], [0, 0, 0, 1], [0, 0, 1, 0], [0, 0, 1, 1], [0, 1, 0, 0], [0, 1, 0, 1], [0, 1, 1, 0], [0, 1, 1, 1]]) On Mon, May 16, 2011 at 4:04 PM, Nikolas Tautenhahn wrote: > Hi, > >> Here >> >> http://code.google.com/p/pylibtiff/source/browse/#svn%2Ftrunk%2Flibtiff%2Fbitarray-0.3.5-numpy >> >> you can find bitarray with numpy support. >> > > Thanks, that looks promising - to get a numpy array, I need to do > > numpy.array(bitarray.bitarray(numpy.binary_repr(i, l))) > > for an integer i and l with i < 2**l, right? > > best regards, > Nik > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From noh.hyunsoo at gmail.com Mon May 16 20:09:31 2011 From: noh.hyunsoo at gmail.com (Hyunsoo Noh) Date: Tue, 17 May 2011 00:09:31 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?PyArray=5FContiguousFromObject_=26_P?= =?utf-8?q?yDECREF?= References: <3FCADC96.3050504@ims.u-tokyo.ac.jp> Message-ID: Hi, I think it should give the memory leaking. I have a similar issue. I am wondering if it works fine when using Py_DECREF() in the code. In my case, I tried to decref the PyArrayObject using Py_DECREF() but it gives a segmentation fault. Instead, if I use the PyArray_XDECREF(), it doesn't give any segmentation fault. However, I am not sure whether it gives any memory leaking issue when I use PyArray_XDECREF(). Thanks. Hyunsoo Noh From pearu.peterson at gmail.com Tue May 17 01:05:17 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 17 May 2011 08:05:17 +0300 Subject: [Numpy-discussion] convert integer into bit array In-Reply-To: <4DD19151.7090601@gmx.de> References: <4DD17322.5080708@gmx.de> <4DD19151.7090601@gmx.de> Message-ID: On Tue, May 17, 2011 at 12:04 AM, Nikolas Tautenhahn wrote: > Hi, > > > Here > > > > > http://code.google.com/p/pylibtiff/source/browse/#svn%2Ftrunk%2Flibtiff%2Fbitarray-0.3.5-numpy > > > > you can find bitarray with numpy support. > > > > Thanks, that looks promising - to get a numpy array, I need to do > > numpy.array(bitarray.bitarray(numpy.binary_repr(i, l))) > > for an integer i and l with i < 2**l, right? > > If l < 64 and little endian is assumed then you can use the fromword(i, l) method: >>> from libtiff import bitarray >>> barr = bitarray.bitarray(0, 'little') >>> barr.fromword(3,4) >>> barr bitarray('1100') that will append 4 bits of the value 3 to the bitarray barr. Also check out various bitarray `to*` and `from*` methods. HTH, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Tue May 17 01:14:25 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Tue, 17 May 2011 08:14:25 +0300 Subject: [Numpy-discussion] convert integer into bit array In-Reply-To: References: <4DD17322.5080708@gmx.de> <4DD19151.7090601@gmx.de> Message-ID: On Tue, May 17, 2011 at 8:05 AM, Pearu Peterson wrote: > > > On Tue, May 17, 2011 at 12:04 AM, Nikolas Tautenhahn wrote: > >> Hi, >> >> > Here >> > >> > >> http://code.google.com/p/pylibtiff/source/browse/#svn%2Ftrunk%2Flibtiff%2Fbitarray-0.3.5-numpy >> > >> > you can find bitarray with numpy support. >> > >> >> Thanks, that looks promising - to get a numpy array, I need to do >> >> numpy.array(bitarray.bitarray(numpy.binary_repr(i, l))) >> >> for an integer i and l with i < 2**l, right? >> >> > If l < 64 and little endian is assumed then you can use the > > fromword(i, l) > > method: > > >>> from libtiff import bitarray > >>> barr = bitarray.bitarray(0, 'little') > >>> barr.fromword(3,4) > >>> barr > bitarray('1100') > > that will append 4 bits of the value 3 to the bitarray barr. > >>> numpy.array(barr) array([ True, True, False, False], dtype=bool) to complete the example... Pearu > Also check out various bitarray `to*` and `from*` methods. > > HTH, > Pearu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue May 17 03:03:25 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 17 May 2011 09:03:25 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> Message-ID: <20110517070325.GA18946@phare.normalesup.org> On Mon, May 16, 2011 at 10:03:09AM -0700, Hoyt Koepke wrote: > > I might go that way: I already have pure-Python code that implements it > > and that I have been using for a year or so. > Fair enough -- though you'd probably get a big speed up moving to cython. Indeed. If this is needed, we'll consider it. > Well, the hungarian algorithm has a theoretical upper bound of O(n^3), > with n being the number of nodes, which is pretty much the best you > can do if you have a dense graph and make no assumptions on > capacities. OK, your input is making my motivation to fight with Jonker-Volgenant go down. I'll see with the other people involved if we still target Jonger-Volgenant, or if we stick with the hungarian algorithm, in which case the problem is solved. > Hope that answers your questions :-). It does. Thanks a lot, it is very useful to have expert knowledge available. Gael From ondrej.marsalek at gmail.com Tue May 17 07:35:22 2011 From: ondrej.marsalek at gmail.com (Ondrej Marsalek) Date: Tue, 17 May 2011 13:35:22 +0200 Subject: [Numpy-discussion] Trying to turn warning to errors In-Reply-To: References: Message-ID: On Fri, May 13, 2011 at 21:15, Ondrej Marsalek wrote: > On Fri, May 13, 2011 at 18:54, Pauli Virtanen wrote: >> Fri, 13 May 2011 17:39:26 +0200, Ondrej Marsalek wrote: >> [clip] >>> while this does not (i.e. still produces just a warning): >>> >>> $ python -W error -c 'import numpy; x=numpy.ones(2); x+=1j' >>> numpy.core.numeric.ComplexWarning: Casting complex values to real >>> discards the imaginary part >>> >>> This is with Python 2.6.6 and numpy 1.5.1, but also confirmed with >>> Python 2.7. >> >> Hmm, I suspect that there is some code in the type casting path >> that swallows exceptions... Seems to also occur on 1.6.0rc3 >> >> Can you file a bug ticket on this on the Numpy Trac, so it won't get lost? >> > > If someone with an account could do this, that would be great. > OK, I made an account and created the ticket. Ondrej From wkerzendorf at googlemail.com Tue May 17 10:03:06 2011 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Wed, 18 May 2011 00:03:06 +1000 Subject: [Numpy-discussion] fitting legendre polynomial Message-ID: <4DD2801A.8070000@gmail.com> Hello, The science package I'm using fits legendre polynomials to data. I heard it is more stable than the normal polynomials for a fit. Is there a polyfit for legendre polynomials? How do I do that with the new legendre polynomials module? Thanks Wolfgang From charlesr.harris at gmail.com Tue May 17 10:37:31 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 May 2011 08:37:31 -0600 Subject: [Numpy-discussion] fitting legendre polynomial In-Reply-To: <4DD2801A.8070000@gmail.com> References: <4DD2801A.8070000@gmail.com> Message-ID: On Tue, May 17, 2011 at 8:03 AM, Wolfgang Kerzendorf < wkerzendorf at googlemail.com> wrote: > Hello, > > The science package I'm using fits legendre polynomials to data. I heard > it is more stable than the normal polynomials for a fit. Is there a > polyfit for legendre polynomials? How do I do that with the new legendre > polynomials module? > > In [1]: from numpy.polynomial import Legendre as L In [2]: x = np.linspace(0,10) In [3]: y = x*(x - 5)*(x - 8) In [4]: p = L.fit(x, y, 3) In [5]: plot(*p.linspace()) Out[5]: [] In [6]: p.coef Out[6]: array([ 1.66666667e+01, -1.36479923e-14, 3.33333333e+01, 5.00000000e+01]) In [7]: p.domain Out[7]: array([ 0., 10.]) Note that the data is mapped to the domain [-1, 1] and then fitted with the Legendre polynomials. The 'p' polynomial keeps track of that. The Chebyshev polynomials probably work as well as the Legendre polynomials for most things. Plot produced is attached. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: legendre-fit.png Type: image/png Size: 22650 bytes Desc: not available URL: From hoytak at stat.washington.edu Tue May 17 12:36:40 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Tue, 17 May 2011 09:36:40 -0700 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110517070325.GA18946@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> <20110517070325.GA18946@phare.normalesup.org> Message-ID: >> Well, the hungarian algorithm has a theoretical upper bound of O(n^3), >> with n being the number of nodes, which is pretty much the best you >> can do if you have a dense graph and make no assumptions on >> capacities. > > OK, your input is making my motivation to fight with Jonker-Volgenant go > down. I'll see with the other people involved if we still target > Jonger-Volgenant, or if we stick with the hungarian algorithm, in which > case the problem is solved. I'd do that route. From my (somewhat anecdotal) observations, a well coded version of the Hungarian algorithm will beat moderately well-coded implementations of other algorithms on most common types of problems; the data structures in other algorithms are harder to get right. Also, they often tend to show their strengths only on sufficiently large problems. I'd guess that for a speed up, your time would be better spent profiling and maybe cythoning your current implementation of the Hungarian algorithm (also note that there's a few bsd/mit C++ implementations floating around, check the wikipedia page). > >> Hope that answers your questions :-). > > It does. Thanks a lot, it is very useful to have expert knowledge > available. Well, I don't know if I'm an "expert" in this area, but I rub shoulders with a few and I'm glad to be of help. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From gael.varoquaux at normalesup.org Tue May 17 13:49:44 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 17 May 2011 19:49:44 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> <20110517070325.GA18946@phare.normalesup.org> Message-ID: <20110517174944.GD9977@phare.normalesup.org> On Tue, May 17, 2011 at 09:36:40AM -0700, Hoyt Koepke wrote: > > OK, your input is making my motivation to fight with Jonker-Volgenant go > > down. I'll see with the other people involved if we still target > > Jonger-Volgenant, or if we stick with the hungarian algorithm, in which > > case the problem is solved. > I'd do that route. From my (somewhat anecdotal) observations, a well > coded version of the Hungarian algorithm will beat moderately > well-coded implementations of other algorithms on most common types of > problems; the data structures in other algorithms are harder to get > right. Also, they often tend to show their strengths only on > sufficiently large problems. I'd guess that for a speed up, your time > would be better spent profiling and maybe cythoning your current > implementation of the Hungarian algorithm. I had some time in a meeting today, and did just that: https://github.com/GaelVaroquaux/scikit-learn/blob/hungarian/scikits/learn/utils/hungarian.py https://github.com/GaelVaroquaux/scikit-learn/blob/hungarian/scikits/learn/utils/tests/test_hungarian.py There's maybe a factor of 2 to be gained in this implementation by writing a few helper functions in Cython. I decided that I would stop optimization it until it became an actual bottleneck somewhere. Cheers, Ga?l From ben.root at ou.edu Tue May 17 13:55:39 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 17 May 2011 12:55:39 -0500 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: <20110517174944.GD9977@phare.normalesup.org> References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> <20110517070325.GA18946@phare.normalesup.org> <20110517174944.GD9977@phare.normalesup.org> Message-ID: On Tue, May 17, 2011 at 12:49 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Tue, May 17, 2011 at 09:36:40AM -0700, Hoyt Koepke wrote: > > > OK, your input is making my motivation to fight with Jonker-Volgenant > go > > > down. I'll see with the other people involved if we still target > > > Jonger-Volgenant, or if we stick with the hungarian algorithm, in which > > > case the problem is solved. > > > I'd do that route. From my (somewhat anecdotal) observations, a well > > coded version of the Hungarian algorithm will beat moderately > > well-coded implementations of other algorithms on most common types of > > problems; the data structures in other algorithms are harder to get > > right. Also, they often tend to show their strengths only on > > sufficiently large problems. I'd guess that for a speed up, your time > > would be better spent profiling and maybe cythoning your current > > implementation of the Hungarian algorithm. > > I had some time in a meeting today, and did just that: > > https://github.com/GaelVaroquaux/scikit-learn/blob/hungarian/scikits/learn/utils/hungarian.py > > https://github.com/GaelVaroquaux/scikit-learn/blob/hungarian/scikits/learn/utils/tests/test_hungarian.py > > There's maybe a factor of 2 to be gained in this implementation by > writing a few helper functions in Cython. I decided that I would stop > optimization it until it became an actual bottleneck somewhere. > > Cheers, > > Ga?l > Is this hungarian method in an official scikits package, or is this just your own thing? I currently use an old tracking algorithm that uses the hungarian method for solving the bipartite graph problem, and I have been unsuccessful in wrapping its C++ library for python. With your module, I should then be able to port a decent implementation of the tracker into Python. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue May 17 14:05:07 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 17 May 2011 20:05:07 +0200 Subject: [Numpy-discussion] Linear assignment problem: Jonker-Volgenant algorithm In-Reply-To: References: <20110516071836.GB14868@phare.normalesup.org> <20110516155517.GN20930@phare.normalesup.org> <20110516162851.GL11221@phare.normalesup.org> <20110517070325.GA18946@phare.normalesup.org> <20110517174944.GD9977@phare.normalesup.org> Message-ID: <20110517180507.GG9977@phare.normalesup.org> On Tue, May 17, 2011 at 12:55:39PM -0500, Benjamin Root wrote: > Is this hungarian method in an official scikits package, or is this just > your own thing? Right now we are playing with the idea of integrating it in the scikits learn, as it would be useful in a couple of places. I don't know whether we will integrate it, on the JV algorithm. Options are under review. However, the hungarian.py that I point to is a single Python file with no dependencies. So if you need it, grab it. In the future Cython helper functions might (or might not) be added. G From bsesar at astro.caltech.edu Tue May 17 15:16:27 2011 From: bsesar at astro.caltech.edu (Branimir Sesar) Date: Tue, 17 May 2011 12:16:27 -0700 Subject: [Numpy-discussion] distutils and SWIG problem Message-ID: <4DD2C98B.2050803@astro.caltech.edu> Dear Numpy users, I've been trying to compile Scikits ANN (http://projects.scipy.org/scikits/wiki/AnnWrapper) with Python 2.7.1, numpy 1.6.0, and SWIG 2.0.3 but the compilation breaks down down with this error: running install running bdist_egg running egg_info running build_src build_src building extension "scikits.ann._ANN" sources Traceback (most recent call last): File "setup.py", line 139, in test_suite = 'nose.collector', File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", line 57, in run r = self.setuptools_run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", line 51, in setuptools_run self.do_egg_install() File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", line 96, in do_egg_install File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 167, in run File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/egg_info.py", line 8, in run self.run_command("build_src") File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 332, in build_extension_sources sources = self.swig_sources(sources, ext) File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 646, in swig_sources extension.swig_opts.remove('-c++') AttributeError: 'str' object has no attribute 'remove' Is this a distutils bug or am I doing something wrong? Thank you very much for your help, Brani From wielandbrendel at gmx.net Tue May 17 20:47:00 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 02:47:00 +0200 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum Message-ID: <4DD31704.1070707@gmx.net> Hello, I am encountering a very strange behaviour of einsum on my machine. I tracked the problem down to the following test code: from numpy import * T = random.random((3,10,10)) W = random.random((3,10,7,275)) print all(einsum('ij...,j...->i...',T[0],W[0]) + einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) - einsum('pij,pjts->its',T,W)< 1e-10) print sum(abs(einsum('ij...,j...->i...',T[0],W[0]) + einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) - einsum('pij,pjts->its',T,W))) On my machine the equality is not fulfilled. However, this depends, strange enough, on the dimensions of W: if the last dimension e.g. is 500 instead of 275, things work again and the equality is fulfilled. Are you encountering similar problems or is this just my machine/installation? Since I am heavily relying on einsum but can't trust my code any more, I am happily waiting for you answers. Thanks a lot! Wieland From charlesr.harris at gmail.com Tue May 17 21:32:25 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 May 2011 19:32:25 -0600 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: <4DD31704.1070707@gmx.net> References: <4DD31704.1070707@gmx.net> Message-ID: On Tue, May 17, 2011 at 6:47 PM, Wieland Brendel wrote: > > Hello, > I am encountering a very strange behaviour of einsum on my machine. I > tracked the problem down to the following test code: > > from numpy import * > > T = random.random((3,10,10)) > W = random.random((3,10,7,275)) > > print all(einsum('ij...,j...->i...',T[0],W[0]) + > einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) > - einsum('pij,pjts->its',T,W)< 1e-10) > print sum(abs(einsum('ij...,j...->i...',T[0],W[0]) + > einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) > - einsum('pij,pjts->its',T,W))) > > On my machine the equality is not fulfilled. However, this depends, > strange enough, on the dimensions of W: if the last dimension e.g. is > 500 instead of 275, things work again and the equality is fulfilled. > > The equality being that the expression should be ~0? I see the problem when the last index is in the range 235 - 390. > Are you encountering similar problems or is this just my > machine/installation? > > Out of curiosity, which machine/OS are you using? I'm on 64 bit fedora 14, AMD 940. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Tue May 17 21:46:21 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 03:46:21 +0200 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum Message-ID: <4DD324ED.7040703@gmx.net> An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Tue May 17 21:47:55 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 03:47:55 +0200 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum References: 4DD31704.1070707@gmx.net Message-ID: <4DD3254B.4090405@gmx.net> An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 17 21:50:20 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 May 2011 19:50:20 -0600 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: References: <4DD31704.1070707@gmx.net> Message-ID: On Tue, May 17, 2011 at 7:32 PM, Charles R Harris wrote: > > > On Tue, May 17, 2011 at 6:47 PM, Wieland Brendel wrote: > >> >> Hello, >> I am encountering a very strange behaviour of einsum on my machine. I >> tracked the problem down to the following test code: >> >> from numpy import * >> >> T = random.random((3,10,10)) >> W = random.random((3,10,7,275)) >> >> print all(einsum('ij...,j...->i...',T[0],W[0]) + >> einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) >> - einsum('pij,pjts->its',T,W)< 1e-10) >> print sum(abs(einsum('ij...,j...->i...',T[0],W[0]) + >> einsum('ij...,j...->i...',T[1],W[1]) + einsum('ij...,j...->i...',T[2],W[2]) >> - einsum('pij,pjts->its',T,W))) >> >> On my machine the equality is not fulfilled. However, this depends, >> strange enough, on the dimensions of W: if the last dimension e.g. is >> 500 instead of 275, things work again and the equality is fulfilled. >> >> > The equality being that the expression should be ~0? > > I see the problem when the last index is in the range 235 - 390. > > >> Are you encountering similar problems or is this just my >> machine/installation? >> >> > Out of curiosity, which machine/OS are you using? I'm on 64 bit fedora 14, > AMD 940. > > Using T = ones((2,8,8)) W = ones((2,8,8,i)) The problem occurs for i in the closed ranges 129..204 and 257..341. The starting points look suspicious but the end points not so much. I think something isn't getting reset to zero. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 17 21:53:50 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 May 2011 19:53:50 -0600 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: <4DD324ED.7040703@gmx.net> References: <4DD324ED.7040703@gmx.net> Message-ID: On Tue, May 17, 2011 at 7:46 PM, Wieland Brendel wrote: > > The equality being that the expression should be ~0? > > Exactly. > > > I see the problem when the last index is in the range 235 - 390. > > Good to see I am not the only one - I was getting crazy. Same range for me by the way.**> Out of curiosity, which machine/OS are you using? I'm on 64 bit fedora 14, > > AMD 940. > > I am using 64 bit Windows 7 / Intel i5 M 540 / Python(x,y) distribution / Numpy 1.6 (latest, no RC) > > Any idea where that might come from? Looks like a very odd problem... > > Wieland > > PS: Its my first time on a mailing list - hope this mail actually replies to the thread. If not: How do I reply? > > > You're doing fine. You have found a bug, not unexpected with einsum not having been tested as much as the rest of the release since it is new and hasn't been used before. Please open a ticket. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Tue May 17 22:09:42 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 04:09:42 +0200 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum References: BANLkTin_f+Ab31duVmoGPwuEWwMfBGUEew@mail.gmail.com Message-ID: <4DD32A66.4070005@gmx.net> An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 17 22:25:55 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 May 2011 20:25:55 -0600 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: <4DD32A66.4070005@gmx.net> References: <4DD32A66.4070005@gmx.net> Message-ID: On Tue, May 17, 2011 at 8:09 PM, Wieland Brendel wrote: > It also fails for > > T = random.random((2,d,d)) > W = random.random((2,d,d,i)) > > and d > 2. For d = 3 it fails for i = 911...1365. > > Should I submit this as a bug (if so, how do I do that?) and/or contact the > author Mark Wiebe? > > Wieland > > PS: How do I reply directly to your messages? > ** > > Just reply in the normal way and it will go to the list so everyone can see it. To open a ticket, go to the scipy site and click on the report bugs icon and follow instructions. You will need to register and it will help if you can produce a simple example the reproduces the problem. I used powers of two for the dimensions because these sort of problems tend to be related to counting in binary. Ones instead of random makes the arithmetic exact so it is easier to see what is going on, etc. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Tue May 17 22:37:30 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 04:37:30 +0200 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum References: 4DD32A66.4070005@gmx.net Message-ID: <4DD330EA.1010807@gmx.net> Thanks for your reply! I managed to open a ticket, http://projects.scipy.org/numpy/ticket/1834 You are actually right, you can also just use zeros instead of random. Maybe I can test a bit more tomorrow... but its 4am in the morning now ;-). Thanks for your help and kindness! Wieland From ndbecker2 at gmail.com Wed May 18 07:39:18 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 18 May 2011 07:39:18 -0400 Subject: [Numpy-discussion] cPickle.load says 'invalid signature' Message-ID: The file is pickle saved on i386 and loaded on x86_64. It contains a numpy array (amoungst other things). On load it says: RuntimeError: invalid signature Is binary format not portable? From pav at iki.fi Wed May 18 07:57:14 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 18 May 2011 11:57:14 +0000 (UTC) Subject: [Numpy-discussion] cPickle.load says 'invalid signature' References: Message-ID: Wed, 18 May 2011 07:39:18 -0400, Neal Becker wrote: > The file is pickle saved on i386 and loaded on x86_64. It contains a > numpy array (amoungst other things). > > On load it says: > > RuntimeError: invalid signature There's no such message in Numpy source code, so the error does not come directly from Numpy. Did you try to isolate the problem a bit more by producing the pickle file without the array? From shish at keba.be Wed May 18 09:57:25 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 18 May 2011 09:57:25 -0400 Subject: [Numpy-discussion] cPickle.load says 'invalid signature' In-Reply-To: References: Message-ID: It's a wild guess, but try to save your pickle with 'wb' as argument of open, and protocol=-1. Then open it with 'rb'. It helped me fix some cross-platform issues in the past. -=- Olivier 2011/5/18 Neal Becker > The file is pickle saved on i386 and loaded on x86_64. It contains a numpy > array (amoungst other things). > > On load it says: > > RuntimeError: invalid signature > > Is binary format not portable? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwwiebe at gmail.com Wed May 18 10:57:35 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Wed, 18 May 2011 09:57:35 -0500 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: References: <4DD324ED.7040703@gmx.net> Message-ID: On Tue, May 17, 2011 at 8:53 PM, Charles R Harris wrote: > On Tue, May 17, 2011 at 7:46 PM, Wieland Brendel wrote: > >> > The equality being that the expression should be ~0? >> >> Exactly. >> >> > I see the problem when the last index is in the range 235 - 390. >> >> Good to see I am not the only one - I was getting crazy. Same range for me by the way.**> Out of curiosity, which machine/OS are you using? I'm on 64 bit fedora 14, >> > AMD 940. >> >> I am using 64 bit Windows 7 / Intel i5 M 540 / Python(x,y) distribution / Numpy 1.6 (latest, no RC) >> >> Any idea where that might come from? Looks like a very odd problem... >> >> Wieland >> >> PS: Its my first time on a mailing list - hope this mail actually replies to the thread. If not: How do I reply? >> >> >> > You're doing fine. You have found a bug, not unexpected with einsum not > having been tested as much as the rest of the release since it is new and > hasn't been used before. Please open a ticket. > > Chuck > I've committed a fix to this, and a crash that I encountered when experimenting with it, to both master and the 1.6.1 branch. It turned out to be an incorrectly handled interaction in the iterator when the inner loop for the reduction case doesn't fit all the way in the buffer. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed May 18 11:27:08 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 May 2011 09:27:08 -0600 Subject: [Numpy-discussion] strange dimension-dependent behaviour of einsum In-Reply-To: References: <4DD324ED.7040703@gmx.net> Message-ID: On Wed, May 18, 2011 at 8:57 AM, Mark Wiebe wrote: > On Tue, May 17, 2011 at 8:53 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Tue, May 17, 2011 at 7:46 PM, Wieland Brendel wrote: >> >>> > The equality being that the expression should be ~0? >>> >>> Exactly. >>> >>> > I see the problem when the last index is in the range 235 - 390. >>> >>> Good to see I am not the only one - I was getting crazy. Same range for me by the way.**> Out of curiosity, which machine/OS are you using? I'm on 64 bit fedora 14, >>> > AMD 940. >>> >>> I am using 64 bit Windows 7 / Intel i5 M 540 / Python(x,y) distribution / Numpy 1.6 (latest, no RC) >>> >>> Any idea where that might come from? Looks like a very odd problem... >>> >>> Wieland >>> >>> PS: Its my first time on a mailing list - hope this mail actually replies to the thread. If not: How do I reply? >>> >>> >>> >> You're doing fine. You have found a bug, not unexpected with einsum not >> having been tested as much as the rest of the release since it is new and >> hasn't been used before. Please open a ticket. >> >> Chuck >> > > I've committed a fix to this, and a crash that I encountered when > experimenting with it, to both master and the 1.6.1 branch. It turned out to > be an incorrectly handled interaction in the iterator when the inner loop > for the reduction case doesn't fit all the way in the buffer. > > Fast work. Looks like we will need to put out a 1.6.1 in a month or so. There is also a new ticket for f2py assumed size arrays. Hopefully (is that the right word?), more bugs will turn up in that time frame. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Wed May 18 13:17:22 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 19:17:22 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 Message-ID: <4DD3FF22.8070205@gmx.net> Hello, I am glad Mark was able to resolve the bug in einsum very fast since I am heavily relying on this function. I am working on a Windows 7 64 bit system with latest Python 2.6. I tried to build numpy from SVN following along http://www.scipy.org/Build_for_Windows After installing numpy I received the error RuntimeError: module compiled against ABI version 1000009 but this version of numpy is 2000000 This seems to be related to my Scipy installation, so I tried to build Scipy as well but couldn't get around installing ATLAS (I am used to using the superpacks...). Could somebody help me out? Maybe even with a 1.6.1 branch dev build? Thanks very much in advance!! Wieland From ralf.gommers at googlemail.com Wed May 18 13:20:28 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 May 2011 19:20:28 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: <4DD3FF22.8070205@gmx.net> References: <4DD3FF22.8070205@gmx.net> Message-ID: On Wed, May 18, 2011 at 7:17 PM, Wieland Brendel wrote: > Hello, > I am glad Mark was able to resolve the bug in einsum very fast since I > am heavily relying on this function. I am working on a Windows 7 64 bit > system with latest Python 2.6. I tried to build numpy from SVN following > along > > http://www.scipy.org/Build_for_Windows > > After installing numpy I received the error > > RuntimeError: module compiled against ABI version 1000009 but this > version of numpy is 2000000 > > This seems to be related to my Scipy installation, so I tried to build > Scipy as well but couldn't get around installing ATLAS (I am used to > using the superpacks...). > > Could somebody help me out? Maybe even with a 1.6.1 branch dev build? > > You have built numpy master. Mark's fix is in the 1.6.x branch, if you build that numpy will be compatible with the scipy superpack. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Wed May 18 13:38:26 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 19:38:26 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 References: 4DD3FF22.8070205@gmx.net Message-ID: <4DD40412.30602@gmx.net> Dear Ralf, thanks for you suggestion - the SVN was of course only the numpy master branch. I switched to the latest 1.6.x branch but the compilation stops with > ValueError: invalid version number '4.' in line 107, C:\Python26\lib\distutils\version.py (before was line 40, self.parse(vstring)). Any idea how I can solve that? Thanks again in advance! Wieland From ralf.gommers at googlemail.com Wed May 18 13:50:08 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 May 2011 19:50:08 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: <4DD40412.30602@gmx.net> References: <4DD40412.30602@gmx.net> Message-ID: On Wed, May 18, 2011 at 7:38 PM, Wieland Brendel wrote: > Dear Ralf, > thanks for you suggestion - the SVN was of course only the numpy master > branch. I switched to the latest 1.6.x branch but the compilation stops > with > > > ValueError: invalid version number '4.' > You have a time machine! Numpy will probably be version 4 around 2018 or so.... > > in line 107, C:\Python26\lib\distutils\version.py > > (before was line 40, self.parse(vstring)). > > Any idea how I can solve that? > Not without more info. Did you clean your build dir? What build command? Normally something like $ git clean -xdf $ python setup.py install should work. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed May 18 13:53:29 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 18 May 2011 11:53:29 -0600 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: References: <4DD40412.30602@gmx.net> Message-ID: On Wed, May 18, 2011 at 11:50 AM, Ralf Gommers wrote: > > > On Wed, May 18, 2011 at 7:38 PM, Wieland Brendel wrote: > >> Dear Ralf, >> thanks for you suggestion - the SVN was of course only the numpy master >> branch. I switched to the latest 1.6.x branch but the compilation stops >> with >> >> > ValueError: invalid version number '4.' >> > > You have a time machine! Numpy will probably be version 4 around 2018 or > so.... > >> >> in line 107, C:\Python26\lib\distutils\version.py >> >> (before was line 40, self.parse(vstring)). >> >> Any idea how I can solve that? >> > > Not without more info. Did you clean your build dir? What build command? > Normally something like > $ git clean -xdf > $ python setup.py install > should work. > > It's usually wise to also remove the previous numpy installation as there may be stray tests left over for things not in the branch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Wed May 18 14:12:51 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 20:12:51 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 References: BANLkTinGU8F=urpz2azTCGv=Y_dC9H5yBw@mail.gmail.com Message-ID: <4DD40C23.6020602@gmx.net> I have deinstalled scipy, numpy and Python(x,y), reinstalled a newer version of Python (x,y) and run the python install again - same result. However, maybe the build cache was still not cleaned? I used the build command > python setup.py config --compiler=mingw32 build --compiler=mingw32 install Thanks for your incredible help! Wieland From aradnix at gmail.com Wed May 18 14:40:31 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Wed, 18 May 2011 13:40:31 -0500 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: <4DD40C23.6020602@gmx.net> References: <4DD40C23.6020602@gmx.net> Message-ID: Hi Wieland: I see you have troubles isnide windows... mmm I don't enjoy to work with that OS but perhaps you should clean the Registry before reinstall your packages. I suggest you tu download and install the last version of Ccleaner, clean your PC and your Registry before try again. I hope it helps you and solve your problem. Good luck: Aradnix -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Wed May 18 14:42:14 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 20:42:14 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 Message-ID: <4DD41306.20008@gmx.net> I succeeded now in installing the latest Numpy version. There was some problem in mingw32ccompiler.py. I had to change the lines elif self.gcc_version < "4.": self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', compiler_so='gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes', linker_exe='g++ -mno-cygwin', linker_so='g++ -mno-cygwin -shared') else: # gcc-4 series releases do not support -mno-cygwin option self.set_executables(compiler='gcc -O2 -Wall', compiler_so='gcc -O2 -Wall -Wstrict-prototypes', linker_exe='g++ ', linker_so='g++ -shared') into else: self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', compiler_so='gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes', linker_exe='g++ -mno-cygwin', linker_so='g++ -mno-cygwin -shared') Thanks very much for your help again! Wieland From mmueller at python-academy.de Wed May 18 16:07:37 2011 From: mmueller at python-academy.de (=?ISO-8859-15?Q?Mike_M=FCller?=) Date: Wed, 18 May 2011 22:07:37 +0200 Subject: [Numpy-discussion] [ANN] Courses in Colorado: "Introduction to Python and Python for Scientists and Engineers" - one day left for early bird rate Message-ID: <4DD42709.1000302@python-academy.de> Python Course in Golden, CO, USA ================================ **There is only one day left to take advantage of the early bird rate.** Introduction to Python and Python for Scientists and Engineers -------------------------------------------------------------- June 3 - 4, 2011 Introduction to Python June 5, 2011 Python for Scientists and Engineers Both courses can be booked individually or together. Venue: Colorado School of Mines, Golden, CO (20 minutes west of Denver) Trainer: Mike M?ller Target Audience --------------- The introductory course is designed for people with basic programming background. Since it is a general introduction to Python it is suitable for everybody interested in Python. The scientist's course assumes a working knowledge of Python. You will be fine if you take the two-day introduction before hand. The topics are of general interest for scientists and engineers. Even though some examples come from the groundwater modeling domain, they are easy to understand for people without prior knowledge in this field. About the Trainer ----------------- Mike M?ller, has been teaching Python since 2004. He is the founder of Python Academy and regularly gives open and in-house Python courses as well as tutorials at PyCon US, OSCON, EuroSciPy and PyCon Asia-Pacific. More Information and Course Registration ---------------------------------------- http://igwmc.mines.edu/short-course/intro_python.html -- Mike mmueller at python-academy.de From ralf.gommers at googlemail.com Wed May 18 16:28:45 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 18 May 2011 22:28:45 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: <4DD41306.20008@gmx.net> References: <4DD41306.20008@gmx.net> Message-ID: On Wed, May 18, 2011 at 8:42 PM, Wieland Brendel wrote: > I succeeded now in installing the latest Numpy version. There was some > problem in mingw32ccompiler.py. I had to change the lines > > elif self.gcc_version < "4.": > self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', > compiler_so='gcc -mno-cygwin -O2 > -Wall -Wstrict-prototypes', > linker_exe='g++ -mno-cygwin', > linker_so='g++ -mno-cygwin -shared') > else: > # gcc-4 series releases do not support -mno-cygwin option > self.set_executables(compiler='gcc -O2 -Wall', > compiler_so='gcc -O2 -Wall > -Wstrict-prototypes', > linker_exe='g++ ', > linker_so='g++ -shared') > > into > > else: > self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', > compiler_so='gcc -mno-cygwin -O2 > -Wall -Wstrict-prototypes', > linker_exe='g++ -mno-cygwin', > linker_so='g++ -mno-cygwin -shared') > > Thanks very much for your help again! > > > Glad you solved it. Can you tell us the details of your setup (gcc version, Cygwin version)? The comment on -mno-cygwin is not completely correct, in gcc 4.0 -mno-cygwin was deprecated and it was only removed in gcc 4.3.2 according to http://www.cygwin.com/ml/cygwin-announce/2008-09/msg00005.html. Excerpt: "Finally, I have removed the controversial "-mno-cygwin" flag. Cause of much debate on the Cygwin mailing list, it is nevertheless the case that this flag has never worked entirely correctly - some cygwin headers are still visible in MinGW mode, which can lead to compilation errors. For the moment, the original gcc-3.4.4 package can be used to compile MinGW applications, but in the near future I will make available a mingw-targeted gcc-4.3.2 cross-compiler." That may mean we should update that version check to <4.3, although I'm not entirely sure that there aren't cygwin gcc packages with lower versions and that flag removed. Any opinions? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed May 18 17:40:54 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 18 May 2011 16:40:54 -0500 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: References: <4DD41306.20008@gmx.net> Message-ID: <4DD43CE6.1060800@gmail.com> On 05/18/2011 03:28 PM, Ralf Gommers wrote: > > > On Wed, May 18, 2011 at 8:42 PM, Wieland Brendel > > wrote: > > I succeeded now in installing the latest Numpy version. There was some > problem in mingw32ccompiler.py. I had to change the lines > > elif self.gcc_version < "4.": > self.set_executables(compiler='gcc -mno-cygwin > -O2 -Wall', > compiler_so='gcc -mno-cygwin -O2 > -Wall -Wstrict-prototypes', > linker_exe='g++ -mno-cygwin', > linker_so='g++ -mno-cygwin > -shared') > else: > # gcc-4 series releases do not support > -mno-cygwin option > self.set_executables(compiler='gcc -O2 -Wall', > compiler_so='gcc -O2 -Wall > -Wstrict-prototypes', > linker_exe='g++ ', > linker_so='g++ -shared') > > into > > else: > self.set_executables(compiler='gcc -mno-cygwin -O2 > -Wall', > compiler_so='gcc -mno-cygwin -O2 > -Wall -Wstrict-prototypes', > linker_exe='g++ -mno-cygwin', > linker_so='g++ -mno-cygwin > -shared') > > Thanks very much for your help again! > > > Glad you solved it. Can you tell us the details of your setup (gcc > version, Cygwin version)? > > The comment on -mno-cygwin is not completely correct, in gcc 4.0 > -mno-cygwin was deprecated and it was only removed in gcc 4.3.2 > according to > http://www.cygwin.com/ml/cygwin-announce/2008-09/msg00005.html. Excerpt: > > "Finally, I have removed the controversial "-mno-cygwin" flag. Cause > of much > debate on the Cygwin mailing list, it is nevertheless the case that > this flag > has never worked entirely correctly - some cygwin headers are still > visible in > MinGW mode, which can lead to compilation errors. For the moment, the > original gcc-3.4.4 package can be used to compile MinGW applications, > but in > the near future I will make available a mingw-targeted gcc-4.3.2 > cross-compiler." > > That may mean we should update that version check to <4.3, although > I'm not entirely sure that there aren't cygwin gcc packages with lower > versions and that flag removed. > > Any opinions? > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Related to another email thread of mine, I would definitely say that bugs on systems with an sufficiently old gcc version relative to the current release are either invalid or closed as won't fix. Especially when older numpy (or even Numeric/numarray) releases are available that worked with these older gcc versions. For those interested, GCC the different release dates are listed at http://gcc.gnu.org/releases.html - 4.0.0 was released in 2005 and 4.3.0 was released in 2008. While not cygwin, I know that numpy 1.6.0 passes the tests on Fedora 8 with GCC 4.1.2 (2007). So I do not know what compiler version should be imposed as a minimum without someone spending the time trying different versions - which seems rather unproductive. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From wielandbrendel at gmx.net Wed May 18 17:49:29 2011 From: wielandbrendel at gmx.net (Wieland Brendel) Date: Wed, 18 May 2011 23:49:29 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 References: BANLkTim-yojjPySFZ2WnMgw5=Kn2GD9vMw@mail.gmail.com Message-ID: <4DD43EE9.8060606@gmx.net> An HTML attachment was scrubbed... URL: From glenn.caltech at gmail.com Wed May 18 18:09:31 2011 From: glenn.caltech at gmail.com (G Jones) Date: Wed, 18 May 2011 15:09:31 -0700 Subject: [Numpy-discussion] Memory leak/fragmentation when using np.memmap In-Reply-To: References: Message-ID: Hello, I need to process several large (~40 GB) files. np.memmap seems ideal for this, but I have run into a problem that looks like a memory leak or memory fragmentation. The following code illustrates the problem import numpy as np x = np.memmap('mybigfile.bin',mode='r',dtype='uint8') print x.shape # prints (42940071360,) in my case ndat = x.shape[0] for k in range(1000): y = x[k*ndat/1000:(k+1)*ndat/1000].astype('float32') #The astype ensures that the data is read in from disk del y One would expect such a program would have a roughly constant memory footprint, but in fact 'top' shows that the RES memory continually increases. I can see that the memory usage is actually occurring because the OS eventually starts to swap to disk. The memory usage does not seem to correspond with the total size of the file. Has anyone see this behavior? Is there a solution? I found this article: http://pushingtheweb.com/2010/06/python-and-tcmalloc/ which sounds similar, but it seems that the ~40 MB chunks I am loading would be using mmap anyway so could be returned to the OS. I am using nearly the latest version of numpy from the git repository (np.__version__ returns 2.0.0.dev-Unknown), Python 2.7.1, and RHEL 5 on x86_64. I appreciate any suggestions. Thanks, Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From aradnix at gmail.com Wed May 18 18:47:43 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Wed, 18 May 2011 17:47:43 -0500 Subject: [Numpy-discussion] How can Import DATA from a Fortran file Message-ID: Hi everybody: I have a file in simple text with information obtained in Fortran 77 and I need to use the data inside for visualize with Mayavi. I was fighting for a while with the VTK simple legacy format. Finally I could run an small example, but now I need to select specific information from that file. Let me explain how is it. I have a header of two lines where tell me the number of domains I have, and later the values of the position and pressure Later for each domain I have two arrays with 125 rows, the first one has another header with two lines where we can find the name of each variable (column) and the dimension of the array for each coordinate (x, y, z). Later we have the array (125 x 8) where the first column is the number or ID for each point (row) the next three contains the coordinates for the point pnx, pny, pnz the following three the displacement Dnx, Dny, Dnz finally the last column contains the value of the pressure field in each point. The second array has only one line as header where specifies the number of points (rows) and the names of each variable (column). This array has the dimension 125 x 11, as in the first array the first column has the number of row, the following columns contain the values of the alpha elements used for the Finite Element Analysis from where I have to find the velocity (vx = 0.5 (alpha1 + alpha2), vy = 0.5 (alpha3 + alpha4), vz = 0.5 (alpha5 + alpha6)) finally the last column is a string that says pos-velocity so we can forget it. In an schematic form: Header: 5 NUMBER OF SUBDOMAINS 4852108.558056722000000 4.858791656580212E+008 POSITION-PRESSURE First array 10.00 10.00 10.00 LX, LY, LZ, ... #header 5 5 5 LS,MS,NS, - NEE,X,Y,Z,DX,DY,DZ,LAMBDA (NEE) #header 1 .000E+00 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 485879165.658 #125 rows with 8 columns 2 .200E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 362994604.232 3 .400E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 287889668.714 4 .600E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 249984468.929 5 .800E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 224851296.708 . . . 125 .800E+01 .800E+01 .800E+01 .200E+01 .200E+01 .200E+01 192572200.800 Second array: 125 L, X, Y, Z, ALPHA(I1), ALPHA(I2), ALPHA(J1), ALPHA(J2), ALPHA(K1), ALPHA(K2), ... #header 1 1.000 1.000 1.000 .000E+00 .845E-04 .000E+00 .826E-04 .000E+00 .828E-04 POS-VELOCITY 2 3.000 1.000 1.000 .845E-04 .308E-04 .000E+00 .267E-04 .000E+00 .269E-04 POS-VELOCITY 3 5.000 1.000 1.000 .308E-04 .177E-04 .000E+00 .633E-05 .000E+00 .666E-05 POS-VELOCITY 4 7.000 1.000 1.000 .177E-04 .122E-04 .000E+00 .246E-05 .000E+00 .297E-05 POS-VELOCITY 5 9.000 1.000 1.000 .122E-04 .908E-05 .000E+00 .114E-05 .000E+00 .183E-05 POS-VELOCITY . . . 125 9.000 9.000 9.000 .102E-04 .160E-04 .133E-05 .000E+00 .457E-05 .000E+00 POS-VELOCITY And both arrays repeat other 4 times, it means I have 5 pairs of arrays, a pair for each domain. I want to divide the file in five pieces one for each domain, but what I really need is can manipulate the arrays by columns. I know that numpy is able to import files in many formats, and I want to believe that inside numpy I can easily manipulate an array by columns, but I don't how, so all this explanation is for ask your helping and find a way to can get from this arrays the information I need to write a new file with the info for export as vtk file and can visualize in Mayavi2 (do you have a better idea or way for visualize this?). Thanks for your helping and your time. Aradnix! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed May 18 19:21:14 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 18 May 2011 23:21:14 +0000 (UTC) Subject: [Numpy-discussion] Memory leak/fragmentation when using np.memmap References: Message-ID: On Wed, 18 May 2011 15:09:31 -0700, G Jones wrote: [clip] > import numpy as np > > x = np.memmap('mybigfile.bin',mode='r',dtype='uint8') print x.shape # > prints (42940071360,) in my case ndat = x.shape[0] > for k in range(1000): > y = x[k*ndat/1000:(k+1)*ndat/1000].astype('float32') #The astype > ensures > that the data is read in from disk > del y > > One would expect such a program would have a roughly constant memory > footprint, but in fact 'top' shows that the RES memory continually > increases. I can see that the memory usage is actually occurring because > the OS eventually starts to swap to disk. The memory usage does not seem > to correspond with the total size of the file. Your OS probably likes to keep the pages touched in memory and in swap, rather than dropping them. This happens at least on Linux. You can check that an equivalent simple C program displays the same behavior (use with file "data" with enough bytes): #include #include #include int main() { unsigned long size = 2000000000; unsigned long i; char *p; int fd; char sum; fd = open("data", O_RDONLY); p = (char*)mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0); sum = 0; for (i = 0; i < size; ++i) { sum += *(p + i); } munmap(p, size); close(fd); return 0; } From glenn.caltech at gmail.com Wed May 18 19:36:31 2011 From: glenn.caltech at gmail.com (G Jones) Date: Wed, 18 May 2011 16:36:31 -0700 Subject: [Numpy-discussion] Memory leak/fragmentation when using np.memmap In-Reply-To: References: Message-ID: Hello, I have seen the effect you describe, I had originally assumed this was the case, but in fact there seems to be more to the problem. If it were only the effect you mention, there should not be any memory error because the OS will drop the pages when the memory is actually needed for something. At least I would hope so. If not, this seems like a huge problem for linux. As a followup, I managed to install tcmalloc as described in the article I mentioned. Running the example I sent now shows a constant memory foot print as expected. I am surprised such a solution was necessary. Certainly others must work with such large datasets using numpy/python? Thanks, Glenn On Wed, May 18, 2011 at 4:21 PM, Pauli Virtanen wrote: > On Wed, 18 May 2011 15:09:31 -0700, G Jones wrote: > [clip] > > import numpy as np > > > > x = np.memmap('mybigfile.bin',mode='r',dtype='uint8') print x.shape # > > prints (42940071360,) in my case ndat = x.shape[0] > > for k in range(1000): > > y = x[k*ndat/1000:(k+1)*ndat/1000].astype('float32') #The astype > > ensures > > that the data is read in from disk > > del y > > > > One would expect such a program would have a roughly constant memory > > footprint, but in fact 'top' shows that the RES memory continually > > increases. I can see that the memory usage is actually occurring because > > the OS eventually starts to swap to disk. The memory usage does not seem > > to correspond with the total size of the file. > > Your OS probably likes to keep the pages touched in memory and in swap, > rather than dropping them. This happens at least on Linux. > > You can check that an equivalent simple C program displays > the same behavior (use with file "data" with enough bytes): > > #include > #include > #include > > int main() > { > unsigned long size = 2000000000; > unsigned long i; > char *p; > int fd; > char sum; > > fd = open("data", O_RDONLY); > p = (char*)mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0); > > sum = 0; > for (i = 0; i < size; ++i) { > sum += *(p + i); > } > munmap(p, size); > close(fd); > > return 0; > } > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed May 18 19:53:30 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 18 May 2011 23:53:30 +0000 (UTC) Subject: [Numpy-discussion] Memory leak/fragmentation when using np.memmap References: Message-ID: On Wed, 18 May 2011 16:36:31 -0700, G Jones wrote: [clip] > As a followup, I managed to install tcmalloc as described in the article > I mentioned. Running the example I sent now shows a constant memory foot > print as expected. I am surprised such a solution was necessary. > Certainly others must work with such large datasets using numpy/python? Well, your example Python code works for me without any other changes, and it shows behavior identical to the C code. Things might depend on the version of the C library and the kernel, so it is quite possible that many do not see these issues. -- Pauli Virtanen From shish at keba.be Wed May 18 20:50:14 2011 From: shish at keba.be (Olivier Delalleau) Date: Wed, 18 May 2011 20:50:14 -0400 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: Seems like something the following would work (it's not particularly efficient nor elegant though, and there may be some stupid bug since I didn't actually try it). Also, there may be some more appropriate numpy structure to handle named columns (the example below computes separately a basic numpy array, and the map from a colunn's name to its index). def to_float(x): try: return float(x) except ValueError: assert x == "POS-VELOCITY" return numpy.nan data = [] for line_idx, line in enumerate(open("data.txt"). readlines()): if line_idx == 0: # header field = dict((field_name, col_idx) for col_idx, field_name in enumerate(line.split())) else: # data line data.append(map(to_float, line.split()) data = numpy.array(data) # Then to get a single column... col_X = data[:, field['X']] -=- Olivier 2011/5/18 Aradenatorix Veckhom Vacelaevus > Hi everybody: > > I have a file in simple text with information obtained in Fortran 77 and I > need to use the data inside for visualize with Mayavi. I was fighting for a > while with the VTK simple legacy format. Finally I could run an small > example, but now I need to select specific information from that file. Let > me explain how is it. > > I have a header of two lines where tell me the number of domains I have, > and later the values of the position and pressure > > Later for each domain I have two arrays with 125 rows, the first one has > another header with two lines where we can find the name of each variable > (column) and the dimension of the array for each coordinate (x, y, z). Later > we have the array (125 x 8) where the first column is the number or ID for > each point (row) the next three contains the coordinates for the point pnx, > pny, pnz the following three the displacement Dnx, Dny, Dnz finally the last > column contains the value of the pressure field in each point. > > The second array has only one line as header where specifies the number of > points (rows) and the names of each variable (column). This array has the > dimension 125 x 11, as in the first array the first column has the number of > row, the following columns contain the values of the alpha elements used for > the Finite Element Analysis from where I have to find the velocity (vx = 0.5 > (alpha1 + alpha2), vy = 0.5 (alpha3 + alpha4), vz = 0.5 (alpha5 + alpha6)) > finally the last column is a string that says pos-velocity so we can forget > it. > > In an schematic form: > > Header: > > 5 NUMBER OF SUBDOMAINS > > 4852108.558056722000000 4.858791656580212E+008 POSITION-PRESSURE > > > First array > > 10.00 10.00 10.00 LX, LY, LZ, ... #header > > 5 5 5 LS,MS,NS, - NEE,X,Y,Z,DX,DY,DZ,LAMBDA > (NEE) #header > > 1 .000E+00 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 > 485879165.658 #125 rows with 8 columns > > 2 .200E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 > 362994604.232 > 3 .400E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 > 287889668.714 > 4 .600E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 > 249984468.929 > 5 .800E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 > 224851296.708 > > . > > . > > . > > 125 .800E+01 .800E+01 .800E+01 .200E+01 .200E+01 .200E+01 > 192572200.800 > > > Second array: > > 125 L, X, Y, Z, ALPHA(I1), ALPHA(I2), ALPHA(J1), ALPHA(J2), ALPHA(K1), > ALPHA(K2), ... #header > 1 1.000 1.000 1.000 .000E+00 .845E-04 .000E+00 > .826E-04 .000E+00 .828E-04 POS-VELOCITY > 2 3.000 1.000 1.000 .845E-04 .308E-04 .000E+00 > .267E-04 .000E+00 .269E-04 POS-VELOCITY > 3 5.000 1.000 1.000 .308E-04 .177E-04 .000E+00 > .633E-05 .000E+00 .666E-05 POS-VELOCITY > 4 7.000 1.000 1.000 .177E-04 .122E-04 .000E+00 > .246E-05 .000E+00 .297E-05 POS-VELOCITY > 5 9.000 1.000 1.000 .122E-04 .908E-05 .000E+00 > .114E-05 .000E+00 .183E-05 POS-VELOCITY > > . > . > . > > 125 9.000 9.000 9.000 .102E-04 .160E-04 .133E-05 .000E+00 > .457E-05 .000E+00 POS-VELOCITY > > > And both arrays repeat other 4 times, it means I have 5 pairs of arrays, a > pair for each domain. > > I want to divide the file in five pieces one for each domain, but what I > really need is can manipulate the arrays by columns. I know that numpy is > able to import files in many formats, and I want to believe that inside > numpy I can easily manipulate an array by columns, but I don't how, so all > this explanation is for ask your helping and find a way to can get from this > arrays the information I need to write a new file with the info for export > as vtk file and can visualize in Mayavi2 (do you have a better idea or way > for visualize this?). Thanks for your helping and your time. > > Aradnix! > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Wed May 18 20:53:08 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 19 May 2011 02:53:08 +0200 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: On 19.05.2011, at 12:47AM, Aradenatorix Veckhom Vacelaevus wrote: > I have a file in simple text with information obtained in Fortran 77 and I need to use the data inside for visualize with Mayavi. I was fighting for a while with the VTK simple legacy format. Finally I could run an small example, but now I need to select specific information from that file. Let me explain how is it. > > I have a header of two lines where tell me the number of domains I have, and later the values of the position and pressure > > Later for each domain I have two arrays with 125 rows, the first one has another header with two lines where we can find the name of each variable (column) and the dimension of the array for each coordinate (x, y, z). Later we have the array (125 x 8) where the first column is the number or ID for each point (row) the next three contains the coordinates for the point pnx, pny, pnz the following three the displacement Dnx, Dny, Dnz finally the last column contains the value of the pressure field in each point. > > The second array has only one line as header where specifies the number of points (rows) and the names of each variable (column). This array has the dimension 125 x 11, as in the first array the first column has the number of row, the following columns contain the values of the alpha elements used for the Finite Element Analysis from where I have to find the velocity (vx = 0.5 (alpha1 + alpha2), vy = 0.5 (alpha3 + alpha4), vz = 0.5 (alpha5 + alpha6)) finally the last column is a string that says pos-velocity so we can forget it. > > In an schematic form: > > Header: > 5 NUMBER OF SUBDOMAINS > 4852108.558056722000000 4.858791656580212E+008 POSITION-PRESSURE > > First array > 10.00 10.00 10.00 LX, LY, LZ, ... #header > 5 5 5 LS,MS,NS, - NEE,X,Y,Z,DX,DY,DZ,LAMBDA (NEE) #header > 1 .000E+00 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 485879165.658 #125 rows with 8 columns > 2 .200E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 362994604.232 > 3 .400E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 287889668.714 > . > . > . > 125 .800E+01 .800E+01 .800E+01 .200E+01 .200E+01 .200E+01 192572200.800 > > Second array: > 125 L, X, Y, Z, ALPHA(I1), ALPHA(I2), ALPHA(J1), ALPHA(J2), ALPHA(K1), ALPHA(K2), ... #header > 1 1.000 1.000 1.000 .000E+00 .845E-04 .000E+00 .826E-04 .000E+00 .828E-04 POS-VELOCITY > 2 3.000 1.000 1.000 .845E-04 .308E-04 .000E+00 .267E-04 .000E+00 .269E-04 POS-VELOCITY > 3 5.000 1.000 1.000 .308E-04 .177E-04 .000E+00 .633E-05 .000E+00 .666E-05 POS-VELOCITY > 4 7.000 1.000 1.000 .177E-04 .122E-04 .000E+00 .246E-05 .000E+00 .297E-05 POS-VELOCITY > 5 9.000 1.000 1.000 .122E-04 .908E-05 .000E+00 .114E-05 .000E+00 .183E-05 POS-VELOCITY > . > . > . > 125 9.000 9.000 9.000 .102E-04 .160E-04 .133E-05 .000E+00 .457E-05 .000E+00 POS-VELOCITY > > And both arrays repeat other 4 times, it means I have 5 pairs of arrays, a pair for each domain. > > I want to divide the file in five pieces one for each domain, but what I really need is can manipulate the arrays by columns. I know that numpy is able to import files in many formats, and I want to believe that inside numpy I can easily manipulate an array by columns, but I don't how, so all this explanation is for ask your helping and find a way to can get from this arrays the information I need to write a new file with the info for export as vtk file and can visualize in Mayavi2 (do you have a better idea or way for visualize this?). Thanks for your helping and your time. > You are right, numpy can manipulate the arrays probably in any way you need once they are read in; but I don't think there are tools to handle files with a complex structure like that right out of the box. You'll probably need to write some code for parsing the file on your own to split the file into the 10 data blocks (by searching for the header lines), but once that's done, there are a number of utilities to read the data columns into numpy arrays - specifically, you want to check the loadtxt(), fromstring() and genfromtxt() functions. As an example, consider you read in your file into a list of text strings by lines = open('your_file', 'r').readlines() and then find the first line of your data block, then array1 = np.loadtxt(lines[start_line:start_line+128]) would read your first block of data, and so on... You might also want to check out the asciitable package at http://cxc.harvard.edu/contrib/asciitable/ which provides a framework to automatically create structured or record arrays using information from the table header (e.g. if you pass it a header line like "L,X,Y,Z,...." you'd be able to access the column data as array1['L'] etc.) HTH, Derek From rblove_lists at comcast.net Wed May 18 21:56:14 2011 From: rblove_lists at comcast.net (Robert Love) Date: Wed, 18 May 2011 20:56:14 -0500 Subject: [Numpy-discussion] Importing a list into a structured numpy array, what is my error? Message-ID: my_list = [[1960, 'W0'],[2001, 'D5']] my_arr = np.array(my_list, dtype=('uint', '|S2')) gives me an error I don't understand: ValueError: mismatch in size of old and new data-descriptor What are the old and new? I get the same problem if try np.asarray() If I make the dtype ('|S4', '|S4') I can read in as strings without errors How do I convert a list of mixed types to a structured array? Thanks for any pointers. From jmarch at enthought.com Wed May 18 22:22:02 2011 From: jmarch at enthought.com (Jonathan March) Date: Wed, 18 May 2011 21:22:02 -0500 Subject: [Numpy-discussion] Importing a list into a structured numpy array, what is my error? In-Reply-To: References: Message-ID: On Wed, May 18, 2011 at 8:56 PM, Robert Love wrote: > > my_list = [[1960, 'W0'],[2001, 'D5']] > > my_arr = np.array(my_list, dtype=('uint', '|S2')) > > gives me an error I don't understand: > > ValueError: mismatch in size of old and new data-descriptor > > What are the old and new? I get the same problem if try np.asarray() > > If I make the dtype ('|S4', '|S4') I can read in as strings without > errors > > How do I convert a list of mixed types to a structured array? > > Thanks for any pointers. > my_list = [(1960, 'W0'),(2001, 'D5')] # tuples inside my_arr = np.array(my_list, dtype='uint, |S2') # one dtype string -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdickinson at enthought.com Thu May 19 07:36:22 2011 From: mdickinson at enthought.com (Mark Dickinson) Date: Thu, 19 May 2011 12:36:22 +0100 Subject: [Numpy-discussion] Conversion from structured dtype to object changed in Numpy 1.6.0. Message-ID: With numpy 1.5.1: Python 2.7.1 |EPD 7.0-2 (32-bit)| (r271:86832, Dec 3 2010, 15:41:32) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import array >>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >>> x.astype(object) array((1.2, 2.3), dtype=object) >>> So the conversion from the structured dtype item to type 'object' produces a tuple, which makes sense. With numpy 1.6.0, the same computation gives a different result, apparently truncating the tuple: Python 2.7.1 |EPD 7.0-2 (32-bit)| (r271:86832, Dec 3 2010, 15:41:32) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import array >>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >>> x.astype(object) array(1.2, dtype=object) >>> Was this change intentional, or should I file a bug? I couldn't find any reference to it in the changelog or release notes. Thanks, Mark From pav at iki.fi Thu May 19 07:59:54 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 19 May 2011 11:59:54 +0000 (UTC) Subject: [Numpy-discussion] Conversion from structured dtype to object changed in Numpy 1.6.0. References: Message-ID: Thu, 19 May 2011 12:36:22 +0100, Mark Dickinson wrote: [clip] >>>> from numpy import array >>>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >>>> x.astype(object) > array(1.2, dtype=object) > > Was this change intentional, or should I file a bug? I couldn't find > any reference to it in the changelog or release notes. Looks like a bug -- the change was not intentional. -- Pauli Virtanen From stephane.raynaud at gmail.com Thu May 19 11:11:05 2011 From: stephane.raynaud at gmail.com (Stephane Raynaud) Date: Thu, 19 May 2011 17:11:05 +0200 Subject: [Numpy-discussion] f2py and lapack DLARUV Message-ID: Hi, trying to build a fortran extension linked to lapack (f77), I faced a problem that I reduced to the following example. ------------ testfile.f90 ------------ subroutine testfunc ??? integer,parameter :: n=2 ??? integer :: iseed(4) ??? real(kind=8) :: x(n) ??? iseed = (/0,1,3,4/) ??? call DLARUV(iseed, n, x) end subroutine testfunc -------------------------------------- Then :? f2py -c -m testmod -llapack -lblas -L/home/raynaud/soft/lapack_gfortran/lib testfile.f90 It rises the following error (using gfortran) : /usr/bin/ld: /home/raynaud/soft/lapack_gfortran/lib/liblapack.a(dlaruv.o): relocation R_X86_64_32S against `.data' can not be used when making a shared object; recompile with -fPIC /home/raynaud/soft/lapack_gfortran/lib/liblapack.a: could not read symbols: Bad value BLAS and LAPACK where compilated with gfortran and options '-fPIC -O3'. I tried to use the DATA statement directly in a subroutine that I converted to a python module, and it worked. Any idea ? Thanks. -- Stephane Raynaud From mat.yeates at gmail.com Thu May 19 14:07:35 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 11:07:35 -0700 Subject: [Numpy-discussion] Windows Registry Keys Message-ID: Hi I have installed a new version of Python27 in a new directory. I want to get this info into the registry so, when I install Numpy, it will use my new Python TIA -Mathew From alan.isaac at gmail.com Thu May 19 14:14:31 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 19 May 2011 14:14:31 -0400 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: References: Message-ID: <4DD55E07.7090700@gmail.com> On 5/19/2011 2:07 PM, Mathew Yeates wrote: > I have installed a new version of Python27 in a new directory. I want to get > this info into the registry so, when I install Numpy, it will use my new > Python > It probably will already. Did you try? (Assumption: you're using Windows installers.) Alan Isaac From mat.yeates at gmail.com Thu May 19 14:15:42 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 11:15:42 -0700 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: <4DD55E07.7090700@gmail.com> References: <4DD55E07.7090700@gmail.com> Message-ID: I *am* using the windows installer. On Thu, May 19, 2011 at 11:14 AM, Alan G Isaac wrote: > On 5/19/2011 2:07 PM, Mathew Yeates wrote: >> I have installed a new version of Python27 in a new directory. I want to get >> this info into the registry so, when I install Numpy, it will use my new >> Python >> > > > It probably will already. > Did you try? > (Assumption: you're using Windows installers.) > > Alan Isaac > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From alan.isaac at gmail.com Thu May 19 14:23:10 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 19 May 2011 14:23:10 -0400 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: References: <4DD55E07.7090700@gmail.com> Message-ID: <4DD5600E.1050005@gmail.com> On 5/19/2011 2:15 PM, Mathew Yeates wrote: > I*am* using the windows installer. And you find that it does not find your most recent Python 2.7 install, for which you also used the Windows installer? Alan Isaac From mat.yeates at gmail.com Thu May 19 14:24:13 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 11:24:13 -0700 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: <4DD5600E.1050005@gmail.com> References: <4DD55E07.7090700@gmail.com> <4DD5600E.1050005@gmail.com> Message-ID: Right. The Registry keys point to the old Python27. On Thu, May 19, 2011 at 11:23 AM, Alan G Isaac wrote: > On 5/19/2011 2:15 PM, Mathew Yeates wrote: >> I*am* ?using the windows installer. > > And you find that it does not find your most recent > Python 2.7 install, for which you also used the > Windows installer? > > Alan Isaac > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From bsesar at astro.caltech.edu Thu May 19 14:28:03 2011 From: bsesar at astro.caltech.edu (Branimir Sesar) Date: Thu, 19 May 2011 11:28:03 -0700 Subject: [Numpy-discussion] AttributeError in numpy.distutils In-Reply-To: <4DD2C98B.2050803@astro.caltech.edu> References: <4DD2C98B.2050803@astro.caltech.edu> Message-ID: <4DD56133.3000405@astro.caltech.edu> Dear Numpy users, I've encountered an AttributeError in numpy.distutils File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 646, in swig_sources extension.swig_opts.remove('-c++') AttributeError: 'str' object has no attribute 'remove' while compiling some code with Python 2.7.1 and Numpy 1.6.0. Is this a known bug? Cheers, Brani From cgohlke at uci.edu Thu May 19 14:47:01 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 19 May 2011 11:47:01 -0700 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: References: <4DD55E07.7090700@gmail.com> <4DD5600E.1050005@gmail.com> Message-ID: <4DD565A5.5060101@uci.edu> On 5/19/2011 11:24 AM, Mathew Yeates wrote: > Right. The Registry keys point to the old Python27. > > On Thu, May 19, 2011 at 11:23 AM, Alan G Isaac wrote: >> On 5/19/2011 2:15 PM, Mathew Yeates wrote: >>> I*am* using the windows installer. >> >> And you find that it does not find your most recent >> Python 2.7 install, for which you also used the >> Windows installer? >> >> Alan Isaac >> Please provide details about your OS and the installers you use. Could it be that you installed a 64 bit Python version and are using the official 32 bit numpy distribution? That will not work. If you are familiar with editing the Windows Registry you might try the SO solution at Christoph From RadimRehurek at seznam.cz Thu May 19 15:10:34 2011 From: RadimRehurek at seznam.cz (RadimRehurek) Date: Thu, 19 May 2011 21:10:34 +0200 (CEST) Subject: [Numpy-discussion] array order in numpy.dot Message-ID: <26084.6906.16693-10923-901923663-1305832234@seznam.cz> Hello, what code paths does `numpy.dot(matrix, matrix)` take with regard to the order of the input matrices (c/fortran contiguous?). In particular, under what conditions does it make a copy of the input matrices, and under which conditions does it set the gemm transposition flags internally? Does this change with different numpy versions, is there a contract re this behaviour? My concern is 1) memory (matrices are large, cannot afford extra copies) and 2) performance. Sorry if I missed this in the documentation. Cheers! Radim From alan.isaac at gmail.com Thu May 19 17:15:12 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 19 May 2011 17:15:12 -0400 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: References: <4DD55E07.7090700@gmail.com> <4DD5600E.1050005@gmail.com> Message-ID: <4DD58860.6090608@gmail.com> On 5/19/2011 2:24 PM, Mathew Yeates wrote: > The Registry keys point to the old Python27. Odd. The default installation settings should have reset this. Or so I believed. Maybe this will help? http://effbot.org/zone/python-register.htm Alan Isaac From ralf.gommers at googlemail.com Thu May 19 17:17:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 19 May 2011 23:17:50 +0200 Subject: [Numpy-discussion] Windows dev-build Numpy 1.6.1 In-Reply-To: <4DD43CE6.1060800@gmail.com> References: <4DD41306.20008@gmx.net> <4DD43CE6.1060800@gmail.com> Message-ID: On Wed, May 18, 2011 at 11:40 PM, Bruce Southey wrote: > On 05/18/2011 03:28 PM, Ralf Gommers wrote: > > > > On Wed, May 18, 2011 at 8:42 PM, Wieland Brendel wrote: > >> I succeeded now in installing the latest Numpy version. There was some >> problem in mingw32ccompiler.py. I had to change the lines >> >> elif self.gcc_version < "4.": >> self.set_executables(compiler='gcc -mno-cygwin -O2 >> -Wall', >> compiler_so='gcc -mno-cygwin -O2 >> -Wall -Wstrict-prototypes', >> linker_exe='g++ -mno-cygwin', >> linker_so='g++ -mno-cygwin -shared') >> else: >> # gcc-4 series releases do not support -mno-cygwin option >> self.set_executables(compiler='gcc -O2 -Wall', >> compiler_so='gcc -O2 -Wall >> -Wstrict-prototypes', >> linker_exe='g++ ', >> linker_so='g++ -shared') >> >> into >> >> else: >> self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', >> compiler_so='gcc -mno-cygwin -O2 >> -Wall -Wstrict-prototypes', >> linker_exe='g++ -mno-cygwin', >> linker_so='g++ -mno-cygwin -shared') >> >> Thanks very much for your help again! >> >> >> Glad you solved it. Can you tell us the details of your setup (gcc > version, Cygwin version)? > > The comment on -mno-cygwin is not completely correct, in gcc 4.0 > -mno-cygwin was deprecated and it was only removed in gcc 4.3.2 according to > http://www.cygwin.com/ml/cygwin-announce/2008-09/msg00005.html. Excerpt: > > "Finally, I have removed the controversial "-mno-cygwin" flag. Cause of > much > debate on the Cygwin mailing list, it is nevertheless the case that this > flag > has never worked entirely correctly - some cygwin headers are still visible > in > MinGW mode, which can lead to compilation errors. For the moment, the > original gcc-3.4.4 package can be used to compile MinGW applications, but > in > the near future I will make available a mingw-targeted gcc-4.3.2 > cross-compiler." > > That may mean we should update that version check to <4.3, although I'm not > entirely sure that there aren't cygwin gcc packages with lower versions and > that flag removed. > > Any opinions? > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion > > Related to another email thread of mine, I would definitely say that bugs > on systems with an sufficiently old gcc version relative to the current > release are either invalid or closed as won't fix. Especially when older > numpy (or even Numeric/numarray) releases are available that worked with > these older gcc versions. > > For those interested, GCC the different release dates are listed at > http://gcc.gnu.org/releases.html - 4.0.0 was released in 2005 and 4.3.0 > was released in 2008. While not cygwin, I know that numpy 1.6.0 passes the > tests on Fedora 8 with GCC 4.1.2 (2007). So I do not know what compiler > version should be imposed as a minimum without someone spending the time > trying different versions - which seems rather unproductive. > > Well, this is slightly different than the RHEL4 bugs IMHO. The numpy/scipy Windows binaries are still made with MinGW-gcc-3.4.5. The default GCC on OS X for Python 2.6 and 32-bit 2.7 is 4.0. Also, not fixing a bug in some random function may be slightly inconvenient for the affected user, not fixing a compile error results in a much bigger problem for that user. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Thu May 19 17:59:28 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 14:59:28 -0700 Subject: [Numpy-discussion] Windows Registry Keys In-Reply-To: <4DD58860.6090608@gmail.com> References: <4DD55E07.7090700@gmail.com> <4DD5600E.1050005@gmail.com> <4DD58860.6090608@gmail.com> Message-ID: cool. just what I was looking for On Thu, May 19, 2011 at 2:15 PM, Alan G Isaac wrote: > On 5/19/2011 2:24 PM, Mathew Yeates wrote: >> The Registry keys point to the old Python27. > > > Odd. ?The default installation settings > should have reset this. ?Or so I believed. > Maybe this will help? > http://effbot.org/zone/python-register.htm > > Alan Isaac > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mat.yeates at gmail.com Thu May 19 18:05:31 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 15:05:31 -0700 Subject: [Numpy-discussion] f2py on Windows, compiler options Message-ID: Hi I am trying to run f2py and link to some libraries. I get a link error LINK : fatal error LNK1104: cannot open file 'LIBC.lib' because (I think) the libraries are compiled with /MT (multithreaded). I tried adding /NODFEAU:TLIB:libc.lib but then I have unresolved dependencies. I want to try compiling the code with /MT. How can tell f2py to use this option when compiling (not linking, this option is passed to cl). -Mathew From ralf.gommers at googlemail.com Thu May 19 19:22:44 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 20 May 2011 01:22:44 +0200 Subject: [Numpy-discussion] Memory leak/fragmentation when using np.memmap In-Reply-To: References: Message-ID: On Thu, May 19, 2011 at 1:53 AM, Pauli Virtanen wrote: > On Wed, 18 May 2011 16:36:31 -0700, G Jones wrote: > [clip] > > As a followup, I managed to install tcmalloc as described in the article > > I mentioned. Running the example I sent now shows a constant memory foot > > print as expected. I am surprised such a solution was necessary. > > Certainly others must work with such large datasets using numpy/python? > > Well, your example Python code works for me without any other changes, > and it shows behavior identical to the C code. > > Yes, the C code does the same thing and yes, technically it's not a memory leak. But that doesn't make it less of a problem. You can map a huge data file but in practice, once you start looping over parts of it, all your memory disappears. And it doesn't come back. So this issue should be documented in the memmap docstring I think. The tcmalloc thing seems like a useful thing to document somewhere too. Things might depend on the version of the C library and the kernel, so it > is quite possible that many do not see these issues. > > I see the same thing on OS X with gcc 4.2. My memory usage increases by almost exactly the amount of Mb's read from disc, and can not be reduced by deleting objects. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu May 19 19:30:47 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 19 May 2011 19:30:47 -0400 Subject: [Numpy-discussion] more python 3.2 fun Message-ID: (just illustrating some porting fun) I was struggling with another python 3.2 bug in scikits.statsmodels (with grunfeld data) with numpy 1.5.1, I'm reading the data with data = recfromtxt(open(filepath + '/grunfeld.csv', 'rb'), delimiter=",", names=True, dtype="f8,f8,f8,a17,f8") >>> grun_exog['firm'][:10] array([b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors'], dtype='|S17') using predefined firms defined as string >>> firms = ['General Motors', 'Chrysler', 'General Electric', 'Westinghouse', 'US Steel'] >>> i 'General Motors' comparing bytes and string produces False not an element wise comparison >>> grun_exog['firm'] == i False >>> grun_exog['firm'][0] == i False >>> grun_exog['firm'][0] , i (b'General Motors', 'General Motors') numpy has a convenient conversion function >>> from numpy.compat import asbytes >>> ( grun_exog['firm'] == asbytes(i) )[:10] array([ True, True, True, True, True, True, True, True, True, True], dtype=bool) Josef From aradnix at gmail.com Thu May 19 20:29:59 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Thu, 19 May 2011 19:29:59 -0500 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: Thanks both for your helping: I was checking the script fro Olivier but doesn't works yet, later I tried with the asciitable package but I also never could read any of my files (finally I decide create a single file for each case, it means to get 2 arrays for each domain). I was reading a while about how to import data with genfromtxt in Numpy, it was very interesting because I have a lot of options for import data, I can skip headers and footers, I can import selected columns and other stuff, so I wanna use it for import my columns in each case and later get an output in a text file, how can I get it? My idea is to make something like: #!/usr/bin/env python #-*- coding:utf-8 -*- import numpy as np from StringIO import StringIO np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, usecols=(1, 2, 3)) np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, usecols=(4, 5, 6)) np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, usecols=(-1)) But I'm not sure how to use it, I have not experience importing that stuff, but also I don't know If I need to add a line at the beginning with data = StringIO(data_file_name) Could you help me once more? Regards: Aradnix -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat.yeates at gmail.com Thu May 19 21:25:54 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 18:25:54 -0700 Subject: [Numpy-discussion] f2py on Windows, compiler options In-Reply-To: References: Message-ID: Solved. Sort of. When I compiled by hand and switched /MD to /MT it worked. It would still be nice if I could control the compiler options f2py passes to cl.exe -Mathew On Thu, May 19, 2011 at 3:05 PM, Mathew Yeates wrote: > Hi > I am trying to run f2py and link to some libraries. > I get a link error > LINK : fatal error LNK1104: cannot open file 'LIBC.lib' because (I > think) the libraries are compiled with /MT (multithreaded). > > I tried adding /NODFEAU:TLIB:libc.lib > > but then I have unresolved dependencies. I want to try compiling the > code with /MT. How can tell f2py to use this option when compiling > (not linking, this option is passed to cl). > > -Mathew > From mat.yeates at gmail.com Thu May 19 22:16:13 2011 From: mat.yeates at gmail.com (Mathew Yeates) Date: Thu, 19 May 2011 19:16:13 -0700 Subject: [Numpy-discussion] f2py on Windows, compiler options In-Reply-To: References: Message-ID: okay. To get it all to work I edited msvc9compiler.py and changed /MD to /MT. This still led to an a different error. having to do with mt.exe which does not come with MSVC 2008 Express. I fixed this commenting out /MANIFEST stuff in msvc9compile.py On Thu, May 19, 2011 at 6:25 PM, Mathew Yeates wrote: > Solved. Sort of. When I compiled by hand and switched /MD to /MT it > worked. It would still be nice if I could control the compiler options > f2py passes to cl.exe > > -Mathew > > On Thu, May 19, 2011 at 3:05 PM, Mathew Yeates wrote: >> Hi >> I am trying to run f2py and link to some libraries. >> I get a link error >> LINK : fatal error LNK1104: cannot open file 'LIBC.lib' because (I >> think) the libraries are compiled with /MT (multithreaded). >> >> I tried adding /NODFEAU:TLIB:libc.lib >> >> but then I have unresolved dependencies. I want to try compiling the >> code with /MT. How can tell f2py to use this option when compiling >> (not linking, this option is passed to cl). >> >> -Mathew >> > From shish at keba.be Fri May 20 09:22:08 2011 From: shish at keba.be (Olivier Delalleau) Date: Fri, 20 May 2011 09:22:08 -0400 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: StringIO(data_file_name) will not read your file, it will just allow you to read the string data_file_name. You can probably just use open(data_file_name) (and yes,you'll probably need to open it for each call to genfromtxt). Sorry my script didn't work, I definitely didn't expect it to work right away, it was more to give you an idea of the kind of stuff you could do manually. I'd expect it should be pretty straightforward to fix it. I personally find it easier to write my own custom code to load such small data files, rather than try to find the right parameters for an existing function ;) (especially with mixed float / string dta). -=- Olivier 2011/5/19 Aradenatorix Veckhom Vacelaevus > Thanks both for your helping: > > I was checking the script fro Olivier but doesn't works yet, later I tried > with the asciitable package but I also never could read any of my files > (finally I decide create a single file for each case, it means to get 2 > arrays for each domain). > > I was reading a while about how to import data with genfromtxt in Numpy, it > was very interesting because I have a lot of options for import data, I can > skip headers and footers, I can import selected columns and other stuff, so > I wanna use it for import my columns in each case and later get an output in > a text file, how can I get it? > > My idea is to make something like: > > #!/usr/bin/env python > #-*- coding:utf-8 -*- > > import numpy as np > from StringIO import StringIO > > np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, > usecols=(1, 2, 3)) > > np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, > usecols=(4, 5, 6)) > > np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, > usecols=(-1)) > > But I'm not sure how to use it, I have not experience importing that stuff, > but also I don't know If I need to add a line at the beginning with > > data = StringIO(data_file_name) > > Could you help me once more? > > Regards: > Aradnix > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri May 20 11:53:32 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 20 May 2011 10:53:32 -0500 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: On Fri, May 20, 2011 at 8:22 AM, Olivier Delalleau wrote: > StringIO(data_file_name) will not read your file, it will just allow you to > read the string data_file_name. You can probably just use > open(data_file_name) (and yes,you'll probably need to open it for each call > to genfromtxt). > Sorry my script didn't work, I definitely didn't expect it to work right > away, it was more to give you an idea of the kind of stuff you could do > manually. I'd expect it should be pretty straightforward to fix it. I > personally find it easier to write my own custom code to load such small > data files, rather than try to find the right parameters for an existing > function ;) (especially with mixed float / string dta). > > -=- Olivier > > 2011/5/19 Aradenatorix Veckhom Vacelaevus >> >> Thanks both for your helping: >> I was checking the script fro Olivier but doesn't works yet, later I tried >> with the asciitable package but I also never could read any of my files >> (finally I decide create a single file for each case, it means to get 2 >> arrays for each domain). >> I was reading a while about how to import data with genfromtxt in Numpy, >> it was very interesting because I have a lot of options for import data, I >> can skip headers and footers, I can import selected columns and other stuff, >> so I wanna use it for import my columns in each case and later get an output >> in a text file, how can I get it? >> My idea is to make something like: >> #!/usr/bin/env python >> #-*- coding:utf-8 -*- >> import numpy as np >> from StringIO import StringIO >> np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, >> usecols=(1, 2, 3)) >> np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, >> usecols=(4, 5, 6)) >> >> np.genfromtxt(StringIO(data_file_name), autostrip=True, skip_header=2, >> usecols=(-1)) >> But I'm not sure how to use it, I have not experience importing that >> stuff, but also I don't know If I need to add a line at the beginning with >> data = StringIO(data_file_name) >> Could you help me once more? >> Regards: >> Aradnix >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> I agree with Olivier that you need to write a special script for this type of file because it contains multiple tables. Most general purpose functions like genfromtxt assume a single 'array' but this will not work here because you have 10 (5 blocks of two arrays) in a single file. If you really need help then you really need to provide the actual file without any changes somewhere. Bruce From aradnix at gmail.com Fri May 20 21:36:46 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Fri, 20 May 2011 20:36:46 -0500 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: Hi again: Thanks Olivier, I understood you only wanna show me a way for solve my problem. I'm reading about the options in python for files and stuff. I think is better to write a code for solve our problems, but I found very useful the genfromtxt tool so I want to use it if I can't I have to parse my file with a lot of loops. And I agree with Bruce also about the complexity of my original data file so I divided the original file in ten files each one with an array for simplify the task, now I need to see how to open each file for can read it and obtain what I need from. I think I need to make two scripts one for the first type arrays and the second for the others where maybe I need to operate over the columns, and I think I really need your helping, but how to provide the actual file? is a little big for attach here. Thanks again for all your helping and advices. Regards: Aradnix -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat May 21 06:40:37 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 21 May 2011 12:40:37 +0200 Subject: [Numpy-discussion] Broken numpy install Message-ID: Hello everyone, I have a problem with my numpy installation. I installed numpy, scipy and matplotlib from source, about one year ago (the macports install wouldn't work for me). This worked fine untill I had some problems two days ago (as it later turned out, completely unrelated to numpy) but my thesis advisor suggested that I update numpy and scipy. Since then I've been trying to make it work (and haven't worked on my thesis at all). So, I updated the git/svn code to the latest versions, re-ran build and install and things don't work any more. I have a script that I am trying to use and here is the error message that I get: RuntimeError: module compiled against ABI version 1000009 but this version of numpy is 2000000 Traceback (most recent call last): File "../ppm/libppm/utils/ppm_dbg/ppmdbg.py", line 2, in from mpl_toolkits.mplot3d import Axes3D File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in from axes3d import Axes3D File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/axes3d.py", line 13, in from matplotlib.axes import Axes, rcParams File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/axes.py", line 12, in import matplotlib.artist as martist File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/artist.py", line 6, in from transforms import Bbox, IdentityTransform, TransformedBbox, TransformedPath File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/transforms.py", line 34, in from matplotlib._path import affine_transform ImportError: numpy.core.multiarray failed to import I have tried googling and followed every piece of advice I could find. Nothing worked. Please let me know what info I need to provide so you can help me. Or tell me a way to completely clean my system and start all over (cause it seems that it is being built for the wrong python). Btw, some people on #scipy on freenode suggested that I try running the script with all interpreters I get from which -a python... they all failed with the same error message. Thanks in advance, Milan Mitrovic From ralf.gommers at googlemail.com Sat May 21 06:54:51 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 21 May 2011 12:54:51 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: On Sat, May 21, 2011 at 12:40 PM, Milan Mitrovic wrote: > Hello everyone, > > I have a problem with my numpy installation. I installed numpy, scipy > and matplotlib from source, about one year ago (the macports install > wouldn't work for me). This worked fine untill I had some problems two > days ago (as it later turned out, completely unrelated to numpy) but > my thesis advisor suggested that I update numpy and scipy. Since then > I've been trying to make it work (and haven't worked on my thesis at > all). > The reason for this is that you recompiled numpy but not matplotlib. You can either use numpy 1.6.0, which will work with the matplotlib you compiled earlier. Or you have to recompile matplotlib against your current numpy (not always easy on OS X). There are also up-to-date binary installers for numpy/scipy/mpl you can use. Ralf > So, I updated the git/svn code to the latest versions, re-ran build > and install and things don't work any more. I have a script that I am > trying to use and here is the error message that I get: > > RuntimeError: module compiled against ABI version 1000009 but this > version of numpy is 2000000 > Traceback (most recent call last): > File "../ppm/libppm/utils/ppm_dbg/ppmdbg.py", line 2, in > from mpl_toolkits.mplot3d import Axes3D > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/__init__.py", > line 1, in > from axes3d import Axes3D > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/axes3d.py", > line 13, in > from matplotlib.axes import Axes, rcParams > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/axes.py", > line 12, in > import matplotlib.artist as martist > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/artist.py", > line 6, in > from transforms import Bbox, IdentityTransform, TransformedBbox, > TransformedPath > File > "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/transforms.py", > line 34, in > from matplotlib._path import affine_transform > ImportError: numpy.core.multiarray failed to import > > I have tried googling and followed every piece of advice I could find. > Nothing worked. Please let me know what info I need to provide so you > can help me. Or tell me a way to completely clean my system and start > all over (cause it seems that it is being built for the wrong python). > > Btw, some people on #scipy on freenode suggested that I try running > the script with all interpreters I get from which -a python... they > all failed with the same error message. > > Thanks in advance, > Milan Mitrovic > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bje at air.net.au Sat May 21 06:55:48 2011 From: bje at air.net.au (Ben Elliston) Date: Sat, 21 May 2011 20:55:48 +1000 Subject: [Numpy-discussion] finding longest contiguous array slices Message-ID: <20110521105548.GA4469@air.net.au> Hi all. I have a 3-d array of booleans (5 x 5 x 1000, say) and would like to return a 5 x 5 of array slices containing the longest contiguous sequence of True values along the z axis. Any tricks that I am missing that will make this easy? Thanks, Ben From milan.v.mitrovic at gmail.com Sat May 21 07:04:57 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 21 May 2011 13:04:57 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: Ok :) I just installed all three packages from the dmg files... and it still fails. same error message. Which python (32bit/64bit) and which matplotlib to use? Thanks again Milan From ralf.gommers at googlemail.com Sat May 21 07:16:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 21 May 2011 13:16:29 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: On Sat, May 21, 2011 at 1:04 PM, Milan Mitrovic wrote: > Ok :) > > I just installed all three packages from the dmg files... and it still > fails. same error message. Which python (32bit/64bit) and which > matplotlib to use? > > If you still get the same error then python is not picking up your newly-installed numpy 1.6.0. You are using Python 2.7, which unfortunately comes in two versions from python.org. Do you know which one you picked, then one ending in macosx10.3 (32-bit version) or macosx10.6 (64-bit version)? Matplotlib only has the 32-bit installer, so this is what you want: ** For python-2.7.1-macosx10.3.dmg ** numpy-1.6.0-py2.7-python.org-macosx10.3.dmg scipy-0.9.0-py2.7-python.org-macosx10.3.dmg matplotlib-1.0.0-python.org-32bit-py2.7-macosx10.3.dmg Or, if you want to stick to 64-bit python and the matplotlib you possibly already have: ** For python-2.7.1-macosx10.6.dmg ** numpy-1.6.0-py2.7-python.org-macosx10.6.dmg scipy-0.9.0-py2.7-python.org-macosx10.6.dmg Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat May 21 07:28:11 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 21 May 2011 13:28:11 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: I have the following: python-2.7.1-macosx10.3.dmg numpy-1.6.0-py2.7-python.org-macosx10.3.dmg scipy-0.9.0-py2.7-python.org-macosx10.3.dmg matplotlib-1.0.1-python.org-32bit-py2.7-macosx10.3.dmg I just installed in that order, and tried again... same error. From ralf.gommers at googlemail.com Sat May 21 08:25:55 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 21 May 2011 14:25:55 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: On Sat, May 21, 2011 at 1:28 PM, Milan Mitrovic wrote: > I have the following: > > python-2.7.1-macosx10.3.dmg > numpy-1.6.0-py2.7-python.org-macosx10.3.dmg > scipy-0.9.0-py2.7-python.org-macosx10.3.dmg > matplotlib-1.0.1-python.org-32bit-py2.7-macosx10.3.dmg > > I just installed in that order, and tried again... same error. > > That shouldn't be possible, maybe you're not running the python/numpy/matplotlib you expect. In your interpreter, do: >>> import os >>> os.__file__ '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc' >>> import numpy >>> numpy.__file__ '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.pyc' >>> numpy.__version__ '1.6.0' and the same for scipy/mpl. If that still gives you the same error, go into /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages, remove numpy/, scipy/, matplotlib/ and reinstall. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.v.mitrovic at gmail.com Sat May 21 08:38:18 2011 From: milan.v.mitrovic at gmail.com (Milan Mitrovic) Date: Sat, 21 May 2011 14:38:18 +0200 Subject: [Numpy-discussion] Broken numpy install In-Reply-To: References: Message-ID: Ah! Thats what I needed... There were some older numpy eggs lying arround there... That fixed it! Thank you very much! If you are coming to Zurich any time soon you have a beer waiting :) Cheers, Milan From mwwiebe at gmail.com Sat May 21 16:25:34 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Sat, 21 May 2011 15:25:34 -0500 Subject: [Numpy-discussion] Conversion from structured dtype to object changed in Numpy 1.6.0. In-Reply-To: References: Message-ID: On Thu, May 19, 2011 at 6:59 AM, Pauli Virtanen wrote: > Thu, 19 May 2011 12:36:22 +0100, Mark Dickinson wrote: > [clip] > >>>> from numpy import array > >>>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) > >>>> x.astype(object) > > array(1.2, dtype=object) > > > > Was this change intentional, or should I file a bug? I couldn't find > > any reference to it in the changelog or release notes. > > Looks like a bug -- the change was not intentional. > I've committed a fix for this. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdickinson at enthought.com Sat May 21 16:28:30 2011 From: mdickinson at enthought.com (Mark Dickinson) Date: Sat, 21 May 2011 21:28:30 +0100 Subject: [Numpy-discussion] Conversion from structured dtype to object changed in Numpy 1.6.0. In-Reply-To: References: Message-ID: On Sat, May 21, 2011 at 9:25 PM, Mark Wiebe wrote: > On Thu, May 19, 2011 at 6:59 AM, Pauli Virtanen wrote: >> >> Thu, 19 May 2011 12:36:22 +0100, Mark Dickinson wrote: >> [clip] >> >>>> from numpy import array >> >>>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >> >>>> x.astype(object) >> > array(1.2, dtype=object) >> > >> > Was this change intentional, or should I file a bug? ?I couldn't find >> > any reference to it in the changelog or release notes. >> >> Looks like a bug -- the change was not intentional. > > I've committed a fix for this. But I hadn't even got around to filing the bug report yet! Thank you. Mark From shish at keba.be Sat May 21 21:53:10 2011 From: shish at keba.be (Olivier Delalleau) Date: Sat, 21 May 2011 21:53:10 -0400 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: There are free file hosting services you can google up. -=- Olivier 2011/5/20 Aradenatorix Veckhom Vacelaevus > Hi again: > > Thanks Olivier, I understood you only wanna show me a way for solve my > problem. I'm reading about the options in python for files and stuff. I > think is better to write a code for solve our problems, but I found very > useful the genfromtxt tool so I want to use it if I can't I have to parse my > file with a lot of loops. > > And I agree with Bruce also about the complexity of my original data file > so I divided the original file in ten files each one with an array for > simplify the task, now I need to see how to open each file for can read it > and obtain what I need from. > > I think I need to make two scripts one for the first type arrays and the > second for the others where maybe I need to operate over the columns, and I > think I really need your helping, but how to provide the actual file? is a > little big for attach here. Thanks again for all your helping and advices. > > Regards: > Aradnix > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aradnix at gmail.com Sat May 21 22:01:36 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Sat, 21 May 2011 21:01:36 -0500 Subject: [Numpy-discussion] How can Import DATA from a Fortran file In-Reply-To: References: Message-ID: Hi: Mmm you're right, but now I'm in a hurry, and it's urgent to finish the visualization, so I could solve my problem now importing data in MS Excel and it worked! Of course I wanna do myself with a script for future cases, but now I have what I need. Thanks everybody for your helping, see you soon. Aradnix -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffspencerd at gmail.com Sun May 22 00:27:11 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Sun, 22 May 2011 14:27:11 +1000 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. Message-ID: <4DD8909F.5040704@gmail.com> I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory back to the OS. I have code here the exhibits the issue: http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g Then I read at http://pushingtheweb.com/2010/06/python-and-tcmalloc/ that this was a problem fixed by compiling your own version of Python using the TCMalloc library which I have now done. The issue is still exhibited so I was keen to know if possibly it was because numpy is compiled and linked to the default memory management library (not sure if this is true or how this works but would I also need to compile numpy linked to the TCMalloc library). Let me know if anyone has any insight on this issue. Or if it isn't related to numpy at all. -- ______________________________________________________ Jeffrey Spencer Department of Electronics and Electrical Engineering Nonlinear Signal Processing Group The University of Melbourne jeffspencerd at gmail.com (Mob.) 044 991 7726 From ralf.gommers at googlemail.com Sun May 22 07:15:11 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 22 May 2011 13:15:11 +0200 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. In-Reply-To: <4DD8909F.5040704@gmail.com> References: <4DD8909F.5040704@gmail.com> Message-ID: On Sun, May 22, 2011 at 6:27 AM, Jeffrey Spencer wrote: > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > back to the OS. I have code here the exhibits the issue: > > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g > > Can you post an actual example that shows your issue? That code doesn't do anything - it only imports some libraries and defines a class which is not even instantiated. > Then I read at http://pushingtheweb.com/2010/06/python-and-tcmalloc/ > that this was a problem fixed by compiling your own version of Python > using the TCMalloc library which I have now done. Funny how such a thing never comes up, and then twice in three days:) http://thread.gmane.org/gmane.comp.python.numeric.general/43965 Ralf > The issue is still > exhibited so I was keen to know if possibly it was because numpy is > compiled and linked to the default memory management library (not sure > if this is true or how this works but would I also need to compile numpy > linked to the TCMalloc library). Let me know if anyone has any insight > on this issue. Or if it isn't related to numpy at all. > > > -- > ______________________________________________________ > Jeffrey Spencer > Department of Electronics and Electrical Engineering > Nonlinear Signal Processing Group > The University of Melbourne > jeffspencerd at gmail.com > (Mob.) 044 991 7726 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun May 22 07:17:55 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 22 May 2011 13:17:55 +0200 Subject: [Numpy-discussion] AttributeError in numpy.distutils In-Reply-To: <4DD56133.3000405@astro.caltech.edu> References: <4DD2C98B.2050803@astro.caltech.edu> <4DD56133.3000405@astro.caltech.edu> Message-ID: On Thu, May 19, 2011 at 8:28 PM, Branimir Sesar wrote: > Dear Numpy users, > > I've encountered an AttributeError in numpy.distutils > > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > line 646, in swig_sources > extension.swig_opts.remove('-c++') > AttributeError: 'str' object has no attribute 'remove' > > while compiling some code with Python 2.7.1 and Numpy 1.6.0. > What are you doing here, compiling numpy? Building some of your own swig-ed code? Please give the details needed to reproduce your issue. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.p.krauss at gmail.com Sun May 22 08:13:28 2011 From: thomas.p.krauss at gmail.com (Tom Krauss) Date: Sun, 22 May 2011 07:13:28 -0500 Subject: [Numpy-discussion] finding longest contiguous array slices In-Reply-To: <20110521105548.GA4469@air.net.au> References: <20110521105548.GA4469@air.net.au> Message-ID: On Sat, May 21, 2011 at 5:55 AM, Ben Elliston wrote: > I have a 3-d array of booleans (5 x 5 x 1000, say) and would like to > return a 5 x 5 of array slices containing the longest contiguous > sequence of True values along the z axis. What do you mean by "5 x 5 of array slices containing the longest contiguous sequence of True values along the z axis"? So you want a 5x5xN slice of the array that is all True? Basically I would first find the index of the longest run of True, then form the slice. Here's an example borrowing heavily from "numpy-native" response here http://stackoverflow.com/questions/1066758/find-length-of-sequences-of-identical-values-in-a-numpy-array x = np.random.rand(5,5,1000)>.01 y = x.all(0).all(0) # <--- is this right? You want all 5x5 values True # make sure all runs of ones are well-bounded bounded = np.hstack(([0], y, [0])) # get 1 at run starts and -1 at run ends difs = np.diff(bounded) run_starts, = np.where(difs > 0) run_ends, = np.where(difs < 0) idx = np.argmax(run_ends - run_starts) x[:,:,run_starts[idx]:run_ends[idx]] # here is your slice If by "5 x 5 of array slices containing the longest contiguous sequence of True values along the z axis" you mean a different slice for each of the 5x5 sub-arrays, then you can use similar idea but find the run_starts and run_ends for each x[i,j]. From jeffspencerd at gmail.com Sun May 22 08:54:48 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Sun, 22 May 2011 22:54:48 +1000 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. In-Reply-To: References: Message-ID: <4DD90798.8060403@gmail.com> Sorry. I attached here the class and script to run it. It is just a minimalistic example of what I'm really doing that exhibits the same behavior. All the script does is run at.createSpecific after instantiating the class. I have seen this exhibited on many other cases but it runs fine in Windows XP. Here is what the script does: import MemoryLeak at = MemoryLeak.Model() at.createSpecific() Thanks, Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: MemoryLeak.py Type: text/x-python Size: 6350 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: runTempLeak.py Type: text/x-python Size: 62 bytes Desc: not available URL: From ralf.gommers at googlemail.com Sun May 22 09:42:41 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 22 May 2011 15:42:41 +0200 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. In-Reply-To: <4DD90798.8060403@gmail.com> References: <4DD90798.8060403@gmail.com> Message-ID: On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer wrote: > Sorry. I attached here the class and script to run it. It is just a > minimalistic example of what I'm really doing that exhibits the same > behavior. All the script does is run at.createSpecific after instantiating > the class. I have seen this exhibited on many other cases but it runs fine > in Windows XP. Here is what the script does: > > import MemoryLeak > at = MemoryLeak.Model() > at.createSpecific() > > That manages to eat up all my cpu cycles for about 5 minutes, but no leak (on OS X). I also don't see anything in the code that can potentially give problems. I suggest you strip this down a lot more so you can isolate the problem. If in your original code you were making a lot of figures in one of those for loops without explicitly closing them, that could perhaps be a problem. But you removed all I/O and matplotlib related code already (except for imports and some keywords). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Sun May 22 10:32:10 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 22 May 2011 09:32:10 -0500 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. In-Reply-To: References: <4DD90798.8060403@gmail.com> Message-ID: On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers wrote: > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer > wrote: >> >> Sorry. I attached here the class and script to run it. It is just a >> minimalistic example of what I'm really doing that exhibits the same >> behavior. All the script does is run at.createSpecific after instantiating >> the class. I have seen this exhibited on many other cases but it runs fine >> in Windows XP. Here is what the script does: >> >> import MemoryLeak >> at = MemoryLeak.Model() >> at.createSpecific() >> > That manages to eat up all my cpu cycles for about 5 minutes, but no leak > (on OS X). I also don't see anything in the code that can potentially give > problems. I suggest you strip this down a lot more so you can isolate the > problem. > > If in your original code you were making a lot of figures in one of those > for loops without explicitly closing them, that could perhaps be a problem. > But you removed all I/O and matplotlib related code already (except for > imports and some keywords). > > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > A good OS is likely to keep Python module in memory just because it knows that it might be requested again. However, from the little I know (too much LWN), if Linux needs that memory it will reclaim that memory only when it needs it or part of some other process. Measuring memory usage using simple tools (like ps or top) are very far from sufficient, so exactly how do you know that the memory has not been released? Bruce From ben_w_123 at yahoo.co.uk Sun May 22 10:49:48 2011 From: ben_w_123 at yahoo.co.uk (Ben Walsh) Date: Sun, 22 May 2011 15:49:48 +0100 Subject: [Numpy-discussion] datarray -- a couple of questions Message-ID: <4DD9228C.3020305@yahoo.co.uk> Hi I've been looking at various labelled-array packages, including Pandas (https://github.com/wesm/pandas) and datarray (https://github.com/fperez/datarray). I was quite interested in the design discussion for datarray, and I had a couple of questions: 1) Is this mailing list the correct place to ask about this? 2) Pandas supports mixed data types (eg. one column of floats, another column of strings). Does the datarray prototype support this? Is it simply a case of using a numpy.ndarray with dtype='object'? Cheers Ben From wesmckinn at gmail.com Sun May 22 12:03:10 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Sun, 22 May 2011 12:03:10 -0400 Subject: [Numpy-discussion] datarray -- a couple of questions In-Reply-To: <4DD9228C.3020305@yahoo.co.uk> References: <4DD9228C.3020305@yahoo.co.uk> Message-ID: On Sun, May 22, 2011 at 10:49 AM, Ben Walsh wrote: > Hi > > I've been looking at various labelled-array packages, including Pandas > (https://github.com/wesm/pandas) and datarray > (https://github.com/fperez/datarray). > > I was quite interested in the design discussion for datarray, and I had > a couple of questions: > > 1) Is this mailing list the correct place to ask about this? > > 2) Pandas supports mixed data types (eg. one column of floats, another > column of strings). Does the datarray prototype support this? Is it > simply a case of using a numpy.ndarray with dtype='object'? > > Cheers > > Ben > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > hi Ben, others can comment, but I think the answer to 1) is yes (if you have more specific usage questions about pandas feel free to ask on pystatsmodels, too). 2) datarray as currently conceived as a homogeneously typed object. So you could use dtype=object but occasionally have computational snafus. pandas tries to avoid these problems by segregating the floating point and non-floating point data. In recent discussions I do not think datarray is intended to be a step-in replacement for something like pandas, which is geared toward solving a bit more domain-specific set of problems (e.g. R-like data manipulations, time series / finance work, etc.). Rather, a future version of pandas will use datarray as its "base layer". So if you're looking for an (R) data.frame - like container, pandas.{DataFrame, DataMatrix} is probably your best bet for now larry is also worth looking at. But like datarray is also does not "do" mixed-type data. cheers, Wes From jeffspencerd at gmail.com Sun May 22 12:55:13 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Mon, 23 May 2011 02:55:13 +1000 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. In-Reply-To: References: <4DD90798.8060403@gmail.com> Message-ID: <4DD93FF1.1000707@gmail.com> Much reduced code exhibiting the issue. This shouldn't use much memory but just keeps going up and up because the copied window I assume gets left around and not taken up by the garbage collector. On 22/05/11 23:42, Ralf Gommers wrote: > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer > > wrote: > > Sorry. I attached here the class and script to run it. It is just > a minimalistic example of what I'm really doing that exhibits the > same behavior. All the script does is run at.createSpecific after > instantiating the class. I have seen this exhibited on many other > cases but it runs fine in Windows XP. Here is what the script does: > > import MemoryLeak > at = MemoryLeak.Model() > at.createSpecific() > > That manages to eat up all my cpu cycles for about 5 minutes, but no > leak (on OS X). I also don't see anything in the code that can > potentially give problems. I suggest you strip this down a lot more so > you can isolate the problem. > > If in your original code you were making a lot of figures in one of > those for loops without explicitly closing them, that could perhaps be > a problem. But you removed all I/O and matplotlib related code already > (except for imports and some keywords). > > Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: runTempLeak1.py Type: text/x-python Size: 723 bytes Desc: not available URL: From jeffspencerd at gmail.com Sun May 22 13:26:17 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Mon, 23 May 2011 03:26:17 +1000 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related. Message-ID: <4DD94739.6@gmail.com> Message: 3 Date: Sun, 22 May 2011 09:32:10 -0500 From: Bruce Southey Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=ISO-8859-1 On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers wrote: > > > > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer > > wrote: >> >> >> >> Sorry. I attached here the class and script to run it. It is just a >> >> minimalistic example of what I'm really doing that exhibits the same >> >> behavior. All the script does is run at.createSpecific after instantiating >> >> the class. I have seen this exhibited on many other cases but it runs fine >> >> in Windows XP. Here is what the script does: >> >> >> >> import MemoryLeak >> >> at = MemoryLeak.Model() >> >> at.createSpecific() >> >> > > That manages to eat up all my cpu cycles for about 5 minutes, but no leak > > (on OS X). I also don't see anything in the code that can potentially give > > problems. I suggest you strip this down a lot more so you can isolate the > > problem. > > > > If in your original code you were making a lot of figures in one of those > > for loops without explicitly closing them, that could perhaps be a problem. > > But you removed all I/O and matplotlib related code already (except for > > imports and some keywords). > > > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > A good OS is likely to keep Python module in memory just because it knows that it might be requested again. However, from the little I know (too much LWN), if Linux needs that memory it will reclaim that memory only when it needs it or part of some other process. Measuring memory usage using simple tools (like ps or top) are very far from sufficient, so exactly how do you know that the memory has not been released? Bruce Bruce, The reason I have the issue is it ends up swapping memory over to the swap. After this, the whole computer runs at a crawl. Even if as your saying it is allocated as free just not reclaimed from Python. The OS should try to get the memory back before swapping everything. I even have my swappiness set lower to avoid sending things to the swap. Jeff -- ________________________ Jeffrey Spencer jeffspencerd at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun May 22 13:32:43 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 22 May 2011 12:32:43 -0500 Subject: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. In-Reply-To: <4DD8909F.5040704@gmail.com> References: <4DD8909F.5040704@gmail.com> Message-ID: On Sat, May 21, 2011 at 23:27, Jeffrey Spencer wrote: > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > back to the OS. I have code here the exhibits the issue: > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g > > Then I read at http://pushingtheweb.com/2010/06/python-and-tcmalloc/ > that this was a problem fixed by compiling your own version of Python > using the TCMalloc library which I have now done. The issue is still > exhibited so I was keen to know if possibly it was because numpy is > compiled and linked to the default memory management library (not sure > if this is true or how this works but would I also need to compile numpy > linked to the TCMalloc library). Let me know if anyone has any insight > on this issue. Or if it isn't related to numpy at all. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bsesar at astro.caltech.edu Sun May 22 14:14:07 2011 From: bsesar at astro.caltech.edu (Branimir Sesar) Date: Sun, 22 May 2011 11:14:07 -0700 Subject: [Numpy-discussion] AttributeError in numpy.distutils In-Reply-To: References: <4DD2C98B.2050803@astro.caltech.edu> <4DD56133.3000405@astro.caltech.edu> Message-ID: <4DD9526F.6030509@astro.caltech.edu> On 05/22/2011 04:17 AM, Ralf Gommers wrote: > > > On Thu, May 19, 2011 at 8:28 PM, Branimir Sesar > > wrote: > > Dear Numpy users, > > I've encountered an AttributeError in numpy.distutils > > File > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > line 646, in swig_sources > extension.swig_opts.remove('-c++') > AttributeError: 'str' object has no attribute 'remove' > > while compiling some code with Python 2.7.1 and Numpy 1.6.0. > > > What are you doing here, compiling numpy? Building some of your own > swig-ed code? Please give the details needed to reproduce your issue. I've been trying to compile Scikits ANN (http://projects.scipy.org/scikits/wiki/AnnWrapper) with Python 2.7.1, numpy 1.6.0, and SWIG 2.0.3 but the compilation breaks down down with the error given below. Previously, I was able to compile Scikits ANN using Enthought Python Distribution 7.0.2 (Python 2.7.1, numpy 1.5.1, swig 1.3.40). running install running bdist_egg running egg_info running build_src build_src building extension "scikits.ann._ANN" sources Traceback (most recent call last): File "setup.py", line 139, in test_suite = 'nose.collector', File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", line 57, in run r = self.setuptools_run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", line 51, in setuptools_run self.do_egg_install() File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", line 96, in do_egg_install File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 167, in run File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/egg_info.py", line 8, in run self.run_command("build_src") File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 332, in build_extension_sources sources = self.swig_sources(sources, ext) File "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", line 646, in swig_sources extension.swig_opts.remove('-c++') AttributeError: 'str' object has no attribute 'remove' From jeffspencerd at gmail.com Sun May 22 14:17:39 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Mon, 23 May 2011 04:17:39 +1000 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related. Message-ID: <4DD95343.1060806@gmail.com> Further isolated the error and it seems to come in from the sum function in numpy. I will probe further later but the code below causes the memory usage to go up and up. If you change sum below to some other function there are no issues. from numpy import arange, sum for x in range(10000000): inhibVal = sum(arange(15)) Cheers, Jeff Message: 3 Date: Sun, 22 May 2011 09:32:10 -0500 From: Bruce Southey Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=ISO-8859-1 On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers wrote: > > > > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer > > wrote: >> >> >> >> Sorry. I attached here the class and script to run it. It is just a >> >> minimalistic example of what I'm really doing that exhibits the same >> >> behavior. All the script does is run at.createSpecific after instantiating >> >> the class. I have seen this exhibited on many other cases but it runs fine >> >> in Windows XP. Here is what the script does: >> >> >> >> import MemoryLeak >> >> at = MemoryLeak.Model() >> >> at.createSpecific() >> >> > > That manages to eat up all my cpu cycles for about 5 minutes, but no leak > > (on OS X). I also don't see anything in the code that can potentially give > > problems. I suggest you strip this down a lot more so you can isolate the > > problem. > > > > If in your original code you were making a lot of figures in one of those > > for loops without explicitly closing them, that could perhaps be a problem. > > But you removed all I/O and matplotlib related code already (except for > > imports and some keywords). > > > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > A good OS is likely to keep Python module in memory just because it knows that it might be requested again. However, from the little I know (too much LWN), if Linux needs that memory it will reclaim that memory only when it needs it or part of some other process. Measuring memory usage using simple tools (like ps or top) are very far from sufficient, so exactly how do you know that the memory has not been released? Bruce -- ________________________ Jeffrey Spencer jeffspencerd at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From efiring at hawaii.edu Sun May 22 16:33:54 2011 From: efiring at hawaii.edu (Eric Firing) Date: Sun, 22 May 2011 10:33:54 -1000 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related. In-Reply-To: <4DD95343.1060806@gmail.com> References: <4DD95343.1060806@gmail.com> Message-ID: <4DD97332.1080608@hawaii.edu> On 05/22/2011 08:17 AM, Jeffrey Spencer wrote: > from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy 1.5.1 package from ubuntu, and using 1.6.1.dev-a265004. efiring at manini:~$ uname -a Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Eric From bsouthey at gmail.com Sun May 22 21:28:16 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sun, 22 May 2011 20:28:16 -0500 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related. In-Reply-To: <4DD95343.1060806@gmail.com> References: <4DD95343.1060806@gmail.com> Message-ID: On Sun, May 22, 2011 at 1:17 PM, Jeffrey Spencer wrote: > Further isolated the error and it seems to come in from the sum function in > numpy. I will probe further later but the code below causes the memory usage > to go up and up. If you change sum below to some other function there are no > issues. > > from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) > > Cheers, > Jeff > > Message: 3 > Date: Sun, 22 May 2011 09:32:10 -0500 > From: Bruce Southey > Subject: Re: [Numpy-discussion] Python memory management issues using > Linux. Maybe Numpy, related. > To: Discussion of Numerical Python > Message-ID: > Content-Type: text/plain; charset=ISO-8859-1 > > On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers > wrote: > >> >> >> On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer >> wrote: > >>> >>> Sorry. I attached here the class and script to run it. It is just a >>> minimalistic example of what I'm really doing that exhibits the same >>> behavior. All the script does is run at.createSpecific after >>> instantiating >>> the class. I have seen this exhibited on many other cases but it runs >>> fine >>> in Windows XP. Here is what the script does: >>> >>> import MemoryLeak >>> at = MemoryLeak.Model() >>> at.createSpecific() >>> > >> That manages to eat up all my cpu cycles for about 5 minutes, but no leak >> (on OS X). I also don't see anything in the code that can potentially give >> problems. I suggest you strip this down a lot more so you can isolate the >> problem. >> >> If in your original code you were making a lot of figures in one of those >> for loops without explicitly closing them, that could perhaps be a >> problem. >> But you removed all I/O and matplotlib related code already (except for >> imports and some keywords). >> >> Ralf >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > A good OS is likely to keep Python module in memory just because it > knows that it might be requested again. However, from the little I > know (too much LWN), if Linux needs that memory it will reclaim that > memory only when it needs it or part of some other process. > > Measuring memory usage using simple tools (like ps or top) are very > far from sufficient, so exactly how do you know that the memory has > not been released? > > Bruce > > > -- > ________________________ > Jeffrey Spencer > jeffspencerd at gmail.com > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > That example is just memory exhaustion not a memory leak because the OS does reclaim that memory. Sure it is not immediate but it does eventually happen otherwise you would end up to no memory left. Please note that numpy requires contiguous memory so allocating huge arrays, relative to the amount of available memory, in such a manner *will* cause thrashing regardless. Also take a look at Pythons' garbage collection module (http://docs.python.org/library/gc.html) and perhaps use manual garbage collection. There is a lot out there on this. As per Wikipedia (http://en.wikipedia.org/wiki/Memory_leak#Other_memory_consumers) "Note that constantly increasing memory usage is not necessarily evidence of a memory leak." But probably more relevant definitions are: http://stackoverflow.com/questions/312069/the-best-memory-leak-definition Bruce From jeffspencerd at gmail.com Sun May 22 23:44:56 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Mon, 23 May 2011 13:44:56 +1000 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related. Message-ID: <4DD9D838.3000602@gmail.com> There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. How do I build against the -ltcmalloc library? Just not sure how to input compile time arguments with the build script. from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy 1.5.1 package from ubuntu, and using 1.6.1.dev-a265004. efiring at manini:~$ uname -a Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Eric Also noticed that the same issue occurs as described with using the prod instead of sum. Other commands like copy and reshape have no issues. I have a machine with 11.04 on it so I will give that a go later to see what goes. Thanks, Jeff Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer wrote: > > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > > back to the OS. I have code here the exhibits the issue: > > > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g > > > > Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/ > > that this was a problem fixed by compiling your own version of Python > > using the TCMalloc library which I have now done. The issue is still > > exhibited so I was keen to know if possibly it was because numpy is > > compiled and linked to the default memory management library (not sure > > if this is true or how this works but would I also need to compile numpy > > linked to the TCMalloc library). Let me know if anyone has any insight > > on this issue. Or if it isn't related to numpy at all. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco -- ________________________ Jeffrey Spencer jeffspencerd at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffspencerd at gmail.com Mon May 23 04:54:09 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Mon, 23 May 2011 18:54:09 +1000 Subject: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related. Message-ID: <4DDA20B1.104@gmail.com> I realized if I re-build numpy it takes the interpreter compile options or you can edit site.cfg. I actually solved the issue by getting the newest svn copy of numpy. It must be something with 1.6b2. The newest SVN has no issues with either library and calls to sum or prod. I was getting memory leak with both the tcmalloc and malloc with the 1.6b2 version of numpy. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. How do I build against the -ltcmalloc library? Just not sure how to input compile time arguments with the build script. from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy 1.5.1 package from ubuntu, and using 1.6.1.dev-a265004. efiring at manini:~$ uname -a Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Eric Also noticed that the same issue occurs as described with using the prod instead of sum. Other commands like copy and reshape have no issues. I have a machine with 11.04 on it so I will give that a go later to see what goes. Thanks, Jeff Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. To: Discussion of Numerical Python Message-ID: Content-Type: text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer wrote: > > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > > back to the OS. I have code here the exhibits the issue: > > > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g > > > > Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/ > > that this was a problem fixed by compiling your own version of Python > > using the TCMalloc library which I have now done. The issue is still > > exhibited so I was keen to know if possibly it was because numpy is > > compiled and linked to the default memory management library (not sure > > if this is true or how this works but would I also need to compile numpy > > linked to the TCMalloc library). Let me know if anyone has any insight > > on this issue. Or if it isn't related to numpy at all. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco -- ________________________ Jeffrey Spencer jeffspencerd at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon May 23 14:33:09 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 May 2011 14:33:09 -0400 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop Message-ID: I have a function in two versions, one vectorized, one with loop the vectorized function gets all randn variables in one big array rvs = distr.rvs(args, **{'size':(nobs, nrep)}) the looping version has: for irep in xrange(nrep): rvs = distr.rvs(args, **{'size':nobs}) the rest should be identical (except for vectorization Is there a guarantee that the 2d arrays are filled up in a specific order so that the loop and vectorized version produce the same result, given the same seed? In my example it doesn't work, ( bootstrap p-values for Anderson-Darling A^2 with estimated loc and scale) >>> np.random.seed(8765679) >>> [bootstrap2(c/ (1 + 4./nobs - 25./nobs**2), NewNorm(), args=(0,1), nobs=nobs, nrep=10000) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14449999999999999, 0.0901, 0.042200000000000001, 0.0212, 0.0076] >>> np.random.seed(8765679) >>> [bootstrap(NewNorm(), args=(0,1), nobs=nobs, nrep=10000, value=c/ (1 + 4./nobs - 25./nobs**2)) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14710000000000001, 0.092899999999999996, 0.044499999999999998, 0.022800000000000001, 0.0080999999999999996] >>> np.array([15.0, 10.0, 5.0, 2.5, 1.0])/100. #Stephens array([ 0.15 , 0.1 , 0.05 , 0.025, 0.01 ]) Josef From kwgoodman at gmail.com Mon May 23 14:42:58 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 23 May 2011 11:42:58 -0700 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: On Mon, May 23, 2011 at 11:33 AM, wrote: > I have a function in two versions, one vectorized, one with loop > > the vectorized function ?gets all randn variables in one big array > rvs = distr.rvs(args, **{'size':(nobs, nrep)}) > > the looping version has: > ? ?for irep in xrange(nrep): > ? ? ? ?rvs = distr.rvs(args, **{'size':nobs}) > > the rest should be identical (except for vectorization > > Is there a guarantee that the 2d arrays are filled up in a specific > order so that the loop and vectorized version produce the same result, > given the same seed? Are you pulling the numbers from rows or columns of the 2d array? Columns seem to work: >> rs = np.random.RandomState([1,2,3]) >> rs.randn(3,3) array([[ 0.89858245, 0.25528877, 0.95172625], [-0.05663392, 0.54721555, 0.11512385], [ 0.82495129, 0.17252144, 0.74570118]]) which gives the same as >> rs = np.random.RandomState([1,2,3]) >> rs.randn(3) array([ 0.89858245, 0.25528877, 0.95172625]) >> rs.randn(3) array([-0.05663392, 0.54721555, 0.11512385]) >> rs.randn(3) array([ 0.82495129, 0.17252144, 0.74570118]) I get similar results with np.random.seed From kwgoodman at gmail.com Mon May 23 14:44:27 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 23 May 2011 11:44:27 -0700 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: On Mon, May 23, 2011 at 11:42 AM, Keith Goodman wrote: > On Mon, May 23, 2011 at 11:33 AM, ? wrote: >> I have a function in two versions, one vectorized, one with loop >> >> the vectorized function ?gets all randn variables in one big array >> rvs = distr.rvs(args, **{'size':(nobs, nrep)}) >> >> the looping version has: >> ? ?for irep in xrange(nrep): >> ? ? ? ?rvs = distr.rvs(args, **{'size':nobs}) >> >> the rest should be identical (except for vectorization >> >> Is there a guarantee that the 2d arrays are filled up in a specific >> order so that the loop and vectorized version produce the same result, >> given the same seed? > > Are you pulling the numbers from rows or columns of the 2d array? > Columns seem to work: Sorry, I meant rows. >>> rs = np.random.RandomState([1,2,3]) >>> rs.randn(3,3) > array([[ 0.89858245, ?0.25528877, ?0.95172625], > ? ? ? [-0.05663392, ?0.54721555, ?0.11512385], > ? ? ? [ 0.82495129, ?0.17252144, ?0.74570118]]) > > which gives the same as > >>> rs = np.random.RandomState([1,2,3]) >>> rs.randn(3) > ? array([ 0.89858245, ?0.25528877, ?0.95172625]) >>> rs.randn(3) > ? array([-0.05663392, ?0.54721555, ?0.11512385]) >>> rs.randn(3) > ? array([ 0.82495129, ?0.17252144, ?0.74570118]) > > I get similar results with np.random.seed > From robert.kern at gmail.com Mon May 23 15:02:35 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 23 May 2011 14:02:35 -0500 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: On Mon, May 23, 2011 at 13:33, wrote: > I have a function in two versions, one vectorized, one with loop > > the vectorized function ?gets all randn variables in one big array > rvs = distr.rvs(args, **{'size':(nobs, nrep)}) > > the looping version has: > ? ?for irep in xrange(nrep): > ? ? ? ?rvs = distr.rvs(args, **{'size':nobs}) > > the rest should be identical (except for vectorization > > Is there a guarantee that the 2d arrays are filled up in a specific > order so that the loop and vectorized version produce the same result, > given the same seed? No general guarantee for all of the scipy distributions, no. I suspect that all of the RandomState methods do work this way, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From bsouthey at gmail.com Mon May 23 15:27:34 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 23 May 2011 14:27:34 -0500 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: <4DDAB526.2050605@gmail.com> On 05/23/2011 02:02 PM, Robert Kern wrote: > On Mon, May 23, 2011 at 13:33, wrote: >> I have a function in two versions, one vectorized, one with loop >> >> the vectorized function gets all randn variables in one big array >> rvs = distr.rvs(args, **{'size':(nobs, nrep)}) >> >> the looping version has: >> for irep in xrange(nrep): >> rvs = distr.rvs(args, **{'size':nobs}) >> >> the rest should be identical (except for vectorization What happened to your 'irep' and 'nrep' variables in the vectorized and looping versions, respectively? The looping version is overwriting 'rvs' but the vectorized version is not unless you are accumulating it elsewhere. (If so then there is another difference..). Also, I try to avoid having variables the same name as a function just in case (old habits die hard). >> Is there a guarantee that the 2d arrays are filled up in a specific >> order so that the loop and vectorized version produce the same result, >> given the same seed? > No general guarantee for all of the scipy distributions, no. I suspect > that all of the RandomState methods do work this way, though. > You have to guarantee that the complete stream of random numbers was not interrupted such as an addition calls or reset between loops. That probably means to generate and store all of the numbers at once and then just access that array as needed. But then again, if you are doing bootstrapping, it really should not matter if you do 'sufficient' resamples. Bruce From josef.pktd at gmail.com Mon May 23 15:34:04 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 May 2011 15:34:04 -0400 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: On Mon, May 23, 2011 at 3:02 PM, Robert Kern wrote: > On Mon, May 23, 2011 at 13:33, ? wrote: >> I have a function in two versions, one vectorized, one with loop >> >> the vectorized function ?gets all randn variables in one big array >> rvs = distr.rvs(args, **{'size':(nobs, nrep)}) >> >> the looping version has: >> ? ?for irep in xrange(nrep): >> ? ? ? ?rvs = distr.rvs(args, **{'size':nobs}) >> >> the rest should be identical (except for vectorization >> >> Is there a guarantee that the 2d arrays are filled up in a specific >> order so that the loop and vectorized version produce the same result, >> given the same seed? > > No general guarantee for all of the scipy distributions, no. I suspect > that all of the RandomState methods do work this way, though. Thanks Keith, Robert, It should work then for some test cases, but no guarantee since the function should eventually take arbitrary distributions. Obviously I was working by columns, using a transpose worked, but rewriting to axis=1 instead of axis=0 which should be more efficient since I had almost all calculations by columns, I needed params = map(lambda x: np.expand_dims(x, 1), params) to get around broadcast errors that work automatically with axis=0 >>> np.random.seed(8765679) >>> [bootstrap(NewNorm(), args=(0,1), nobs=nobs, nrep=10000, value=c/ (1 + 4./nobs - 25./nobs**2)) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14449999999999999, 0.0901, 0.042200000000000001, 0.0212, 0.0076] >>> np.random.seed(8765679) >>> [bootstrap2(c/ (1 + 4./nobs - 25./nobs**2), NewNorm(), args=(0,1), nobs=nobs, nrep=10000) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14449999999999999, 0.0901, 0.042200000000000001, 0.0212, 0.0076] Josef > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Mon May 23 15:41:55 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 23 May 2011 12:41:55 -0700 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: References: Message-ID: On Mon, May 23, 2011 at 12:34 PM, wrote: > Obviously I was working by columns, using a transpose worked, but > rewriting to axis=1 instead of axis=0 which should be more efficient > since I had almost all calculations by columns, I needed > params = map(lambda x: np.expand_dims(x, 1), params) > to get around broadcast errors that work automatically with axis=0 Here's another way to work by columns (note the input parameter oder='F'): >> rs = np.random.RandomState([1,2,3]) >> rs.randn(9).reshape(3,3,order='F') array([[ 0.89858245, -0.05663392, 0.82495129], [ 0.25528877, 0.54721555, 0.17252144], [ 0.95172625, 0.11512385, 0.74570118]]) >> rs = np.random.RandomState([1,2,3]) >> rs.randn(3) array([ 0.89858245, 0.25528877, 0.95172625]) >> rs.randn(3) array([-0.05663392, 0.54721555, 0.11512385]) >> rs.randn(3) array([ 0.82495129, 0.17252144, 0.74570118]) From josef.pktd at gmail.com Mon May 23 15:47:59 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 May 2011 15:47:59 -0400 Subject: [Numpy-discussion] random seed replicate 2d randn with 1d loop In-Reply-To: <4DDAB526.2050605@gmail.com> References: <4DDAB526.2050605@gmail.com> Message-ID: On Mon, May 23, 2011 at 3:27 PM, Bruce Southey wrote: > On 05/23/2011 02:02 PM, Robert Kern wrote: >> On Mon, May 23, 2011 at 13:33, ?wrote: >>> I have a function in two versions, one vectorized, one with loop >>> >>> the vectorized function ?gets all randn variables in one big array >>> rvs = distr.rvs(args, **{'size':(nobs, nrep)}) >>> >>> the looping version has: >>> ? ? for irep in xrange(nrep): >>> ? ? ? ? rvs = distr.rvs(args, **{'size':nobs}) >>> >>> the rest should be identical (except for vectorization > What happened to your 'irep' and 'nrep' variables in the vectorized and > looping versions, respectively? > The looping version is overwriting 'rvs' but the vectorized version is > not unless you are accumulating it elsewhere. (If so then there is > another difference..). Yes in the loop case, I'm accumulating only what I need and throw away the random numbers immediately. In the vectorized version, I build and use the entire (nrep, nobs) = (10000, 200) or (10000, 1000) array at once, which is faster than the loop. > Also, I try to avoid having variables the same name as a function just > in case (old habits die hard). rvs is a method, I never have a function called rvs, and it's one of my favorite names for an array of random variables during testing. > >>> Is there a guarantee that the 2d arrays are filled up in a specific >>> order so that the loop and vectorized version produce the same result, >>> given the same seed? >> No general guarantee for all of the scipy distributions, no. I suspect >> that all of the RandomState methods do work this way, though. >> > You have to guarantee that the complete stream of random numbers was not > interrupted such as an addition calls or reset between loops. That > probably means to generate and store all of the numbers at once and then > just access that array as needed. > > But then again, if you are doing bootstrapping, it really should not > matter if you do 'sufficient' resamples. I won't rely on it later on, and I don't think it makes a difference, but getting identical results across different implementations is very useful for testing. Josef > > > Bruce > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From goocreations at gmail.com Tue May 24 03:29:42 2011 From: goocreations at gmail.com (Goo Creations) Date: Tue, 24 May 2011 09:29:42 +0200 Subject: [Numpy-discussion] 2D Fourier Transform with 1D functions Message-ID: Hi all I know there exists a function for Fourier transforming 2D : fft2 But I'm trying to understand how this function works. As far as I know, first a column based transform is done and then a row based transform. This is what I've created for a test: *test = np.array([[1+0j,2+0j], [3+0j, 4+0j]]) own = np.array([[1+0j, 2+0j, 3+0j, 4+0j]]) testRes = np.fft.fft2(test, (2,2)) print testRes ownRes = np.fft.fft(own,4) * At this point I'm stuck, since I'm not sure how to do the row based transform on ownRes. Is there anyone out there who knows how to do this? Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.wuyts at gmail.com Tue May 24 05:41:28 2011 From: jan.wuyts at gmail.com (Jan Wuyts) Date: Tue, 24 May 2011 11:41:28 +0200 Subject: [Numpy-discussion] filtering rows from a numpy.ndarray Message-ID: I've read in a datafile to an ndarray with this command: data=np.genfromtxt('n.txt', delimiter='\t', dtype=[('paramname','S26'), ('study','S26'), ('class','S10'), ('count','f4'), ('average','f4'), ('sd','f4')) I would like to filter the array to only rows where paramname == 'bmi'. What would be the most elegant way of doing this? I found 1 way for doing this but it doesn't look very efficient nor elegant. bmidata=np.asarray([x for x in data if x['paramname']=='bmi'], dtype=data.dtype) any simpler ways? Thanks, Jan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue May 24 05:48:36 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 24 May 2011 09:48:36 +0000 (UTC) Subject: [Numpy-discussion] filtering rows from a numpy.ndarray References: Message-ID: Tue, 24 May 2011 11:41:28 +0200, Jan Wuyts wrote: > bmidata=np.asarray([x for x in data if x['paramname']=='bmi'], > dtype=data.dtype) Use indexing: bmidata = data[data['paramname'] == 'bmi'] From tmp50 at ukr.net Tue May 24 06:22:47 2011 From: tmp50 at ukr.net (Dmitrey) Date: Tue, 24 May 2011 13:22:47 +0300 Subject: [Numpy-discussion] [ANN] Guaranteed solution of nonlinear equation(s) Message-ID: Hi all, I have made my free solver interalg (http://openopt.org/interalg) be capable of solving nonlinear equations and systems of them. Unlike scipy optimize fsolve it doesn't matter which functions are involved - convex, nonconvex, multiextremum etc. Even some discontinuous funcs can be handled. If no solution exists, interalg determines it rather quickly. For more info see http://forum.openopt.org/viewtopic.php?id=423 Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue May 24 09:28:42 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 24 May 2011 15:28:42 +0200 Subject: [Numpy-discussion] 2D Fourier Transform with 1D functions In-Reply-To: References: Message-ID: Hi Chris On Tue, May 24, 2011 at 9:29 AM, Goo Creations wrote: > At this point I'm stuck, since I'm not sure how to do the row based > transform on ownRes. > Is there anyone out there who knows how to do this? NumPy's FFT supports an "axis" argument, so you can do In [29]: own = np.array([[1+0j, 2+0j], [3+0j, 4+0j]]) In [30]: fft_step1 = np.fft.fft(own, axis=0) In [31]: fft_step2 = np.fft.fft(fft_step1, axis=1) In [32]: fft_step2 Out[32]: array([[ 10.+0.j, -2.+0.j], [ -4.+0.j, 0.+0.j]]) In [33]: np.fft.fft2(own) Out[33]: array([[ 10.+0.j, -2.+0.j], [ -4.+0.j, 0.+0.j]]) Cheers St?fan From goocreations at gmail.com Tue May 24 09:53:34 2011 From: goocreations at gmail.com (Goo Creations) Date: Tue, 24 May 2011 15:53:34 +0200 Subject: [Numpy-discussion] 2D Fourier Transform with 1D functions In-Reply-To: References: Message-ID: Thank you very much St?fan!! 2011/5/24 St?fan van der Walt > Hi Chris > > On Tue, May 24, 2011 at 9:29 AM, Goo Creations > wrote: > > At this point I'm stuck, since I'm not sure how to do the row based > > transform on ownRes. > > Is there anyone out there who knows how to do this? > > NumPy's FFT supports an "axis" argument, so you can do > > In [29]: own = np.array([[1+0j, 2+0j], [3+0j, 4+0j]]) > > In [30]: fft_step1 = np.fft.fft(own, axis=0) > > In [31]: fft_step2 = np.fft.fft(fft_step1, axis=1) > > In [32]: fft_step2 > Out[32]: > array([[ 10.+0.j, -2.+0.j], > [ -4.+0.j, 0.+0.j]]) > > In [33]: np.fft.fft2(own) > Out[33]: > array([[ 10.+0.j, -2.+0.j], > [ -4.+0.j, 0.+0.j]]) > > Cheers > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From craigyk at me.com Tue May 24 13:39:02 2011 From: craigyk at me.com (Craig Yoshioka) Date: Tue, 24 May 2011 10:39:02 -0700 Subject: [Numpy-discussion] labeled axes Message-ID: Hi all, I've read some discussions about adding labeled axes, and even ticks, to numpy arrays (such as in Luis' dataarray). I have recently found that the ability to label axes would be very helpful to me, but I'd like to keep the implementation as lightweight as possible. The reason I would find this useful is because I am writing a ndarray subclass that loads image/volume file formats into numpy arrays. Some of these files might have multiple images/volumes, I'll call them channels, and also may have an additional dimension for vectors associated with each pixel/voxel, like color. The max dims of the array would then be 5. Example: data = ndarray([1023,128,128,128,3]) might mean (channels,z,y,x,rgb) for one array. Now I want to keep as much of the fancy indexing capabilities of numpy as I can, but I am finding it difficult to track the removal of axes that can occur from indexing. For example data[2,2] would return an array of shape (128,128,3), or the third slice through the third volume in the dataset, but the returned array has lost the meaning associated with its axes, so saving it back out would require manual relabeling of the axes. I'd like to be able to track the axes as metadata and retain all the fancy numpy indexing. There are two ways I could accomplish this with minimal code on the python side: One would be if indexing of the array always returned an array of the same dimensionality, that is data[2,2] returned an array of shape (1,1,128,128,3). I could then delete the degenerate axes labels from the metadata, and return the compressed array, resulting in the same output: class Data(np.ndarray): def __getitem__(self,indices): data = np.ndarray.__getitem__(self,indices,donotcompress=True) # as an example data.axeslabels = [label for label,dim in zip(self.axeslabels,data.shape) if dim > 1] return data.compress() def __getslice__(self,s1,s2,step): # trivial case Another approach would be if there is some function in the numpy internals that I could use to get the needed information before calling the ndarray's __getitem__ function: class Data(np.ndarray): def __getitem__(self,indices): unique = np.uniqueIndicesPerDimension(indices) data = np.ndarray.__getitem__(self,indices) data.axeslabels = [label for label,dim in zip(self.axeslabels, unique) if dim > 1] return data Finally, I could implement my own parser for the passed indices to figure this out myself. This would be bad since I'd have to recreate a lot of the same code that must go on inside numpy, and it would be slower, error-prone, etc. : class Data(np.ndarray): def __getitem__(self,indices): indices = self.uniqueDimensionIndices(indices) data = np.ndarray.__getitem__(self,indices) data.axeslabels = [label for label,dim in zip(self.axeslabels,indices) if dim > 1] return data def uniqueDimensionIndices(self,indices): if isinstance(indices,int): indices = (indices,) if isinstance(indices,tuple): .... elif isinstance(indices,list): ... Is there anything in the numpy internals already that would allow me to do #1 or #2?, I don't think #3 is a very good option. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kathleen.M.Tacina at nasa.gov Tue May 24 19:22:33 2011 From: Kathleen.M.Tacina at nasa.gov (Kathleen M Tacina) Date: Tue, 24 May 2011 23:22:33 +0000 Subject: [Numpy-discussion] Runtime error in numpy development version Message-ID: <1306279353.28355.105.camel@MOSES.grc.nasa.gov> I get the following runtime error when I import numpy: "module compiled against ABI version 2000000 but this version of numpy is 1000009" (see below). Python 2.7.1 (r271:86832, May 20 2011, 14:41:42) [GCC 4.4.0 20090514 (Red Hat 4.4.0-6)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np RuntimeError: module compiled against ABI version 2000000 but this version of numpy is 1000009 >>> np.version.version '2.0.0.dev-3071eab' >>> This error doesn't seem to effect the operation of numpy --- most tests in np.test('full') pass and most of the numpy functions I used regularly work --- but it causes the import of opencv to fail. >>> import cv RuntimeError: module compiled against ABI version 2000000 but this version of numpy is 1000009 Traceback (most recent call last): File "", line 1, in ImportError: numpy.core.multiarray failed to import >>> import numpy.core.multiarray >>> numpy.core.multiarray.arange(5) array([0, 1, 2, 3, 4]) >>> Thanks for any help you can give me with this. Kathy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffspencerd at gmail.com Tue May 24 21:00:50 2011 From: jeffspencerd at gmail.com (Jeffrey Spencer) Date: Wed, 25 May 2011 11:00:50 +1000 Subject: [Numpy-discussion] Runtime error in numpy development version In-Reply-To: <1306279353.28355.105.camel@MOSES.grc.nasa.gov> References: <1306279353.28355.105.camel@MOSES.grc.nasa.gov> Message-ID: <4DDC54C2.5090106@gmail.com> Try removing numpy and openCV (and associated files) from /usr/local/lib/python2.7/dist-packages or wherever you originally compiled it too. Then rebuild numpy and openCV. Probably openCV was compiled with a different version of numpy at the time of compilation. Just a guess but removing and rebuild should work. Cheers, Jeff On 05/25/2011 09:22 AM, Kathleen M Tacina wrote: > I get the following runtime error when I import numpy: "module > compiled against ABI version 2000000 but this version of numpy is > 1000009" (see below). > > Python 2.7.1 (r271:86832, May 20 2011, 14:41:42) > [GCC 4.4.0 20090514 (Red Hat 4.4.0-6)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > RuntimeError: module compiled against ABI version 2000000 but this > version of numpy is 1000009 > >>> np.version.version > '2.0.0.dev-3071eab' > >>> > > This error doesn't seem to effect the operation of numpy --- most > tests in np.test('full') pass and most of the numpy functions I used > regularly work --- but it causes the import of opencv to fail. > > >>> import cv > RuntimeError: module compiled against ABI version 2000000 but this > version of numpy is 1000009 > Traceback (most recent call last): > File "", line 1, in > ImportError: numpy.core.multiarray failed to import > >>> import numpy.core.multiarray > >>> numpy.core.multiarray.arange(5) > array([0, 1, 2, 3, 4]) > >>> > > Thanks for any help you can give me with this. > > Kathy > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmp50 at ukr.net Wed May 25 03:28:56 2011 From: tmp50 at ukr.net (Dmitrey) Date: Wed, 25 May 2011 10:28:56 +0300 Subject: [Numpy-discussion] [SciPy-User] [ANN] Guaranteed solution of nonlinear equation(s) In-Reply-To: <201105250854.16814.yosefmel@post.tau.ac.il> References: <201105250854.16814.yosefmel@post.tau.ac.il> Message-ID: --- ???????? ????????? --- ?? ????: "Yosef Meller" ????: scipy-user at scipy.org ????: 25 ??? 2011, 08:54:16 ????: Re: [SciPy-User] [ANN] Guaranteed solution of nonlinear equation(s) On ??? ????? 24 ??? 2011 13:22:47 Dmitrey wrote: > Hi all, > I have made my free solver interalg ( http://openopt.org/interalg ) be > capable of solving nonlinear equations and systems of them. Unlike > scipy optimize fsolve it doesn't matter which functions are involved - > convex, nonconvex, multiextremum etc. Even some discontinuous funcs > can be handled. If no solution exists, interalg determines it rather > quickly. > > For more info see http://forum.openopt.org/viewtopic.php?id=423 Interesting. Is there any description of the actual algorithm? I tried looking for it in the link and the openopt site, but couldn't find it. Algorithm belongs to family of interval methods. Those ones, along with Lipschitz methods, are capable of doing the task (searching extremum with guaranteed precision), but require extremely much computational time and memory, thus are very rarely used. Some ideas created and programmed by me increased speed in many orders (see that benchmark vs Direct, intsolver and commercial BARON), memory consumption also isn't very huge. As for solving equations system, currently |f1| + |f2| + |f3| + ... + |f_k| is minimized (| . | means abs( . )). I know better way of handling nonlinear systems by interalg, but it would take much time to write those enhancements (maybe some weeks or even more), my current work doesn't allow me to spend so much time for interalg and other openopt items development. Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Wed May 25 21:29:10 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 25 May 2011 18:29:10 -0700 Subject: [Numpy-discussion] Py3: numpy.rec.array fails with open file Message-ID: <4DDDACE6.6000004@uci.edu> Hello, the following code raises a ValueError on Python 3, but works as expected with Python 2: >>> import sys, numpy >>> fd = open(sys.executable, 'rb') >>> numpy.rec.array(fd, dtype=[('_', 'b')]) Traceback (most recent call last): File "", line 1, in File "C:\Python32\lib\site-packages\numpy\core\records.py", line 799, in array raise ValueError("Unknown input type") ValueError: Unknown input type The problem seems to be the test for file objects in numpy/compat/py3k.py: def isfileobj(f): return isinstance(f, io.FileIO) If this is updated to include _io.BufferedReader, the type of `fd`, numpy.rec.array works with file objects opened in 'rb' mode: def isfileobj(f): return isinstance(f, (io.FileIO, io._io.BufferedReader)) Tested with numpy 1.6.0 on Windows. Christoph From charlesr.harris at gmail.com Wed May 25 22:23:14 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 25 May 2011 20:23:14 -0600 Subject: [Numpy-discussion] Py3: numpy.rec.array fails with open file In-Reply-To: <4DDDACE6.6000004@uci.edu> References: <4DDDACE6.6000004@uci.edu> Message-ID: On Wed, May 25, 2011 at 7:29 PM, Christoph Gohlke wrote: > Hello, > > the following code raises a ValueError on Python 3, but works as > expected with Python 2: > > >>> import sys, numpy > >>> fd = open(sys.executable, 'rb') > >>> numpy.rec.array(fd, dtype=[('_', 'b')]) > Traceback (most recent call last): > File "", line 1, in > File "C:\Python32\lib\site-packages\numpy\core\records.py", line 799, > in array > raise ValueError("Unknown input type") > ValueError: Unknown input type > > > The problem seems to be the test for file objects in numpy/compat/py3k.py: > > def isfileobj(f): > return isinstance(f, io.FileIO) > > If this is updated to include _io.BufferedReader, the type of `fd`, > numpy.rec.array works with file objects opened in 'rb' mode: > > def isfileobj(f): > return isinstance(f, (io.FileIO, io._io.BufferedReader)) > > Tested with numpy 1.6.0 on Windows. > > Seems reasonable. Can you open a ticket and make a test for this? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkerzendorf at googlemail.com Wed May 25 22:27:09 2011 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Thu, 26 May 2011 12:27:09 +1000 Subject: [Numpy-discussion] weird TypeError: only length-1 arrays can be converted to Python scalars Message-ID: <4DDDBA7D.9020303@gmail.com> Dear all, I have come across a weird error, in which the traceback doesn't really reflect the problem (ah osx 10.6.7, python2.6, np 1.6, sp 0.9, ipython 0.10.1, mpl 1.0.1): ---- File "weird_error.py", line 8, in fmin.simplex() File "/Users/wkerzend/scripts/python/pyspecgrid/specgrid.py", line 134, in __call__ gridSpec = self.specGrid.getSpec(*args) File "/Users/wkerzend/scripts/python/pyspecgrid/specgrid.py", line 235, in getSpec return oned.onedspec(self.wave, self.interpGrid(args), mode='waveflux') File "interpnd.pyx", line 120, in interpnd.NDInterpolatorBase.__call__ (scipy/interpolate/interpnd.c:1896) File "interpnd.pyx", line 142, in interpnd._ndim_coords_from_arrays (scipy/interpolate/interpnd.c:2145) File "/Library/Python/2.6/site-packages/numpy/lib/stride_tricks.py", line 69, in broadcast_arrays args = map(np.asarray, args) File "/Library/Python/2.6/site-packages/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) TypeError: only length-1 arrays can be converted to Python scalars ------ I have debugged it here: --> 235 return array(a, dtype, copy=False, order=order) 236 ipdb> a a = 0.1 dtype = None order = None ipdb> type(a) ---------------- It just doesn't make sense to me, please help, Wolfgang From cgohlke at uci.edu Thu May 26 00:38:36 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 25 May 2011 21:38:36 -0700 Subject: [Numpy-discussion] Py3: numpy.rec.array fails with open file In-Reply-To: References: <4DDDACE6.6000004@uci.edu> Message-ID: <4DDDD94C.2070502@uci.edu> On 5/25/2011 7:23 PM, Charles R Harris wrote: > > > On Wed, May 25, 2011 at 7:29 PM, Christoph Gohlke > wrote: > > Hello, > > the following code raises a ValueError on Python 3, but works as > expected with Python 2: > > > >> import sys, numpy > > >> fd = open(sys.executable, 'rb') > > >> numpy.rec.array(fd, dtype=[('_', 'b')]) > Traceback (most recent call last): > File "", line 1, in > File "C:\Python32\lib\site-packages\numpy\core\records.py", line 799, > in array > raise ValueError("Unknown input type") > ValueError: Unknown input type > > > The problem seems to be the test for file objects in > numpy/compat/py3k.py: > > def isfileobj(f): > return isinstance(f, io.FileIO) > > If this is updated to include _io.BufferedReader, the type of `fd`, > numpy.rec.array works with file objects opened in 'rb' mode: > > def isfileobj(f): > return isinstance(f, (io.FileIO, io._io.BufferedReader)) > > Tested with numpy 1.6.0 on Windows. > > > Seems reasonable. Can you open a ticket and make a test for this? > > Chuck > > Opened ticket #1843 with test case and possible fix attached. Anyway, this is easy to work around by using numpy.rec.fromfile() instead of numpy.rec.array(). Christoph From jtalla at gmail.com Thu May 26 04:37:38 2011 From: jtalla at gmail.com (Talla) Date: Thu, 26 May 2011 11:37:38 +0300 Subject: [Numpy-discussion] Numpy question Message-ID: Hello I installed python2.7 and numpy on windows XP and the installation goes very smooth, now when I am trying to run a python script to get the band structure I got the following message: C:\Python27>python AbinitBandStructureMaker.py Traceback (most recent call last): File "AbinitBandStructureMaker.py", line 16, in from Numeric import * ImportError: No module named Numeric C:\Python27> In addition when I run import command I got ('import' is not recognized as an internal or external command, operable program or batch file.) Therefore, It seems to me it is not getting the correct path for numpy and therefore "AbinitBandStructureMaker.py" can not be executed properly , I tried to set the path variables with no luck. I need to get over this as it consumes all my time. Your help is really appreciated. Talla -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu May 26 04:44:44 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 26 May 2011 08:44:44 +0000 (UTC) Subject: [Numpy-discussion] Numpy question References: Message-ID: Thu, 26 May 2011 11:37:38 +0300, Talla wrote: [clip] > C:\Python27>python AbinitBandStructureMaker.py Traceback (most recent > call last): > File "AbinitBandStructureMaker.py", line 16, in > from Numeric import * > ImportError: No module named Numeric > > C:\Python27> > In addition when I run import command I got ('import' is not recognized > as an internal or external command, operable program or batch file.) > > Therefore, It seems to me it is not getting the correct path for numpy > and therefore "AbinitBandStructureMaker.py" can not be executed properly > , I tried to set the path variables with no luck. I need to get over > this as it consumes all my time. The script that you are trying does not use Numpy, but Numeric. Numeric is an obsolete predecessor for Numpy, and its development was discontinued in 2005. You can find it here: http://sourceforge.net/projects/numpy/files/Old%20Numeric/24.2/ No pre-made binaries for Python 2.7, for obvious reasons, however. -- Pauli Virtanen From scott.sinclair.za at gmail.com Thu May 26 04:50:58 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 26 May 2011 10:50:58 +0200 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: Message-ID: On 26 May 2011 10:37, Talla wrote: > C:\Python27> > In addition when I run import command I got ('import' is not recognized as > an internal or external command, operable program or batch file.) import is not a command you can run at your command line, it's part of Python. Do something like this instead: C:\Python27> cd C:\ C:\>python # Should start the Python interpreter, print python version etc. >>> import numpy >>> print(numpy.arange(3)) It will be worth taking some time to read a basic Python tutorial. Cheers, Scott From jtalla at gmail.com Thu May 26 05:04:29 2011 From: jtalla at gmail.com (Talla) Date: Thu, 26 May 2011 12:04:29 +0300 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: Message-ID: Thank you very much, In the script manual it mentioned that I need to install numpy and python (I am attaching the manual), however I tried to install numeric and python 2.4 as you suggested and I got a new error message: C:\Python24>python AbinitBandStructureMaker.py 'import site' failed; use -v for traceback Traceback (most recent call last): File "AbinitBandStructureMaker.py", line 13, in ? import os File "C:\Program Files\Scienomics\SciPython-2.6.2\\lib\os.py", line 757 bs = b"" ^ SyntaxError: invalid syntax C:\Python24 On Thu, May 26, 2011 at 11:44 AM, Pauli Virtanen wrote: > Thu, 26 May 2011 11:37:38 +0300, Talla wrote: > [clip] > > C:\Python27>python AbinitBandStructureMaker.py Traceback (most recent > > call last): > > File "AbinitBandStructureMaker.py", line 16, in > > from Numeric import * > > ImportError: No module named Numeric > > > > C:\Python27> > > In addition when I run import command I got ('import' is not recognized > > as an internal or external command, operable program or batch file.) > > > > Therefore, It seems to me it is not getting the correct path for numpy > > and therefore "AbinitBandStructureMaker.py" can not be executed properly > > , I tried to set the path variables with no luck. I need to get over > > this as it consumes all my time. > > The script that you are trying does not use Numpy, but Numeric. > Numeric is an obsolete predecessor for Numpy, and its development > was discontinued in 2005. > > You can find it here: > > http://sourceforge.net/projects/numpy/files/Old%20Numeric/24.2/ > > No pre-made binaries for Python 2.7, for obvious reasons, however. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Dr. Jamal A Talla Assistant professor Physics Department College of Science, Building 43, Rm 1047 King Faisal University P.O. Box 380, Al-Ahsa - 31982 Kingdom of Saudi Arabia Cell Phone: +966544623057 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: AbinitBandStructureMaker_manual.pdf Type: application/pdf Size: 57001 bytes Desc: not available URL: From jtalla at gmail.com Thu May 26 05:17:46 2011 From: jtalla at gmail.com (Talla) Date: Thu, 26 May 2011 12:17:46 +0300 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: Message-ID: Hello Below is the output of the coammands you mentioned. C:\>python Python 2.6.2 (r262:71600, Jul 7 2009, 20:21:09) [MSC v.1500 32 bit (Intel)] on win3 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print(numpy.arange(3)) [0 1 2] >>> Which looks to me it is working just fine. so what is the problem though? On Thu, May 26, 2011 at 11:50 AM, Scott Sinclair wrote: > On 26 May 2011 10:37, Talla wrote: > > C:\Python27> > > In addition when I run import command I got ('import' is not recognized > as > > an internal or external command, operable program or batch file.) > > import is not a command you can run at your command line, it's part of > Python. Do something like this instead: > > C:\Python27> cd C:\ > C:\>python > # Should start the Python interpreter, print python version etc. > >>> import numpy > >>> print(numpy.arange(3)) > > It will be worth taking some time to read a basic Python tutorial. > > Cheers, > Scott > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu May 26 06:15:28 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 26 May 2011 10:15:28 +0000 (UTC) Subject: [Numpy-discussion] Numpy question References: Message-ID: Thu, 26 May 2011 12:04:29 +0300, Talla wrote: [clip] > however I tried to > install numeric and python 2.4 as you suggested and I got a new error > message: > C:\Python24>python AbinitBandStructureMaker.py 'import site' failed; use > -v for traceback Traceback (most recent call last): > File "AbinitBandStructureMaker.py", line 13, in ? > import os > File "C:\Program Files\Scienomics\SciPython-2.6.2\\lib\os.py", line > 757 > bs = b"" Apparently, your two Python installations conflict in some way. If so, you should get the same error just by typing "python". You can install several Python versions from Python.org at the same time, but apparently the Scienomics package does something strange. You can try doing unset PYTHONPATH python AbinitBandStructureMaker.py If you still get the same error, you can try asking the Scienomics support how to have another Python version installed at the same time as their product. -- Pauli Virtanen From wesmckinn at gmail.com Thu May 26 07:53:24 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Thu, 26 May 2011 12:53:24 +0100 Subject: [Numpy-discussion] labeled axes In-Reply-To: References: Message-ID: On Tue, May 24, 2011 at 6:39 PM, Craig Yoshioka wrote: > Hi all, > I've read some discussions about adding labeled axes, and even ticks, to > numpy arrays ?(such as in Luis' dataarray). > I have recently found that the ability to label axes would be very helpful > to me, but I'd like to keep the implementation as lightweight as possible. > The reason I would find this useful is because I am writing a ndarray > subclass that?loads image/volume file formats into numpy arrays. ?Some of > these files might have multiple images/volumes, I'll call them channels, and > also may have an additional dimension for vectors associated with each > pixel/voxel, like color. ?The max dims of the array would then be 5. > Example: data = ndarray([1023,128,128,128,3]) might mean > (channels,z,y,x,rgb) for one array. ?Now I want to keep as much of the fancy > indexing capabilities of numpy as I can, but I am finding it difficult to > track the removal of axes that can occur from indexing. ?For example > data[2,2] would return an array of shape (128,128,3), or the third slice > through the third volume in the dataset, but the returned array has lost the > meaning associated with its axes, so saving it back out would require manual > relabeling of the axes. ? I'd like to be able to track the axes as metadata > and retain all the fancy numpy indexing. > There are two ways I could accomplish this with minimal code on the python > side: > ?One would be if indexing of the array always returned an array of the same > dimensionality, that is data[2,2] returned an array of shape > (1,1,128,128,3). ?I could then delete the degenerate axes labels from the > metadata, and return the compressed array, resulting in the same output: > > class Data(np.ndarray): > def __getitem__(self,indices): > data = np.ndarray.__getitem__(self,indices,donotcompress=True) # as an > example > data.axeslabels = [label for label,dim in zip(self.axeslabels,data.shape) if > dim > 1] > return data.compress() > def __getslice__(self,s1,s2,step): > # trivial case > > Another approach would be if there is some function in the numpy internals > that I could use to get the needed information before calling the ndarray's > __getitem__ function: > > class Data(np.ndarray): > def __getitem__(self,indices): > unique =?np.uniqueIndicesPerDimension(indices) > data =?np.ndarray.__getitem__(self,indices) > data.axeslabels =?[label for label,dim in zip(self.axeslabels,?unique) if > dim > 1] > return data > > Finally, I could implement my own parser for the passed indices to figure > this out myself. ?This would be bad since I'd have to recreate a lot of the > same code that must go on inside numpy, and it would be slower, error-prone, > etc. : > > class Data(np.ndarray): > def __getitem__(self,indices): > indices =?self.uniqueDimensionIndices(indices) > data =?np.ndarray.__getitem__(self,indices) > data.axeslabels =?[label for label,dim in zip(self.axeslabels,indices) if > dim > 1] > return data > def uniqueDimensionIndices(self,indices): > if isinstance(indices,int): > indices = (indices,) > if isinstance(indices,tuple): > .... > elif isinstance(indices,list): > ... > > Is there anything in the numpy internals already that would allow me to do > #1 or #2?, I don't think #3 is a very good option. > Thanks! > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > I would recommend joining or at least following the datarray project-- I believe it will do what you want, and we are working actively to build out this functionality. Here are some links: Datarray Github: https://github.com/fperez/datarray docs: http://fperez.github.com/datarray-doc/ Podcast we did about recent meeting about datarray http://inscight.org/2011/05/18/episode_13/ Other projects to consider using: larry and pandas-- these support data alignment which you may not care about. In pandas I'm only concerned with data with ndim <= 3, a bit specific to statistics/econometrics/finance applications. - Wes From derek at astro.physik.uni-goettingen.de Thu May 26 11:13:54 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Thu, 26 May 2011 17:13:54 +0200 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: Message-ID: On 26 May 2011, at 11:17, Talla wrote: > Below is the output of the coammands you mentioned. > > C:\>python > Python 2.6.2 (r262:71600, Jul 7 2009, 20:21:09) [MSC v.1500 32 bit > (Intel)] on win3 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> print(numpy.arange(3)) > [0 1 2] > >>> > > Which looks to me it is working just fine. > > so what is the problem though? Numpy is working with your Python2.6 and 2.7 installations, but as Pauli pointed out, the script you want to run instead is using instead a different and very ancient Python library, Numeric (which in its days occasionally was referred to as "NumPy", too, just to add confusion ;-). This means you could either try to - get Python2.4 to work on your machine and hope the Numeric binary works with it - compile Numeric from source for your Python2.6 or Python2.7 installation (it actually still builds with Python2.7), but this might be challenging for a beginner - try to modify your script to work with numpy instead. Numpy provides all the functionality Numeric has ever offered afaik; its syntax is merely slightly different in a few cases. You should find some helpful information here: http://www.scipy.org/Converting_from_Numeric (please note the caveats about the automatic converter!) It might even be enough to replace the line in the script File "AbinitBandStructureMaker.py", line 16, in from Numeric import * with from numpy.oldnumeric import * ymmv - if you can contact the maintainer/original author of the script, you might ask them for assistance - it does not make much sense to rely on a software package that has not been maintained for half a decade, after all. HTH, Derek From bsouthey at gmail.com Thu May 26 11:50:18 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 26 May 2011 10:50:18 -0500 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: Message-ID: <4DDE76BA.40203@gmail.com> On 05/26/2011 10:13 AM, Derek Homeier wrote: > On 26 May 2011, at 11:17, Talla wrote: > >> Below is the output of the coammands you mentioned. >> >> C:\>python >> Python 2.6.2 (r262:71600, Jul 7 2009, 20:21:09) [MSC v.1500 32 bit >> (Intel)] on win3 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy >>>>> print(numpy.arange(3)) >> [0 1 2] >> Which looks to me it is working just fine. >> >> so what is the problem though? > Numpy is working with your Python2.6 and 2.7 installations, but as > Pauli pointed out, > the script you want to run instead is using instead a different and > very ancient Python library, > Numeric (which in its days occasionally was referred to as "NumPy", > too, just to add > confusion ;-). > This means you could either try to > > - get Python2.4 to work on your machine and hope the Numeric binary > works with it > > - compile Numeric from source for your Python2.6 or Python2.7 > installation (it actually > still builds with Python2.7), but this might be challenging for a > beginner > > - try to modify your script to work with numpy instead. Numpy provides > all the functionality > Numeric has ever offered afaik; its syntax is merely slightly > different in a few cases. > You should find some helpful information here: > > http://www.scipy.org/Converting_from_Numeric > > (please note the caveats about the automatic converter!) > It might even be enough to replace the line in the script > File "AbinitBandStructureMaker.py", line 16, in > from Numeric import * > with > from numpy.oldnumeric import * > > ymmv - if you can contact the maintainer/original author of the > script, you might ask > them for assistance - it does not make much sense to rely on a > software package that > has not been maintained for half a decade, after all. > > HTH, > Derek > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion The source code does not appears to use many Numeric/numpy functions so it should be easy to convert to numpy. You can probably get away with by just importing what it needs: from numpy import array The 'AbinitStructureViewer.py' is a little more complex because it uses the old dtype with zeros(). So you will also have to import zeros as well ('from numpy import array, zeros') *and* change those calls such as: 'zeros(3,Int)' to 'zeros(3,int)' For more details about converting see: http://www.scipy.org/Converting_from_Numeric Bruce From matthew.brett at gmail.com Thu May 26 17:03:04 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 26 May 2011 14:03:04 -0700 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Hi, On Wed, May 4, 2011 at 9:24 AM, Robert Kern wrote: > On Wed, May 4, 2011 at 11:14, Matthew Brett wrote: >> Hi, >> >> On Tue, May 3, 2011 at 7:58 PM, Robert Kern wrote: > >>> I can't speak for the rest of the group, but as for myself, if you >>> would like to draft such a letter, I'm sure I will agree with its >>> contents. >> >> Thank you - sadly I am not confident in deserving your confidence, but >> I will do my best to say something sensible. ? Any objections to a >> public google doc? > > Even better! I've put up a draft here: numpy-whaley-support - https://docs.google.com/document/d/1gPhUUjWqNpRatw90kCqL1WPWvn1yicf2VAowWSyHlno/edit?hl=en_US&authkey=CPv49_cK I didn't know who to put as signatories. Maybe an extended steering group like (from http://scipy.org/Developer_Zone): Jarrod Millman Eric Jones Robert Kern Travis Oliphant Stefan van der Walt plus: Pauli Ralf Chuck or something like that? Anyone else care to sign / edit? Mark W for example? Sorry, I haven't been following the numpy commits very carefully of late. Best, Matthew From akshar.bhosale at gmail.com Thu May 26 20:45:26 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Fri, 27 May 2011 06:15:26 +0530 Subject: [Numpy-discussion] problem with installation of numpy+scipy on intel platform Message-ID: Hi, we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 x86_64 with kernel 2.6.18-92.el5. We have following configuration : /opt/intel/Compiler/11.0/069/mkl/lib/em64t Now we want to install numpy and scipy as an user in my home directory. Following are the libraries build inside MKL. libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so libfftw2xf_intel.a libmkl_blas95.a libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so libfftw3xc_intel.a libmkl_cdft.a libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so libfftw3xf_intel.a libmkl_cdft_core.a libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale libmkl_blacs_ilp64.a libmkl_core.a libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod libmkl_blacs_intelmpi20_lp64.a libmkl_def.so libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a libmkl_lapack95.a libmkl.so mkl95_blas.mod libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so libmkl_lapack.so libmkl_solver_ilp64.a mkl95_precision.mod libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a libmkl_mc3.so libmkl_solver_ilp64_sequential.a libmkl_blacs_lp64.a libmkl_gf_lp64.so libmkl_mc.so libmkl_solver_lp64.a libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a libmkl_p4n.so libmkl_solver_lp64_sequential.a libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so libmkl_pgi_thread.a libmkl_vml_def.so Version we are trying to build of numpy and scipy are : numpy : 1.6.0b2 and scipy : 0.9.0 We have configured python as a user in my home directory with version : 2.6.6 our machine has python : 2.4.3 We want to know the exact procedure for installing it using MKL as we are facing lot of issues while installing the same. Please help us. -akshar -------------- next part -------------- An HTML attachment was scrubbed... URL: From craigyk at me.com Thu May 26 21:36:23 2011 From: craigyk at me.com (Craig Yoshioka) Date: Thu, 26 May 2011 18:36:23 -0700 Subject: [Numpy-discussion] labeled axes In-Reply-To: References: Message-ID: <6134933D-6F0D-4937-9B0B-39E76F852E80@me.com> Thanks, I will. I was just seeing if there was any intention of adding this to type of support to numpy directly. It would be faster, and I'm sure it would make projects like dataarray much simpler to implement (dataarray does a lot more than my suggestion). On May 26, 2011, at 4:53 AM, Wes McKinney wrote: > > I would recommend joining or at least following the datarray project-- > I believe it will do what you want, and we are working actively to > build out this functionality. Here are some links: > > Datarray > Github: https://github.com/fperez/datarray > docs: http://fperez.github.com/datarray-doc/ > > Podcast we did about recent meeting about datarray > http://inscight.org/2011/05/18/episode_13/ > > Other projects to consider using: larry and pandas-- these support > data alignment which you may not care about. In pandas I'm only > concerned with data with ndim <= 3, a bit specific to > statistics/econometrics/finance applications. > > - Wes > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jsalvati at u.washington.edu Thu May 26 22:48:34 2011 From: jsalvati at u.washington.edu (John Salvatier) Date: Thu, 26 May 2011 19:48:34 -0700 Subject: [Numpy-discussion] labeled axes In-Reply-To: <6134933D-6F0D-4937-9B0B-39E76F852E80@me.com> References: <6134933D-6F0D-4937-9B0B-39E76F852E80@me.com> Message-ID: I'm glad datarray is still active. :) On Thu, May 26, 2011 at 6:36 PM, Craig Yoshioka wrote: > Thanks, I will. I was just seeing if there was any intention of adding this > to type of support to numpy directly. It would be faster, and I'm sure it > would make projects like dataarray much simpler to implement (dataarray does > a lot more than my suggestion). > > On May 26, 2011, at 4:53 AM, Wes McKinney wrote: > > > > > I would recommend joining or at least following the datarray project-- > > I believe it will do what you want, and we are working actively to > > build out this functionality. Here are some links: > > > > Datarray > > Github: https://github.com/fperez/datarray > > docs: http://fperez.github.com/datarray-doc/ > > > > Podcast we did about recent meeting about datarray > > http://inscight.org/2011/05/18/episode_13/ > > > > Other projects to consider using: larry and pandas-- these support > > data alignment which you may not care about. In pandas I'm only > > concerned with data with ndim <= 3, a bit specific to > > statistics/econometrics/finance applications. > > > > - Wes > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mesanthu at gmail.com Thu May 26 22:52:17 2011 From: mesanthu at gmail.com (santhu kumar) Date: Thu, 26 May 2011 21:52:17 -0500 Subject: [Numpy-discussion] Extending dimensions of a matrix Message-ID: Hello All, I am trying to extend a piece of matlab code into numpy. And I am stuck at converting, ( x is a nX3 matrix) Matlab : dim = size(x,1) % would be n nx = x(:,:,ones(1,dim)) % would be nX3Xn matrix .. Now the same in Numpy seems to be tricky to me .. Can somebody help me in writing this piece of code. Thanks Santhosh -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu May 26 23:22:58 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 26 May 2011 22:22:58 -0500 Subject: [Numpy-discussion] Extending dimensions of a matrix In-Reply-To: References: Message-ID: On Thu, May 26, 2011 at 21:52, santhu kumar wrote: > Hello All, > > I am trying to extend a piece of matlab code into numpy. > > And I am stuck at converting, ( x is a nX3 matrix) > Matlab : > dim = size(x,1) % would be n > nx = x(:,:,ones(1,dim)) % would be nX3Xn matrix .. > > Now the same in Numpy seems to be tricky to me .. Can somebody help me in > writing this piece of code. [~] |1> n = 10 [~] |2> x = np.arange(n*3).reshape((n,3)) [~] |3> nx = x[:,:,np.newaxis][:,:,np.zeros(n, dtype=int)] [~] |4> nx.shape (10, 3, 10) [~] |5> nx array([[[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]], [[ 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], [ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4], [ 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]], [[ 6, 6, 6, 6, 6, 6, 6, 6, 6, 6], [ 7, 7, 7, 7, 7, 7, 7, 7, 7, 7], [ 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]], [[ 9, 9, 9, 9, 9, 9, 9, 9, 9, 9], [10, 10, 10, 10, 10, 10, 10, 10, 10, 10], [11, 11, 11, 11, 11, 11, 11, 11, 11, 11]], [[12, 12, 12, 12, 12, 12, 12, 12, 12, 12], [13, 13, 13, 13, 13, 13, 13, 13, 13, 13], [14, 14, 14, 14, 14, 14, 14, 14, 14, 14]], [[15, 15, 15, 15, 15, 15, 15, 15, 15, 15], [16, 16, 16, 16, 16, 16, 16, 16, 16, 16], [17, 17, 17, 17, 17, 17, 17, 17, 17, 17]], [[18, 18, 18, 18, 18, 18, 18, 18, 18, 18], [19, 19, 19, 19, 19, 19, 19, 19, 19, 19], [20, 20, 20, 20, 20, 20, 20, 20, 20, 20]], [[21, 21, 21, 21, 21, 21, 21, 21, 21, 21], [22, 22, 22, 22, 22, 22, 22, 22, 22, 22], [23, 23, 23, 23, 23, 23, 23, 23, 23, 23]], [[24, 24, 24, 24, 24, 24, 24, 24, 24, 24], [25, 25, 25, 25, 25, 25, 25, 25, 25, 25], [26, 26, 26, 26, 26, 26, 26, 26, 26, 26]], [[27, 27, 27, 27, 27, 27, 27, 27, 27, 27], [28, 28, 28, 28, 28, 28, 28, 28, 28, 28], [29, 29, 29, 29, 29, 29, 29, 29, 29, 29]]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pearu.peterson at gmail.com Fri May 27 00:32:21 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 27 May 2011 07:32:21 +0300 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Hi, Would it be possible to setup a signing system where anyone who would like to support Clint could sign and advertise the system on relevant mailing lists? This would provide larger body of supporters for this letter and perhaps will have greater impact to whom the letter will be addressed. Personally, I would be happy to sign to such a letter. On the letter: the letter should also mention scipy community as they benefit most from the ATLAS speed. Best regards, Pearu On Fri, May 27, 2011 at 12:03 AM, Matthew Brett wrote: > Hi, > > On Wed, May 4, 2011 at 9:24 AM, Robert Kern wrote: > > On Wed, May 4, 2011 at 11:14, Matthew Brett > wrote: > >> Hi, > >> > >> On Tue, May 3, 2011 at 7:58 PM, Robert Kern > wrote: > > > >>> I can't speak for the rest of the group, but as for myself, if you > >>> would like to draft such a letter, I'm sure I will agree with its > >>> contents. > >> > >> Thank you - sadly I am not confident in deserving your confidence, but > >> I will do my best to say something sensible. Any objections to a > >> public google doc? > > > > Even better! > > I've put up a draft here: > > numpy-whaley-support - > > https://docs.google.com/document/d/1gPhUUjWqNpRatw90kCqL1WPWvn1yicf2VAowWSyHlno/edit?hl=en_US&authkey=CPv49_cK > > I didn't know who to put as signatories. Maybe an extended steering > group like (from http://scipy.org/Developer_Zone): > > Jarrod Millman > Eric Jones > Robert Kern > Travis Oliphant > Stefan van der Walt > > plus: > > Pauli > Ralf > Chuck > > or something like that? Anyone else care to sign / edit? Mark W for > example? Sorry, I haven't been following the numpy commits very > carefully of late. > > Best, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri May 27 00:39:54 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 26 May 2011 21:39:54 -0700 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Hi, On Thu, May 26, 2011 at 9:32 PM, Pearu Peterson wrote: > Hi, > > Would it be possible to setup a signing system where anyone who would like > to support Clint could sign and advertise the system on relevant mailing > lists? > This would provide larger body of supporters for this letter and perhaps > will have greater impact to whom the letter will be > addressed. Personally, I would be happy to sign to such a letter. > > On the letter: the letter should also mention scipy community as they > benefit > most from the ATLAS speed. Maybe it would be best phrased then as 'numpy and scipy developers' instead of the steering group? I'm not sure how this kind of thing works for tenure letters, I would guess that if there are a very large number of signatures it might be difficult to see who is being represented... I'm open to suggestions. I can also ask Clint. I've added you as an editor - would you consider adding your name at the end, and maybe something about scipy? - you know the scipy blas / lapack stuff much better than I do. Cheers, Matthew From matthew.brett at gmail.com Fri May 27 00:44:53 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 26 May 2011 21:44:53 -0700 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: Oh sorry and: On Thu, May 26, 2011 at 2:03 PM, Matthew Brett wrote: > Hi, > > On Wed, May 4, 2011 at 9:24 AM, Robert Kern wrote: >> On Wed, May 4, 2011 at 11:14, Matthew Brett wrote: >>> Hi, >>> >>> On Tue, May 3, 2011 at 7:58 PM, Robert Kern wrote: >> >>>> I can't speak for the rest of the group, but as for myself, if you >>>> would like to draft such a letter, I'm sure I will agree with its >>>> contents. >>> >>> Thank you - sadly I am not confident in deserving your confidence, but >>> I will do my best to say something sensible. ? Any objections to a >>> public google doc? >> >> Even better! > > I've put up a draft here: > > numpy-whaley-support - > https://docs.google.com/document/d/1gPhUUjWqNpRatw90kCqL1WPWvn1yicf2VAowWSyHlno/edit?hl=en_US&authkey=CPv49_cK > > I didn't know who to put as signatories. ?Maybe an extended steering > group like (from http://scipy.org/Developer_Zone): > > Jarrod Millman > Eric Jones > Robert Kern > Travis Oliphant > Stefan van der Walt > > plus: > > Pauli > Ralf > Chuck David C... Sorry - I was up very late last night. Cheers, Matthew From pearu.peterson at gmail.com Fri May 27 01:19:12 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Fri, 27 May 2011 08:19:12 +0300 Subject: [Numpy-discussion] Numpy steering group? In-Reply-To: References: Message-ID: On Fri, May 27, 2011 at 7:39 AM, Matthew Brett wrote: > Hi, > > On Thu, May 26, 2011 at 9:32 PM, Pearu Peterson > wrote: > > Hi, > > > > Would it be possible to setup a signing system where anyone who would > like > > to support Clint could sign and advertise the system on relevant mailing > > lists? > > This would provide larger body of supporters for this letter and perhaps > > will have greater impact to whom the letter will be > > addressed. Personally, I would be happy to sign to such a letter. > > > > On the letter: the letter should also mention scipy community as they > > benefit > > most from the ATLAS speed. > > Maybe it would be best phrased then as 'numpy and scipy developers' > instead of the steering group? > > I'm not sure how this kind of thing works for tenure letters, I would > guess that if there are a very large number of signatures it might be > difficult to see who is being represented... I'm open to suggestions. > I can also ask Clint. > > I've added you as an editor - would you consider adding your name at > the end, and maybe something about scipy? - you know the scipy blas / > lapack stuff much better than I do. > Done for adding the name. The document is currently numpy oriented and I am not sure where to enter with scipy.. Technical summary of the situation with scipy blas/lapack stuff: The main difference in between numpy and scipy lapack-wise is that numpy has a lite C version of few lapack routines in case the lapack libraries are not available for building it while for scipy the lapack libraries are prerequisites as scipy provides interfaces to a much larger number of lapack routines. Having ATLAS in addition would greatly increase the performance of these routines. Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From srean.list at gmail.com Fri May 27 11:36:42 2011 From: srean.list at gmail.com (srean) Date: Fri, 27 May 2011 10:36:42 -0500 Subject: [Numpy-discussion] Adding the arrays in an array iterator Message-ID: Hi List, I have to sum up an unknown number of ndarrays of the same size. These arrays, possibly thousands in number, are provided by an iterator. Right now I use python reduce with operator.add. Does that invoke the corresponding ufunc internally, I want to avoid creating temporaries. With ufunc I know how to do it, but not sure how to make use of that in reduce. It is not essential that I use reduce though, so I would welcome idiomatic and efficient way of executing this. So far I have stayed away from building an ndarray object and summing across the relevant dimension. Is that what I should be doing ? Different invocations of this function has different number of arrays, so I cannot pre-compile away this into a numexpr. Thanks and regards srean -------------- next part -------------- An HTML attachment was scrubbed... URL: From michaeladamkatz at yahoo.com Fri May 27 12:48:15 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Fri, 27 May 2011 09:48:15 -0700 (PDT) Subject: [Numpy-discussion] finding elements that match any in a set Message-ID: <21804.15125.qm@web161313.mail.bf1.yahoo.com> I have a numpy array, records, with named fields including a field named "integer_field". I have an array (or list) of values of interest, and I want to get the indexes where integer_field has any of those values. Because I can do indexes = np.where( records.integer_field > 5 ) I thought I could do indexes = np.where( records.integer_field in values ) But that doesn't work. (As a side question I'm interested in why that doesn't work, when values is a python list.) How can I get those indexes? I though perhaps I need a nested np.where, or some other two-step process, but it wasn't clear to me how to do it. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Fri May 27 12:55:07 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Fri, 27 May 2011 12:55:07 -0400 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <21804.15125.qm@web161313.mail.bf1.yahoo.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> Message-ID: On Fri, May 27, 2011 at 12:48 PM, Michael Katz wrote: > I have a numpy array, records, with named fields including a field named > "integer_field". I have an array (or list) of values of interest, and I want > to get the indexes where integer_field has any of those values. > > Because I can do > > ??? indexes = np.where( records.integer_field > 5 ) > > I thought I could do > > ??? indexes = np.where( records.integer_field in values ) > > But that doesn't work. (As a side question I'm interested in why that > doesn't work, when values is a python list.) > > How can I get those indexes? I though perhaps I need a nested np.where, or > some other two-step process, but it wasn't clear to me how to do it. > Check out this recent thread. I think the proposed class does what you want. It's more efficient than in1d, if values is small compared to the length of records. http://thread.gmane.org/gmane.comp.python.scientific.user/29035/ Skipper From ralf.gommers at googlemail.com Fri May 27 12:55:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 27 May 2011 18:55:39 +0200 Subject: [Numpy-discussion] problem with installation of numpy+scipy on intel platform In-Reply-To: References: Message-ID: On Fri, May 27, 2011 at 2:45 AM, akshar bhosale wrote: > Hi, > we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 > x86_64 with kernel 2.6.18-92.el5. > We have following configuration : > /opt/intel/Compiler/11.0/069/mkl/lib/em64t > Now we want to install numpy and scipy as an user in my home directory. > > Following are the libraries build inside MKL. > > libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a > libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so > libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a > libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so > libfftw2xf_intel.a libmkl_blas95.a > libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so > libfftw3xc_intel.a libmkl_cdft.a > libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so > libfftw3xf_intel.a libmkl_cdft_core.a > libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale > libmkl_blacs_ilp64.a libmkl_core.a > libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod > libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so > libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod > libmkl_blacs_intelmpi20_lp64.a libmkl_def.so > libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod > libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a > libmkl_lapack95.a libmkl.so mkl95_blas.mod > libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a > libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod > libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so > libmkl_lapack.so libmkl_solver_ilp64.a > mkl95_precision.mod > libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a > libmkl_mc3.so libmkl_solver_ilp64_sequential.a > libmkl_blacs_lp64.a libmkl_gf_lp64.so > libmkl_mc.so libmkl_solver_lp64.a > libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a > libmkl_p4n.so libmkl_solver_lp64_sequential.a > libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so > libmkl_pgi_thread.a libmkl_vml_def.so > > Version we are trying to build of numpy and scipy are : > numpy : 1.6.0b2 and scipy : 0.9.0 > Better to use numpy 1.6.0, some bugs got fixed since beta 2. > We have configured python as a user in my home directory with version : > 2.6.6 our machine has python : 2.4.3 > > We want to know the exact procedure for installing it using MKL as we are > facing lot of issues while installing the same. > Please help us. > Did you try the instructions at http://scipy.org/Installing_SciPy/Linux, section "Any distribution with Intel C compiler and MKL"? Any specific problems with those? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitriy.rybalkin at gmail.com Fri May 27 13:47:24 2011 From: dmitriy.rybalkin at gmail.com (Dmitriy Rybalkin) Date: Fri, 27 May 2011 21:47:24 +0400 Subject: [Numpy-discussion] Contribution Message-ID: Greeting the community of Numpy project My name is Dmitriy and I'am a computer science master student of Australian National University and, to be honest, not a good programmer, though I found that Numpy/Scipy project has a good Cookbook and I thought I could contribute to the Numpy project by showing some examples of simple classification techniques (Fisher discriminant function, Naive Bayes) which use numpy functions. So, I'am inquiring about such contribution and some details: can I do that and, if yes, then how can I submit such example or should I only modify the Wiki page and request the permission? With best regards Dmitriy Rybalkin From jtalla at gmail.com Fri May 27 14:06:45 2011 From: jtalla at gmail.com (Talla) Date: Fri, 27 May 2011 21:06:45 +0300 Subject: [Numpy-discussion] Numpy question In-Reply-To: <4DDE76BA.40203@gmail.com> References: <4DDE76BA.40203@gmail.com> Message-ID: Hello Sir: I am using python24 and Numaric under windows XP and I am trying to create *.agr file from *.dbs file, I followed the instructions very carefully and every time I am getting the following message: Traceback (most recent call last): File "AbinitBandStructuremaker.py", line 1274, in ? data.fermienergy = max(maxlist) ValueError: max() arg is an empty sequence Nothing wrong in the .dbs file Your help is really appreciated -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri May 27 14:41:36 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 27 May 2011 18:41:36 +0000 (UTC) Subject: [Numpy-discussion] Contribution References: Message-ID: Hi, On Fri, 27 May 2011 21:47:24 +0400, Dmitriy Rybalkin wrote: > My name is Dmitriy and I'am a computer science master student of > Australian National University and, to be honest, not a good > programmer, though I found that Numpy/Scipy project has a good > Cookbook and I thought I could contribute to the Numpy project by > showing some examples of simple classification techniques (Fisher > discriminant function, Naive Bayes) which use numpy functions. So, > So, I'am inquiring about such contribution and some details: > can I do that and, if yes, then how can I submit such example or should > I only modify the Wiki page and request the permission? Yes, further examples in the Cookbook are very welcome. You do not need a separate permission: the Cookbook is open for everyone to edit. You only need to create an account first: http://scipy.org/UserPreferences To add an entry to the Cookbook, first edit the main page http://scipy.org/Cookbook and add a link to a sub-page there to a suitable section. E.g. * [:Cookbook/Simple Classification:Simple Classification] or someting similar under the "Scientific Scripts" section. After saving the edit, click this new link to create the new page. -- Pauli Virtanen From dmitriy.rybalkin at gmail.com Fri May 27 15:11:09 2011 From: dmitriy.rybalkin at gmail.com (Dmitriy Rybalkin) Date: Fri, 27 May 2011 23:11:09 +0400 Subject: [Numpy-discussion] Contribution In-Reply-To: References: Message-ID: All right, thank you very much for this information On Fri, May 27, 2011 at 10:41 PM, Pauli Virtanen wrote: > Hi, > > On Fri, 27 May 2011 21:47:24 +0400, Dmitriy Rybalkin wrote: >> My name is Dmitriy and I'am a computer science master student of >> Australian National University and, to be honest, not a good >> programmer, though I found that Numpy/Scipy project has a good >> Cookbook and I thought I could contribute to the Numpy project by >> showing some examples of simple classification techniques (Fisher >> discriminant function, Naive Bayes) which use numpy functions. So, >> So, I'am inquiring about such contribution and some details: >> can I do that and, if yes, then how can I submit such example or should >> I only modify the Wiki page and request the permission? > > Yes, further examples in the Cookbook are very welcome. > > You do not need a separate permission: the Cookbook is open for > everyone to edit. You only need to create an account first: > http://scipy.org/UserPreferences > > To add an entry to the Cookbook, first edit the main page > > ? ?http://scipy.org/Cookbook > > and add a link to a sub-page there to a suitable section. E.g. > > ?* [:Cookbook/Simple Classification:Simple Classification] > > or someting similar under the "Scientific Scripts" section. After > saving the edit, click this new link to create the new page. > > -- > Pauli Virtanen > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Fri May 27 20:28:07 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 May 2011 17:28:07 -0700 Subject: [Numpy-discussion] Numpy question In-Reply-To: References: <4DDE76BA.40203@gmail.com> Message-ID: <4DE04197.5030505@noaa.gov> On 5/27/11 11:06 AM, Talla wrote: > I am using python24 and Numeric under windows XP and I am trying to > create *.agr file from *.dbs file, I followed the instructions very > carefully and every time I am getting the following message: > Traceback (most recent call last): > File "AbinitBandStructuremaker.py", line 1274, in ? > data.fermienergy = max(maxlist) > ValueError: max() arg is an empty sequence > > Nothing wrong in the .dbs file This is really a question for the author of AbinitBandStructuremaker.py. If there is no one familiar with that code tha can help you'll need to debug it yourself. That error means that in this case, "maxlist" is an empty list, so tring to find teh maximum value in it is not possible. It also looks like that is the built-in python max(0, not the Numeric one, so this isn't really a numpy-related question anyway. without more context, we can't even begin to figure out what's wrong. Good luck, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Fri May 27 20:48:37 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 27 May 2011 17:48:37 -0700 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <21804.15125.qm@web161313.mail.bf1.yahoo.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> Message-ID: <4DE04665.7030105@noaa.gov> On 5/27/11 9:48 AM, Michael Katz wrote: > I have a numpy array, records, with named fields including a field named > "integer_field". I have an array (or list) of values of interest, and I > want to get the indexes where integer_field has any of those values. > > Because I can do > > indexes = np.where( records.integer_field > 5 ) > > I thought I could do > > indexes = np.where( records.integer_field in values ) > > But that doesn't work. (As a side question I'm interested in why that > doesn't work, when values is a python list.) that doesn't work because the python list "in" operator doesn't understand arrays -- so it is looking ot see if the entire array is in the list. actually, it doesn't even get that far: In [16]: a in l --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/chris.barker/ in () ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() The ValueError results because it was decided that numpy array should not have a boolean value to avoid confusion -- i.e. is na array true whenever it is non-empty (like a list), or when all it's elements are true, or???? When I read this question, I thought -- hmmm, numpy needs something like "in", as the usual way: np.any(), would require a loop in this case. Then I read Skipper's message: On 5/27/11 9:55 AM, Skipper Seabold wrote: > Check out this recent thread. I think the proposed class does what you > want. It's more efficient than in1d, if values is small compared to > the length of records. So that class may be worthwhile, but I think np.in1d is exactly what you are looking for: indexes = np.in1d( records.integer_field, values ) Funny I'd never noticed that before. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aradnix at gmail.com Fri May 27 22:07:35 2011 From: aradnix at gmail.com (Aradenatorix Veckhom Vacelaevus) Date: Fri, 27 May 2011 21:07:35 -0500 Subject: [Numpy-discussion] How to fix my data for visualize with Mayavi2 and to use TVTK? Message-ID: Hi again: Well, finally I solved my troubles with the importation of data from the Fortran files I've mentioned here few weeks ago... but the importation using the VTK legacy formats doesn't works... I have not idea how to build the LUT (Look up table) for my case and... I'm not sure if that's the only way for can fix my data and visualize them inside mayavi2. Well... although I don't know if this is the appropriate list for ask about mayavi2 and TVTK. Well sorry if it's the wrong place. I need to build and visualize an structured grid object, I have the data but I don't know how to insert the data in mayavi for visualize. I was checking the documentation, the cookbook but I can't find a clear answer. I know I can create an script in python and use the nyumpy arrays for it, but not more. I hope someone can help me. Regards: Aradnix -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Fri May 27 22:51:01 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 27 May 2011 21:51:01 -0500 Subject: [Numpy-discussion] How to fix my data for visualize with Mayavi2 and to use TVTK? In-Reply-To: References: Message-ID: On Fri, May 27, 2011 at 9:07 PM, Aradenatorix Veckhom Vacelaevus < aradnix at gmail.com> wrote: > Hi again: > > Well, finally I solved my troubles with the importation of data from the > Fortran files I've mentioned here few weeks ago... but the importation using > the VTK legacy formats doesn't works... I have not idea how to build the LUT > (Look up table) for my case and... I'm not sure if that's the only way for > can fix my data and visualize them inside mayavi2. > > Well... although I don't know if this is the appropriate list for ask about > mayavi2 and TVTK. Well sorry if it's the wrong place. > > Try the enthought-dev mailing list: https://mail.enthought.com/mailman/listinfo/enthought-dev Warren > I need to build and visualize an structured grid object, I have the data > but I don't know how to insert the data in mayavi for visualize. I was > checking the documentation, the cookbook but I can't find a clear answer. I > know I can create an script in python and use the nyumpy arrays for it, but > not more. > > I hope someone can help me. > > Regards: > Aradnix > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat May 28 01:58:13 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 27 May 2011 22:58:13 -0700 Subject: [Numpy-discussion] labeled axes In-Reply-To: <6134933D-6F0D-4937-9B0B-39E76F852E80@me.com> References: <6134933D-6F0D-4937-9B0B-39E76F852E80@me.com> Message-ID: On Thu, May 26, 2011 at 6:36 PM, Craig Yoshioka wrote: > Thanks, I will. I was just seeing if there was any intention of adding this to type of support to numpy directly. ?It would be faster, and I'm sure it would make projects like dataarray much simpler to implement (dataarray does a lot more than my suggestion). Datarray was deliberately implemented as a subclass of the numpy array so that it would be as easy as possible to eventually merge things into numpy proper. We simply wanted to experiment with the api and code in a separate project for simplicity of development, but once things are sufficiently baked out and validated by others, there's every intention of pushing this into numpy itself. And yes, we are indeed trying to make some progress on it, we spent some time at Berkeley today on it, and a new grad student from the stats department, Jonathan Terhorst, is putting some solid work into it over the next few days. I really hope it won't be much longer before this is finally ready for production use, I know we need it badly in multiple places. Best, f From akshar.bhosale at gmail.com Sat May 28 03:54:34 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Sat, 28 May 2011 13:24:34 +0530 Subject: [Numpy-discussion] Problem with installation of numpy and scipy using MKL Message-ID: Hi, we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 x86_64 with kernel 2.6.18-92.el5. We have following configuration : /opt/intel/Compiler/11.0/069/ mkl/lib/em64t Now we want to install numpy and scipy as an user in my home directory. Following are the libraries build inside MKL. libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so libfftw2xf_intel.a libmkl_blas95.a libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so libfftw3xc_intel.a libmkl_cdft.a libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so libfftw3xf_intel.a libmkl_cdft_core.a libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale libmkl_blacs_ilp64.a libmkl_core.a libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod libmkl_blacs_intelmpi20_lp64.a libmkl_def.so libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a libmkl_lapack95.a libmkl.so mkl95_blas.mod libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so libmkl_lapack.so libmkl_solver_ilp64.a mkl95_precision.mod libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a libmkl_mc3.so libmkl_solver_ilp64_sequential.a libmkl_blacs_lp64.a libmkl_gf_lp64.so libmkl_mc.so libmkl_solver_lp64.a libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a libmkl_p4n.so libmkl_solver_lp64_sequential.a libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so libmkl_pgi_thread.a libmkl_vml_def.so Version we are trying to build of numpy and scipy are : numpy : 1.6.0b2 and scipy : 0.9.0 We have configured python as a user in my home directory with version : 2.6.6 our machine has python : 2.4.3 We want to know the exact procedure for installing it using MKL as we are facing lot of issues while installing the same. Please help us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sat May 28 05:15:37 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 28 May 2011 11:15:37 +0200 Subject: [Numpy-discussion] Contribution In-Reply-To: References: Message-ID: <20110528091537.GC2799@phare.normalesup.org> I Dmitriy, Welcome. This is a great initiative. You'll learn by doing, and I am willing to bet that contributing examples can make you a good programmer. On Fri, May 27, 2011 at 09:47:24PM +0400, Dmitriy Rybalkin wrote: > I thought I could contribute to the Numpy project by > showing some examples of simple classification techniques (Fisher > discriminant function, Naive Bayes) which use numpy functions. So, > I'am inquiring about such contribution and some details: can I do that > and, if yes, then how can I submit such example or should I only > modify the Wiki page and request the permission? It might be interesting to stress on the corresponding wiki pages that these methods are available in readily coded package, such as the scikits.learn. The reason I stress this is that code on a wiki page tends to go unmaintained, while a great deal of quality assurance goes in the standard software packages. Ga?l From emmanuelle.gouillart at normalesup.org Sat May 28 05:22:13 2011 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Sat, 28 May 2011 11:22:13 +0200 Subject: [Numpy-discussion] [ANN] PyPhy - Python in Physics - Satellite of Euroscipy 2011 Message-ID: <20110528092213.GA24443@phare.normalesup.org> On behalf of the organizers, it is my pleasure to announce PyPhy, a satellite meeting of Euroscipy 2011 on **Python in Physics**. Date and venue -------------- August 29th, 2011 (full day) Department of Physics, Ecole normale sup?rieure, Paris Main topics ------------ * Python for teaching physics * Python for research in physics * developing Python modules for applications in physics Organizers ---------- * Alberto Rosso (LPTMS, Universit? Paris-Sud - alberto.rosso at u-psud.fr) * Werner Krauth (Physics Department, ENS Paris - werner.krauth at ens.fr) In physics, Python is widely used as a scripting and programming language for data processing, but also for numerical computing where it is a viable alternative to compiled languages, with the advantage of being easy to learn, to write, to read and to modify. For this reason, Python, which is entirely free, is an ideal tool for teaching algorithmic contents in physics. A number of initiatives integrating Python into curricula have been developed in the last few years, largely independently in Physics departments all over the world. At the same time, Python libraries for numerical analysis, combinatorics, graphics, interfacing other languages, etc. have either reached maturity, or are developing very rapidly. This makes Python an exciting option for research applications. A growing number of research programs now write code directly in Python. This informal workshop on teaching and research using Python in Physics will be a forum for coordinating these initiatives, for sharing different experiences, and for exchanging with the key developers of the scientific Python modules present at the EuroSciPy 2011 conference. Beginners and Students are welcome to attend. People interested in presenting a contribution should contact the organizers and send a short abstract before June 30, 2011. We welcome contributions on your experience in teaching Python to physicists and engineers, developing a Python module for applications in physics, or using Python for your everyday research work. Participation in this workshop will be free of charge but participants should also make themselves known by email (alberto.rosso at u-psud.fr) before August 28, 2011. Important dates --------------- * June 30: deadline for contributions * August 28: deadline for (free) registration Invited Speakers ---------------- * Samuel Bottani (MSC, Paris Diderot) * Gianfranco Durin (ISI, Turin) * Emmanuelle Gouillart (Saint-Gobain Research) * Konrad Hinsen (CBM Universit? d'Orleans) * Vivien Lecomte (LPMA Paris Diderot) * Chris Myers (Department of Physics, Cornell University) * Michael Schindler (LPCT, EPSCI, Paris) * Georg von Hippel (INP Universit?t Mainz) How to get there ---------------- See the instructions on http://www.phys.ens.fr/spip.php?rubrique42&lang=en Website ------- http://www.euroscipy.org/card/pyphy2011 --------- Apologies if you receive this announcement more than once! Emmanuelle From akshar.bhosale at gmail.com Sat May 28 10:32:16 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Sat, 28 May 2011 20:02:16 +0530 Subject: [Numpy-discussion] Is this right forum for doubts abt numpy/scipy installation Message-ID: Hi, Is this right forum for doubts abt numpy/scipy installation? Please find our issue below .. we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 x86_64 with kernel 2.6.18-92.el5. We have following configuration : /opt/intel/Compiler/11.0/069/ mkl/lib/em64t Now we want to install numpy and scipy as an user in my home directory. Following are the libraries build inside MKL. libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so libfftw2xf_intel.a libmkl_blas95.a libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so libfftw3xc_intel.a libmkl_cdft.a libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so libfftw3xf_intel.a libmkl_cdft_core.a libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale libmkl_blacs_ilp64.a libmkl_core.a libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod libmkl_blacs_intelmpi20_lp64.a libmkl_def.so libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a libmkl_lapack95.a libmkl.so mkl95_blas.mod libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so libmkl_lapack.so libmkl_solver_ilp64.a mkl95_precision.mod libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a libmkl_mc3.so libmkl_solver_ilp64_sequential.a libmkl_blacs_lp64.a libmkl_gf_lp64.so libmkl_mc.so libmkl_solver_lp64.a libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a libmkl_p4n.so libmkl_solver_lp64_sequential.a libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so libmkl_pgi_thread.a libmkl_vml_def.so Version we are trying to build of numpy and scipy are : numpy : 1.6.0b2 and scipy : 0.9.0 We have configured python as a user in my home directory with version : 2.6.6 our machine has python : 2.4.3 We want to know the exact procedure for installing it using MKL as we are facing lot of issues while installing the same. Please help us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michaeladamkatz at yahoo.com Sat May 28 15:18:11 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Sat, 28 May 2011 12:18:11 -0700 (PDT) Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <4DE04665.7030105@noaa.gov> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> Message-ID: <141131.25608.qm@web161315.mail.bf1.yahoo.com> Yes, thanks, np.in1d is what I needed. I didn't know how to find that. It still seems counterintuitive to me that indexes = np.where( records.integer_field in values ) does not work whereas indexes = np.where( records.integer_field > 5 ) does. In one case numpy is overriding the > operator; it's not checking if an array is greater than 5, but whether each element in the array is greater than 5. >From a naive user's point of view, not knowing much about the difference between > and in from a python point of view, it seems like in would get overridden the same way. ________________________________ From: Christopher Barker To: Discussion of Numerical Python Sent: Fri, May 27, 2011 5:48:37 PM Subject: Re: [Numpy-discussion] finding elements that match any in a set On 5/27/11 9:48 AM, Michael Katz wrote: > I have a numpy array, records, with named fields including a field named > "integer_field". I have an array (or list) of values of interest, and I > want to get the indexes where integer_field has any of those values. > > Because I can do > > indexes = np.where( records.integer_field > 5 ) > > I thought I could do > > indexes = np.where( records.integer_field in values ) > > But that doesn't work. (As a side question I'm interested in why that > doesn't work, when values is a python list.) that doesn't work because the python list "in" operator doesn't understand arrays -- so it is looking ot see if the entire array is in the list. actually, it doesn't even get that far: In [16]: a in l --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /Users/chris.barker/ in () ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() The ValueError results because it was decided that numpy array should not have a boolean value to avoid confusion -- i.e. is na array true whenever it is non-empty (like a list), or when all it's elements are true, or???? When I read this question, I thought -- hmmm, numpy needs something like "in", as the usual way: np.any(), would require a loop in this case. Then I read Skipper's message: On 5/27/11 9:55 AM, Skipper Seabold wrote: > Check out this recent thread. I think the proposed class does what you > want. It's more efficient than in1d, if values is small compared to > the length of records. So that class may be worthwhile, but I think np.in1d is exactly what you are looking for: indexes = np.in1d( records.integer_field, values ) Funny I'd never noticed that before. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat May 28 15:30:05 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 28 May 2011 14:30:05 -0500 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <141131.25608.qm@web161315.mail.bf1.yahoo.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> Message-ID: On Sat, May 28, 2011 at 14:18, Michael Katz wrote: > Yes, thanks, np.in1d is what I needed. I didn't know how to find that. > > It still seems counterintuitive to me that > > ??? indexes = np.where( records.integer_field in values ) > > does not work whereas > > ??? indexes = np.where( records.integer_field > 5 ) > does. > > In one case numpy is overriding the > operator; it's not checking if an > array is greater than 5, but whether each element in the array is greater > than 5. > > From a naive user's point of view, not knowing much about the difference > between > and in from a python point of view, it seems like in would get > overridden the same way. The Python operators are turned into special method calls on one of the objects. Most of the special methods that define the mathematical operators come in pairs: __lt__ and __gt__, __add__ and __radd__, etc. So if we have (x > y) then x.__gt__(y) is checked first. If x does not know about the type of y, then y.__lt__(x) is checked. Similarly, for (x + y), x.__add__(y) is checked first, then y.__radd__(x) is checked. (myarray > 5), myarray.__gt__(5) is checked. numpy arrays do know about ints, so that works. However, (myarray in mylist) turns into mylist.__contains__(myarray). Only the list object is ever checked for this method. There is no paired method myarray.__rcontains__(mylist) so there is nothing that numpy can override to make this operation do anything different from what lists normally do, which is check if the given object is equal to one of the items in the list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From michaeladamkatz at yahoo.com Sat May 28 15:40:32 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Sat, 28 May 2011 12:40:32 -0700 (PDT) Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> Message-ID: <431116.28334.qm@web161304.mail.bf1.yahoo.com> Thanks for the explanation. It sounds like Python needs __rcontains__. ________________________________ From: Robert Kern To: Discussion of Numerical Python Sent: Sat, May 28, 2011 12:30:05 PM Subject: Re: [Numpy-discussion] finding elements that match any in a set On Sat, May 28, 2011 at 14:18, Michael Katz wrote: > Yes, thanks, np.in1d is what I needed. I didn't know how to find that. > > It still seems counterintuitive to me that > > indexes = np.where( records.integer_field in values ) > > does not work whereas > > indexes = np.where( records.integer_field > 5 ) > does. > > In one case numpy is overriding the > operator; it's not checking if an > array is greater than 5, but whether each element in the array is greater > than 5. > > From a naive user's point of view, not knowing much about the difference > between > and in from a python point of view, it seems like in would get > overridden the same way. The Python operators are turned into special method calls on one of the objects. Most of the special methods that define the mathematical operators come in pairs: __lt__ and __gt__, __add__ and __radd__, etc. So if we have (x > y) then x.__gt__(y) is checked first. If x does not know about the type of y, then y.__lt__(x) is checked. Similarly, for (x + y), x.__add__(y) is checked first, then y.__radd__(x) is checked. (myarray > 5), myarray.__gt__(5) is checked. numpy arrays do know about ints, so that works. However, (myarray in mylist) turns into mylist.__contains__(myarray). Only the list object is ever checked for this method. There is no paired method myarray.__rcontains__(mylist) so there is nothing that numpy can override to make this operation do anything different from what lists normally do, which is check if the given object is equal to one of the items in the list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat May 28 15:46:32 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 28 May 2011 14:46:32 -0500 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <431116.28334.qm@web161304.mail.bf1.yahoo.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> Message-ID: On Sat, May 28, 2011 at 14:40, Michael Katz wrote: > Thanks for the explanation. It sounds like Python needs __rcontains__. Not really. For the mathematical operators, there are good ways for the operands to "know" what types they can deal with and which they can't. For mylist.__contains__(x), it should treat all objects exactly the same: check if it equals any item that it contains. There is no way for it to say, "Oh, I don't know how to deal with this type, so I'll pass it over to x.__contains__()". A function call is the best place for this operation, not syntax. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Sat May 28 15:47:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 28 May 2011 21:47:50 +0200 Subject: [Numpy-discussion] Is this right forum for doubts abt numpy/scipy installation In-Reply-To: References: Message-ID: On Sat, May 28, 2011 at 4:32 PM, akshar bhosale wrote: > Hi, > Is this right forum for doubts abt numpy/scipy installation? > It is. Please don't keep resending your email. This is your third email with the exact same content, while I replied to your first one already. Ralf > Please find our issue below .. > we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 > x86_64 with kernel 2.6.18-92.el5. > We have following configuration : > /opt/intel/Compiler/11.0/069/ > mkl/lib/em64t > Now we want to install numpy and scipy as an user in my home directory. > > Following are the libraries build inside MKL. > > libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a > libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so > libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a > libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so > libfftw2xf_intel.a libmkl_blas95.a > libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so > libfftw3xc_intel.a libmkl_cdft.a > libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so > libfftw3xf_intel.a libmkl_cdft_core.a > libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale > libmkl_blacs_ilp64.a libmkl_core.a > libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod > libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so > libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod > libmkl_blacs_intelmpi20_lp64.a libmkl_def.so > libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod > libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a > libmkl_lapack95.a libmkl.so mkl95_blas.mod > libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a > libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod > libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so > libmkl_lapack.so libmkl_solver_ilp64.a > mkl95_precision.mod > libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a > libmkl_mc3.so libmkl_solver_ilp64_sequential.a > libmkl_blacs_lp64.a libmkl_gf_lp64.so > libmkl_mc.so libmkl_solver_lp64.a > libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a > libmkl_p4n.so libmkl_solver_lp64_sequential.a > libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so > libmkl_pgi_thread.a libmkl_vml_def.so > > Version we are trying to build of numpy and scipy are : > numpy : 1.6.0b2 and scipy : 0.9.0 > We have configured python as a user in my home directory with version : > 2.6.6 our machine has python : 2.4.3 > > We want to know the exact procedure for installing it using MKL as we are > facing lot of issues while installing the same. > Please help us. > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitriy.rybalkin at gmail.com Sat May 28 23:09:42 2011 From: dmitriy.rybalkin at gmail.com (Dmitriy Rybalkin) Date: Sun, 29 May 2011 07:09:42 +0400 Subject: [Numpy-discussion] Contribution In-Reply-To: <20110528091537.GC2799@phare.normalesup.org> References: <20110528091537.GC2799@phare.normalesup.org> Message-ID: Thank you, Gael, I really hope that I'll improve my programming skills. Also I have a question: Will the page with my example be deleted in case the code is not efficient or smth else has been found to be wrong? On Sat, May 28, 2011 at 1:15 PM, Gael Varoquaux wrote: > I Dmitriy, > > Welcome. > > This is a great initiative. You'll learn by doing, and I am willing to > bet that contributing examples can make you a good programmer. > > On Fri, May 27, 2011 at 09:47:24PM +0400, Dmitriy Rybalkin wrote: >> I thought I could contribute to the Numpy project by >> showing some examples of simple classification techniques (Fisher >> discriminant function, Naive Bayes) which use numpy functions. So, >> I'am inquiring about such contribution and some details: can I do that >> and, if yes, then how can I submit such example or should I only >> modify the Wiki page and request the permission? > > It might be interesting to stress on the corresponding wiki pages that > these methods are available in readily coded package, such as the > scikits.learn. The reason I stress this is that code on a wiki page tends > to go unmaintained, while a great deal of quality assurance goes in the > standard software packages. > > Ga?l > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dmitriy.rybalkin at gmail.com Sat May 28 23:34:04 2011 From: dmitriy.rybalkin at gmail.com (Dmitriy Rybalkin) Date: Sun, 29 May 2011 07:34:04 +0400 Subject: [Numpy-discussion] Contribution In-Reply-To: References: <20110528091537.GC2799@phare.normalesup.org> Message-ID: I added the page LinearClassification to Numpy/Scipy Cookbook http://www.scipy.org/Cookbook/LinearClassification, hope that it won't be deleted On Sun, May 29, 2011 at 7:09 AM, Dmitriy Rybalkin wrote: > Thank you, Gael, I really hope that I'll improve my programming skills. > Also I have a question: Will the page with my example be deleted in > case the code is not efficient or smth else has been found to be > wrong? > > On Sat, May 28, 2011 at 1:15 PM, Gael Varoquaux > wrote: >> I Dmitriy, >> >> Welcome. >> >> This is a great initiative. You'll learn by doing, and I am willing to >> bet that contributing examples can make you a good programmer. >> >> On Fri, May 27, 2011 at 09:47:24PM +0400, Dmitriy Rybalkin wrote: >>> I thought I could contribute to the Numpy project by >>> showing some examples of simple classification techniques (Fisher >>> discriminant function, Naive Bayes) which use numpy functions. So, >>> I'am inquiring about such contribution and some details: can I do that >>> and, if yes, then how can I submit such example or should I only >>> modify the Wiki page and request the permission? >> >> It might be interesting to stress on the corresponding wiki pages that >> these methods are available in readily coded package, such as the >> scikits.learn. The reason I stress this is that code on a wiki page tends >> to go unmaintained, while a great deal of quality assurance goes in the >> standard software packages. >> >> Ga?l >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > From gael.varoquaux at normalesup.org Sun May 29 03:55:21 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 29 May 2011 09:55:21 +0200 Subject: [Numpy-discussion] Contribution In-Reply-To: References: <20110528091537.GC2799@phare.normalesup.org> Message-ID: <20110529075521.GA9026@phare.normalesup.org> On Sun, May 29, 2011 at 07:09:42AM +0400, Dmitriy Rybalkin wrote: > Thank you, Gael, I really hope that I'll improve my programming skills. > Also I have a question: Will the page with my example be deleted in > case the code is not efficient or smth else has been found to be > wrong? I cannot really answer this question, as there is no rule: it depends on the goodwill of whoever reads and tries out the page. It's a wiki. In my experience, people are reluctant to delete other's work. Sometimes they fix it, which is the best solution. Gael From alan.isaac at gmail.com Sun May 29 10:13:31 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 29 May 2011 10:13:31 -0400 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <431116.28334.qm@web161304.mail.bf1.yahoo.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> Message-ID: <4DE2548B.5080803@gmail.com> On 5/28/2011 3:40 PM, Robert wrote: > (myarray in mylist) turns into mylist.__contains__(myarray). > Only the list object is ever checked for this method. There is no > paired method myarray.__rcontains__(mylist) so there is nothing that > numpy can override to make this operation do anything different from > what lists normally do, which is check if the given object is equal to > one of the items in the list. This seems to me to slightly miscast the problem. How would an __rcontains__ method really fix things? Would the list type check against a table of stuff that it knows how to contain? That seems horrible. And even if possible, NumPy would then have to break the rule that ``in`` tests for equality, because (I believe) the real problem in this case is that np equality testing does not return a bool. From this perspective, what is missing is not __rcontains__ (since the list already knows what to do) but rather __eeq__ for element-by-element comparison (ideally, along with an element-by-element operator such as say .==). In the meantime the OP could use any(all(a==x) for x in lst) fwiw, Alan Isaac From alan.isaac at gmail.com Sun May 29 10:14:58 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Sun, 29 May 2011 10:14:58 -0400 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> Message-ID: <4DE254E2.4020402@gmail.com> On 5/28/2011 3:46 PM, Robert Kern wrote: > mylist.__contains__(x), it should treat all objects exactly > the same: check if it equals any item that it contains. There is no > way for it to say, "Oh, I don't know how to deal with this type, so > I'll pass it over to x.__contains__()". Which makes my comment redundant ... From neilcrighton at gmail.com Sun May 29 13:03:25 2011 From: neilcrighton at gmail.com (Neil Crighton) Date: Sun, 29 May 2011 17:03:25 +0000 (UTC) Subject: [Numpy-discussion] finding elements that match any in a set References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> Message-ID: Michael Katz yahoo.com> writes: > Yes, thanks, np.in1d is what I needed. I didn't know how to find that. Did you check in the documentation? If so, where did you check? Would you have found it if it was in the 'See also' section of where()? (http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) I ask because people often post to the list needing in1d() after not being able to find it via the docs, so it would be nice to add references in the places people go looking for it. Neil From ralf.gommers at googlemail.com Sun May 29 14:19:54 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 29 May 2011 20:19:54 +0200 Subject: [Numpy-discussion] AttributeError in numpy.distutils In-Reply-To: <4DD9526F.6030509@astro.caltech.edu> References: <4DD2C98B.2050803@astro.caltech.edu> <4DD56133.3000405@astro.caltech.edu> <4DD9526F.6030509@astro.caltech.edu> Message-ID: On Sun, May 22, 2011 at 8:14 PM, Branimir Sesar wrote: > On 05/22/2011 04:17 AM, Ralf Gommers wrote: > > > > > > On Thu, May 19, 2011 at 8:28 PM, Branimir Sesar > > > wrote: > > > > Dear Numpy users, > > > > I've encountered an AttributeError in numpy.distutils > > > > File > > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > > line 646, in swig_sources > > extension.swig_opts.remove('-c++') > > AttributeError: 'str' object has no attribute 'remove' > > > > while compiling some code with Python 2.7.1 and Numpy 1.6.0. > > > > > > What are you doing here, compiling numpy? Building some of your own > > swig-ed code? Please give the details needed to reproduce your issue. > > I've been trying to compile Scikits ANN > (http://projects.scipy.org/scikits/wiki/AnnWrapper) with Python 2.7.1, > numpy 1.6.0, and SWIG 2.0.3 but the compilation breaks down down with > the error given below. Previously, I was able to compile Scikits ANN > using Enthought Python Distribution 7.0.2 (Python 2.7.1, numpy 1.5.1, > swig 1.3.40). > > running install > running bdist_egg > running egg_info > running build_src > build_src > building extension "scikits.ann._ANN" sources > Traceback (most recent call last): > File "setup.py", line 139, in > test_suite = 'nose.collector', > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/core.py", > > line 186, in setup > return old_setup(**new_attr) > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/core.py", line > 152, in setup > dist.run_commands() > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line > 953, in run_commands > self.run_command(cmd) > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line > 972, in run_command > cmd_obj.run() > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", > > line 57, in run > r = self.setuptools_run() > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", > > line 51, in setuptools_run > self.do_egg_install() > File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", > line 96, in do_egg_install > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line > 326, in run_command > self.distribution.run_command(command) > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line > 972, in run_command > cmd_obj.run() > File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", > line 167, in run > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line > 326, in run_command > self.distribution.run_command(command) > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line > 972, in run_command > cmd_obj.run() > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/egg_info.py", > > line 8, in run > self.run_command("build_src") > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line > 326, in run_command > self.distribution.run_command(command) > File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line > 972, in run_command > cmd_obj.run() > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > > line 152, in run > self.build_sources() > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > > line 169, in build_sources > self.build_extension_sources(ext) > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > > line 332, in build_extension_sources > sources = self.swig_sources(sources, ext) > File > > "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", > > line 646, in swig_sources > extension.swig_opts.remove('-c++') > AttributeError: 'str' object has no attribute 'remove' > > Looks like this is a bug introduced in numpy 1.6.0 by commit ff0822c4. Right above this line (numpy/distutils/command/build_src.py, line 646) add this: if isinstance(extension.swig_opts, basestring): extension.swig_opts = extension.swig_opts.split() Then you should be able to compile scikits.ann. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Sun May 29 15:58:41 2011 From: Chris.Barker at noaa.gov (Chris Barker) Date: Sun, 29 May 2011 12:58:41 -0700 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <4DE2548B.5080803@gmail.com> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> <4DE2548B.5080803@gmail.com> Message-ID: <4DE2A571.8040107@noaa.gov> > On 5/28/2011 3:40 PM, Robert wrote: >> (myarray in mylist) turns into mylist.__contains__(myarray). >> Only the list object is ever checked for this method. There is no >> paired method myarray.__rcontains__(mylist) so there is nothing that >> numpy can override to make this operation do anything different from >> what lists normally do, however, numpy arrays should be able to override "in" be defining their own.__contains__ method, so you could do: something in an_array and get a useful, vectorized result. So I thought I'd see what currently happens when I try that: In [24]: a Out[24]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) In [25]: 3 in a Out[25]: True So the simple case works just like a list. But what If I want what the OP wants: In [26]: b Out[26]: array([3, 6, 4]) In [27]: b in a Out[27]: False OK, so the full b array is not in a, and it doesn't "vectorize" it, either. But: In [29]: a Out[29]: array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]]) In [30]: b in a Out[30]: True HUH? I'm not sure by what definition we would say that b is contained in a. but maybe.. In [41]: b Out[41]: array([ 4, 2, 345]) In [42]: b in a Out[42]: False so it's "are all of the elements in b in a somewhere?" but only for 2-d arrays? So what does it mean? The docstring is not helpful: In [58]: np.ndarray.__contains__? Type: wrapper_descriptor Base Class: String Form: Namespace: Interactive Docstring: x.__contains__(y) <==> y in x If nothing useful, maybe it could provide a vectorized version of "in" for this sort of use case. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From michaeladamkatz at yahoo.com Sun May 29 16:02:12 2011 From: michaeladamkatz at yahoo.com (Michael Katz) Date: Sun, 29 May 2011 13:02:12 -0700 (PDT) Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> Message-ID: <590521.36890.qm@web161308.mail.bf1.yahoo.com> Yes, in this case I definitely would have found in1d() if it was referenced in the where() section, either as a "see also" or even better as an example where where() is combined with np.in1d(): indexes_of_interest = np.where( np.in1d( my_records.integer_field, my_values_of_interest ) ) I think the where() documentation page must be a place where a lot of people/newbies spend a lot of time. Perhaps like me they are focusing on the solution being "where() + some python stuff I already know", instead of thinking of other numpy functions, like in1d(), that might come into play. It makes sense that in1d() is under the "Set" section. However (just to try to explain further why I didn't look and find it there), somehow I think of "set" when I am focused on having a list without duplicates. In my case I wasn't worried about duplicates, just about "I want all the guys that match any of these other guys". I did google for "numpy member", "numpy membership", "numpy in", but none led me to in1d(). Also, it's worth saying that, as a newcomer to numpy and relative newcomer to python, I often think that what I'm looking for isn't going to end up being a function with a name -- often some use of slices or (fancy) indexing, or some other "pure syntax" mechanism, ends up doing what you want. So that's one reason I didn't simply scan all the available numpy function names. ________________________________ From: Neil Crighton To: numpy-discussion at scipy.org Sent: Sun, May 29, 2011 10:03:25 AM Subject: Re: [Numpy-discussion] finding elements that match any in a set Michael Katz yahoo.com> writes: > Yes, thanks, np.in1d is what I needed. I didn't know how to find that. Did you check in the documentation? If so, where did you check? Would you have found it if it was in the 'See also' section of where()? (http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) I ask because people often post to the list needing in1d() after not being able to find it via the docs, so it would be nice to add references in the places people go looking for it. Neil _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Sun May 29 17:50:29 2011 From: e.antero.tammi at gmail.com (eat) Date: Mon, 30 May 2011 00:50:29 +0300 Subject: [Numpy-discussion] finding elements that match any in a set In-Reply-To: <4DE2A571.8040107@noaa.gov> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> <4DE2548B.5080803@gmail.com> <4DE2A571.8040107@noaa.gov> Message-ID: On Sun, May 29, 2011 at 10:58 PM, Chris Barker wrote: > > On 5/28/2011 3:40 PM, Robert wrote: > >> (myarray in mylist) turns into mylist.__contains__(myarray). > >> Only the list object is ever checked for this method. There is no > >> paired method myarray.__rcontains__(mylist) so there is nothing that > >> numpy can override to make this operation do anything different from > >> what lists normally do, > > however, numpy arrays should be able to override "in" be defining their > own.__contains__ method, so you could do: > > something in an_array > > and get a useful, vectorized result. > > So I thought I'd see what currently happens when I try that: > > In [24]: a > Out[24]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) > > In [25]: 3 in a > Out[25]: True > > So the simple case works just like a list. But what If I want what the > OP wants: > > In [26]: b > Out[26]: array([3, 6, 4]) > > In [27]: b in a > Out[27]: False > > OK, so the full b array is not in a, and it doesn't "vectorize" it, > either. But: > > In [29]: a > Out[29]: > array([[ 0, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8], > [ 9, 10, 11]]) > > In [30]: b in a > Out[30]: True > > HUH? > > I'm not sure by what definition we would say that b is contained in a. > > but maybe.. > > In [41]: b > Out[41]: array([ 4, 2, 345]) > > In [42]: b in a > Out[42]: False > > so it's "are all of the elements in b in a somewhere?" but only for 2-d > arrays? > > > So what does it mean? > FWIW, a short prelude on the theme seems quite promising, indeed: In []: A Out[]: array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) In []: [0, 1, 2] in A Out[]: True In []: [0, 3, 6] in A Out[]: True In []: [0, 4, 8] in A Out[]: True In []: [8, 4, 0] in A Out[]: True In []: [2, 4, 6] in A Out[]: True In []: [6, 4, 2] in A Out[]: True In []: [3, 1, 5] in A Out[]: True In [1061]: [3, 1, 4] in A Out[1061]: True But In []: [1, 2, 3] in A Out[]: False In []: [3, 2, 1] in A Out[]: True So, obviously the logic behind __contains__ is not so very straightforward. Perhaps just a bug? Regards, eat > > The docstring is not helpful: > > In [58]: np.ndarray.__contains__? > Type: wrapper_descriptor > Base Class: > String Form: > Namespace: Interactive > Docstring: > x.__contains__(y) <==> y in x > > > If nothing useful, maybe it could provide a vectorized version of "in" > for this sort of use case. > > -Chris > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Mon May 30 03:54:21 2011 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Mon, 30 May 2011 10:54:21 +0300 Subject: [Numpy-discussion] AttributeError in numpy.distutils In-Reply-To: References: <4DD2C98B.2050803@astro.caltech.edu> <4DD56133.3000405@astro.caltech.edu> <4DD9526F.6030509@astro.caltech.edu> Message-ID: On Sun, May 29, 2011 at 9:19 PM, Ralf Gommers wrote: > > > On Sun, May 22, 2011 at 8:14 PM, Branimir Sesar wrote: > >> On 05/22/2011 04:17 AM, Ralf Gommers wrote: >> > >> > >> > On Thu, May 19, 2011 at 8:28 PM, Branimir Sesar >> > > wrote: >> > >> > Dear Numpy users, >> > >> > I've encountered an AttributeError in numpy.distutils >> > >> > File >> > >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", >> > line 646, in swig_sources >> > extension.swig_opts.remove('-c++') >> > AttributeError: 'str' object has no attribute 'remove' >> > >> > while compiling some code with Python 2.7.1 and Numpy 1.6.0. >> > >> > >> > What are you doing here, compiling numpy? Building some of your own >> > swig-ed code? Please give the details needed to reproduce your issue. >> >> I've been trying to compile Scikits ANN >> (http://projects.scipy.org/scikits/wiki/AnnWrapper) with Python 2.7.1, >> numpy 1.6.0, and SWIG 2.0.3 but the compilation breaks down down with >> the error given below. Previously, I was able to compile Scikits ANN >> using Enthought Python Distribution 7.0.2 (Python 2.7.1, numpy 1.5.1, >> swig 1.3.40). >> >> running install >> running bdist_egg >> running egg_info >> running build_src >> build_src >> building extension "scikits.ann._ANN" sources >> Traceback (most recent call last): >> File "setup.py", line 139, in >> test_suite = 'nose.collector', >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/core.py", >> >> line 186, in setup >> return old_setup(**new_attr) >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/core.py", line >> 152, in setup >> dist.run_commands() >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line >> 953, in run_commands >> self.run_command(cmd) >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line >> 972, in run_command >> cmd_obj.run() >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", >> >> line 57, in run >> r = self.setuptools_run() >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/install.py", >> >> line 51, in setuptools_run >> self.do_egg_install() >> File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", >> line 96, in do_egg_install >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line >> 326, in run_command >> self.distribution.run_command(command) >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line >> 972, in run_command >> cmd_obj.run() >> File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", >> line 167, in run >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line >> 326, in run_command >> self.distribution.run_command(command) >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line >> 972, in run_command >> cmd_obj.run() >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/egg_info.py", >> >> line 8, in run >> self.run_command("build_src") >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/cmd.py", line >> 326, in run_command >> self.distribution.run_command(command) >> File "/home/bsesar/usr/pydebug/lib/python2.7/distutils/dist.py", line >> 972, in run_command >> cmd_obj.run() >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", >> >> line 152, in run >> self.build_sources() >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", >> >> line 169, in build_sources >> self.build_extension_sources(ext) >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", >> >> line 332, in build_extension_sources >> sources = self.swig_sources(sources, ext) >> File >> >> "/home/bsesar/usr/pydebug/lib/python2.7/site-packages/numpy/distutils/command/build_src.py", >> >> line 646, in swig_sources >> extension.swig_opts.remove('-c++') >> AttributeError: 'str' object has no attribute 'remove' >> >> Looks like this is a bug introduced in numpy 1.6.0 by commit ff0822c4. > > This is not a bug as explained in http://projects.scipy.org/numpy/ticket/1851 > Right above this line (numpy/distutils/command/build_src.py, line 646) add > this: > > if isinstance(extension.swig_opts, basestring): > extension.swig_opts = extension.swig_opts.split() > > Then you should be able to compile scikits.ann. > > The bug is in scikits/ann/setup.py. Applying the following patch --- scikits/ann/setup.py (revision 2267) +++ scikits/ann/setup.py (working copy) @@ -45,11 +45,11 @@ library_dirs = [ann_library_dir], libraries = ['ANN'], language='c++', - swig_opts='-c++', + swig_opts=['-c++'], ) config.add_subpackage('tests') return config will make the build successful. HTH, Pearu -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon May 30 06:17:08 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 30 May 2011 10:17:08 +0000 (UTC) Subject: [Numpy-discussion] Abinit question References: <4DDE76BA.40203@gmail.com> Message-ID: Fri, 27 May 2011 21:06:45 +0300, Talla wrote: [clip: AbinitBandStructuremaker.py] FYI: try getting a newer version of the abinit software from abinit.org They seem to have an updated version of this particular script in their source package. Any further discussion should probably go to their forums: http://www.abinit.org/community From akshar.bhosale at gmail.com Sun May 29 09:05:02 2011 From: akshar.bhosale at gmail.com (akshar bhosale) Date: Sun, 29 May 2011 18:35:02 +0530 Subject: [Numpy-discussion] numpy,scipy installation using mkl Message-ID: Hi, Is this right forum for doubts abt numpy/scipy installation? Please find our issue below .. we have machine having intel xeon x7350 processors(8 nos) and RHEL 5.2 x86_64 with kernel 2.6.18-92.el5. We have following configuration : /opt/intel/Compiler/11.0/069/ mkl/lib/em64t Now we want to install numpy and scipy as an user in my home directory. Following are the libraries build inside MKL. libfftw2x_cdft_DOUBLE.a libmkl_blacs_sgimpt_ilp64.a libmkl_intel_ilp64.a libmkl_pgi_thread.so libmkl_vml_mc2.so libfftw2xc_intel.a libmkl_blacs_sgimpt_lp64.a libmkl_intel_ilp64.so libmkl_scalapack.a libmkl_vml_mc3.so libfftw2xf_intel.a libmkl_blas95.a libmkl_intel_lp64.a libmkl_scalapack_ilp64.a libmkl_vml_mc.so libfftw3xc_intel.a libmkl_cdft.a libmkl_intel_lp64.so libmkl_scalapack_ilp64.so libmkl_vml_p4n.so libfftw3xf_intel.a libmkl_cdft_core.a libmkl_intel_sp2dp.a libmkl_scalapack_lp64.a locale libmkl_blacs_ilp64.a libmkl_core.a libmkl_intel_sp2dp.so libmkl_scalapack_lp64.so mkl77_blas.mod libmkl_blacs_intelmpi20_ilp64.a libmkl_core.so libmkl_intel_thread.a libmkl_sequential.a mkl77_lapack1.mod libmkl_blacs_intelmpi20_lp64.a libmkl_def.so libmkl_intel_thread.so libmkl_sequential.so mkl77_lapack.mod libmkl_blacs_intelmpi_ilp64.a libmkl_em64t.a libmkl_lapack95.a libmkl.so mkl95_blas.mod libmkl_blacs_intelmpi_ilp64.so libmkl_gf_ilp64.a libmkl_lapack.a libmkl_solver.a mkl95_lapack.mod libmkl_blacs_intelmpi_lp64.a libmkl_gf_ilp64.so libmkl_lapack.so libmkl_solver_ilp64.a mkl95_precision.mod libmkl_blacs_intelmpi_lp64.so libmkl_gf_lp64.a libmkl_mc3.so libmkl_solver_ilp64_sequential.a libmkl_blacs_lp64.a libmkl_gf_lp64.so libmkl_mc.so libmkl_solver_lp64.a libmkl_blacs_openmpi_ilp64.a libmkl_gnu_thread.a libmkl_p4n.so libmkl_solver_lp64_sequential.a libmkl_blacs_openmpi_lp64.a libmkl_gnu_thread.so libmkl_pgi_thread.a libmkl_vml_def.so Version we are trying to build of numpy and scipy are : numpy : 1.6.0b2 and scipy : 0.9.0 We have configured python as a user in my home directory with version : 2.6.6 our machine has python : 2.4.3 We want to know the exact procedure for installing it using MKL as we are facing lot of issues while installing the same. Please help us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From edo.paskino at gmail.com Tue May 31 05:04:38 2011 From: edo.paskino at gmail.com (Edoardo Pasca) Date: Tue, 31 May 2011 11:04:38 +0200 Subject: [Numpy-discussion] local variable referenced before assignment Message-ID: Dear all, sometimes I encounter the problem that calling many times a function it happens that some local variables are not defined and the procedure crashes. For example I have a function defined as def procedure(tt, Ctissue, WeinmannFit, bodyWeight): for maxFastSampling in range(1,len(tt)): if tt[maxFastSampling] - tt[maxFastSampling-1] > 10: break if numpy.equal(Ctissue[:maxFastSampling] , 0 ).all(): # do something Now the inputs are all numpy.array except bodyWeight that is a number. The program calls the function procedure many times but at some point fails at the evaluation of numpy.equal with UnboundLocalError: local variable 'maxFastSampling' referenced before assignment I wonder whether you see any way that the maxFastSampling variable might not be set. Python 2.6.4 numpy 1.5.0 Thanks Edo -- Edo weblog? http://edo.imanetti.net/wordpress podcast http://sieegiueeaccaso.blogspot.com cooking http://edo.imanetti.net/cucina :wq From schut at sarvision.nl Tue May 31 05:34:19 2011 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 31 May 2011 11:34:19 +0200 Subject: [Numpy-discussion] local variable referenced before assignment In-Reply-To: References: Message-ID: On 05/31/2011 11:04 AM, Edoardo Pasca wrote: > Dear all, > > sometimes I encounter the problem that calling many times a function > it happens that some local variables are not defined and the procedure > crashes. > > For example I have a function defined as > > def procedure(tt, Ctissue, WeinmannFit, bodyWeight): > for maxFastSampling in range(1,len(tt)): > if tt[maxFastSampling] - tt[maxFastSampling-1]> 10: > break > > if numpy.equal(Ctissue[:maxFastSampling] , 0 ).all(): > # do something > > Now the inputs are all numpy.array except bodyWeight that is a number. > > The program calls the function procedure many times but at some point > fails at the evaluation of numpy.equal with > > UnboundLocalError: local variable 'maxFastSampling' referenced before assignment > > I wonder whether you see any way that the maxFastSampling variable > might not be set. Edo, When len(tt) is <2, range(1, len(tt)) gives an emtpy list, the for loop will never execute and maxFastSampling will not be set. VS. > > Python 2.6.4 > numpy 1.5.0 > > Thanks > > Edo > From edo.paskino at gmail.com Tue May 31 05:48:47 2011 From: edo.paskino at gmail.com (Edoardo Pasca) Date: Tue, 31 May 2011 11:48:47 +0200 Subject: [Numpy-discussion] local variable referenced before assignment In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 11:34 AM, Vincent Schut wrote: > On 05/31/2011 11:04 AM, Edoardo Pasca wrote: >> UnboundLocalError: local variable 'maxFastSampling' referenced before assignment >> >> I wonder whether you see any way that the maxFastSampling variable >> might not be set. > > Edo, > > When len(tt) is <2, range(1, len(tt)) gives an emtpy list, the for loop > will never execute and maxFastSampling will not be set. > > VS. Dear Vincent, thank you for the prompt response. That's indeed the cause. Now I am investigating what happens in the procedure that calls the function passing tt. Thanks Edo >> >> Python 2.6.4 >> numpy 1.5.0 >> >> Thanks >> >> Edo >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Edo weblog? http://edo.imanetti.net/wordpress podcast http://sieegiueeaccaso.blogspot.com cooking http://edo.imanetti.net/cucina :wq From christoph.graves at gmail.com Tue May 31 07:56:47 2011 From: christoph.graves at gmail.com (cgraves) Date: Tue, 31 May 2011 04:56:47 -0700 (PDT) Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> Message-ID: <31740152.post@talk.nabble.com> Ralf Gommers-2 wrote: > > On Fri, May 6, 2011 at 12:57 PM, Derek Homeier < > derek at astro.physik.uni-goettingen.de> wrote: > >> >> On 6 May 2011, at 07:53, Ralf Gommers wrote: >> >> > >> > >> Looks okay, and I agree that it's better to fix it now. The timing >> > >> is a bit unfortunate though, just after RC2. I'll have closer look >> > >> tomorrow and if it can go in, probably tag RC3. >> > >> >> > >> If in the meantime a few more people could test this, that would be >> > >> helpful. >> > >> >> > >> Ralf >> > > >> > > I agree, wish I had time to push this before rc2. I could add the >> > > explanatory comments >> > > mentioned above and switch to use the atleast_[12]d() solution, test >> > > that and push it >> > > in a couple of minutes, or should I better leave it as is now for >> > > testing? >> > >> > Quick follow-up: I just applied the above changes, added some tests to >> > cover Ben's test cases and tested this with 1.6.0rc2 on OS X 10.5 >> > i386+ppc >> > + 10.6 x86_64 (Python2.7+3.2). So I'd be ready to push it to my repo >> > and do >> > my (first) pull request... >> > >> > Go ahead, I'll have a look at it tonight. Thanks for testing on >> > several Pythons, that definitely helps. >> >> >> Done, the request only appears on my repo >> https://github.com/dhomeier/numpy/ >> >> is that correct? If someone could test it on Linux and Windows as >> well... >> > > Committed, thanks for all the work. > > The pull request was in the wrong place, that's a minor flaw in the github > UI. After you press "Pull Request" you need to read the small print to see > where it's going. > > Cheers, > Ralf > > Dear all, I've downloaded the latest numpy (1.6.0) and loadtxt has the ndmin option, however neither genfromtxt nor recfromtxt, which use loadtxt, have it. Should they have inherited the option? Who can make it happen? Best, Chris -- View this message in context: http://old.nabble.com/loadtxt-savetxt-tickets-tp31238871p31740152.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From pgmdevlist at gmail.com Tue May 31 10:20:03 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 16:20:03 +0200 Subject: [Numpy-discussion] Changing milestones on tickets Message-ID: All, Could somebody tell me the advantage of changing the milestone of tickets that have been closed for more than 10 months ? I'm genuinely curious. P. From mwwiebe at gmail.com Tue May 31 10:25:07 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 31 May 2011 09:25:07 -0500 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: Message-ID: It's so we can see what bugs are actually fixed for 2.0 (as opposed to a prior release), and a bug that's marked 'closed' but Unscheduled simply doesn't make sense to me. -Mark On Tue, May 31, 2011 at 9:20 AM, Pierre GM wrote: > All, > Could somebody tell me the advantage of changing the milestone of tickets > that have been closed for more than 10 months ? I'm genuinely curious. > P. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Tue May 31 10:53:26 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 May 2011 16:53:26 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <31740152.post@talk.nabble.com> References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> Message-ID: Hi Chris, On 31 May 2011, at 13:56, cgraves wrote: > I've downloaded the latest numpy (1.6.0) and loadtxt has the ndmin > option, > however neither genfromtxt nor recfromtxt, which use loadtxt, have it. > Should they have inherited the option? Who can make it happen? you are mistaken, genfromtxt is not using loadtxt (and could not possibly, since it has the more complex parser to handle missing data); thus ndmin could not be inherited automatically. It certainly would make sense to provide the same functionality for genfromtxt (which should then be inherited by [nd,ma,rec]fromtxt), so I'd go ahead and file a feature (enhancement) request. I can't promise I can take care of it myself, as I am less familiar with genfromtxt, but I'd certainly have a look at it. Does anyone have an opinion whether this is a case for reopening (yet again...) http://projects.scipy.org/numpy/ticket/1562 or create a new ticket? Cheers, Derek From pgmdevlist at gmail.com Tue May 31 11:09:57 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 17:09:57 +0200 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: Message-ID: On May 31, 2011, at 4:25 PM, Mark Wiebe wrote: > It's so we can see what bugs are actually fixed for 2.0 (as opposed to a prior release), and a bug that's marked 'closed' but Unscheduled simply doesn't make sense to me. I'm sorry, I'm still failing to see the logic. * You're not sure whether the bug has been actually fixed ? Reopen the ticket, and set it a milestone to the next release. * When a ticket has been closed with a "fixed" resolution, that's it, it's over, regardless of whether it had been scheduled for a specific release at the time it was open. Why changing its milestone to a version that postdate it by, say, a couple of releases? * If you need to find the bugs that have been fixed for 2.0 but not for 1.6, check the time when their ticket was closed. Everything after 05/14/2011 is a good bet... From pgmdevlist at gmail.com Tue May 31 11:18:43 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 17:18:43 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com> <5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de> <40DA9A71-A1B7-4F0E-A013-925C2414155C@astro.physik.uni-goettingen.de> <38117E8E-ADDB-4B1F-A797-04A6D4B6DEF4@gmail.com> <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> Message-ID: <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> On May 31, 2011, at 4:53 PM, Derek Homeier wrote: > Hi Chris, > > On 31 May 2011, at 13:56, cgraves wrote: > >> I've downloaded the latest numpy (1.6.0) and loadtxt has the ndmin >> option, >> however neither genfromtxt nor recfromtxt, which use loadtxt, have it. >> Should they have inherited the option? Who can make it happen? > > you are mistaken, genfromtxt is not using loadtxt (and could not > possibly, since it has the more complex parser to handle missing > data); thus ndmin could not be inherited automatically. > It certainly would make sense to provide the same functionality for > genfromtxt (which should then be inherited by [nd,ma,rec]fromtxt), so > I'd go ahead and file a feature (enhancement) request. I can't promise > I can take care of it myself, as I am less familiar with genfromtxt, > but I'd certainly have a look at it. Oh, that shouldn't be too difficult: 'ndmin' tells whether the array must be squeezed before being returned, right ? You can add some test at the very end of genfromtxt to check what to do with the output (whether to squeeze it or not, whether to transpose it or not)... If you don't mind doing it, I'd be quite grateful (I don't have time to work on numpy these days, much to my regret). Don't forget to change the user manual as well... From mwwiebe at gmail.com Tue May 31 11:33:02 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 31 May 2011 10:33:02 -0500 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 10:09 AM, Pierre GM wrote: > > On May 31, 2011, at 4:25 PM, Mark Wiebe wrote: > > > It's so we can see what bugs are actually fixed for 2.0 (as opposed to a > prior release), and a bug that's marked 'closed' but Unscheduled simply > doesn't make sense to me. > > I'm sorry, I'm still failing to see the logic. > * You're not sure whether the bug has been actually fixed ? Reopen the > ticket, and set it a milestone to the next release. * When a ticket has been closed with a "fixed" resolution, that's it, it's > over, regardless of whether it had been scheduled for a specific release at > the time it was open. Why changing its milestone to a version that postdate > it by, say, a couple of releases? > Because if a bug is closed, then it's scheduled for the next release by definition. Having bugs closed and unscheduled is an inconsistency in the bug tracker, and adjusting it to a release that it was at least fixed by is putting it closer to consistency than leaving it as unscheduled. Actually tracking down the release it went out in is too much work especially when the comments don't reference that. * If you need to find the bugs that have been fixed for 2.0 but not for 1.6, > check the time when their ticket was closed. Everything after 05/14/2011 is > a good bet... > Then what's the point of the Milestone field? I think it should reflect the earliest milestone the bugfix is targeted at, or the earliest milestone it was fixed in. Having it reflect the random setting that happened to be chosen at some time doesn't seem like a good policy to me. At least changing it as I have makes it easier to be more consistent in the future. The date can't be used for such a query, because after the 1.6 branch some fixes went into both 1.6 and 2.0, and some went into just 2.0. I expect this to be the case fairly often, and any closed ticket right now should be marked with milestone 1.6.1 or 2.0 depending on whether it is committed to the 1.6.x branch. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue May 31 11:33:10 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 31 May 2011 10:33:10 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> Message-ID: <4DE50A36.4020402@gmail.com> On 05/31/2011 10:18 AM, Pierre GM wrote: > On May 31, 2011, at 4:53 PM, Derek Homeier wrote: > >> Hi Chris, >> >> On 31 May 2011, at 13:56, cgraves wrote: >> >>> I've downloaded the latest numpy (1.6.0) and loadtxt has the ndmin >>> option, >>> however neither genfromtxt nor recfromtxt, which use loadtxt, have it. >>> Should they have inherited the option? Who can make it happen? >> you are mistaken, genfromtxt is not using loadtxt (and could not >> possibly, since it has the more complex parser to handle missing >> data); thus ndmin could not be inherited automatically. >> It certainly would make sense to provide the same functionality for >> genfromtxt (which should then be inherited by [nd,ma,rec]fromtxt), so >> I'd go ahead and file a feature (enhancement) request. I can't promise >> I can take care of it myself, as I am less familiar with genfromtxt, >> but I'd certainly have a look at it. > Oh, that shouldn't be too difficult: 'ndmin' tells whether the array must be squeezed before being returned, right ? You can add some test at the very end of genfromtxt to check what to do with the output (whether to squeeze it or not, whether to transpose it or not)... If you don't mind doing it, I'd be quite grateful (I don't have time to work on numpy these days, much to my regret). Don't forget to change the user manual as well... > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion (Different function so different ticket.) Sure you can change the end of the code but that may hide various problem. Unlike loadtxt, genfromtxt has a lot of flexibility especially handling missing values and using converter functions. So I think that some examples must be provided that can not be handled by providing a suitable converter or that require multiple assumptions about input file (such as having more than one delimiter). Bruce From ben.root at ou.edu Tue May 31 11:45:48 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 31 May 2011 10:45:48 -0500 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <4DE50A36.4020402@gmail.com> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> Message-ID: On Tue, May 31, 2011 at 10:33 AM, Bruce Southey wrote: > On 05/31/2011 10:18 AM, Pierre GM wrote: > > On May 31, 2011, at 4:53 PM, Derek Homeier wrote: > > > >> Hi Chris, > >> > >> On 31 May 2011, at 13:56, cgraves wrote: > >> > >>> I've downloaded the latest numpy (1.6.0) and loadtxt has the ndmin > >>> option, > >>> however neither genfromtxt nor recfromtxt, which use loadtxt, have it. > >>> Should they have inherited the option? Who can make it happen? > >> you are mistaken, genfromtxt is not using loadtxt (and could not > >> possibly, since it has the more complex parser to handle missing > >> data); thus ndmin could not be inherited automatically. > >> It certainly would make sense to provide the same functionality for > >> genfromtxt (which should then be inherited by [nd,ma,rec]fromtxt), so > >> I'd go ahead and file a feature (enhancement) request. I can't promise > >> I can take care of it myself, as I am less familiar with genfromtxt, > >> but I'd certainly have a look at it. > > Oh, that shouldn't be too difficult: 'ndmin' tells whether the array must > be squeezed before being returned, right ? You can add some test at the very > end of genfromtxt to check what to do with the output (whether to squeeze it > or not, whether to transpose it or not)... If you don't mind doing it, I'd > be quite grateful (I don't have time to work on numpy these days, much to my > regret). Don't forget to change the user manual as well... > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > (Different function so different ticket.) > > Sure you can change the end of the code but that may hide various > problem. Unlike loadtxt, genfromtxt has a lot of flexibility especially > handling missing values and using converter functions. So I think that > some examples must be provided that can not be handled by providing a > suitable converter or that require multiple assumptions about input file > (such as having more than one delimiter). > > Bruce > At this point, I wonder if it might be smarter to create a .atleast_Nd() function and use that everywhere it is needed. Having similar logic tailored for each loading function might be a little dangerous if bug fixes are made to one, but not the others. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Tue May 31 11:52:26 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 May 2011 17:52:26 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <4DE50A36.4020402@gmail.com> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> Message-ID: <1B6FBF74-D6E9-4A8E-9641-B7FEE54C814E@astro.physik.uni-goettingen.de> On 31 May 2011, at 17:33, Bruce Southey wrote: >>> It certainly would make sense to provide the same functionality for >>> genfromtxt (which should then be inherited by [nd,ma,rec]fromtxt), >>> so >>> I'd go ahead and file a feature (enhancement) request. I can't >>> promise >>> I can take care of it myself, as I am less familiar with genfromtxt, >>> but I'd certainly have a look at it. >> Oh, that shouldn't be too difficult: 'ndmin' tells whether the >> array must be squeezed before being returned, right ? You can add >> some test at the very end of genfromtxt to check what to do with >> the output (whether to squeeze it or not, whether to transpose it >> or not)... If you don't mind doing it, I'd be quite grateful (I >> don't have time to work on numpy these days, much to my regret). >> Don't forget to change the user manual as well... >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > (Different function so different ticket.) > > Sure you can change the end of the code but that may hide various > problem. Unlike loadtxt, genfromtxt has a lot of flexibility > especially > handling missing values and using converter functions. So I think that > some examples must be provided that can not be handled by providing a > suitable converter or that require multiple assumptions about input > file > (such as having more than one delimiter). I think stuff like multiple delimiters should have been dealt with before, as the right place to insert the ndmin code (which includes the decision to squeeze or not to squeeze as well as to add additional dimensions, if required) would be right at the end before the 'unpack' switch, or rather replacing the bit: if usemask: output = output.view(MaskedArray) output._mask = outputmask if unpack: return output.squeeze().T return output.squeeze() But there it's already not clear to me how to deal with the MaskedArray case... Cheers, Derek From derek at astro.physik.uni-goettingen.de Tue May 31 12:04:58 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 May 2011 18:04:58 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> Message-ID: <47DC652A-B6F1-40EC-8616-6DEAD959B2DE@astro.physik.uni-goettingen.de> On 31 May 2011, at 17:45, Benjamin Root wrote: > > At this point, I wonder if it might be smarter to create > a .atleast_Nd() function and use that everywhere it is needed. > Having similar logic tailored for each loading function might be a > little dangerous if bug fixes are made to one, but not the others. Like a generalised version of .atleast_1d / .atleast_2d? It would also have to include an .atmost_Nd functionality of some sorts, to replace the .squeeze(), generally a good idea (e.g. something like np.atleast_Nd(X, ndmin=0, ndmax=-1), where the default is not to reduce the maximum number of dimensions...). But for the io routines the situation is a bit more complex, since different shapes are expected to be returned depending on the text input (e.g. (1, N) for a single row vs. (N, 1) for a single column of data). Cheers, Derek From bsouthey at gmail.com Tue May 31 12:06:10 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 31 May 2011 11:06:10 -0500 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: Message-ID: <4DE511F2.8060103@gmail.com> On 05/31/2011 10:33 AM, Mark Wiebe wrote: > On Tue, May 31, 2011 at 10:09 AM, Pierre GM > wrote: > > > On May 31, 2011, at 4:25 PM, Mark Wiebe wrote: > > > It's so we can see what bugs are actually fixed for 2.0 (as > opposed to a prior release), and a bug that's marked 'closed' but > Unscheduled simply doesn't make sense to me. > > I'm sorry, I'm still failing to see the logic. > * You're not sure whether the bug has been actually fixed ? Reopen > the ticket, and set it a milestone to the next release. > > * When a ticket has been closed with a "fixed" resolution, that's > it, it's over, regardless of whether it had been scheduled for a > specific release at the time it was open. Why changing its > milestone to a version that postdate it by, say, a couple of releases? > > Because if a bug is closed, then it's scheduled for the next release > by definition. Having bugs closed and unscheduled is an inconsistency > in the bug tracker, and adjusting it to a release that it was at least > fixed by is putting it closer to consistency than leaving it as > unscheduled. Actually tracking down the release it went out in is too > much work especially when the comments don't reference that. > > * If you need to find the bugs that have been fixed for 2.0 but > not for 1.6, check the time when their ticket was closed. > Everything after 05/14/2011 is a good bet... > > Then what's the point of the Milestone field? I think it should > reflect the earliest milestone the bugfix is targeted at, or the > earliest milestone it was fixed in. Having it reflect the random > setting that happened to be chosen at some time doesn't seem like a > good policy to me. At least changing it as I have makes it easier to > be more consistent in the future. > > The date can't be used for such a query, because after the 1.6 branch > some fixes went into both 1.6 and 2.0, and some went into just 2.0. I > expect this to be the case fairly often, and any closed ticket right > now should be marked with milestone 1.6.1 or 2.0 depending on whether > it is committed to the 1.6.x branch. > > -Mark > I think using 'milestone' in this way is misleading because it is a field that the bug reporter enters when creating a new ticket. The actual value is usually wishful thinking by the reporter since it rarely appears to be updated. To be fair, most reports appear to be addressed very quickly so that only the difficult ones remain. If you want to use it in that way then I think that it should not be present in the initial ticket. Rather the assigned developer has to assign the milestone. There is probably also a requirement that milestone automatically updates when after each release. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue May 31 12:14:54 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 18:14:54 +0200 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: <4DE511F2.8060103@gmail.com> References: <4DE511F2.8060103@gmail.com> Message-ID: <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> On May 31, 2011, at 6:06 PM, Bruce Southey wrote: > On 05/31/2011 10:33 AM, Mark Wiebe wrote: >> On Tue, May 31, 2011 at 10:09 AM, Pierre GM wrote: >> >> On May 31, 2011, at 4:25 PM, Mark Wiebe wrote: >> >> > It's so we can see what bugs are actually fixed for 2.0 (as opposed to a prior release), and a bug that's marked 'closed' but Unscheduled simply doesn't make sense to me. >> >> I'm sorry, I'm still failing to see the logic. >> * You're not sure whether the bug has been actually fixed ? Reopen the ticket, and set it a milestone to the next release. >> * When a ticket has been closed with a "fixed" resolution, that's it, it's over, regardless of whether it had been scheduled for a specific release at the time it was open. Why changing its milestone to a version that postdate it by, say, a couple of releases? >> Because if a bug is closed, then it's scheduled for the next release by definition. Having bugs closed and unscheduled is an inconsistency in the bug tracker, and adjusting it to a release that it was at least fixed by is putting it closer to consistency than leaving it as unscheduled. Actually tracking down the release it went out in is too much work especially when the comments don't reference that. >> >> * If you need to find the bugs that have been fixed for 2.0 but not for 1.6, check the time when their ticket was closed. Everything after 05/14/2011 is a good bet... >> Then what's the point of the Milestone field? I think it should reflect the earliest milestone the bugfix is targeted at, or the earliest milestone it was fixed in. Having it reflect the random setting that happened to be chosen at some time doesn't seem like a good policy to me. At least changing it as I have makes it easier to be more consistent in the future. >> >> The date can't be used for such a query, because after the 1.6 branch some fixes went into both 1.6 and 2.0, and some went into just 2.0. I expect this to be the case fairly often, and any closed ticket right now should be marked with milestone 1.6.1 or 2.0 depending on whether it is committed to the 1.6.x branch. >> >> -Mark >> > I think using 'milestone' in this way is misleading because it is a field that the bug reporter enters when creating a new ticket. The actual value is usually wishful thinking by the reporter since it rarely appears to be updated. To be fair, most reports appear to be addressed very quickly so that only the difficult ones remain. > > If you want to use it in that way then I think that it should not be present in the initial ticket. Rather the assigned developer has to assign the milestone. There is probably also a requirement that milestone automatically updates when after each release. I'm OK with that. Mark, you could you add some note on the trac system to remind the person that closes a ticket that s/he needs to update the milestone field ? I find very commendable to strive for consistency, mind you. I'm just not not very comfortable with the idea of modifying old records a posteriori to adjust to new policies... From Chris.Barker at noaa.gov Tue May 31 12:25:04 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 31 May 2011 09:25:04 -0700 Subject: [Numpy-discussion] what does "in" do with numpy arrays? In-Reply-To: References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> <4DE2548B.5080803@gmail.com> <4DE2A571.8040107@noaa.gov> Message-ID: <4DE51660.8070705@noaa.gov> Hi folks, I've re-titled this thread, as it's about a new question, now: What does: something in a_numpy_array mean? i.e. how has __contains__ been defined? A couple of us have played with it, and can't make sense of it: > In [24]: a > Out[24]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) > > In [25]: 3 in a > Out[25]: True > > So the simple case works just like a list. But what If I look for an array in another array? > In [26]: b > Out[26]: array([3, 6, 4]) > > In [27]: b in a > Out[27]: False > > OK, so the full b array is not in a, and it doesn't "vectorize" it, > either. But: > > In [29]: a > Out[29]: > array([[ 0, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8], > [ 9, 10, 11]]) > > In [30]: b in a > Out[30]: True > > HUH? > > I'm not sure by what definition we would say that b is contained in a. > > but maybe.. > > In [41]: b > Out[41]: array([ 4, 2, 345]) > > In [42]: b in a > Out[42]: False > > so it's "are all of the elements in b in a somewhere?" but only for 2-d > arrays? > > > So what does it mean? > > The docstring is not helpful: > > In [58]: np.ndarray.__contains__? > Type: wrapper_descriptor > Base Class: > String Form: > Namespace: Interactive > Docstring: > x.__contains__(y)<==> y in x On 5/29/11 2:50 PM, eat wrote: > FWIW, a short prelude on the theme seems quite promising, indeed: > In []: A > Out[]: > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > In []: [0, 1, 2] in A > Out[]: True > In []: [0, 3, 6] in A > Out[]: True > In []: [0, 4, 8] in A > Out[]: True > In []: [8, 4, 0] in A > Out[]: True > In []: [2, 4, 6] in A > Out[]: True > In []: [6, 4, 2] in A > Out[]: True > In []: [3, 1, 5] in A > Out[]: True > In [1061]: [3, 1, 4] in A > Out[1061]: True > But > In []: [1, 2, 3] in A > Out[]: False > In []: [3, 2, 1] in A > Out[]: True > > So, obviously the logic behind __contains__ is not so very > straightforward. Perhaps just a bug? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pgmdevlist at gmail.com Tue May 31 12:25:06 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 18:25:06 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <1B6FBF74-D6E9-4A8E-9641-B7FEE54C814E@astro.physik.uni-goettingen.de> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> <1B6FBF74-D6E9-4A8E-9641-B7FEE54C814E@astro.physik.uni-goettingen.de> Message-ID: <1FAE9453-604F-4E83-B79F-307D5B4C5CF7@gmail.com> On May 31, 2011, at 5:52 PM, Derek Homeier wrote: > > I think stuff like multiple delimiters should have been dealt with > before, as the right place to insert the ndmin code (which includes > the decision to squeeze or not to squeeze as well as to add additional > dimensions, if required) would be right at the end before the 'unpack' > switch, or rather replacing the bit: > > if usemask: > output = output.view(MaskedArray) > output._mask = outputmask > if unpack: > return output.squeeze().T > return output.squeeze() > > But there it's already not clear to me how to deal with the > MaskedArray case... Oh, easy. You need to replace only the last three lines of genfromtxt with the ones from loadtxt (808-833). Then, if usemask is True, you need to use ma.atleast_Xd instead of np.atleast_Xd. Et voil?. Comments: * I would raise an exception if ndmin isn't correct *before* trying to read the file... * You could define a `collapse_function` that would be `np.atleast_1d`, `np.atleast_2d`, `ma.atleast_1d`... depending on the values of `usemask` and `ndmin`... If you have any question about numpy.ma, don't hesitate to contact me directly. From derek at astro.physik.uni-goettingen.de Tue May 31 12:37:36 2011 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 May 2011 18:37:36 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <1FAE9453-604F-4E83-B79F-307D5B4C5CF7@gmail.com> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> <1B6FBF74-D6E9-4A8E-9641-B7FEE54C814E@astro.physik.uni-goettingen.de> <1FAE9453-604F-4E83-B79F-307D5B4C5CF7@gmail.com> Message-ID: <0C7E9797-A7FD-4CA2-BBA9-A0613D023D76@astro.physik.uni-goettingen.de> On 31 May 2011, at 18:25, Pierre GM wrote: > On May 31, 2011, at 5:52 PM, Derek Homeier wrote: >> >> I think stuff like multiple delimiters should have been dealt with >> before, as the right place to insert the ndmin code (which includes >> the decision to squeeze or not to squeeze as well as to add >> additional >> dimensions, if required) would be right at the end before the >> 'unpack' >> switch, or rather replacing the bit: >> >> if usemask: >> output = output.view(MaskedArray) >> output._mask = outputmask >> if unpack: >> return output.squeeze().T >> return output.squeeze() >> >> But there it's already not clear to me how to deal with the >> MaskedArray case... > > Oh, easy. > You need to replace only the last three lines of genfromtxt with the > ones from loadtxt (808-833). Then, if usemask is True, you need to > use ma.atleast_Xd instead of np.atleast_Xd. Et voil?. > Comments: > * I would raise an exception if ndmin isn't correct *before* trying > to read the file... > * You could define a `collapse_function` that would be > `np.atleast_1d`, `np.atleast_2d`, `ma.atleast_1d`... depending on > the values of `usemask` and `ndmin`... > If you have any question about numpy.ma, don't hesitate to contact > me directly. Thanks for the directions! I was not sure about the usemask case because it presently does not invoke .squeeze() either... On a possibly related note, genfromtxt also treats the 'unpack'ing of structured arrays differently from loadtxt (which returns a list of arrays in that case) - do you know if this is on purpose, or also rather missing functionality (I guess it might break recfromtxt()...)? Cheers, Derek From mwwiebe at gmail.com Tue May 31 12:44:15 2011 From: mwwiebe at gmail.com (Mark Wiebe) Date: Tue, 31 May 2011 11:44:15 -0500 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> References: <4DE511F2.8060103@gmail.com> <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> Message-ID: On Tue, May 31, 2011 at 11:14 AM, Pierre GM wrote: > > On May 31, 2011, at 6:06 PM, Bruce Southey wrote: > > > On 05/31/2011 10:33 AM, Mark Wiebe wrote: > >> On Tue, May 31, 2011 at 10:09 AM, Pierre GM > wrote: > >> > >> On May 31, 2011, at 4:25 PM, Mark Wiebe wrote: > >> > >> > It's so we can see what bugs are actually fixed for 2.0 (as opposed to > a prior release), and a bug that's marked 'closed' but Unscheduled simply > doesn't make sense to me. > >> > >> I'm sorry, I'm still failing to see the logic. > >> * You're not sure whether the bug has been actually fixed ? Reopen the > ticket, and set it a milestone to the next release. > >> * When a ticket has been closed with a "fixed" resolution, that's it, > it's over, regardless of whether it had been scheduled for a specific > release at the time it was open. Why changing its milestone to a version > that postdate it by, say, a couple of releases? > >> Because if a bug is closed, then it's scheduled for the next release by > definition. Having bugs closed and unscheduled is an inconsistency in the > bug tracker, and adjusting it to a release that it was at least fixed by is > putting it closer to consistency than leaving it as unscheduled. Actually > tracking down the release it went out in is too much work especially when > the comments don't reference that. > >> > >> * If you need to find the bugs that have been fixed for 2.0 but not for > 1.6, check the time when their ticket was closed. Everything after > 05/14/2011 is a good bet... > >> Then what's the point of the Milestone field? I think it should reflect > the earliest milestone the bugfix is targeted at, or the earliest milestone > it was fixed in. Having it reflect the random setting that happened to be > chosen at some time doesn't seem like a good policy to me. At least changing > it as I have makes it easier to be more consistent in the future. > >> > >> The date can't be used for such a query, because after the 1.6 branch > some fixes went into both 1.6 and 2.0, and some went into just 2.0. I expect > this to be the case fairly often, and any closed ticket right now should be > marked with milestone 1.6.1 or 2.0 depending on whether it is committed to > the 1.6.x branch. > >> > >> -Mark > >> > > I think using 'milestone' in this way is misleading because it is a field > that the bug reporter enters when creating a new ticket. The actual value is > usually wishful thinking by the reporter since it rarely appears to be > updated. To be fair, most reports appear to be addressed very quickly so > that only the difficult ones remain. > > > > If you want to use it in that way then I think that it should not be > present in the initial ticket. Rather the assigned developer has to assign > the milestone. There is probably also a requirement that milestone > automatically updates when after each release. > > I'm OK with that. Mark, you could you add some note on the trac system to > remind the person that closes a ticket that s/he needs to update the > milestone field ? > I tried to find a place to do this in the trac system, but couldn't find an option like this. There are some validator plugins on the 'trac-hack' website, but they look very primitive and probably wouldn't give something reasonable either. It looks like a trac guru is needed to do something like this. > I find very commendable to strive for consistency, mind you. I'm just not > not very comfortable with the idea of modifying old records a posteriori to > adjust to new policies... I was under the impression this already was the policy, and the only reason it wasn't followed and the existing bugs hadn't been updated was the fact that trac has no nice mass-editing functionality. In particular, the 'roadmap' view (a prominent link at the top of the trac) suggests this by showing the bugs fixed for every unfinished milestone, and doing this required that someone insert custom trac markup into the milestones. If there is a bug policy written up somewhere it should probably be linked from main trac wiki page. -Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue May 31 13:11:37 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 31 May 2011 17:11:37 +0000 (UTC) Subject: [Numpy-discussion] Changing milestones on tickets References: <4DE511F2.8060103@gmail.com> <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> Message-ID: Tue, 31 May 2011 11:44:15 -0500, Mark Wiebe wrote: [clip] >> I find very commendable to strive for consistency, mind you. I'm just >> not not very comfortable with the idea of modifying old records a >> posteriori to adjust to new policies... > > I was under the impression this already was the policy, and the only > reason it wasn't followed and the existing bugs hadn't been updated was > the fact that trac has no nice mass-editing functionality. In > particular, the 'roadmap' view (a prominent link at the top of the trac) > suggests this by showing the bugs fixed for every unfinished milestone, > and doing this required that someone insert custom trac markup into the > milestones. If there is a bug policy written up somewhere it should > probably be linked from main trac wiki page. As far as I know, there simply has been no clear policy to the use of the milestone field. But at least I have the same idea as you here about how it should be used --- tickets should initially go into Unscheduled, and from there moved into a milestone in which they are (or are planned to be) fixed. The reason why so many bugs went into 2.0.0 is that this was the default value earlier, and most of the time the milestone was not updated when the tickets were closed. Anyway, it makes sense to have closed bugs appear under the milestone they were actually fixed in. I see no harm in changing this, and cleaning it up is a good thing. Pauli From ralf.gommers at googlemail.com Tue May 31 15:06:04 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 31 May 2011 21:06:04 +0200 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: <4DE511F2.8060103@gmail.com> <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> Message-ID: On Tue, May 31, 2011 at 7:11 PM, Pauli Virtanen wrote: > Tue, 31 May 2011 11:44:15 -0500, Mark Wiebe wrote: > [clip] > >> I find very commendable to strive for consistency, mind you. I'm just > >> not not very comfortable with the idea of modifying old records a > >> posteriori to adjust to new policies... > > > > I was under the impression this already was the policy, and the only > > reason it wasn't followed and the existing bugs hadn't been updated was > > the fact that trac has no nice mass-editing functionality. In > > particular, the 'roadmap' view (a prominent link at the top of the trac) > > suggests this by showing the bugs fixed for every unfinished milestone, > > and doing this required that someone insert custom trac markup into the > > milestones. If there is a bug policy written up somewhere it should > > probably be linked from main trac wiki page. > > As far as I know, there simply has been no clear policy to the use > of the milestone field. But at least I have the same idea as you here > about how it should be used --- tickets should initially go into > Unscheduled, and from there moved into a milestone in which they > are (or are planned to be) fixed. > > The reason why so many bugs went into 2.0.0 is that this was the default > value earlier, and most of the time the milestone was not updated when > the tickets were closed. > > Anyway, it makes sense to have closed bugs appear under the milestone > they were actually fixed in. I see no harm in changing this, and cleaning > it up is a good thing. > > It is helpful to have this cleaned up, thanks Mark for taking the time for this. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue May 31 15:28:28 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 21:28:28 +0200 Subject: [Numpy-discussion] loadtxt ndmin option In-Reply-To: <0C7E9797-A7FD-4CA2-BBA9-A0613D023D76@astro.physik.uni-goettingen.de> References: <0D212BB0-437C-4312-9CE1-BF982F3BAC40@astro.physik.uni-goettingen.de> <31740152.post@talk.nabble.com> <6224395B-AE11-424B-98EA-F9C88F50AE18@gmail.com> <4DE50A36.4020402@gmail.com> <1B6FBF74-D6E9-4A8E-9641-B7FEE54C814E@astro.physik.uni-goettingen.de> <1FAE9453-604F-4E83-B79F-307D5B4C5CF7@gmail.com> <0C7E9797-A7FD-4CA2-BBA9-A0613D023D76@astro.physik.uni-goettingen.de> Message-ID: On May 31, 2011, at 6:37 PM, Derek Homeier wrote: > On 31 May 2011, at 18:25, Pierre GM wrote: > >> On May 31, 2011, at 5:52 PM, Derek Homeier wrote: >>> >>> I think stuff like multiple delimiters should have been dealt with >>> before, as the right place to insert the ndmin code (which includes >>> the decision to squeeze or not to squeeze as well as to add >>> additional >>> dimensions, if required) would be right at the end before the >>> 'unpack' >>> switch, or rather replacing the bit: >>> >>> if usemask: >>> output = output.view(MaskedArray) >>> output._mask = outputmask >>> if unpack: >>> return output.squeeze().T >>> return output.squeeze() >>> >>> But there it's already not clear to me how to deal with the >>> MaskedArray case... >> >> Oh, easy. >> You need to replace only the last three lines of genfromtxt with the >> ones from loadtxt (808-833). Then, if usemask is True, you need to >> use ma.atleast_Xd instead of np.atleast_Xd. Et voil?. >> Comments: >> * I would raise an exception if ndmin isn't correct *before* trying >> to read the file... >> * You could define a `collapse_function` that would be >> `np.atleast_1d`, `np.atleast_2d`, `ma.atleast_1d`... depending on >> the values of `usemask` and `ndmin`... >> If you have any question about numpy.ma, don't hesitate to contact >> me directly. > > Thanks for the directions! I was not sure about the usemask case > because it presently does not invoke .squeeze() either... The idea is that if `usemask` is True, you build a second array (the mask), that you attach to your main array at the very end (in the `output=output.view(MaskedArray), output._mask = mask` combo...). Afterwards, it's a regular MaskedArray that supports the .squeeze() method... > On a > possibly related note, genfromtxt also treats the 'unpack'ing of > structured arrays differently from loadtxt (which returns a list of > arrays in that case) - do you know if this is on purpose, or also > rather missing functionality (I guess it might break recfromtxt()...)? Keep in mind that I haven't touched genfromtxt since 8-10 months or so. I wouldn't be surprised that it were lagging a bit behind loadtxt in terms of development. Yes, there'll be some tweaking to do for recfromtxt (it's OK for now if `ndmin` and `unpack` are the defaults) and others, but nothing major. From pgmdevlist at gmail.com Tue May 31 15:32:18 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 31 May 2011 21:32:18 +0200 Subject: [Numpy-discussion] Changing milestones on tickets In-Reply-To: References: <4DE511F2.8060103@gmail.com> <7B384569-0C31-4BEA-AC16-D8015F0F6C19@gmail.com> Message-ID: <3A84E12C-2CC8-4ADE-9623-7DC4E5515946@gmail.com> On May 31, 2011, at 9:06 PM, Ralf Gommers wrote: > > It is helpful to have this cleaned up, thanks Mark for taking the time for this. Mind you, I do agree with y'all for the cleaning up. I just had a shock when I received the batch of what I thought were brand new bugs that turned up to have been fixed and closed for months... Anyhow, I got all the answers I needed :) From robert.kern at gmail.com Tue May 31 20:31:21 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 31 May 2011 19:31:21 -0500 Subject: [Numpy-discussion] what does "in" do with numpy arrays? In-Reply-To: <4DE51660.8070705@noaa.gov> References: <21804.15125.qm@web161313.mail.bf1.yahoo.com> <4DE04665.7030105@noaa.gov> <141131.25608.qm@web161315.mail.bf1.yahoo.com> <431116.28334.qm@web161304.mail.bf1.yahoo.com> <4DE2548B.5080803@gmail.com> <4DE2A571.8040107@noaa.gov> <4DE51660.8070705@noaa.gov> Message-ID: On Tue, May 31, 2011 at 11:25, Christopher Barker wrote: > Hi folks, > > I've re-titled this thread, as it's about a new question, now: > > What does: > > something in a_numpy_array > > mean? i.e. how has __contains__ been defined? > > A couple of us have played with it, and can't make sense of it: > >> In [24]: a >> Out[24]: array([ 0, ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, 10, 11]) >> >> In [25]: 3 in a >> Out[25]: True >> >> So the simple case works just like a list. But what If I look for an array in another array? > >> In [26]: b >> Out[26]: array([3, 6, 4]) >> >> In [27]: b in a >> Out[27]: False >> >> OK, so the full b array is not in a, and it doesn't "vectorize" it, >> either. But: >> >> In [29]: a >> Out[29]: >> array([[ 0, ?1, ?2], >> ? ? ? ? ?[ 3, ?4, ?5], >> ? ? ? ? ?[ 6, ?7, ?8], >> ? ? ? ? ?[ 9, 10, 11]]) >> >> In [30]: b in a >> Out[30]: True >> >> HUH? >> >> I'm not sure by what definition we would say that b is contained in a. >> >> but maybe.. >> >> In [41]: b >> Out[41]: array([ ?4, ? 2, 345]) >> >> In [42]: b in a >> Out[42]: False >> >> so it's "are all of the elements in b in a somewhere?" but only for 2-d >> arrays? It dates back to Numeric's semantics for bool(some_array), which would be True if any of the elements were nonzero. Just like any other iterable container in Python, `x in y` will essentially do for row in y: if x == row: return True return False Iterate along the first axis of y and compare by boolean equality. In Numeric/numpy's case, this comparison is broadcasted. So that's why [3,6,4] works, because there is one row where 3 is in the first column. [4,2,345] doesn't work because the 4 and the 2 are not in those columns. Probably, this should be considered a mistake during the transition to numpy's semantics of having bool(some_array) raise an exception. `scalar in array` should probably work as-is for an ND array, but there are several different possible semantics for `array in array` that should be explicitly spelled out, much like bool(some_array). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From charlesr.harris at gmail.com Tue May 31 21:08:18 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 May 2011 19:08:18 -0600 Subject: [Numpy-discussion] New functions. Message-ID: Hi All, I've been contemplating new functions that could be added to numpy and thought I'd run them by folks to see if there is any interest. 1) Modified sort/argsort functions that return the maximum k values. This is easy to do with heapsort and almost as easy with mergesort. 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a faster version of nansum possible. 3) Fast medians. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue May 31 21:15:51 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 31 May 2011 20:15:51 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 20:08, Charles R Harris wrote: > Hi All, > > I've been contemplating new functions that could be added to numpy and > thought I'd run them by folks to see if there is any interest. > > 1) Modified sort/argsort functions that return the maximum k values. > ??? This is easy to do with heapsort and almost as easy with mergesort. > > 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a faster > version of nansum possible. > > 3) Fast medians. +3 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From warren.weckesser at enthought.com Tue May 31 21:18:28 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 31 May 2011 20:18:28 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:08 PM, Charles R Harris wrote: > Hi All, > > I've been contemplating new functions that could be added to numpy and > thought I'd run them by folks to see if there is any interest. > > 1) Modified sort/argsort functions that return the maximum k values. > This is easy to do with heapsort and almost as easy with mergesort. > > While you're at, how about a function that finds both the max and min in one pass? (Mentioned previously in this thread: http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a faster > version of nansum possible. > > 3) Fast medians. > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue May 31 21:26:51 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 May 2011 19:26:51 -0600 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 7:18 PM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > > > On Tue, May 31, 2011 at 8:08 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Hi All, >> >> I've been contemplating new functions that could be added to numpy and >> thought I'd run them by folks to see if there is any interest. >> >> 1) Modified sort/argsort functions that return the maximum k values. >> This is easy to do with heapsort and almost as easy with mergesort. >> >> > > While you're at, how about a function that finds both the max and min in > one pass? (Mentioned previously in this thread: > http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) > > What should it be called? minmax? This also brings suggests a function that returns both mean and standard deviation, something that could also be implemented using a more stable algorithm than the current one. > > 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a >> faster version of nansum possible. >> >> 3) Fast medians. >> >> >> Other suggestions are welcome. Most of these are of the low hanging fruit variety and shouldn't be too much work. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.root at ou.edu Tue May 31 21:31:10 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 31 May 2011 20:31:10 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:18 PM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > > > On Tue, May 31, 2011 at 8:08 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Hi All, >> >> I've been contemplating new functions that could be added to numpy and >> thought I'd run them by folks to see if there is any interest. >> >> 1) Modified sort/argsort functions that return the maximum k values. >> This is easy to do with heapsort and almost as easy with mergesort. >> >> > > While you're at, how about a function that finds both the max and min in > one pass? (Mentioned previously in this thread: > http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) > > > +1 from myself and probably just about anybody in matplotlib. If both the maxs and mins are searched during the same run through an array, I would imagine that would result in a noticeable speedup with automatic range finding. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Tue May 31 21:33:24 2011 From: david at silveregg.co.jp (David) Date: Wed, 01 Jun 2011 10:33:24 +0900 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: <4DE596E4.8080506@silveregg.co.jp> On 06/01/2011 10:08 AM, Charles R Harris wrote: > Hi All, > > I've been contemplating new functions that could be added to numpy and > thought I'd run them by folks to see if there is any interest. > > 1) Modified sort/argsort functions that return the maximum k values. > This is easy to do with heapsort and almost as easy with mergesort. > > 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a > faster version of nansum possible. > > 3) Fast medians. +1 for fast median as well, and more generally fast "linear" (O(kN)) order statistics would be nice. cheers, David From charlesr.harris at gmail.com Tue May 31 21:34:55 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 May 2011 19:34:55 -0600 Subject: [Numpy-discussion] New functions. In-Reply-To: <4DE596E4.8080506@silveregg.co.jp> References: <4DE596E4.8080506@silveregg.co.jp> Message-ID: On Tue, May 31, 2011 at 7:33 PM, David wrote: > On 06/01/2011 10:08 AM, Charles R Harris wrote: > > Hi All, > > > > I've been contemplating new functions that could be added to numpy and > > thought I'd run them by folks to see if there is any interest. > > > > 1) Modified sort/argsort functions that return the maximum k values. > > This is easy to do with heapsort and almost as easy with mergesort. > > > > 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a > > faster version of nansum possible. > > > > 3) Fast medians. > > +1 for fast median as well, and more generally fast "linear" (O(kN)) > order statistics would be nice. > > OK, noob question. What are order statistics? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Tue May 31 21:36:35 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 31 May 2011 21:36:35 -0400 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 9:31 PM, Benjamin Root wrote: > > > On Tue, May 31, 2011 at 8:18 PM, Warren Weckesser > wrote: >> >> >> On Tue, May 31, 2011 at 8:08 PM, Charles R Harris >> wrote: >>> >>> Hi All, >>> >>> I've been contemplating new functions that could be added to numpy and >>> thought I'd run them by folks to see if there is any interest. >>> >>> 1) Modified sort/argsort functions that return the maximum k values. >>> ??? This is easy to do with heapsort and almost as easy with mergesort. >>> >> >> >> While you're at, how about a function that finds both the max and min in >> one pass?? (Mentioned previously in this thread: >> http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) >> >> > > +1 from myself and probably just about anybody in matplotlib.? If both the > maxs and mins are searched during the same run through an array, I would > imagine that would result in a noticeable speedup with automatic range > finding. > I don't know if it's one pass off the top of my head, but I've used percentile for interpercentile ranges. [docs] [1]: X = np.random.random(1000) [docs] [2]: np.percentile(X,[0,100]) [2]: [0.00016535235312509222, 0.99961513543316571] [docs] [3]: X.min(),X.max() [3]: (0.00016535235312509222, 0.99961513543316571) Skipper From david at silveregg.co.jp Tue May 31 21:39:36 2011 From: david at silveregg.co.jp (David) Date: Wed, 01 Jun 2011 10:39:36 +0900 Subject: [Numpy-discussion] New functions. In-Reply-To: References: <4DE596E4.8080506@silveregg.co.jp> Message-ID: <4DE59858.10309@silveregg.co.jp> On 06/01/2011 10:34 AM, Charles R Harris wrote: > > > On Tue, May 31, 2011 at 7:33 PM, David > wrote: > > On 06/01/2011 10:08 AM, Charles R Harris wrote: > > Hi All, > > > > I've been contemplating new functions that could be added to > numpy and > > thought I'd run them by folks to see if there is any interest. > > > > 1) Modified sort/argsort functions that return the maximum k values. > > This is easy to do with heapsort and almost as easy with > mergesort. > > > > 2) Ufunc fadd (nanadd?) Treats nan as zero in addition. Should make a > > faster version of nansum possible. > > > > 3) Fast medians. > > +1 for fast median as well, and more generally fast "linear" (O(kN)) > order statistics would be nice. > > > OK, noob question. What are order statistics? In statistics, order statistics are statistics based on sorted samples, median, min and max being the most common: http://en.wikipedia.org/wiki/Order_statistic Concretely here, I meant a fast way to compute any rank of a given data set, e.g. with the select algorithm. I wanted to do that for some time, but never took the time for it, David From ben.root at ou.edu Tue May 31 21:48:54 2011 From: ben.root at ou.edu (Benjamin Root) Date: Tue, 31 May 2011 20:48:54 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:26 PM, Charles R Harris wrote: > > > On Tue, May 31, 2011 at 7:18 PM, Warren Weckesser < > warren.weckesser at enthought.com> wrote: > >> >> >> On Tue, May 31, 2011 at 8:08 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> Hi All, >>> >>> I've been contemplating new functions that could be added to numpy and >>> thought I'd run them by folks to see if there is any interest. >>> >>> 1) Modified sort/argsort functions that return the maximum k values. >>> This is easy to do with heapsort and almost as easy with mergesort. >>> >>> >> >> While you're at, how about a function that finds both the max and min in >> one pass? (Mentioned previously in this thread: >> http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) >> >> > What should it be called? minmax? > That's what it is called in IDL. Don't know of any other tool that does anything similar. > > This also brings suggests a function that returns both mean and standard > deviation, something that could also be implemented using a more stable > algorithm than the current one. > Yeah, it does. An implementation I did once for a project of mine in Matlab was to have a function that took an index that indicated that I wanted first/second/third/etc - order moments and it used that to figure out that it needed to save the sum, sum of squares, sum of cubes, etc. in order to calculate the mean, std, kurtosis, and so on. Just a suggestion on how to implement a useful function for stats. Of course, there are all sorts of issues with summation here, but my arrays were small enough to not worry in that project. Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Tue May 31 21:53:01 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 31 May 2011 20:53:01 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:36 PM, Skipper Seabold wrote: > On Tue, May 31, 2011 at 9:31 PM, Benjamin Root wrote: > > > > > > On Tue, May 31, 2011 at 8:18 PM, Warren Weckesser > > wrote: > >> > >> > >> On Tue, May 31, 2011 at 8:08 PM, Charles R Harris > >> wrote: > >>> > >>> Hi All, > >>> > >>> I've been contemplating new functions that could be added to numpy and > >>> thought I'd run them by folks to see if there is any interest. > >>> > >>> 1) Modified sort/argsort functions that return the maximum k values. > >>> This is easy to do with heapsort and almost as easy with mergesort. > >>> > >> > >> > >> While you're at, how about a function that finds both the max and min in > >> one pass? (Mentioned previously in this thread: > >> http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051072.html) > >> > >> > > > > +1 from myself and probably just about anybody in matplotlib. If both > the > > maxs and mins are searched during the same run through an array, I would > > imagine that would result in a noticeable speedup with automatic range > > finding. > > > > I don't know if it's one pass off the top of my head, but I've used > percentile for interpercentile ranges. > > [docs] > [1]: X = np.random.random(1000) > > [docs] > [2]: np.percentile(X,[0,100]) > [2]: [0.00016535235312509222, 0.99961513543316571] > > [docs] > [3]: X.min(),X.max() > [3]: (0.00016535235312509222, 0.99961513543316571) > > percentile() isn't one pass; using percentile like that is much slower: In [25]: %timeit np.percentile(X,[0,100]) 10000 loops, best of 3: 103 us per loop In [26]: %timeit X.min(),X.max() 100000 loops, best of 3: 11.8 us per loop Warren > Skipper > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Tue May 31 22:00:25 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 31 May 2011 22:00:25 -0400 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 9:53 PM, Warren Weckesser wrote: > > > On Tue, May 31, 2011 at 8:36 PM, Skipper Seabold > wrote: >> I don't know if it's one pass off the top of my head, but I've used >> percentile for interpercentile ranges. >> >> [docs] >> [1]: X = np.random.random(1000) >> >> [docs] >> [2]: np.percentile(X,[0,100]) >> [2]: [0.00016535235312509222, 0.99961513543316571] >> >> [docs] >> [3]: X.min(),X.max() >> [3]: (0.00016535235312509222, 0.99961513543316571) >> > > > percentile() isn't one pass; using percentile like that is much slower: > > In [25]: %timeit np.percentile(X,[0,100]) > 10000 loops, best of 3: 103 us per loop > > In [26]: %timeit X.min(),X.max() > 100000 loops, best of 3: 11.8 us per loop > Probably should've checked that before opening my mouth. Never actually used it for a minmax, but it is faster than two calls to scipy.stats.scoreatpercentile. Guess I'm +1 to fast order statistics. Skipper From charlesr.harris at gmail.com Tue May 31 22:26:50 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 May 2011 20:26:50 -0600 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:00 PM, Skipper Seabold wrote: > On Tue, May 31, 2011 at 9:53 PM, Warren Weckesser > wrote: > > > > > > On Tue, May 31, 2011 at 8:36 PM, Skipper Seabold > > wrote: > >> I don't know if it's one pass off the top of my head, but I've used > >> percentile for interpercentile ranges. > >> > >> [docs] > >> [1]: X = np.random.random(1000) > >> > >> [docs] > >> [2]: np.percentile(X,[0,100]) > >> [2]: [0.00016535235312509222, 0.99961513543316571] > >> > >> [docs] > >> [3]: X.min(),X.max() > >> [3]: (0.00016535235312509222, 0.99961513543316571) > >> > > > > > > percentile() isn't one pass; using percentile like that is much slower: > > > > In [25]: %timeit np.percentile(X,[0,100]) > > 10000 loops, best of 3: 103 us per loop > > > > In [26]: %timeit X.min(),X.max() > > 100000 loops, best of 3: 11.8 us per loop > > > > Probably should've checked that before opening my mouth. Never > actually used it for a minmax, but it is faster than two calls to > scipy.stats.scoreatpercentile. Guess I'm +1 to fast order statistics. > > So far the biggest interest seems to be in order statistics of various sorts, so to speak. *Order Statistics* minmax median k'th element largest/smallest k elements *Other Statistics* mean/std *Nan functions* nanadd Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue May 31 22:50:48 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 31 May 2011 21:50:48 -0500 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 9:26 PM, Charles R Harris wrote: > > > On Tue, May 31, 2011 at 8:00 PM, Skipper Seabold > wrote: >> >> On Tue, May 31, 2011 at 9:53 PM, Warren Weckesser >> wrote: >> > >> > >> > On Tue, May 31, 2011 at 8:36 PM, Skipper Seabold >> > wrote: >> >> I don't know if it's one pass off the top of my head, but I've used >> >> percentile for interpercentile ranges. >> >> >> >> [docs] >> >> [1]: X = np.random.random(1000) >> >> >> >> [docs] >> >> [2]: np.percentile(X,[0,100]) >> >> [2]: [0.00016535235312509222, 0.99961513543316571] >> >> >> >> [docs] >> >> [3]: X.min(),X.max() >> >> [3]: (0.00016535235312509222, 0.99961513543316571) >> >> >> > >> > >> > percentile() isn't one pass; using percentile like that is much slower: >> > >> > In [25]: %timeit np.percentile(X,[0,100]) >> > 10000 loops, best of 3: 103 us per loop >> > >> > In [26]: %timeit X.min(),X.max() >> > 100000 loops, best of 3: 11.8 us per loop >> > >> >> Probably should've checked that before opening my mouth. Never >> actually used it for a minmax, but it is faster than two calls to >> scipy.stats.scoreatpercentile. Guess I'm +1 to fast order statistics. >> > > So far the biggest interest seems to be in order statistics of various > sorts, so to speak. > > Order Statistics > > minmax > median > k'th element > largest/smallest k elements > > Other Statistics > > mean/std > > Nan functions > > nanadd > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > How about including all or some of Keith's Bottleneck package? He has tried to include some of the discussed functions and tried to make them very fast. Also, this Wikipedia "Algorithms for calculating variance" (http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance) has some basic info on calculating the variance as well as higher order moments.However, there are probably more efficient algorithms available. Bruce From charlesr.harris at gmail.com Tue May 31 23:41:46 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 31 May 2011 21:41:46 -0600 Subject: [Numpy-discussion] New functions. In-Reply-To: References: Message-ID: On Tue, May 31, 2011 at 8:50 PM, Bruce Southey wrote: > On Tue, May 31, 2011 at 9:26 PM, Charles R Harris > wrote: > > > > > > On Tue, May 31, 2011 at 8:00 PM, Skipper Seabold > > wrote: > >> > >> On Tue, May 31, 2011 at 9:53 PM, Warren Weckesser > >> wrote: > >> > > >> > > >> > On Tue, May 31, 2011 at 8:36 PM, Skipper Seabold > > >> > wrote: > >> >> I don't know if it's one pass off the top of my head, but I've used > >> >> percentile for interpercentile ranges. > >> >> > >> >> [docs] > >> >> [1]: X = np.random.random(1000) > >> >> > >> >> [docs] > >> >> [2]: np.percentile(X,[0,100]) > >> >> [2]: [0.00016535235312509222, 0.99961513543316571] > >> >> > >> >> [docs] > >> >> [3]: X.min(),X.max() > >> >> [3]: (0.00016535235312509222, 0.99961513543316571) > >> >> > >> > > >> > > >> > percentile() isn't one pass; using percentile like that is much > slower: > >> > > >> > In [25]: %timeit np.percentile(X,[0,100]) > >> > 10000 loops, best of 3: 103 us per loop > >> > > >> > In [26]: %timeit X.min(),X.max() > >> > 100000 loops, best of 3: 11.8 us per loop > >> > > >> > >> Probably should've checked that before opening my mouth. Never > >> actually used it for a minmax, but it is faster than two calls to > >> scipy.stats.scoreatpercentile. Guess I'm +1 to fast order statistics. > >> > > > > So far the biggest interest seems to be in order statistics of various > > sorts, so to speak. > > > > Order Statistics > > > > minmax > > median > > k'th element > > largest/smallest k elements > > > > Other Statistics > > > > mean/std > > > > Nan functions > > > > nanadd > > > > Chuck > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > How about including all or some of Keith's Bottleneck package? > He has tried to include some of the discussed functions and tried to > make them very fast. > I don't think they are sufficiently general as they are limited to 2 dimensions. However, I think the moving filters should go into scipy, either in ndimage or maybe signals. Some of the others we can still speed of significantly, for instance nanmedian, by using the new functionality in numpy, i.e., numpy sort has worked with nans for a while now. It looks like call overhead dominates the nanmax times for small arrays and this might improve if the ufunc machinery is cleaned up a bit more, I don't know how far Mark got with that. One bit of infrastructure that could be generally helpful is low-level support for masked arrays, but that is a larger topic. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: