Mailman 3 June 2012 - NumPy-Discussion

Bizarre errors with byteswapping, complex256, PPC
by Matthew Brett 21 Jun '12

21 Jun '12

Hi, Our Debian friends were hammering our code tests before the upcoming freeze, and found the following very odd thing on 32-bit PPC running Debian squeeze and numpy 1.6.2 or current trunk. Consider the following script: <script> import numpy as np arr = np.arange(10, dtype=np.complex256) bs_arr = arr.byteswap().newbyteorder('S') print arr print bs_arr print arr == bs_arr print arr == bs_arr print arr == bs_arr </script> Here is the output from some example runs of this script: np-devel)[mb312@joshlegacy ~/tmp]$ python funny_bs.py [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ True False False False False False False False False False] [ True False False False False False False False False False] [ True False False False False False False False False False] (that's the most common result) (np-devel)[mb312@joshlegacy ~/tmp]$ python funny_bs.py [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ True True True True True True True True True True] [ True True True True True True True True True True] [ True True True True True True True True True True] (this happens maybe 10% of the time) (np-devel)[mb312@joshlegacy ~/tmp]$ python funny_bs.py [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ 0.0+0.0j 1.0+0.0j 2.0+0.0j 3.0+0.0j 4.0+0.0j 5.0+0.0j 6.0+0.0j 7.0+0.0j 8.0+0.0j 9.0+0.0j] [ True True True True True True True True True True] [ True False False False False False False False False False] [ True True True True True True True True True True] (less than 10% of the time - order of True, False prints _of the same comparison_ is random. This only seems to happen with complex256. Is there anything I can do to debug this further? Does anyone want a login to this machine to have a look? See you, Matthew

3 11

frompyfunc question
by Larsen, Brian A 20 Jun '12

20 Jun '12

Hello all, I was looking to wrap hasattr in a numpy ufunc and got some weird behavior. Here is a transcript: In [5]: import numpy as np In [6]: np.__version__ Out[6]: '1.6.2' In [7]: b = [1,2,3] In [7]: nphas = np.frompyfunc(hasattr, 2, 1) In [8]: hasattr(b, 'extend') Out[8]: True In [9]: nphas(b, 'extend') Out[9]: NotImplemented In [10]: nphas(b, ['extend']*2) Out[10]: NotImplemented In [11]: type(nphas(b, ['extend']*2)) Out[11]: NotImplementedType This isn't an exception but an object returned, what what or why is not implemented? What I really want to do is be able to run hasattr like isinstance is run, isinstance(b, (list, str)) Anyone have any thoughts/advice here? Cheers, Brian -- Brian A. Larsen ISR-1 Space Science and Applications Los Alamos National Laboratory PO Box 1663, MS-D466 Los Alamos, NM 87545 USA (For overnight add: SM-30, Bikini Atoll Road) Phone: 505-665-7691 Fax: 505-665-7395 email: balarsen(a)lanl.gov<mailto:balarsen@lanl.gov> Correspondence / Technical data or Software Publicly Available

1 0

PyArray_MapIter{Reset, Next, New, Bind} not exposed?
by John Salvatier 20 Jun '12

20 Jun '12

Hello, I wanted to create a function that visits elements in an array using the same rules as advanced indexing (with integer and boolean arrays) but does addition instead of assignment (discussed more here http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062687.html). I looked at the code for advanced indexing and tried to copy that, but it uses PyArrayMapIterObject and the functions for using that object (PyArray_MapIterReset, and the equivalent Next,New and Bind functions as well as _swap_axes) are not exposed by the C API as far as I can tell. Is that on purpose? Is there an easy way to gain access to these functions? Should I give up on this approach? (The function I was trying to build is index_inc in here ( https://github.com/jsalvatier/advinc/blob/master/advinc/advinc.c)) Cheers, John

1 0

convert any non square matrix in to square matrix using numpy
by bob tnur 18 Jun '12

18 Jun '12

Hi, how I can convert (by adding zero) of any non-square numpy matrix in to square matrix using numpy? then how to find the minimum number in each row except the zeros added(for making square matrix)? ;)

3 2

numpy allocation event hooks
by Thouis (Ray) Jones 18 Jun '12

18 Jun '12

Based on some previous discussion on the numpy list [1] and in now-cancelled PRs [2,3], I'd like to solicit opinions on adding an interface for numpy memory allocation event tracking, as implemented in this PR: https://github.com/numpy/numpy/pull/309 A brief summary of the changes: - PyDataMem_NEW/FREE/RENEW become functions in the numpy API. (they used to be macros for malloc/free/realloc) These are the functions used to manage allocations for array's internal data. Most other numpy data is allocated through Python's allocator. - PyDataMem_NEW/RENEW return void* instead of char*. - Adds PyDataMem_SetEventHook() to the API, with this description: * Sets the allocation event hook for numpy array data. * Takes a PyDataMem_EventHookFunc *, which has the signature: * void hook(void *old, void *new, size_t size, void *user_data). * Also takes a void *user_data, and void **old_data. * * Returns a pointer to the previous hook or NULL. If old_data is * non-NULL, the previous user_data pointer will be copied to it. * * If not NULL, hook will be called at the end of each PyDataMem_NEW/FREE/RENEW: * result = PyDataMem_NEW(size) -> (*hook)(NULL, result, size, user_data) * PyDataMem_FREE(ptr) -> (*hook)(ptr, NULL, 0, user_data) * result = PyDataMem_RENEW(ptr, size) -> (*hook)(ptr, result, size, user_data) * * When the hook is called, the GIL will be held by the calling * thread. The hook should be written to be reentrant, if it performs * operations that might cause new allocation events (such as the * creation/descruction numpy objects, or creating/destroying Python * objects which might cause a gc) The PR also includes an example using the hook functions to track allocation via Python callback funcions (in tools/allocation_tracking). Why I think this is worth adding to numpy, even though other tools may be able to provide similar functionality: - numpy arrays use orders of magnitude more memory than most python objects, and this is often a limiting factor in algorithms. - numpy can behave in complicated ways with regards to memory management, e.g., views, OWNDATA, temporaries, etc., making it sometimes difficult to know where memory usage problems are happening and why. - numpy attracts a large number of programmers with limited low-level programming expertise, and who don't have the skills to use external tools (or time/motivation to acquire those skills), but still need to be able to diagnose these sorts of problems. - Other tools are not well integrated with Python, and vary a great deal between OS and compiler setup. I appreciate any feedback. Ray Jones [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062373.html [2] (python callbacks) https://github.com/numpy/numpy/pull/284 [3] (C-level logging) https://github.com/numpy/numpy/pull/301

2 2

Enum/Factor NEP (now with code)
by Bryan Van de Ven 17 Jun '12

17 Jun '12

Hi all, It has been some time, but I do have an update regarding this proposed feature. I thought it would be helpful to flesh out some parts of a possible implementation to learn what can be spelled reasonably in NumPy. Mark Wiebe helped out greatly in navigating the NumPy code codebase. Here is a link to my branch with this code; https://github.com/bryevdv/numpy/tree/enum and the updated NEP: https://github.com/bryevdv/numpy/blob/enum/doc/neps/enum.rst Not everything in the NEP is implemented (integral levels and natural naming in particular) and some parts definitely need more fleshing out. However, things currently work basically as described in the NEP, and there is also a small set of tests that demonstrate current usage. A few things will crash python (astype especially). More tests are needed. I would appreciate as much feedback and discussion as you can provide! Thanks, Bryan Van de Ven

6 18

Pull request: Split maskna support out of mainline into a branch
by Nathaniel Smith 17 Jun '12

17 Jun '12

Just submitted this pull request for discussion: https://github.com/numpy/numpy/pull/297 As per earlier discussion on the list, this PR attempts to remove exactly and only the maskna-related code from numpy mainline: http://mail.scipy.org/pipermail/numpy-discussion/2012-May/062417.html The suggestion is that we merge this to master for the 1.7 release, and immediately "git revert" it on a branch so that it can be modified further without blocking the release. The first patch does the actual maskna removal; the second and third rearrange things so that PyArray_ReduceWrapper does not end up in the public API, for reasons described therein. All tests pass with Python 2.4, 2.5, 2.6, 2.7, 3.1, 3.2 on 64-bit Ubuntu. The docs also appear to build. Before I re-based this I also tested against Scipy, matplotlib, and pandas, and all were fine. -- Nathaniel

4 8

travis-ci support for numpy
by Nathaniel Smith 16 Jun '12

16 Jun '12

Thanks to Marc Abramowitz[1], Numpy commits are now being tested by Travis-CI: http://travis-ci.org/#!/numpy/numpy As discussed on the numfocus list[2], this isn't really a complete CI solution, because it only gives test coverage on 64-bit Ubuntu. But, it does cover all supported versions of Python, which should at least catch the very common Python 2.4 compatibility bugs, so that's useful until someone gets a proper buildbot set up. And what will be useful even then is that travis-ci can automatically test all pull requests and report the results in the PR discussion directly[3], sort of like Fernando's script, but without human intervention. To get this turned on, someone needs to: -- donate some arbitrarily small (or large) amount: https://love.travis-ci.org/ -- email support(a)travis-ci.org and say "Hi, I donated and here is my order # to confirm; could you please turn on PR testing for the numpy/numpy repository, which was connected to travis by user 'njsmith'" I nominate Travis for the "someone", both on grounds of name collision amusingness and on grounds of him having a budget for such things :-). (Though literally, a $1 donation would seem to suffice from what the developers have said.) -n [1] https://github.com/numpy/numpy/pull/292 [2] https://groups.google.com/d/msg/numfocus/I_kmL4FUGaY/f2tO7IEU-l4J [3] http://about.travis-ci.org/blog/announcing-pull-request-support/

2 1

[ANN} OpenOpt / FuncDesigner release 0.39
by Dmitrey 15 Jun '12

15 Jun '12

Hi all, I'm glad to inform you about new OpenOpt Suite release 0.39 (2012-June-15): interalg: add categorical variables and general logical constraints, many other improvements Some improvements for automatic differentiation DerApproximator and some OpenOpt / FuncDesigner functionality now works with PyPy New solver lsmr for dense / sparse LLSP oovar constructors now can handle parameters lb and ub, e.g. a = oovar('a', lb=-1, ub=[1,2,3]) (this oovar should have size 3) or x = oovars(10, lb=-1, ub=1) New FuncDesigner function hstack, similar syntax to numpy.hstack, e.g. f = hstack((a,b,c,d)) Some bugfixes I have some progress toward solving in FuncDesigner linear DAE (differential algebraic equations, example) and Stochastic Opimization (example), but this is too premature yet to be released, there is 60-70% probability it will be properly implemented in next OpenOpt release. In our website you could vote for most required OpenOpt Suite development direction(s). -------------------- Regards, D. http://openopt.org/Dmitrey

1 0

automatic differentiation with PyAutoDiff
by James Bergstra 14 Jun '12

14 Jun '12

Further to the recent discussion on lazy evaluation & numba, I moved what I was doing into a new project: PyAutoDiff: https://github.com/jaberg/pyautodiff It currently works by executing CPython bytecode with a numpy-aware engine that builds a symbolic expression graph with Theano... so you can do for example: >>> import autodiff, numpy as np >>> autodiff.fmin_l_bfgs_b(lambda x: (x + 1) ** 2, [np.zeros(())]) ... and you'll see `[array(-1.0)]` printed out. In the future, I think it should be able to export the gradient-computing function as bytecode, which could then be optimized by e.g. numba or a theano bytecode front-end. For now it just compiles and runs the Theano graph that it built. It's still pretty rough (you'll see if you look at the code!) but I'm excited about it. - James

6 16