A little help please?
My user-defined type project has mostly gone well, but I'm stuck on mixed-type arithmetic. I have 2 types: cmplx_int32 and cmplx_int64. I have added basic arithmetic for those types, and for mix of those arrays and their respective scalars. But mixed arithmetic only partly works. In [2]: a Out[2]: array([(0,0), (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (9,0)], dtype=cmplx_int32) In [3]: b Out[3]: array([(0,0), (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (9,0)], dtype=cmplx_int64) In [4]: a+b --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/nbecker/numpy/<ipython console> in <module>() TypeError: function not supported for these types, and can't coerce safely to supported types In [5]: b+a Out[5]: array([(0,0), (2,0), (4,0), (6,0), (8,0), (10,0), (12,0), (14,0), (16,0), (18,0)], dtype=cmplx_int64) What I did: d1 is dtype cmplx_int32, d2 is dtype cmplx_int64. 1. PyArray_RegisterCastFunc (d1, d2->type_num, &cmplx_to_cmplx<cmplx_int64_t,cmplx_int32_t>); That registers a conversion from cmplx_int32->cmplx_int64. (Docs never explain when this conversion is used, BTW) 2. PyArray_RegisterCanCast (d1, d2->type_num, PyArray_NOSCALAR); 3. d1->f->castdict = PyDict_New(); PyObject *key = PyInt_FromLong (d2->type_num); PyObject *cobj = PyCObject_FromVoidPtr ((void*)(void(* (void*,void*,npy_intp,void*,void*))&cmplx_to_cmplx<cmplx_int64_t,cmplx_int32_t>, 0); PyDict_SetItem (d1->f->castdict, key, cobj); So now add (cmplx_int64, cmplx_int32) is OK, the 2nd arg is converted, but no attempt is made at radd. Obviously, I don't want to convert 64->32. Any clues what's missing here?
Neal Becker wrote:
My user-defined type project has mostly gone well, but I'm stuck on mixed-type arithmetic.
I have 2 types: cmplx_int32 and cmplx_int64. I have added basic arithmetic for those types, and for mix of those arrays and their respective scalars. But mixed arithmetic only partly works This is an area that needs testing and possible fixes. The relevant code is in ufuncobject.c (select_types) and in multiarraymodule.c (PyArray_CanCoerceScalar). If you can go through that code you may be able to see what the problem is and let us know.
I tried to support this kind of thing you are doing, but I'm not sure how well I succeeded because I didn't have time or the code to test it with. Thus, there is still some work to do. The fact that radd is not called is because ufuncs try to handle everything (the ufunc is more general than just the functions with "r" prefixes. I think one problem may be due to the fact that the first argument to a ufunc is the one that defines the search for the correctly registered function and there may be no code to allow other arguments to direct the search should that one fail. I'm actually pleased you've gotten this far. I'll keep trying to help as I get time. -Travis O.
Travis E. Oliphant wrote:
Neal Becker wrote:
My user-defined type project has mostly gone well, but I'm stuck on mixed-type arithmetic.
I have 2 types: cmplx_int32 and cmplx_int64. I have added basic arithmetic for those types, and for mix of those arrays and their respective scalars. But mixed arithmetic only partly works This is an area that needs testing and possible fixes. The relevant code is in ufuncobject.c (select_types) and in multiarraymodule.c (PyArray_CanCoerceScalar). If you can go through that code you may be able to see what the problem is and let us know.
I tried to support this kind of thing you are doing, but I'm not sure how well I succeeded because I didn't have time or the code to test it with. Thus, there is still some work to do.
The fact that radd is not called is because ufuncs try to handle everything (the ufunc is more general than just the functions with "r" prefixes. I think one problem may be due to the fact that the first argument to a ufunc is the one that defines the search for the correctly registered function and there may be no code to allow other arguments to direct the search should that one fail.
I'm actually pleased you've gotten this far. I'll keep trying to help as I get time.
The code for this is a bit hard to understand. It does appear that it only searches for a conversion on the 2nd argument. I don't think that's desirable behavior. What I'm wondering is, this works fine for builtin types. What is different in the handling of builtin types?
Neal Becker wrote:
Travis E. Oliphant wrote:
The code for this is a bit hard to understand. It does appear that it only searches for a conversion on the 2nd argument. I don't think that's desirable behavior.
What I'm wondering is, this works fine for builtin types. What is different in the handling of builtin types?
There are quite a few differences which lead to the current issues. 1) For built-in types there is a coercion order that can be searched more intelligently which does not exist for user-defined types. 2) For built-in types all the 1d loops are stored in a single C-array in the same order as the signatures. The entire signature list is scanned until a signature to which all inputs can be cast is found. 3) For user-defined types the 1d loops (functions) for a particular user-defined type are stored in a linked-list that itself is stored in a Python dictionary (as a C-object) attached to the ufunc and keyed by the user-defined type (of the first argument). Thus, what is missing is code to search all the linked lists in all the entries of all the user-defined types on input (only the linked-list keyed by the first user-defined type is searched at the moment). This would allow similar behavior to the built-in types (but a bit more expensive searching). -Travis O.
Travis E. Oliphant wrote:
Neal Becker wrote:
Travis E. Oliphant wrote:
The code for this is a bit hard to understand. It does appear that it only searches for a conversion on the 2nd argument. I don't think that's desirable behavior.
...
Thus, what is missing is code to search all the linked lists in all the entries of all the user-defined types on input (only the linked-list keyed by the first user-defined type is searched at the moment). This would allow similar behavior to the built-in types (but a bit more expensive searching).
Sounds like this needs a bit of re-thinking. Given a set of function signatures: F(a,b,c) F(d,e,f) ... The user calls: F(A,B,C) (no relation between a,A ,etc) How do we find the 'best' match? I think we can start with: Rules: 1) Only allowed (at most) 1 conversion on each argument But what is the 'best' match mean? I think that can't be decided without some sort of hierarchical relation of the types. Now given a hierarchy, I still don't know the solution, but it sounds like some kind of graph algorithm.
Neal Becker wrote:
Sounds like this needs a bit of re-thinking.
Given a set of function signatures: F(a,b,c) F(d,e,f) ...
The user calls: F(A,B,C) (no relation between a,A ,etc)
How do we find the 'best' match?
I think we can start with: Rules: 1) Only allowed (at most) 1 conversion on each argument
But what is the 'best' match mean?
I think that can't be decided without some sort of hierarchical relation of the types.
Now given a hierarchy, I still don't know the solution, but it sounds like some kind of graph algorithm.
Dear Neal, This response is probably a bit off-topic but I thought I'd provide this nugget to help guide your search for information. This topic has been heavily studied by theoretical computer scientists. Benjamin Pierce's Types and Programming Languages (MIT Press, 2002) is an excellent text gives a detailed review of the state of the art in the field of Type Systems. Finding the "best" match or an unambiguous one can be difficult depending on how the language is defined. Many type systems are defined by a set of typing rules and axioms. Types of expressions are often resolved by generating a proof that these rules are met. I've always found the subject of static typing interesting because it deals with how to make type guarantees of a program before it is evaluated or compiled. The subject to which you're referring is called dispatch. Many common OOP languages (e.g. C++, Java) support only a rudimentary dispatch algorithm where the method branch chosen for dispatch is determined at run-time by examining the type of the target object on which the method is invoked. The types of the arguments are only examined statically at compile or type-check time. Languages supporting multiple dispatch (variants of Java like MultiJava, Charming Python, etc.) examine the types of the arguments at run-time, and some multimethod type checkers can prove whether a program will ever encounter an ambiguous choice between branches before the program is ever run or compiled. I hope this helps. Damian
Travis E. Oliphant wrote:
Neal Becker wrote:
Travis E. Oliphant wrote:
The code for this is a bit hard to understand. It does appear that it only searches for a conversion on the 2nd argument. I don't think that's desirable behavior.
What I'm wondering is, this works fine for builtin types. What is different in the handling of builtin types?
3) For user-defined types the 1d loops (functions) for a particular user-defined type are stored in a linked-list that itself is stored in a Python dictionary (as a C-object) attached to the ufunc and keyed by the user-defined type (of the first argument).
Thus, what is missing is code to search all the linked lists in all the entries of all the user-defined types on input (only the linked-list keyed by the first user-defined type is searched at the moment). This would allow similar behavior to the built-in types (but a bit more expensive searching).
This code is now in place in current SVN. Could you re-try your example with the current code-base to see if it is fixed. Thanks, -Travis
Travis E. Oliphant wrote:
Travis E. Oliphant wrote:
Neal Becker wrote:
Travis E. Oliphant wrote:
The code for this is a bit hard to understand. It does appear that it only searches for a conversion on the 2nd argument. I don't think that's desirable behavior.
What I'm wondering is, this works fine for builtin types. What is different in the handling of builtin types?
3) For user-defined types the 1d loops (functions) for a particular user-defined type are stored in a linked-list that itself is stored in a Python dictionary (as a C-object) attached to the ufunc and keyed by the user-defined type (of the first argument).
Thus, what is missing is code to search all the linked lists in all the entries of all the user-defined types on input (only the linked-list keyed by the first user-defined type is searched at the moment). This would allow similar behavior to the built-in types (but a bit more expensive searching).
This code is now in place in current SVN. Could you re-try your example with the current code-base to see if it is fixed.
Thanks,
-Travis It seems to have broken 1 test:
FAIL: Test of inplace operations and rich comparisons ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.5/site-packages/numpy/ma/tests/test_old_ma.py", line 480, in check_testInplace assert id1 == id(x.data) AssertionError ---------------------------------------------------------------------- Ran 801 tests in 1.229s FAILED (failures=1) But looks like my test is working. BTW, don't forget the patch I sent diff --git a/numpy/core/src/ufuncobject.c b/numpy/core/src/ufuncobject.c --- a/numpy/core/src/ufuncobject.c +++ b/numpy/core/src/ufuncobject.c @@ -3434,10 +3434,10 @@ static int cmp_arg_types(int *arg1, int *arg2, int n) { - while (n--) { - if (PyArray_EquivTypenums(*arg1, *arg2)) continue; - if (PyArray_CanCastSafely(*arg1, *arg2)) - return -1; + for (;n > 0; n--, ++arg1, ++arg2) { + if (PyArray_EquivTypenums(*arg1, *arg2) || + PyArray_CanCastSafely(*arg1, *arg2)) + continue; return 1; } return 0;
participants (3)
-
Damian Eads -
Neal Becker -
Travis E. Oliphant