[Cython] array expressions

[mark florisson]

On 21 October 2012 13:39, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 14.10.2012 13:59:
>>>>> The problem with minivect as a package is that it caters to different
>>>>> projects, which have different requirements. Cython and minivect are
>>>>> quite closely coupled, and any future change, or in the future any
>>>>> older version may not have the functionality Cython needs, it's not
>>>>> exactly a stable API at this point.
>>>>
>>>> Ok, understood.
>>>>
>>>>
>>>>> For instance Numba needs python
>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>
>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>> compatibility. And that is really because not all use cases have yet
>>>>> been anticipated, and some really require a change, as I've already
>>>>> seen with Numba.
>>>>>
>>>>> We could list minivect as a dependency, which works for
>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>> into problems that didn't install with pip, and I don't think an
>>>>> exclusion of a 300kb addition is worth any of that.
>>>>
>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>> but including it directly and considering it a part of Cython (and Numba
>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>> package, or at least to support a separate installation and independent
>>>> update. Basically, if you can't update it separately, there's no use in
>>>> installing it separately.
>>>>
>>>> As long as we handle this so, we should take care to keep the generic parts
>>>> in their separate package directory and the Cython specific parts in
>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>> as possible, so that we can actually reach a point where both have an
>>>> interface. I would guess that the need to support Numba from the same
>>>> source base will encourage this kind of separation anyway.
>>>
>>> Yes, definitely.
>>>
>>>> Note that this means that minivect will fall under the release schedules of
>>>> Cython and Numba (independently), instead of really having its own releases.
>>>
>>> It can have its own releases as well, but currently there isn't much
>>> point :) Minivect can be developed independent of the releases, since
>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>> habit of squashing the minivect pulls to avoid its history.
>>>
>>> I'll also wait for Dag and Robert to see if they have a (final)
>>> opinion before merging the subtree.
>>
>> As I mentioned on the pull request, looks good to me. Given the
>> (somewhat) tight coupling with the AST but the desire to use the
>> codebase for multiple projects, a subtree seems to make the most sense
>> (until/if we have some kind of a plugin system).
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Great, thanks for the review Robert. There seems to be a refcount
> issue with python 3:
>
> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from Cython.Compiler import Main
> [98632 refs]
>>>> ^D
> [98632 refs]
> python: Modules/gcmodule.c:327: visit_decref: Assertion
> `gc->gc.gc_refs != 0' failed.
>
> The object being visited is a UtilityCode object. I think the error
> means the object is being visited more often than it's refcount value,
> which means it's not increffed properly somewhere. I'm not sure how/if
> my changes introduced this, has this problem been encountered recently
> in Cython's master?

This is running with -m trace --trace:

<snip>
Builtin.py(396):     for bf in builtin_function_table:
Builtin.py(397):         bf.declare_in_scope(builtin_scope)
 --- modulename: Builtin, funcname: declare_in_scope
Builtin.py(163):         func_type, sig = self.func_type, self.sig
Builtin.py(164):         if func_type is None:
Builtin.py(165):             if sig is None:
Builtin.py(166):                 sig = Signature(self.args, self.ret_type)
 --- modulename: TypeSlots, funcname: __init__
TypeSlots.py(85):         self.has_dummy_arg = 0
TypeSlots.py(86):         self.has_generic_args = 0
TypeSlots.py(87):         if arg_format[:1] == '-':
TypeSlots.py(90):         if arg_format[-1:] == '*':
TypeSlots.py(93):         self.fixed_arg_format = arg_format
TypeSlots.py(94):         self.ret_format = ret_format
TypeSlots.py(95):         self.error_value =
self.error_value_map.get(ret_format, None)
TypeSlots.py(96):         self.is_staticmethod = False
Builtin.py(167):             func_type = sig.function_type()
 --- modulename: TypeSlots, funcname: function_type
TypeSlots.py(125):         args = []
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
 --- modulename: TypeSlots, funcname: num_fixed_args
TypeSlots.py(99):         return len(self.fixed_arg_format)
TypeSlots.py(127):             if self_arg_override is not None and
self.is_self_arg(i):
TypeSlots.py(131):                 arg_type = self.fixed_arg_type(i)
 --- modulename: TypeSlots, funcname: fixed_arg_type
TypeSlots.py(110):         return self.format_map[self.fixed_arg_format[i]]
TypeSlots.py(132):
args.append(PyrexTypes.CFuncTypeArg("", arg_type, None))
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2966):         self.name = name
PyrexTypes.py(2967):         if cname is not None:
PyrexTypes.py(2970):             self.cname = Naming.var_prefix + name
PyrexTypes.py(2971):         self.type = type
PyrexTypes.py(2972):         self.pos = pos
PyrexTypes.py(2973):         self.needs_type_test = False # TODO:
should these defaults be set in analyse_types()?
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
TypeSlots.py(127):             if self_arg_override is not None and
self.is_self_arg(i):
TypeSlots.py(131):                 arg_type = self.fixed_arg_type(i)
 --- modulename: TypeSlots, funcname: fixed_arg_type
TypeSlots.py(110):         return self.format_map[self.fixed_arg_format[i]]
TypeSlots.py(132):
args.append(PyrexTypes.CFuncTypeArg("", arg_type, None))
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2966):         self.name = name
PyrexTypes.py(2967):         if cname is not None:
PyrexTypes.py(2970):             self.cname = Naming.var_prefix + name
PyrexTypes.py(2971):         self.type = type
PyrexTypes.py(2972):         self.pos = pos
PyrexTypes.py(2973):         self.needs_type_test = False # TODO:
should these defaults be set in analyse_types()?
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
TypeSlots.py(133):         if self_arg_override is not None and
self.returns_self_type():
TypeSlots.py(136):             ret_type = self.return_type()
 --- modulename: TypeSlots, funcname: return_type
TypeSlots.py(113):         return self.format_map[self.ret_format]
TypeSlots.py(137):         exc_value = self.exception_value()
 --- modulename: TypeSlots, funcname: exception_value
TypeSlots.py(121):         return self.error_value_map.get(self.ret_format)
TypeSlots.py(138):         return PyrexTypes.CFuncType(ret_type, args,
exception_value = exc_value)
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2531):         self.return_type = return_type
python: Modules/gcmodule.c:327: visit_decref: Assertion
`gc->gc.gc_refs != 0' failed.
Aborted

Maybe I'll try rebasing on the latest master again.