[Cython] array expressions

[mark florisson]

On 14 October 2012 09:23, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote:
>>
>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>
>>> mark florisson, 13.10.2012 20:30:
>>>>
>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>
>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>
>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>
>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>
>>>>>>>>
>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>> requires a bit of adaptation.
>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>
>>>>>>
>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>> interface than the implementation).
>>>>>
>>>>>
>>>>> Thanks for doing that, it'd be great to get this in (but myself I've
>>>>> got
>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>> code and
>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>
>>>>>> I don't see a reason for a new pull request.
>>>>
>>>>
>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>> to split of, merge, push, and pull subdirectories.
>>>>
>>>> This means when users pull the master project, they get the
>>>> sub-projects as well (without themselves needing newer git versions).
>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>> the subproject's history).
>>>>
>>>> What about using this approach? That way Cython remains stable and
>>>> pinned on the right minivect version now and in the future, with no
>>>> burden on users.
>>>
>>>
>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>> either, even though a lot of people use Cython together with it.
>>
>>
>> This is a very bad comparison. NumPy is not used by Cython at all, but
>> by Cython-generated modules! Whereas minivect is a tool used in the
>> compiler itself and working on the AST level.
>>
>> Plex would be a better comparison (though bad as well, since Plex is not
>> optional while minivect is).
>>
>>> Keeping the two packages separate helps in keeping the interface between
>>> both clean. I wouldn't want to end up with Cython shipping some
>>> patched up
>>> version of minivect just because it's so easy, and I would like to allow
>>> users to install a new version of either Cython or minivect at any time.
>>
>>
>> I think this goal (allowing separate upgrades of Cython and/or minivect)
>> is unrealistic and pointless.
>>
>> I think you should look at minivect as "some AST transform algorithms
>> which numba and Cython are able to share". It doesn't really have a life
>> on its own, it's just a means for Cython and numba to cooperate. (Really
>> long-term then hopefully NumPy, numexpr, Theano etc. would jump on too,
>> but that won't happen just yet. If it does, we can revisit this.)
>
>
> Also, I don't even know when/if numba will use it... Mark, are you able to
> fill us in on that?
>

Yes, numba uses minivect, currently as a subtree (did that yesterday).
Submodules are very bad, especially if you develop the submodule
itself, and switch branches in the superproject (it will just forget
about unpinned changes, which means you just lose your work without so
much as a warning).

Numba mostly uses and extends the minivect type system and related
type conversions (to/from numpy dtypes, to/from ctypes, etc). NumbaPro
uses it for array expressions and ufuncs, which will at some point
make it into numba.

You're entirely right in saying that it breathes only through other
projects. I'll add the rest of the response in answer to Stefan's
question.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel