[Cython] array expressions

[Stefan Behnel]

Dag Sverre Seljebotn, 14.10.2012 11:03:
> On 10/14/2012 10:32 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 14.10.2012 10:18:
>>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>> mark florisson, 13.10.2012 20:30:
>>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>>
>>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>>> requires a bit of adaptation.
>>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>>
>>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>>> interface than the implementation).
>>>>>>
>>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>>> code and
>>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>>
>>>>>>> I don't see a reason for a new pull request.
>>>>>
>>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>>> to split of, merge, push, and pull subdirectories.
>>>>>
>>>>> This means when users pull the master project, they get the
>>>>> sub-projects as well (without themselves needing newer git versions).
>>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>>> the subproject's history).
>>>>>
>>>>> What about using this approach? That way Cython remains stable and
>>>>> pinned on the right minivect version now and in the future, with no
>>>>> burden on users.
>>>>
>>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>>> either, even though a lot of people use Cython together with it.
>>>
>>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>>> itself and working on the AST level.
>>>
>>> Plex would be a better comparison (though bad as well, since Plex is not
>>> optional while minivect is).
>>>
>>>> Keeping the two packages separate helps in keeping the interface between
>>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>>> version of minivect just because it's so easy, and I would like to allow
>>>> users to install a new version of either Cython or minivect at any time.
>>>
>>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>>> unrealistic and pointless.
>>>
>>> I think you should look at minivect as "some AST transform algorithms which
>>> numba and Cython are able to share". It doesn't really have a life on its
>>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>>> won't happen just yet. If it does, we can revisit this.)
>>
>> Ok, so I gave bad examples, fair enough. But I still don't see why we
>> should integrate minivect with Cython more deeply than necessary. Could you
>> explain what the advantage would be over having two separate packages?
> 
> Mark already raised the issue and we discussed in through in another thread

This is still the same thread, at least for me. I replied to it when asking
for progress.


> where you participated, and me, Mark and Robert at least agreed on
> something very similar to what Mark is proposing to do now.
> 
> If you want to reopen the discussion I wish you had rather dug up the old
> thread again and respond to arguments in it, rather than us having to
> repeat the arguments here.

The single argument for git based integration was that it simplifies the
workflow during development. Meaning, it's a development-only thing and
deployments would use separate package installations.

I think that's ok as long as it's actually helpful for development and
doesn't start getting in the way. I can't comment on that yet, but I
wouldn't want to have to deal with it.


> (Though please also consider what it does to discussion climate to reopen
> discussions about trivial details that can also trivially be changed later
> if better arguments crop up.

It may be a trivial detail as long as it only has a reasonable impact on
the development. If it has a user impact, it's no longer immediately
obvious that it's trivial.


> Mark had a weekend of spare time to get this
> merged (yay!), and now who-knows-what happens because you decided to
> bikeshed something that had, at least to the eyes of rest of us, achieved
> consensus on this list.)

Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of
what he's doing now just wasn't clear enough to me.

Mark, could you explain in a bit more detail in which cases the git version
will be used and in which cases users would (or can) install minivect
separately?

Stefan