[Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy

[Sebastian Berg]

On Mo, 2016-09-12 at 20:22 +0000, Pauli Virtanen wrote:
> Mon, 12 Sep 2016 11:31:07 +0200, Sebastian Berg kirjoitti:
> > 
> > > 
> > > * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME
> > >   flags for NpyIter_New.
> > > 
> > > * New API function PyArray_MapIterArrayCopyIfOverlap,
> > >   as ufunc.at needs to check overlaps for index arrays before
> > >   constructing iterators, and the parsing is done in multiarray.
> > I think here Nathaniels point might be right. It could be we can
> > assume
> > that copying is always fine, there is probably only one or two
> > downstream projects using this function, plus it seems harder to
> > create
> > abusing structures that actually do something useful.
> > It was only exposed for usage in `ufunc.at` if I remember right. I
> > know
> > theano uses it though, but not sure about anyone else, maybe numba.
> > On
> > the other hand.... It is not the worst API clutter in history.
> Do you suggest that I break the PyArray_MapIterArray API?
> 
> One issue here is that the function doesn't make distinction between
> read-
> only access and read-write access, so copying may give unnecessary 
> slowdown. The second thing is that it will result to a bit uglier
> code, as 
> I need to manage the overlap with the second operation in ufunc_at.
> 

Yeah, was only wondering about MapIterArray, because I might get away
with the API break in the case that it works everywhere for our
internal usage. But if its not quite straight forward, there is no
point in thinking about it.

> For NpyIter, I'd still be wary about copying by default, because it's
> not 
> needed everywhere (the may_share_memory checks are better done
> earlier), 
> and since the semantic change can break things inside Numpy.
> 

Yes, I tend to agree here about it. You can always wonder whether its
still the most convenient place to do the checks (at least for a few
places), but from glancing at the code, it still seems elegant to me.
If we are concerned about making the iterator more and more complex,
maybe we can really do something else about it as well.

I am not sure whether I will manage to look at it very closely soon, so
would invite anyone to take a look; this is definitely a highlight!

- Sebastian


> 	Pauli
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160912/a8d30f3c/attachment.sig>