[Numpy-discussion] Documentation question.

[Travis Oliphant]

Hey Mark, 

I spent some quality time with your iterator docs tonight and look forward to getting into the code a bit more soon.   I wanted to get your general impressions about what it would take to extend the iterator API to handle iterating over "regions" of the inputs --- i.e. to support generalized ufuncs. 

Also, on my todo list is to compare generalized ufuncs with "threading" in PDL (Perl Data Language) and see if we can support that in NumPy.   Threading is the word that PDL uses to describe "broadcasting"  --- but it does more than ufuncs.   

Here is some information on it. 

http://marketingstartups.com/2009/05/28/the-first-six-steps-of-getting-your-startup-noticed/

-Travis



On Feb 1, 2012, at 8:31 PM, Mark Wiebe wrote:

> On Wed, Feb 1, 2012 at 6:14 PM, Travis Oliphant <travis at continuum.io> wrote:
> 
> On Feb 1, 2012, at 7:04 PM, Mark Wiebe wrote:
> 
>> On Wed, Feb 1, 2012 at 3:29 PM, Charles R Harris <charlesr.harris at gmail.com> wrote:
>> The macro PyArray_RemoveLargest has been replaced by PyArray_RemoveSmallest (which seems strange), but I wonder if this documentation still makes sense.
>> 
>> My impression about this code is that it went through a number of rounds trying to choose an iteration order heuristic that has improved performance over C-order. The change of Largest to Smallest probably reflects one of these heuristic changes. I think it's safe to say that the nditer introduced in 1.6 completely removes the need for this functionality. I did a grep for this function in the master branch, and it is no longer used by NumPy internally.
> 
> There is a common need to iterate over all but one dimension of a NumPy array.   The final dimension is iterated over in an "internal" loop.   This is the essence of how ufuncs work and avoid the possibly expensive overhead of a C-call during each iteration.
> 
> This use-case is handled by the flag NPY_ITER_EXTERNAL_LOOP (http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html#NPY_ITER_EXTERNAL_LOOP) in the nditer.
>  
> Initially, it seemed prudent to remove the dimension that had the largest size (so that the final inner-iteration was the largest number).   Later, timings revealed that that the 'inner' dimension should be the one with the smallest *striding*.   I have not looked at nditer in detail, but would appreciate seeing an explanation of how the nditer approach removes the need for this macro.   When that is clear, then this macro can and should be deprecated. 
> 
> To see the full list of what to use in the nditer versus the older iterators, I created a table:
> 
> http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html#converting-from-previous-numpy-iterators
> 
> Only PyArray_BroadcastToShape and PyArray_MultiIter_NEXTi don't have a nice correspondence, because they refer to implementation details in the previous iterators which are done differently in the nditer.
> 
> -Mark
>  
> 
> -Travis
> 
> 
> 
> 
>> 
>> -Mark
>>  
>> diff --git a/doc/source/user/c-info.beyond-basics.rst b/doc/source/user/c-info.beyond-basics.rs
>> index 9ed2ab3..3437985 100644
>> --- a/doc/source/user/c-info.beyond-basics.rst
>> +++ b/doc/source/user/c-info.beyond-basics.rst
>> @@ -189,7 +189,7 @@ feature follows.
>>          PyArray_MultiIter_NEXT(mobj);
>>      }
>>  
>> -The function :cfunc:`PyArray_RemoveLargest` ( ``multi`` ) can be used to
>> +The function :cfunc:`PyArray_RemoveSmallest` ( ``multi`` ) can be used to
>>  take a multi-iterator object and adjust all the iterators so that
>>  iteration does not take place over the largest dimension (it makes
>>  that dimension of size 1). The code being looped over that makes use
>> 
>> Chuck
>> 
>> 
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> 
>> 
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120201/b6aeb2a8/attachment.html>