[Numpy-discussion] Raveling, reshape order keyword unnecessarily confuses index and memory ordering

Tue Apr 2 13:59:01 EDT 2013

Hi,

On Tue, Apr 2, 2013 at 7:32 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Sat, Mar 30, 2013 at 2:08 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>> Hi,
>>
>> We were teaching today, and found ourselves getting very confused
>> about ravel and shape in numpy.
>>
>> Summary
>> --------------
>>
>> There are two separate ideas needed to understand ordering in ravel and
>> reshape:
>>
>> Idea 1): ravel / reshape can proceed from the last axis to the first,
>> or the first to the last.  This is "ravel index ordering"
>> Idea 2) The physical layout of the array (on disk or in memory) can be
>> "C" or "F" contiguous or neither.
>> This is "memory ordering"
>>
>> The index ordering is usually (but see below) orthogonal to the memory
>> ordering.
>>
>> The 'ravel' and 'reshape' commands use "C" and "F" in the sense of
>> index ordering, and this mixes the two ideas and is confusing.
>>
>> What the current situation looks like
>> ----------------------------------------------------
>>
>> Specifically, we've been rolling this around 4 experienced numpy users
>> and we all predicted at least one of the results below wrongly.
>>
>> This was what we knew, or should have known:
>>
>> In [2]: import numpy as np
>>
>> In [3]: arr = np.arange(10).reshape((2, 5))
>>
>> In [5]: arr.ravel()
>> Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>
>> So, the 'ravel' operation unravels over the last axis (1) first,
>> followed by axis 0.
>>
>> So far so good (even if the opposite to MATLAB, Octave).
>>
>> Then we found the 'order' flag to ravel:
>>
>> In [10]: arr.flags
>> Out[10]:
>>   C_CONTIGUOUS : True
>>   F_CONTIGUOUS : False
>>   OWNDATA : False
>>   WRITEABLE : True
>>   ALIGNED : True
>>   UPDATEIFCOPY : False
>>
>> In [11]: arr.ravel('C')
>> Out[11]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>
>> But we soon got confused.  How about this?
>>
>> In [12]: arr_F = np.array(arr, order='F')
>>
>> In [13]: arr_F.flags
>> Out[13]:
>>   C_CONTIGUOUS : False
>>   F_CONTIGUOUS : True
>>   OWNDATA : True
>>   WRITEABLE : True
>>   ALIGNED : True
>>   UPDATEIFCOPY : False
>>
>> In [16]: arr_F
>> Out[16]:
>> array([[0, 1, 2, 3, 4],
>>        [5, 6, 7, 8, 9]])
>>
>> In [17]: arr_F.ravel('C')
>> Out[17]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>
>> Right - so the flag 'C' to ravel, has got nothing to do with *memory*
>> ordering, but is to do with *index* ordering.
>>
>> And in fact, we can ask for memory ordering specifically:
>>
>> In [22]: arr.ravel('K')
>> Out[22]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>
>> In [23]: arr_F.ravel('K')
>> Out[23]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9])
>>
>> In [24]: arr.ravel('A')
>> Out[24]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>
>> In [25]: arr_F.ravel('A')
>> Out[25]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9])
>>
>> There are some confusions to get into with the 'order' flag to reshape
>> as well, of the same type.
>>
>> Ravel and reshape use the tems 'C' and 'F" in the sense of index ordering.
>>
>> This is very confusing.  We think the index ordering and memory
>> ordering ideas need to be separated, and specifically, we should avoid
>> using "C" and "F" to refer to index ordering.
>>
>> Proposal
>> -------------
>>
>> * Deprecate the use of "C" and "F" meaning backwards and forwards
>> index ordering for ravel, reshape
>> * Prefer "Z" and "N", being graphical representations of unraveling in
>> 2 dimensions, axis1 first and axis0 first respectively (excellent
>> naming idea by Paul Ivanov)
>>
>> What do y'all think?
>
> Surely it should be "Z" and "ᴎ"? ;-)
>
> I knew what your examples would produce, but only because I've bumped into
> this before. When you do reshapes of various sorts (ravel() ==
> reshape((-1,))), then, like you say, there are two totally different sets of
> coordinate mapping in play:
>
> chunk of memory  <-1->  virtual array layout  <-2->  new array layout
>   (C pointers)   <--->    (Python indexes)    <--->  (Python indexes)
>
> Mapping (1) is determined by the array strides, and you have to think about
> it when you interface with C code, but at the Python level it's pretty much
> irrelevant; all operations are defined at the "virtual array layout" level.
>
> Further confusing the issue is the fact that the vast majority of legal
> memory<->virtual array mappings are *neither* C- nor F-ordered. Strides are
> very flexible.
>
> Further further confusing the issue is that mapping (2) actually consists of
> two mappings: if you have an array with shape (3, 4, 5) and reshape it to
> (4, 15), then the way you work out the overall mapping is by first mapping
> the (3, 4, 5) onto a flat 1-d space with 60 elements, and then mapping
> *that* to the (4, 15) space.
>
> Anyway, I agree that this is very confusing; certainly it confused me. If
> you bump into these two mappings just in passing, and separately, then it's
> very easy to miss the fact that they have nothing to do with each other. And
> I agree that using exactly the same terminology for both of them is part of
> what causes this. I even kind of like the "Z"/"N" naming scheme (I still
> have to look up what C/F actually mean every time, I'm ashamed to say).
>
> But I don't see how the proposed solution helps, because the problem isn't
> that mapping (1) and (2) use different ordering schemes -- the
> column-major/row-major distinction really does apply to both equally. Using
> different names for those seems like it will confuse the issue further, if
> anything. The problem IMHO is that sometimes "order=" is used to specify
> mapping (1), and sometimes it's used to specify mapping (2), when in fact
> these are totally orthogonal.

Yes.  Of course ravel is the perfect storm because it refers to order
in both senses.

> To see this, note that semantically it would be perfectly possible for
> .reshape() to take *two* order= arguments: one to specify the coordinate
> space mapping (2), and the other to specify the desired memory layout used
> by the result array (1). Of course we shouldn't actually do this, because in
> the unlikely event that someone actually wanted both of these they could
> just call asarray() on the output of reshape().

Yes.

> Maybe we should go through and rename "order" to something more descriptive
> in each case, so we'd have
>   a.reshape(..., index_order="C")
>   a.copy(memory_order="F")
> etc.?

That seems like a good idea.  If you are proposing it, I am "+1".

> This way if you just bumped into these while reading code, it would still be
> immediately obvious that they were dealing with totally different concepts.
> Compare to reading along without the docs and seeing
>   a.reshape(..., order="Z")
>   a.copy(order="C")
> That'd just leave me even more baffled than the current system -- I'd start
> thinking that "Z" and "C" somehow were different options for the same order=
> option, so they must somehow mean ways of ordering elements?

I don't think you'd be more baffled than the current system, which, as
you say, conflates two orthogonal concepts.  Rather, I think it would
cause the user to stop, as they should, and consider what concept
order is using in this case.

I don't find it difficult to explain this:

There are two different but related concepts of 'order'

1) The memory layout of the array
2) The index ordering used to unravel the array

If you see 'Z' or 'N" for 'order' - that refers to index ordering.
If you see 'C' or 'F" for order - that refers to memory layout.

Cheers,

Matthew