[Numpy-discussion] Raveling, reshape order keyword unnecessarily confuses index and memory ordering

Tue Apr 2 07:32:15 EDT 2013

On Sat, Mar 30, 2013 at 2:08 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:
> Hi,
>
> We were teaching today, and found ourselves getting very confused
> about ravel and shape in numpy.
>
> Summary
> --------------
>
> There are two separate ideas needed to understand ordering in ravel and
reshape:
>
> Idea 1): ravel / reshape can proceed from the last axis to the first,
> or the first to the last.  This is "ravel index ordering"
> Idea 2) The physical layout of the array (on disk or in memory) can be
> "C" or "F" contiguous or neither.
> This is "memory ordering"
>
> The index ordering is usually (but see below) orthogonal to the memory
ordering.
>
> The 'ravel' and 'reshape' commands use "C" and "F" in the sense of
> index ordering, and this mixes the two ideas and is confusing.
>
> What the current situation looks like
> ----------------------------------------------------
>
> Specifically, we've been rolling this around 4 experienced numpy users
> and we all predicted at least one of the results below wrongly.
>
> This was what we knew, or should have known:
>
> In [2]: import numpy as np
>
> In [3]: arr = np.arange(10).reshape((2, 5))
>
> In [5]: arr.ravel()
> Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> So, the 'ravel' operation unravels over the last axis (1) first,
> followed by axis 0.
>
> So far so good (even if the opposite to MATLAB, Octave).
>
> Then we found the 'order' flag to ravel:
>
> In [10]: arr.flags
> Out[10]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [11]: arr.ravel('C')
> Out[11]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> But we soon got confused.  How about this?
>
> In [12]: arr_F = np.array(arr, order='F')
>
> In [13]: arr_F.flags
> Out[13]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [16]: arr_F
> Out[16]:
> array([[0, 1, 2, 3, 4],
>        [5, 6, 7, 8, 9]])
>
> In [17]: arr_F.ravel('C')
> Out[17]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> Right - so the flag 'C' to ravel, has got nothing to do with *memory*
> ordering, but is to do with *index* ordering.
>
> And in fact, we can ask for memory ordering specifically:
>
> In [22]: arr.ravel('K')
> Out[22]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> In [23]: arr_F.ravel('K')
> Out[23]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9])
>
> In [24]: arr.ravel('A')
> Out[24]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>
> In [25]: arr_F.ravel('A')
> Out[25]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9])
>
> There are some confusions to get into with the 'order' flag to reshape
> as well, of the same type.
>
> Ravel and reshape use the tems 'C' and 'F" in the sense of index ordering.
>
> This is very confusing.  We think the index ordering and memory
> ordering ideas need to be separated, and specifically, we should avoid
> using "C" and "F" to refer to index ordering.
>
> Proposal
> -------------
>
> * Deprecate the use of "C" and "F" meaning backwards and forwards
> index ordering for ravel, reshape
> * Prefer "Z" and "N", being graphical representations of unraveling in
> 2 dimensions, axis1 first and axis0 first respectively (excellent
> naming idea by Paul Ivanov)
>
> What do y'all think?

Surely it should be "Z" and "ᴎ"? ;-)

I knew what your examples would produce, but only because I've bumped into
this before. When you do reshapes of various sorts (ravel() ==
reshape((-1,))), then, like you say, there are two totally different sets
of coordinate mapping in play:

chunk of memory  <-1->  virtual array layout  <-2->  new array layout
  (C pointers)   <--->    (Python indexes)    <--->  (Python indexes)

Mapping (1) is determined by the array strides, and you have to think about
it when you interface with C code, but at the Python level it's pretty much
irrelevant; all operations are defined at the "virtual array layout" level.

Further confusing the issue is the fact that the vast majority of legal
memory<->virtual array mappings are *neither* C- nor F-ordered. Strides are
very flexible.

Further further confusing the issue is that mapping (2) actually consists
of two mappings: if you have an array with shape (3, 4, 5) and reshape it
to (4, 15), then the way you work out the overall mapping is by first
mapping the (3, 4, 5) onto a flat 1-d space with 60 elements, and then
mapping *that* to the (4, 15) space.

Anyway, I agree that this is very confusing; certainly it confused me. If
you bump into these two mappings just in passing, and separately, then it's
very easy to miss the fact that they have nothing to do with each other.
And I agree that using exactly the same terminology for both of them is
part of what causes this. I even kind of like the "Z"/"N" naming scheme (I
still have to look up what C/F actually mean every time, I'm ashamed to
say).

But I don't see how the proposed solution helps, because the problem isn't
that mapping (1) and (2) use different ordering schemes -- the
column-major/row-major distinction really does apply to both equally. Using
different names for those seems like it will confuse the issue further, if
anything. The problem IMHO is that sometimes "order=" is used to specify
mapping (1), and sometimes it's used to specify mapping (2), when in fact
these are totally orthogonal.

To see this, note that semantically it would be perfectly possible for
.reshape() to take *two* order= arguments: one to specify the coordinate
space mapping (2), and the other to specify the desired memory layout used
by the result array (1). Of course we shouldn't actually do this, because
in the unlikely event that someone actually wanted both of these they could
just call asarray() on the output of reshape().

Maybe we should go through and rename "order" to something more descriptive
in each case, so we'd have
  a.reshape(..., index_order="C")
  a.copy(memory_order="F")
etc.?

This way if you just bumped into these while reading code, it would still
be immediately obvious that they were dealing with totally different
concepts. Compare to reading along without the docs and seeing
  a.reshape(..., order="Z")
  a.copy(order="C")
That'd just leave me even more baffled than the current system -- I'd start
thinking that "Z" and "C" somehow were different options for the same
order= option, so they must somehow mean ways of ordering elements?

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130402/df80969b/attachment.html>