[PYTHON MATRIX-SIG] Yorick, Python and Array.py

[Chris Chase S1A]

>>>>> "D" == Dave Munro

D> Here is Dave's suggestion:

[deleted material about pseudo indexes and comformability in Yorick]

Two weeks ago someone pointed me toward the Yorick language for its
array capabilities.  After studying it, I found that Yorick has the
most flexible array subscripting syntax of any language that I have
seen.

The advantages of Yorick can be added to Python but it requires some
major work on some internals.  In a reply to one of my previous posts,
Guido already said he doesn't mind someone submitting internal changes
that keep the language backward compatible.  I have been working on
these modifications and hope to finish them soon (I don't have much
time available for this SIG and tinkering around with Python).  I will
post a summary of the changes in a separate message that I attempting
to get some immediate feedback before I finish my mods.

D> In order for this to work, you need to broaden the definition of when
D> two arrays are conformable; instead of requiring an exact match for
D> the shape of the largest common cells of the two operands, you need to
D> treat 1-length dimensions as "wildcards" matching any length.  

I already have tried this by making a small modification to Array.py,
the prototype from Konrad Hinsen <hinsenk@ere.umontreal.ca>.  If
anyone is interested I can send you the modified version.

Array.py already has a concept of wildcard for extending a shorter
frame for one argument to match a longer frame of another argument.
It does this by combining the lower-rank array will then be combined
with each matching ranked cell in the higher ranked array.  Yorick
calls this broadcasting because the lower ranked cell is "broadcasted"
or replicated along the missing higher dimensions.

Yorick extends this one step further.  Yorick will broadcast along any
size 1 dimension to match the common dimension of the other arguments.
When combining a lower rank array with a higher rank we can think of
the missing higher dimensions as being size 1.  (i.e. equivalent to
appending 1's to the shape of the shorter frame).  Then the
broadcasting method will have the same behavior already exhibited by
Array.py.

However, it has the additional benefit of broadcasting other unit
dimensions (a simple viewpoint unit dimensions have similar behave
like a scalar.)  

As an example, using the modified Array.py:
b = Array([ [2,3,7], [9,8,2] ])
d = Array([1,2,3])
e = Array([[1],[2]])

# This could already be done.  It adds to each row.
>>> b+d
 3  5 10 
10 10  5 

# This is new. It adds a column vector containing 1,2 to each column.
>>> b+e
 3  4  8 
11 10  4 

It makes outer products extremely simple:

>>> d+e
2 3 4 
3 4 5 
>>> d*e
1 2 3 
2 4 6 

Pseudo indexes make this type of thing even easier.

In the above if e=Array([1,2]) then I could have added e along the
columns using: b+e(-,)

This is more useful in even higher dimensions.  It saves pairs of
transposition operators and/or reshaping which can be confusing.

D> However, I imagine that the implementor will be loathe to change his
D> notion of array conformability at this late date.  Nevertheless, you
D> might as well ask to see what he says; probably the change wouldn't
D> actually break any existing correct code.

There is no single "implementor" here.  My understanding is that the
Matrix module concept is still in flux - hence the need for this SIG.
There have been several implementations and design concepts suggested
on this list.  I have not seen any proposal to accept as a standard a
particular Array module yet.

D> In Yorick, the combination of the relaxed conformability rule,
D> pseudo-indices and a general transpose operator that can produce any
D> permutation of dimensions allows for nearly all meaningful
D> combinations of arrays.

I do like Yorick's completely general transpose() operator which uses a
list of permutation cycles combined with circular shifts and negative
dimension numbers (taken relative to the rank) to conveniently specify
any arbitrary transposition on dimensions.

D> Yorick has one other indexing syntax which has proven useful, which I
D> call "rubber indices".  

This is another very useful feature.  The idea is that you can have
left or right justified index lists relative to array
rank. Unspecified dimensions indexes default to the entire dimension.
Optionally, the unspecified dimensions can be collapsed into a single
dimension that is the product of the collapsed dimensions.  Yorick
provides a very nice syntax for a rubber index ".." and "*" to
collapse the missing indexes.  This becomes very useful when the
dimension is unknown to the user and provides much simpler and clearer
code than a complex sequence of take() and reshape() operators.

b(..,2,3) or b(*,2,3) index along the last two dimensions.  b(1,..,5)
indexes along the first and last dimension.  Indexes before the rubber
index are left-justified while those following the rubber index are
right-justified with respect to the shape of the indexed array.

One other subscripting feature that Yorick (Matlab has this also)
has is the ability to specify a stride with slice notation.
E.g. 1:10:2 specifies a range the has increments of 2.  The stride can
also be negative.  This is like the range() function.

I list the additions that I am experimenting with in another post.

Chris

------- end -------

=================
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
=================