Hi Ralf,

I realize you feel strongly that this whole thread is rehashing history, but I think it is worth pointing out that many seem to consider that the criterion for allowing backward incompatible changes, i.e., that "existing code is buggy or is consistently confusing many users", is actually fulfilled here.

Indeed, this appears true to such an extent that even those among the steering council do not agree: while the topic of this thread was about introducing *new* properties (because in the relevant issue I had suggested to Steward it was not possible to change .T), it was Eric who brought up the question whether we shouldn't just change `.T` after all. And in the relevant issue, Sebastian noted that "I am not quite convinced that we cannot change .T (at least in the sense of deprecation) myself", with Chuck chiming in that "I don't recall being in opposition, and I also think the current transpose is not what we want."

That makes three of your fellow steering council members who are not sure, despite all the previous discussions (of which Chuck surely has seen most - sorry, Chuck!).

It seems to me the only sure way in which we can avoid future discussions is to actually address the underlying problem. E.g., is the cost of deprecating & changing .T truly that much more than even having this discussion?

All the best,

Marten


On Wed, Jun 26, 2019 at 4:18 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:


On Wed, Jun 26, 2019 at 10:04 PM Kirill Balunov <kirillbalunov@gmail.com> wrote:
Only concerns #4 from Ilhan's list.

ср, 26 июн. 2019 г. в 00:01, Ralf Gommers <ralf.gommers@gmail.com>:

[....]

Perhaps not full consensus between the many people with different opinions and interests. But for the first one, arr.T change: it's clear that this won't happen.

To begin with, I must admit that I am not familiar with the accepted policy of introducing changes to NumPy. But I find it quite nonconstructive just to say - it will not happen. What then is the point in the discussion?

There has been a *very* long discussion already, and several others on the same topic before. There are also long-standing ways of dealing with backwards compatibility - e.g. what Matthew said is not new, it's an agreed upon way of working. http://www.numpy.org/neps/nep-0023-backwards-compatibility.html lists some principles. That NEP is not yet accepted (it needs rework), but it gives a good idea of what does and does not go.
 
 
Between Juan's examples of valid use, and what Stephan and Matthew said, there's not much more to add. We're not going to change correct code for minor benefits.

I fully agree that any feature can find its use, valid or not is another question. Juan did not present these examples, but I will allow myself to assume that it is more correct to describe what is being done there as a permutation, and not a transpose. In addition, in the very next sentence, Juan adds that "These could be easily changed to .transpose() (honestly they probably should!)"

We're not going to change correct code for minor benefits.

It's fair, I personally have no preferences in both cases, the most important thing for me is that in the 2d case it works correctly. To be honest, until today, I thought that `.T` will raise for` ndim > 2`. At least that's what my experience told me. For example in

    Matlab - Error using  .' Transpose on ND array is not defined. Use PERMUTE instead.

    Julia - transpose not defined for Array(Float64, 3). Consider using permutedims for higher-dimensional arrays.

    Sympy - raise ValueError("array rank not 2")

Here, I agree with the authors that, to begin with, `transpose` is not the best name, since in general it doesn’t fit as an any mathematical definition (of course it will depend on what we take as an element) or a definition from linear algebra. Thus the name `transpose` only leads to confusion.

For a note about another suggestion - `.T` to mean a transpose of the last two dimensions, in Mathematica authors for some reason did the opposite (personally, I could not understand why they made such a choice :) ):

    Transpose[list]
        transposes the first two levels in list.

    I feel strongly that we should have the following policy:

    * Under no circumstances should we make changes that mean that correct
    old code will give different results with new Numpy.

I find this overly strict rules that do not allow to evolve. I completely agree that a silent change in behavior is a disaster, that changing behavior (if it is not an error) in the same minor version (1.X.Y) is not acceptable, but I see no reason to extend this rule for a major version bump (2.A.B.),  especially if it allows something to improve.

I'm sorry, you'll have to live with this rule. We've had lots of discussion about this rule in many concrete cases. When existing code is buggy or is consistently confusing many users, we can discuss. But in general changing old code to do something else is a terrible idea.


I would see such a rough version of a roadmap of change (I foresee my loneliness in this :)) Also considering this comment  

    Personally I would find any divergence between a.T and a.transpose()
    to be rather surprising.

it will be as follows:

1. in 1.18 add the `.permute` method to the array, with the same semantics as `.transpose`. 
2. Starting from 1.18, emit  `FutureWarning`, ` DeprectationWarning` for `.transpose` and advise replacing it with `.permute`.
3. Starting from 1.18 for `.T` with` ndim> 2`, emit a `FutureWarning`, with a note that in future versions the behavior will change.
4. In version 2, remove the `.transpose` and change the behavior for `.T`.

This is simply not enough. Many users will skip versions when upgrading. There must be an exceptionally good reason to change numerical results, and this simply is not one.

Cheers,
Ralf


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion