Konrad Hinsen <hinsen@cnrs-orleans.fr> writes:
[did you mean this to be off-list? If not, please just forward it to the list.]
No, I sent the mail to the list as well, but one out of three mails I send to the list never arrive there at first try... In this case, the copy sent to myself got lost as well, so I don't have any copy left, sorry.
<rant>
I don't know about the others out there, but I have 30000 lines of published Python code plus a lot of unpublished code (scripts), all of which use NumPy arrays almost everywhere. There are also a few places where views are created intentionally, which are then passed around to other code and can end up anywhere. The time required to update all that to new slicing semantics would be enormous, and I don't see how I could justify it to myself or to my employer. I'd also have to stop advertising Python as a time-efficient development tool.
I sympathize with this view. However, I think the solution to this problem should be a compatibility wrapper rather than a design compromise.
There are at least 2 reasons why:
1. Numarray has quite a few incompatibilities to Numeric anyway, so even without this change you'd be forced to rewrite all or most of those scripts
The question is how much effort it is to update code. If it is easy, most people will do it sooner or later. If it is difficult, they won't. And that will lead to a split in the user community, which I think is highly detrimental to the further development of NumPy and Numarray. A compatibility wrapper won't change this. Assume that I have tons of code that I can't update because it's too much effort. Instead I use the compatbility wrapper. When I add a line or a function to that code, it will of course stick to the old conventions. When I add a new module, I will also prefer the old conventions, for consistency. And other people working with the code will pick up the old conventions as well. At the same time, other people will use the new conventions. There will be two parts of the community that cannot easily read each other's code. So unless we can reach a concensus that will guarantee that 90% of existing code will be adapted to the new interfaces, there will be a split.
(or use the wrapper), but none of the incompatibilities I'm currently aware of would, in my eyes, buy one as much as introducing copy-indexing semantics would. So if things get broken anyway, one might as well take
I agree, but it also comes at the highest cost. There is absolute no way to identify automatically the code that needs to be adapted, and there is no run-time error message in case of failure - just a wrong result. None of the other proposed changes is as risky as this one.
this step (especially since intentional views are, on the whole, used rather sparingly -- although tracking down these uses in retrospect might admittedly be unpleasant).
It is not merely unpleasant, the cost is simply prohibitive.
2. Numarray is supposed to be incorporated into the core. Compromising the consistency of core python (and code that depends on it) is in my eyes worse than compromising code written for Numeric.
I don't see view behaviour as inconsistent with Python. Python has one mutable sequence type, the list, with copy behaviour. One type is hardly enough to establish a rule.
As a third reason I could claim that there is some hope of a much more widespread adoption of Numeric/numarray as an alternative to matlab etc. in the next couple of years, so that it might be wise to fix things now, but I'd understand if you'd remain unimpressed by that :)
I'd like to see any supporting evidence. I think this argument is based on the reasoning "I would prefer it to be this way, so many others would certainly also prefer it, so they would start using NumPy if only these changes were made." This is not how decision processes work in real life. On the contrary, people might look at the history of NumPy and decide that it is too unreliable to base a serious project on - if they changed the interface once, they might do it again. This is a particularly important aspect in the OpenSource universe, where there are no contracts that promise anything. If you want people to use your code, you have to demonstrate that it is reliable, and that applies to both the code and the interfaces. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
Konrad Hinsen <hinsen@cnrs-orleans.fr> writes: [Konrad wants to keep alias-slicing behavior for backward-compatibility]
I sympathize with this view. However, I think the solution to this problem should be a compatibility wrapper rather than a design compromise.
There are at least 2 reasons why:
1. Numarray has quite a few incompatibilities to Numeric anyway, so even without this change you'd be forced to rewrite all or most of those scripts
The question is how much effort it is to update code. If it is easy, most people will do it sooner or later. If it is difficult, they won't. And that will lead to a split in the user community, which I think is highly detrimental to the further development of NumPy and Numarray.
I agree that avoiding a split of the Numeric user community is a crucial issue and that efforts have to be taken to make transition painless enough to happen (in most cases; maybe it needs to be even 90% or more as you say).
A compatibility wrapper won't change this. Assume that I have tons of code that I can't update because it's too much effort. Instead I use the compatbility wrapper. When I add a line or a function to that code, it will of course stick to the old conventions. When I add a new module, I will also prefer the old conventions, for consistency. And other people working with the code will pick up the old conventions as well. At the same time, other people will use the new conventions. There will be two parts of the community that cannot easily read each other's code.
I don't think the situation is quite so bleak. Yes, library code should be converted, and although a compatibility wrapper might be helpful in the process, I agree that it isn't a full solution for the reasons you cite above. But there is plenty of code that is mainly used internally and no longer changes (much), for which I think a compatibility wrapper is a fine solution (and might be preferable to conversion, even if it involves little effort). If I had some matlab (or C) code that fulfills similar criteria, I'd also rather wrap it somehow rather than to convert it to python.
So unless we can reach a concensus that will guarantee that 90% of existing code will be adapted to the new interfaces, there will be a split.
(or use the wrapper), but none of the incompatibilities I'm currently aware of would, in my eyes, buy one as much as introducing copy-indexing semantics would. So if things get broken anyway, one might as well take
I agree, but it also comes at the highest cost. There is absolute no way to identify automatically the code that needs to be adapted, and there is no run-time error message in case of failure - just a wrong result. None of the other proposed changes is as risky as this one.
Wouldn't an (almost) automatic solution be to simply replace (almost) all instances of a[b:c] with a.view[b:c] in your legacy code? Even for unusual cases (like if you heavily mix arrays and lists) you could still autoconvert by inserting ``if type(foo) == ArrayType:...``, although this would admittedly be rather messy. The unnecessary ``.view``s can be eliminated over time and even if they aren't, no one would have to learn or switch between two libraries.
this step (especially since intentional views are, on the whole, used rather sparingly -- although tracking down these uses in retrospect might admittedly be unpleasant).
It is not merely unpleasant, the cost is simply prohibitive.
See above. I personally hope that even without resorting to something like the above, converting my code to copy behavior wouldn't be too much of an effort, but my code-base is much smaller than yours and I can't currently recall more than one case of intended aliasing that would require a couple of changes and my estimate might also prove quite wrong. I have no idea which scenario is typical.
2. Numarray is supposed to be incorporated into the core. Compromising the consistency of core python (and code that depends on it) is in my eyes worse than compromising code written for Numeric.
I don't see view behaviour as inconsistent with Python. Python has one mutable sequence type, the list, with copy behaviour. One type is hardly enough to establish a rule.
Well, AFAIK there are actually three mutable sequence types in python core and all have copy-slicing behavior: list, UserList and array: >>> import array >>> aa = array.array('d', [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]) >>> bb = aa[:] >>> bb is aa 0 I would suppose that in the grand scheme of things numarray.array is intended as an eventual replacement for array.array, or not? Furthermore list is such a fundamental data type in python that I think it is actually enough to establish a rule (if the vast majority of 3rd party modules sequence types don't have the same semantics, I'd regard it as a strong argument for your position, but I haven't checked).
As a third reason I could claim that there is some hope of a much more widespread adoption of Numeric/numarray as an alternative to matlab etc. in the next couple of years, so that it might be wise to fix things now, but I'd understand if you'd remain unimpressed by that :)
I'd like to see any supporting evidence. I think this argument is based on the reasoning "I would prefer it to be this way, so many others would certainly also prefer it, so they would start using NumPy if only these changes were made." This is not how decision processes work in real life.
Sure, but I didn't try to imply this causality anyway:) My argument wasn't so much "lets make it really good (where good is what *I* say) then loads of people will adopt it", it was more: "Numeric has a good chance to grow considerably in popularity over the next years, so it will be much easier to fix things now than later" (for slicing behavior, now is likely to be the last chance). The fact that matlab users are used to copy-on-demand and the fact that many people, (including you if I understand you correctly) think that copy-slicing semantics as such (without backward compatibility concerns) are preferable, might have a small influence on people's decision to adopt Numeric, but I perfectly agree that this influence will be minor compared to other issues.
On the contrary, people might look at the history of NumPy and decide that it is too unreliable to base a serious project on - if they changed the interface once, they might do it again. This is a particularly important aspect in the OpenSource universe, where there are no contracts that promise anything. If you want people to use your
I don't think matlab or similar alternatives make legally binding promises about backwards compatibility, or do they? It guess it is actually more difficult to *force* incompatible changes on people with an open source project than with commercial software, but I agree that splitting or lighthearted sacrifices of backwards compatibility are more of a temptation with open source, for one thing because there are usually less financial stakes involved for the authors.
code, you have to demonstrate that it is reliable, and that applies to both the code and the interfaces.
Yes, this is very important and I very much appreciate that you stress these and similar points in your postings. But reliability to me also includes the ability for growth -- I not only want my old code to work in a couple of years, I also want the tool I wrote it in to remain competitive and this can conflict with backwards-compatibility. I like the balance python strikes here so far -- the language has improved significantly (and in my eyes has remained superior to newer competitors such as ruby) but at the same time for me and most other people transitions between versions haven't caused too much trouble. This increases the value of my code-base to me: I can assume that it will still work (or be adapted without too much effort) in years to come and yet be written in an excellent language for the job. Striking this balance is however quite difficult (as can be seen by the heated discussions in c.l.p), so getting it right will most likely involve considerable effort (and controversy) within the Numeric community. alex -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.Schmolck@gmx.net http://www.dcs.ex.ac.uk/people/aschmolc/
participants (2)
-
Alexander Schmolck -
Konrad Hinsen