Python array multiplication for negative values should throw an error
Hopefully this is not a duplicate of an existing thread or in the wrong section, first time posting here. In Python, array multiplication is quite useful to repeat an existing array, e.g. [1,2,3] * 2 becomes [1,2,3,4,5,6]. Multiplication by 0 yields an empty array: [1,2,3] * 0 becomes []. However, operations such as [numpy array] * -1 are very common to get the inverse of an array of numbers. The confusion here stems from the lack of type checking: while the programmer should check whether the array is a NumPy array or a Python array, this is not always done, giving rise to difficult to trace cases where [1,2,3] * -1 yields [] instead of [-1,-2,-3]. I can not think of good reasons why Python array multiplication should not throw an error for negative multipliers, because it is meaningless to have array multiplication by negative value in the way it is intended in Python. Instead I would propose that array multiplication by negative value throws an error. I would like to hear your opinions on this matter.
On Tue, May 31, 2022 at 5:54 AM fjwillemsen--- via Python-ideas < python-ideas@python.org> wrote:
I can not think of good reasons why Python array multiplication should not throw an error for negative multipliers, because it is meaningless to have array multiplication by negative value in the way it is intended in Python.
I can think of several reasons, but the big one is "because there is a lot of production code out there that depends on this behavior." Maybe if Python were adding this feature today, the implementers might take your suggestion to heart and raise on negative length array construction. But, it did not and now there is a lot of code that already exists and would fail if this were changed. Consider it water under the bridge and move on. Eric
Thanks for your response. While legacy production code is always an issue with potentially breaking changes, this line of thought would mean that future versions of Python could never introduce breaking changes. This should not be the case and is not the case now: for new Python versions, developers are expected to test and check for breaking changes, not just update production environments to a new version and hope for the best. The fact that this would throw an error instead of the expected empty list should then quickly lead to the right solution, whereas the other way around, when you expect an error but get an empty list, is hard to debug. That is a tradeoff with a clear winner in my opinion.
Two contradictory opinions from me :-) 1) multiplying a list or a bumpy array be an integer are very different operations — and in most cases, the result will be a different size (and type), which is the case in this example. So it’s rarely a hard to find bug. So that’s not a very compelling argument for this change. 2) this is surprising result that I’d pretty surprised that much code is using on purpose. So I’d think that it would make sense to have it raise an exception. Unless of course, someone does come up with some evidence that there is code in the wild that is counting on it working this way. BTW: to the OP: yes, Python can change, but we do try to keep breaking changes to an absolute minimum, and any breaking change needs to be very compelling— this really isn’t that. In fact, I don’t think there’s been a breaking change since the Py3 transition. I think PEP 563 was going to be the first — and that’s been delayed. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, 1 Jun 2022 at 03:41, Christopher Barker
In fact, I don’t think there’s been a breaking change since the Py3 transition.
I think PEP 563 was going to be the first — and that’s been delayed.
Depends on your definition of "breaking". For instance, this code is legal in Python 3.9, gives a DeprecationWarning in 3.10, and will eventually be illegal:
5in range(10) True
Technically, that's a breaking change. It's currently legal, and eventually won't. But (a) very little code will use this intentionally, and (b) it's trivially easy to fix it in a way that works on older versions (since the more normal way of writing it, with a space before the word "in", is compatible with every version of Python known to mankind, or thereabouts). Similarly, collections.Mapping has been deprecated since 3.3 (use collections.abc.Mapping instead), and that's no longer legal. It, too, gave a warning for a couple of versions. Generally, with every new Python version, it's a good idea to check the What's New for anything that's been removed, but for the most part, code won't break, or if it does, it'll be only in very minor ways. (One thing that Python really tries extremely hard to avoid is behavioural changes, where the same syntax is legal in multiple versions, but does different things. Those are extremely difficult to cope with in cross-version code.) Making something illegal that was previously valid is definitely possible, but it needs strong justification. Given that the "was this a list or a numpy array?" question can never be answered safely when you multiply by a positive integer, I'm dubious of the value of detecting an error when you multiply by a negative integer. It would be a backward compatibility break for fairly low value, in my opinion. ChrisA
Here is another problem in this general area. Multiplying an array by a float already raises a type error. >>> []*0.0 Traceback (most recent call last): TypeError: can't multiply sequence by non-int of type 'float' This problem can arrse 'by accident'. For example, consider
x = {0.0} x.update(range(3)) list(x) [0.0, 1, 2]
Let's now return to the feature (of the builtin list, tuple and str classes): >>> [1] * (-1) == [] True On the one hand I expect there's code that relies on the feature to run correctly. On the other hand I expect there's code that should raise an exception but doesn't, because of this feature. @ericfahlgren It would be most helpful if you could provide or create a reference to support your claim, that there is already a lot of code that would fail if this change was made. @fjwillemsen It would be most helpful if you could provide real-world examples of code where raising an exception would enable better code. Examples relating this are of course welcome from all, not just the two named contributors. -- Jonathan
Thank you, that is a good point! I would expect a similar error message for multiplication with negative integers. A real-world example could be the scenario where I encountered this: arrays were consistently assumed to be NumPy arrays, but in some cases cached versions of these arrays would be loaded from a JSON file without casting them to a NumPy array. Because the function signatures all state NumPy arrays, I applied an array * -1 operation before interpolation and other operations, in the expectation that this would yield the inverse of the elements instead of an empty list. It took me a while to find out that the problem was not in the interpolation or other operations following it, but in the array type. Of course, such bugs do not occur solely due to the lack of a raised errors on negative multiplication: in this case, a combination of a faulty assumption on the programmers' part and Python's lack of strict typing. However, raising an error on negative multiplication would immediately make it clear what is wrong, instead of hiding it.
fjwillemsen--- via Python-ideas writes:
Of course, such bugs do not occur solely due to the lack of a raised errors on negative multiplication: in this case, a combination of a faulty assumption on the programmers' part and Python's lack of strict typing.
I'll take that bait: did mypy miss this?
On Mon, May 30, 2022 at 02:31:35PM -0000, fjwillemsen--- via Python-ideas wrote:
In Python, array multiplication is quite useful to repeat an existing array, e.g. [1,2,3] * 2 becomes [1,2,3,4,5,6].
It certainly does not do that. >>> [1, 2, 3]*2 [1, 2, 3, 1, 2, 3] Also, that's a list, not an array. Python does have arrays, from the `array` module. And of course there are numpy arrays, which behave completely differently, performing scalar multiplication rather than sequence replication: >>> from numpy import array >>> array([1, 2, 3])*2 array([2, 4, 6])
However, operations such as [numpy array] * -1 are very common to get the inverse of an array of numbers.
If that is common, there's a lot of buggy code out there! *wink* Multiplying a numpy array by the scalar -1 performs scalar multiplication, same as any other scalar. To get the inverse of a numpy array, you need to use numpy.linalg.inv: >>> import numpy.linalg >>> arr = array([[1, 2], [3, 4]]) >>> numpy.linalg.inv(arr) array([[-2. , 1. ], [ 1.5, -0.5]])
The confusion here stems from the lack of type checking: while the programmer should check whether the array is a NumPy array or a Python array, this is not always done, giving rise to difficult to trace cases where [1,2,3] * -1 yields [] instead of [-1,-2,-3].
This confusion has nothing to do with multiplication by -1. As the earlier example above shows, scalar multiplication on a numpy array and sequence replication on a list always give different results, not just for -1. (The only exception is multiplication by 1.) I am afraid that this invalidates your argument from Numpy arrays. It simply isn't credible that people are accidentally passing lists instead of numpy arrays, and then getting surprised by the result **only** when multiplying by a negative value. Its not just negatives that are different.
I can not think of good reasons why Python array multiplication should not throw an error for negative multipliers, because it is meaningless to have array multiplication by negative value in the way it is intended in Python.
Its not meaningless, it is far more *useful* than an unnecessary and annoying exception would be. For example, here is how I might pad a list to some minimum length with zeroes: mylist.extend([0]*(minlength - len(mylist))) If this was 1991 and Python was brand new, then the behaviour of sequence replication for negative values would be up for debate. But Python is 31 years old and there is 31 years worth of code that relies on this behaviour, so we would need **extraordinarily strong** reasons to break all that code. Not an extraordinarily weak argument based on confusion between numpy array scalar multiplication and list replication. Sorry to be blunt. -- Steve
participants (7)
-
Chris Angelico -
Christopher Barker -
Eric Fahlgren -
fjwillemsen＠icloud.com -
Jonathan Fine -
Stephen J. Turnbull -
Steven D'Aprano