Deprecate Promotion of numbers to strings?
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
Hi all, in https://github.com/numpy/numpy/pull/15925 I propose to deprecate promotion of strings and numbers. I have to double check whether this has a large effect on pandas, but it currently seems to me that it will be reasonable. This means that `np.promote_types("S", "int8")`, etc. will lead to an error instead of returning `"S4"`. For the user, I believe the two main visible changes are that: np.array(["string", 0]) will stop creating a string array and return either an `object` array or give an error (object array would be the default currently). Another larger visible change will be code such as: np.concatenate([np.array(["string"]), np.array([2])]) will result in an error instead of returning a string array. (Users will have to cast manually here.) The alternative is to return an object array also for the concatenate example. I somewhat dislike that because `object` is not homogeneously typed and we thus lose type information. This also affects functions that wish to cast inputs to a common type (ufuncs also do this sometimes). A further example of this and discussion is at the end of the mail [1]. So the first question is whether we can form an agreement that an error is the better choice for `concatenate` and `np.promote_types()`. I.e. there is no one dtype that can faithfully represent both strings and integers. (This is currently the case e.g. for datetime64 and float64.) The second question is what to do for: np.array(["string", 0]) which currently always returns strings. Arguably, it must also either return an `object` array, or raise an error (requiring the user to pick string or object using `dtype=object`). The default would be to create a FutureWarning that an `object` array will be returned for `np.asarray(["string", 0])` in the future. But if we know already that we prefer an error, it would be better to give a DeprecationWarning right away. (It just does not seem nice to change the same thing twice even if the workaround is identical.) Cheers, Sebastian [1] A second more in-depth point is that code such as: common_dtype = np.result_type(arr1, arr2) # or promote_types arr1 = arr1.astype(common_dtype, copy=False) arr2 = arr2.astype(common_dtype, copy=False) will currently use `string` in this case while it would error in the future. This already fails with other type combinations such as `datetime64` and `float64` at the moment. The main alternative to this proposal is to return `object` for the common dtype, since an object array is not homogeneously typed, it arguably can represent both inputs. I do not quite like this choice personally because in the above example, it may be that the next line is something like: return arr1 * arr2 in which case, the preferred return may be `str` and not `object`. We currently never promote to `object` unless one of the arrays is already an `object` array, and that seems like the right choice to me.
![](https://secure.gravatar.com/avatar/209654202cde8ec709dee0a4d23c717d.jpg?s=120&d=mm&r=g)
I wonder if we can lessen the blow by allowing `np.concatenate([np.array(["string"]), np.array([2])], casting='unsafe', dtype=str)` or similar in its place. It seems a little unfortunate that with this change, we lose the ability to concatenate numbers to strings without making intermediate copies. Eric On Thu, 30 Apr 2020 at 18:32, Sebastian Berg <sebastian@sipsolutions.net> wrote:
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Thu, 2020-04-30 at 18:47 +0100, Eric Wieser wrote:
I agree we can do that for concatenate and am happy to add just add it. Adding the dtype argument (maybe for now only force-casting is fine?) to `np.concatenate` seems like a reasonable extension of concatenate even without the loss of this potential use-case. - Sebastian
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Apr 30, 2020 at 10:32 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Sebastian -- thanks for driving this forward! Pandas and Xarray already override these casting rules, so I think this is generally a good idea: https://github.com/pydata/xarray/blob/3820fb77256682d909c1e41d962e29bec0edd6... Note that Xarray also overrides np.promote_types(np.bytes_, np.unicode_) to object. This means that `np.promote_types("S", "int8")`, etc. will lead to an
In the long term, I guess this would error as part of the plan to require explicitly writing dtype=object to get object arrays?
I agree, it is better to raise an error than inadvertently cast to object dtype. This can make errors appear later in strange ways. We would need to make this change slowly over several releases, e.g., by issuing a warning first.
![](https://secure.gravatar.com/avatar/209654202cde8ec709dee0a4d23c717d.jpg?s=120&d=mm&r=g)
I wonder if we can lessen the blow by allowing `np.concatenate([np.array(["string"]), np.array([2])], casting='unsafe', dtype=str)` or similar in its place. It seems a little unfortunate that with this change, we lose the ability to concatenate numbers to strings without making intermediate copies. Eric On Thu, 30 Apr 2020 at 18:32, Sebastian Berg <sebastian@sipsolutions.net> wrote:
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Thu, 2020-04-30 at 18:47 +0100, Eric Wieser wrote:
I agree we can do that for concatenate and am happy to add just add it. Adding the dtype argument (maybe for now only force-casting is fine?) to `np.concatenate` seems like a reasonable extension of concatenate even without the loss of this potential use-case. - Sebastian
![](https://secure.gravatar.com/avatar/93a76a800ef6c5919baa8ba91120ee98.jpg?s=120&d=mm&r=g)
On Thu, Apr 30, 2020 at 10:32 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Sebastian -- thanks for driving this forward! Pandas and Xarray already override these casting rules, so I think this is generally a good idea: https://github.com/pydata/xarray/blob/3820fb77256682d909c1e41d962e29bec0edd6... Note that Xarray also overrides np.promote_types(np.bytes_, np.unicode_) to object. This means that `np.promote_types("S", "int8")`, etc. will lead to an
In the long term, I guess this would error as part of the plan to require explicitly writing dtype=object to get object arrays?
I agree, it is better to raise an error than inadvertently cast to object dtype. This can make errors appear later in strange ways. We would need to make this change slowly over several releases, e.g., by issuing a warning first.
participants (3)
-
Eric Wieser
-
Sebastian Berg
-
Stephan Hoyer