Can you raise an issue on the Github issue tracker for NumPy? These issues will be looked at more closely. This kind of change should not have made it in to the release. <off-topic> Given the lack of availability of time from enough experts in NumPy, this is the sort of thing that can happen. I was not able to guide development of NumPy appropriately at my old job. That's a big reason I left. I still have more to do than just guide NumPy now, but making sure NumPy is maintained is a big part of what I am doing and why both NumFOCUS and Continuum Analytics exist. I am very hopeful that we can avoid this sort of regression in the future. More tests will help. <off-topic> I think it's important to note that there are many people who will be in the same boat of upgrading to 1.6 over the coming year and there are going to be other little issues like this we will need to address. -Travis On Jun 4, 2012, at 4:12 PM, Dag Sverre Seljebotn wrote:

On 06/04/2012 09:06 PM, Mike Hansen wrote:

...

On Mon, May 28, 2012 at 3:15 AM, Mike Hansen wrote:

...

In trying to upgrade NumPy within Sage, we notices some differences in behavior between 1.5 and 1.6. In particular, in 1.5, we have

sage: f = 0.5 sage: f.__array_interface__ {'typestr': '=f8'} sage: numpy.array(f) array(0.5) sage: numpy.array(float(f)) array(0.5)

In 1.6, we get the following,

sage: f = 0.5 sage: f.__array_interface__ {'typestr': '=f8'} sage: numpy.array(f) array(0.500000000000000, dtype=object)

This seems to be do to the changes in PyArray_FromAny introduced in https://github.com/mwhansen/numpy/commit/2635398db3f26529ce2aaea4028a8118844... . In particular, _array_find_type used to be used to query our __array_interface__ attribute, and it no longer seems to work. Is there a way to get the old behavior with the current code?

No idea. If you want to spend the time to fix this properly, you could implement PEP 3118 and use that instead to export your array data (which can be done from Cython using __getbuffer__ on a Cython class).

Dag _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Mon, Jun 4, 2012 at 10:12 PM, Dag Sverre Seljebotn wrote:

On 06/04/2012 09:06 PM, Mike Hansen wrote:

...

On Mon, May 28, 2012 at 3:15 AM, Mike Hansen  wrote:

...

In trying to upgrade NumPy within Sage, we notices some differences in behavior between 1.5 and 1.6.  In particular, in 1.5, we have

sage: f = 0.5 sage: f.__array_interface__ {'typestr': '=f8'} sage: numpy.array(f) array(0.5) sage: numpy.array(float(f)) array(0.5)

In 1.6, we get the following,

sage: f = 0.5 sage: f.__array_interface__ {'typestr': '=f8'} sage: numpy.array(f) array(0.500000000000000, dtype=object)

This seems to be do to the changes in PyArray_FromAny introduced in https://github.com/mwhansen/numpy/commit/2635398db3f26529ce2aaea4028a8118844... .  In particular, _array_find_type used to be used to query our __array_interface__ attribute, and it no longer seems to work.  Is there a way to get the old behavior with the current code?

No idea. If you want to spend the time to fix this properly, you could implement PEP 3118 and use that instead to export your array data (which can be done from Cython using __getbuffer__ on a Cython class).

I don't think that would work, because looking more closely, I don't think they're actually doing anything like what __array_interface__/PEP3118 are designed for. They just have some custom class ("sage.rings.real_mpfr.RealLiteral", I guess an arbitrary precision floating point of some sort?), and they want instances that are passed to np.array() to be automatically coerced to another type (float64) by default. But there's no buffer sharing or anything like that going on at all. Mike, does that sound right? This automagic coercion seems... in very dubious taste to me. (Why does creating an array object imply that you want to throw away precision? You can already throw away precision explicitly by doing np.array(f, dtype=float).) But if this automatic coercion feature is useful, then wouldn't it be better to have a different interface instead of kluging it into __array_interface__, like we should check for an attribute called __numpy_preferred_dtype__ or something? -n

It isn't just the array() calls which end up getting problems. For example, in 1.5.x

sage: f = 10; type(f) sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64

while in 1.6.x

sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object)

We also see problems with calls like

sage: scipy.stats.uniform(0,15).ppf([0.5,0.7]) array([ 7.5, 10.5])

which work in 1.5.x, but fail with a traceback "TypeError: array cannot be safely cast to required type" in 1.6.x.

I'm getting problems like this after a 1.6 upgrade as well. Lots of object arrays being created when previously there would either be an error, or an array of floats. Also, lots of the "TypeError: array cannot be safely cast to required type" are cropping up. Honestly, most of these are in places where my code was lax and so I just cleaned things up to use the right dtypes etc. But still a bit unexpected in terms of having more code to fix than I was used to for 0.X numpy revisions. Just another data-point, though. Not really a complaint. Zach

There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable.

Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that "big changes in how arrays get constructed and typecast are ahead that may require code fixes". That was my main point, but probably a PEBCAK issue more than anything. Zach

On Tue, Jun 5, 2012 at 7:47 PM, Ralf Gommers wrote:

On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus wrote:

...

...

There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable.

Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that "big changes in how arrays get constructed and typecast are ahead that may require code fixes". That was my main point, but probably a PEBCAK issue more than anything.

It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point.

Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6. -n

During the original discussion, Gael pointed out that the changes would probably break some code (which might need to be cleaned up but still). I think it was underestimated how quickly people would upgrade and see the changes and therefore be able to report problems. We are talking about a 1.7 release, but there are still people who have not upgraded their code to use 1.6 (when some of the big changes occurred). This should probably guide our view of how long it takes to migrate behavior in NumPy and minimize migration difficulties for users. -Travis On Jun 5, 2012, at 2:01 PM, Zachary Pincus wrote:

...

...

On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus wrote:

...

...

There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable.

Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that "big changes in how arrays get constructed and typecast are ahead that may require code fixes". That was my main point, but probably a PEBCAK issue more than anything.

It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point.

Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6.

Yes, unless I'm wrong I experienced these problems from 1.5.something to 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so I just fixed the code and moved on (long, stupid story about why the upgrade before a deadline...). It's just that the issues mentioned above seem to have hit me too and I wanted to mention that. But unhelpfully, I think, without code, and now I've hijacked this thread! Sorry.

Zach _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

I don't think that would work, because looking more closely, I don't think they're actually doing anything like what __array_interface__/PEP3118 are designed for. They just have some custom class ("sage.rings.real_mpfr.RealLiteral", I guess an arbitrary precision floating point of some sort?), and they want instances that are passed to np.array() to be automatically coerced to another type (float64) by default. But there's no buffer sharing or anything like that going on at all. Mike, does that sound right?

This automagic coercion seems... in very dubious taste to me. (Why does creating an array object imply that you want to throw away precision? You can already throw away precision explicitly by doing np.array(f, dtype=float).) But if this automatic coercion feature is useful, then wouldn't it be better to have a different interface instead of kluging it into __array_interface__, like we should check for an attribute called __numpy_preferred_dtype__ or something?

Interesting. It does look like off-label use of the __array_interface__ attribute. Given that "array" used to query the __array_interface__ attribute for type discovery, I still wonder why it was disabled in 1.6? -Travis