reshape and ravel methods of arrays now return views or raise error
It look's like Tim's arguments have won out for the most part. The .reshape and .ravel methods of arrays now return views only or raise an error if that is not possible. However, both of these methods still have an order argument because I don't think it is prudent to have a.transpose().ravel() automatically column-scan --- you have to tell ravel you want a Fortran-order result. The reshape and ravel functions behave exactly as they used to (view if possible, copy otherwise). I don't think these functions should be deprecated. I don't think any of the functional interfaces should be deprecated, either. Part of the reason for the change is that it was not disruptive. None of the NumPy code broke (except for a small piece of arrayprint) after the change. None of the SciPy code broke either (as far as I can tell). Hopefully we can put this to rest and move on to more important issues... -Travis
Travis Oliphant wrote:
It look's like Tim's arguments have won out for the most part.
Hooray. Now let's hope I'm right!
The .reshape and .ravel methods of arrays now return views only or raise an error if that is not possible. However, both of these methods still have an order argument because I don't think it is prudent to have a.transpose().ravel() automatically column-scan --- you have to tell ravel you want a Fortran-order result.
OK, I think I understand why you want the order flag now. But isn't flat in a weird state now? All of the other reshape methods take an order parameter, but flat is inherently C-ordered, is it not? Also, is it useful to view an array as fortran ordered? I suppose if you are working with multidimensional Fortran data you want to keep it in the correct order. Still, something seems not quite right about the current state of affairs. Let me ask a question about the order flag: does it mean anything for a noncontiguous array? As I interpret it, right now: CONTIGUOUS = True, FORTRAN = True => Strides are Fortran order CONTIGUOUS = True, FORTRAN = False=> Strides are C order CONTIGUOUS = False, FORTRAN = Any=> Matrix is Discontiguous Is that correct? Or is it possible to have strides that neither C-order not Fortran-Order in a contiguous matrix? I'm trying to find a way to shoehorn an order flag in here without breaking anything, but I don't think it'll fit. So you'd need a separate flag to indicate the reshape-order if one wanted to do that. Mostly this would make life simpler for everyone, in normal use, the reshape-order flag gets set at array creation and everything just works. flat works properly, the order flag is only needed in construction. However, since the behaviour of the array is different with respect to reshape, you'd probably want to tag the repr somehow ('farray' instead of 'array' perhaps). In practice, this would be essentially the same as having a sister class to ndarray that worked on Fortran ordered data. Everything would behave the same except for how reshape works. I'm not sure if any of that is a good idea, but I'm not entirely convinced that the order flag, although a big improvement over what it replaced, is the best approach, so I'm musing.
The reshape and ravel functions behave exactly as they used to (view if possible, copy otherwise). I don't think these functions should be deprecated. I don't think any of the functional interfaces should be deprecated, either.
The good thing about functions is that I can always hot patch numpy with safer versions if I like. So I'll call this good. When I get around to it, I'll probably try patching in versions of reshape and transpose that lock either views or copies and see how that works out. I'll report back on how it goes.
Part of the reason for the change is that it was not disruptive. None of the NumPy code broke (except for a small piece of arrayprint) after the change. None of the SciPy code broke either (as far as I can tell). Hopefully we can put this to rest and move on to more important issues...
Hooray! -tim
On 3/28/06, Tim Hochberg <tim.hochberg@cox.net> wrote:
Travis Oliphant wrote:
It look's like Tim's arguments have won out for the most part.
Hooray. Now let's hope I'm right!
+1. Now I'm nervous.
<snip>
Part of the reason for the change is that it was not disruptive. None of the NumPy code broke (except for a small piece of arrayprint) after the change. None of the SciPy code broke either (as far as I can tell). Hopefully we can put this to rest and move on to more important issues...
But what could be more important to safe programming than having a sure grasp of when you are working with a view or a copy? O_- Chuck
Uh oh, Just updated to new numpy so that dot() wouldn't be broken. What fun to play with the new order/ravel/reshape stuff. Unfortunately, the ravel/reshape methods seem to not be quite worked out properly yet. Basically (and as expected) the ravel method doesn't work anymore on fortran-strided arrays. But since fortran- strided arrays happen *all-the-time* (e.g. after a transpose operation), things get confusing fast. In fact, absolutely no reshape operation (other than the identity reshape) will work on fortran-strided arrays (see below). Also, the errors could be more helpful, and anyway ravel shouldn't fail with an error about reshape fortran-strided. This can't be the optimal situation, since it makes the method interface next-to-useless (because exceptions might be thrown at any moment) unless a *lot* of care is taken to properly sanitize the arrays that are used (in which case any speed gains are probably lost). In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.7.2291' In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) In [4]: b = a.transpose() In [5]: a.ravel() Out[5]: array([1, 2, 3, 4, 5, 6]) In [6]: b.ravel() ValueError: cannot return a view from reshape. In [7]: b.shape Out[7]: (2, 3) In [8]: b.reshape((1,6)) ValueError: cannot return a view from reshape. In [9]: b.reshape((6,1)) ValueError: cannot return a view from reshape. In [10]: b.reshape((3,2)) ValueError: cannot return a view from reshape. This state of affairs seems pretty hostile to both advanced users and basic ones. Now I need to go through all of my code and eliminate the use of array methods, since about half of the time I'll be calling things like ravel on arrays that have been transposed. What a waste -- is there something better I can do or something better numpy can do? At the very least, a strong convention really needs to be established and made clear about what the methods do (never return copies) and what the functions do (sometimes return copies? hardly a strong convention), and when to choose one or the other. Zach
Let me give a more concrete example, pulled from some code I was actually using. The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 ) It appears that to be safe, a function like this must be rewritten as: def distance_squared(a, b): return numpy.sum( (numpy.ravel(a) - numpy.ravel(b))**2 ) At the very least, it seems surprising (to a novice especially, but to me too, and I don't consider myself a total newb at this) that for something as simple as this, that the ravel method might not work in some cases, especially since one of the exciting new features of numpy is (finally!) having useful array methods (makes life in ipython with tab completion much easier). Zach PS. I know that the "proper" solution would look more like the below. I guess the question is how tolerant should numpy be of code that deviates from "proper". def distance_squared(a, b): return ((a - b)**2).sum() On Mar 28, 2006, at 5:35 PM, Zachary Pincus wrote:
Uh oh,
Just updated to new numpy so that dot() wouldn't be broken. What fun to play with the new order/ravel/reshape stuff.
Unfortunately, the ravel/reshape methods seem to not be quite worked out properly yet. Basically (and as expected) the ravel method doesn't work anymore on fortran-strided arrays. But since fortran-strided arrays happen *all-the-time* (e.g. after a transpose operation), things get confusing fast.
In fact, absolutely no reshape operation (other than the identity reshape) will work on fortran-strided arrays (see below). Also, the errors could be more helpful, and anyway ravel shouldn't fail with an error about reshape fortran-strided.
This can't be the optimal situation, since it makes the method interface next-to-useless (because exceptions might be thrown at any moment) unless a *lot* of care is taken to properly sanitize the arrays that are used (in which case any speed gains are probably lost).
In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.7.2291' In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) In [4]: b = a.transpose() In [5]: a.ravel() Out[5]: array([1, 2, 3, 4, 5, 6]) In [6]: b.ravel() ValueError: cannot return a view from reshape. In [7]: b.shape Out[7]: (2, 3) In [8]: b.reshape((1,6)) ValueError: cannot return a view from reshape. In [9]: b.reshape((6,1)) ValueError: cannot return a view from reshape. In [10]: b.reshape((3,2)) ValueError: cannot return a view from reshape.
This state of affairs seems pretty hostile to both advanced users and basic ones. Now I need to go through all of my code and eliminate the use of array methods, since about half of the time I'll be calling things like ravel on arrays that have been transposed. What a waste -- is there something better I can do or something better numpy can do?
At the very least, a strong convention really needs to be established and made clear about what the methods do (never return copies) and what the functions do (sometimes return copies? hardly a strong convention), and when to choose one or the other.
Zach
------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel? cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On 3/28/06, Zachary Pincus <zpincus@stanford.edu> wrote:
Let me give a more concrete example, pulled from some code I was actually using.
The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 )
def distance_squared(a,b) return ((a - b)**2).sum() The function should probably throw an error if a and b don't have the same shape and order and now it does. I note that your original function was written for the old functional sum and attempts to be efficient. The new Numpy sum method make it much easier to do what you want. Thanks, Travis. Chuck
Charles R Harris wrote:
On 3/28/06, *Zachary Pincus* <zpincus@stanford.edu <mailto:zpincus@stanford.edu>> wrote:
Let me give a more concrete example, pulled from some code I was actually using.
The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 )
def distance_squared(a,b) return ((a - b)**2).sum()
The function should probably throw an error if a and b don't have the same shape and order and now it does.
Does it? In [55]: def distance_squared(a, b): ....: return ((a - b) ** 2).sum() ....: In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) Out[56]: 55 -- Robert Kern robert.kern@gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
Charles R Harris wrote:
On 3/28/06, *Zachary Pincus* <zpincus@stanford.edu <mailto:zpincus@stanford.edu>> wrote:
Let me give a more concrete example, pulled from some code I was actually using.
The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 )
def distance_squared(a,b) return ((a - b)**2).sum()
The function should probably throw an error if a and b don't have the same shape and order and now it does.
Does it?
In [55]: def distance_squared(a, b): ....: return ((a - b) ** 2).sum() ....:
In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) Out[56]: 55
Right. It will work with any two arrays that can be broadcast to the same shape. Given numpy's rules, that's arguably the right thing to do in general although whether it's the right thing to do here depends on the specific application, which I don't know. Still, this is almost certainly better than the original case, which will blithely compute the distance_squared for any two arrays with the same total size. Then again, it's possible that's what the author really wanted. Unlikely, I think, but possible. Regards, -tim
Robert, On 3/28/06, Robert Kern <robert.kern@gmail.com> wrote:
Charles R Harris wrote:
On 3/28/06, *Zachary Pincus* <zpincus@stanford.edu <mailto:zpincus@stanford.edu>> wrote:
Let me give a more concrete example, pulled from some code I was actually using.
The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 )
def distance_squared(a,b) return ((a - b)**2).sum()
The function should probably throw an error if a and b don't have the same shape and order and now it does.
Does it?
In [55]: def distance_squared(a, b): ....: return ((a - b) ** 2).sum() ....:
In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) Out[56]: 55
Oops! Guess there should be a test for shape equality in there somewhere. Chuck
Zachary Pincus wrote:
Uh oh,
Just updated to new numpy so that dot() wouldn't be broken. What fun to play with the new order/ravel/reshape stuff.
Unfortunately, the ravel/reshape methods seem to not be quite worked out properly yet. Basically (and as expected) the ravel method doesn't work anymore on fortran-strided arrays. But since fortran-strided arrays happen *all-the-time* (e.g. after a transpose operation), things get confusing fast.
In fact, absolutely no reshape operation (other than the identity reshape) will work on fortran-strided arrays (see below). Also, the errors could be more helpful, and anyway ravel shouldn't fail with an error about reshape fortran-strided. There is one more problem with what I checked in. The PyArray_Ravel() function in C is now completely different and is used in _check_axis and
You've demonstrated my biggest concern with adopting Tim's approach. Because you can very easily get arrays that are not C-contiguous, you end up with a stituation in which the reshape and ravel methods are not very useful in my mind. therefore creates a segfault. So, I've decided to compromise. .reshape maintains it's view-only behavior .ravel() goes back to copy-if-necessary behavior (If you don't like it then use .reshape(-1) ). This should fix most of Zach's concerns and also fixes the segfault you get with rand(10,3).transpose().sum() -Travis
On 3/28/06, Travis Oliphant <oliphant.travis@ieee.org> wrote:
rand(10,3).transpose().sum()
Shouldn't sum be fixed instead? I admit that a quick look at the code leaves me no more enlightened. It is going to take more than a little time to figure out how numpy is put together. Chuck
Charles R Harris wrote:
On 3/28/06, *Travis Oliphant* <oliphant.travis@ieee.org <mailto:oliphant.travis@ieee.org>> wrote:
rand(10,3).transpose().sum()
Shouldn't sum be fixed instead? I admit that a quick look at the code leaves me no more enlightened. It is going to take more than a little time to figure out how numpy is put together.
FWIW: I also fixed the _check_axis code (it was ignoring errors and shouldn't have been). -Travis
Since fortran-strided arrays happen *all-the-time* (e.g. after a transpose operation),
I have to wonder if it's really a good idea to do this. Why not just make a re-ordered copy on transpose, rather than making it Fortran strided? Is the optimization really that important? I had always thought of Fortran strided arrays as being a special case exception (for working with external Fortran codes), rather than something that happens all the time. I suppose if all these issues are worked out, it'd be transparent to me, but it strikes me as more complication that necessary. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
Since fortran-strided arrays happen *all-the-time* (e.g. after a transpose operation),
I have to wonder if it's really a good idea to do this. Why not just make a re-ordered copy on transpose, rather than making it Fortran strided? Is the optimization really that important?
And optimization is important. The _dotblas.c function for example now works much faster with very common transpose operations because the underlying blas knows how to deal with striding so it was a complete wasted to always use C-ordering and get a copy of a Fortran-strided array. On large arrays, avoiding the unnecessary copy can be very significant. -Travis
participants (7)
-
Charles R Harris -
Christopher Barker -
Robert Kern -
Tim Hochberg -
Travis Oliphant -
Travis Oliphant -
Zachary Pincus