Hi, Just a heads up that right now views of recarrays seem to be problematic, this doesn't work anymore:
import statsmodels.api as sm dta = sm.datasets.macrodata.load() # returns a record array with 14 fields dta.data[['infl', 'realgdp']].view((float,2))
I opened http://projects.scipy.org/numpy/ticket/2187 for this. Probably a blocker for 1.7.0. Question: is that really the recommended way to get an (N, 2) size float array from two columns of a larger record array? If so, why isn't there a better way? If you'd want to write to that (N, 2) array you have to append a copy, making it even uglier. Also, then there really should be tests for views in test_records.py. Ralf
On Sun, Jul 22, 2012 at 2:15 PM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
Hi,
Just a heads up that right now views of recarrays seem to be problematic, this doesn't work anymore:
import statsmodels.api as sm dta = sm.datasets.macrodata.load() # returns a record array with 14 fields dta.data[['infl', 'realgdp']].view((float,2))
I opened http://projects.scipy.org/numpy/ticket/2187 for this. Probably a blocker for 1.7.0.
Question: is that really the recommended way to get an (N, 2) size float array from two columns of a larger record array? If so, why isn't there a better way? If you'd want to write to that (N, 2) array you have to append a copy, making it even uglier. Also, then there really should be tests for views in test_records.py.
Any comments on this? I have a lot of broken code to deal with if ((float, shape[1])) is no longer allowed on structured and rec arrays. Skipper
I'm actively looking at this issue since it was my pull request that broke this (https://github.com/numpy/numpy/pull/350). We definitely don't want to break this functionality for 1.7. The problem is that even though indexing with a subset of fields still returns a copy (for now), it now returns a copy of a view of the original array. When you call copy() on a view, it copies the entire original structured array with the view dtype. A short term fix would be to "manually" create a proper copy to return similar to what _index_fields() did before my change, but since the idea is to eventually return the view instead of a copy, long term we need a way to do a proper copy of a structured array view that doesn't copy the unwanted fields. -Jay On Wed, Jul 25, 2012 at 10:15 AM, Skipper Seabold <jsseabold@gmail.com>wrote:
On Sun, Jul 22, 2012 at 2:15 PM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
Hi,
Just a heads up that right now views of recarrays seem to be problematic, this doesn't work anymore:
import statsmodels.api as sm dta = sm.datasets.macrodata.load() # returns a record array with 14 fields dta.data[['infl', 'realgdp']].view((float,2))
I opened http://projects.scipy.org/numpy/ticket/2187 for this. Probably a blocker for 1.7.0.
Question: is that really the recommended way to get an (N, 2) size float array from two columns of a larger record array? If so, why isn't there a better way? If you'd want to write to that (N, 2) array you have to append a copy, making it even uglier. Also, then there really should be tests for views in test_records.py.
Any comments on this? I have a lot of broken code to deal with if ((float, shape[1])) is no longer allowed on structured and rec arrays.
Skipper _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Wed, Jul 25, 2012 at 10:29 AM, Jay Bourque <jay.bourque@continuum.io> wrote:
I'm actively looking at this issue since it was my pull request that broke this (https://github.com/numpy/numpy/pull/350). We definitely don't want to break this functionality for 1.7. The problem is that even though indexing with a subset of fields still returns a copy (for now), it now returns a copy of a view of the original array. When you call copy() on a view, it copies the entire original structured array with the view dtype. A short term fix would be to "manually" create a proper copy to return similar to what _index_fields() did before my change, but since the idea is to eventually return the view instead of a copy, long term we need a way to do a proper copy of a structured array view that doesn't copy the unwanted fields.
This should be fixed for 1.7.0. However, I am going to release beta now, and then see what we can do about this. Ondrej
Jay, On Mon, Aug 20, 2012 at 12:40 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
On Wed, Jul 25, 2012 at 10:29 AM, Jay Bourque <jay.bourque@continuum.io> wrote:
I'm actively looking at this issue since it was my pull request that broke this (https://github.com/numpy/numpy/pull/350). We definitely don't want to break this functionality for 1.7. The problem is that even though indexing with a subset of fields still returns a copy (for now), it now returns a copy of a view of the original array. When you call copy() on a view, it copies the entire original structured array with the view dtype. A short term fix would be to "manually" create a proper copy to return similar to what _index_fields() did before my change, but since the idea is to eventually return the view instead of a copy, long term we need a way to do a proper copy of a structured array view that doesn't copy the unwanted fields.
This should be fixed for 1.7.0. However, I am going to release beta now, and then see what we can do about this.
What would be the best "short term" fix, so that we can release 1.7.0? I am still trying to understand what exactly the problem with dtype is in _index_fields(). Would you suggest to keep using the view, or somehow revert to the old behavior while still trying to pass all the new tests in your PR 350? If you have any hints, it would save me some time. Thanks, Ondrej
Ondrej, Sorry for the delay in getting back to this. I have some free time today to get this resolved if you haven't already fixed it. -Jay On Wed, Aug 29, 2012 at 7:19 PM, Ondřej Čertík <ondrej.certik@gmail.com>wrote:
Jay,
On Mon, Aug 20, 2012 at 12:40 PM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
On Wed, Jul 25, 2012 at 10:29 AM, Jay Bourque <jay.bourque@continuum.io> wrote:
I'm actively looking at this issue since it was my pull request that broke this (https://github.com/numpy/numpy/pull/350). We definitely don't want to break this functionality for 1.7. The problem is that even though indexing with a subset of fields still returns a copy (for now), it now returns a copy of a view of the original array. When you call copy() on a view, it copies the entire original structured array with the view dtype. A short term fix would be to "manually" create a proper copy to return similar to what _index_fields() did before my change, but since the idea is to eventually return the view instead of a copy, long term we need a way to do a proper copy of a structured array view that doesn't copy the unwanted fields.
This should be fixed for 1.7.0. However, I am going to release beta now, and then see what we can do about this.
What would be the best "short term" fix, so that we can release 1.7.0?
I am still trying to understand what exactly the problem with dtype is in _index_fields(). Would you suggest to keep using the view, or somehow revert to the old behavior while still trying to pass all the new tests in your PR 350? If you have any hints, it would save me some time.
Thanks, Ondrej
On Fri, Aug 31, 2012 at 6:15 AM, Jay Bourque <jay.bourque@continuum.io> wrote:
Ondrej,
Sorry for the delay in getting back to this. I have some free time today to get this resolved if you haven't already fixed it.
I haven't. If you can look at it, that would be absolutely awesome. If you don't manage to fix it, if you can give me some hints what's going on, that would also be a huge help. Many thanks! Ondrej
Ondrej, Just submitted the following pull request for this: https://github.com/numpy/numpy/pull/401 -Jay On Fri, Aug 31, 2012 at 12:09 PM, Ondřej Čertík <ondrej.certik@gmail.com>wrote:
On Fri, Aug 31, 2012 at 6:15 AM, Jay Bourque <jay.bourque@continuum.io> wrote:
Ondrej,
Sorry for the delay in getting back to this. I have some free time today to get this resolved if you haven't already fixed it.
I haven't. If you can look at it, that would be absolutely awesome. If you don't manage to fix it, if you can give me some hints what's going on, that would also be a huge help.
Many thanks! Ondrej
participants (4)
-
Jay Bourque -
Ondřej Čertík -
Ralf Gommers -
Skipper Seabold