copy/deepcopy pull request, Travis build failure
I submitted a pull request and one of the Travis builds is failing: https://travis-ci.org/#!/numpy/numpy/jobs/2933551 Given my changes, https://github.com/dwf/numpy/commit/4c88fdafc003397d6879f81bf59f68adeeb59f2b I don't see how the masked array module (responsible for the failing test) could possibly be affected in this manner (RuntimeWarning raised by encountering an invalid value in power()). Moreover, I cannot reproduce it after setting NPY_SEPARATE_COMPILATION on my own machine (whatever that does -- I'm not quite clear). Does anyone have any insight into what's going on here? Can anyone else reproduce it outside of Travis? Thanks, David
On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
I submitted a pull request and one of the Travis builds is failing:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing. I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that? Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess. Regards, Sebastian
Given my changes,
https://github.com/dwf/numpy/commit/4c88fdafc003397d6879f81bf59f68adeeb59f2b
I don't see how the masked array module (responsible for the failing test) could possibly be affected in this manner (RuntimeWarning raised by encountering an invalid value in power()).
Moreover, I cannot reproduce it after setting NPY_SEPARATE_COMPILATION on my own machine (whatever that does -- I'm not quite clear).
Does anyone have any insight into what's going on here? Can anyone else reproduce it outside of Travis?
Thanks, David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg
On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing.
Ah, okay. I figured it was something like that.
I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that?
Oops, you're right. I assumed I was changing __copy__ only. Pull request updated. Given that behaviour is documented it really ought to be tested. I'll add one.
Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess.
Agreed, but for now the .copy() method's default shouldn't change. I think the scikit-learn usecase I described is a good reason why the copy protocol methods should maintain data ordering, though. David
On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley
On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg
wrote: On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing.
Ah, okay. I figured it was something like that.
I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that?
Oops, you're right. I assumed I was changing __copy__ only. Pull request updated.
Given that behaviour is documented it really ought to be tested. I'll add one.
Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess.
Agreed, but for now the .copy() method's default shouldn't change. I think the scikit-learn usecase I described is a good reason why the copy protocol methods should maintain data ordering, though.
I think this might be something that could wait for a big numpy version change. At least I always assumed that a new array created by copying is the default numpy C order (unless otherwise requested). I never rely on it except sometimes when I think about speed of operation in fortran versus c order. np.copy says: This is equivalent to
np.array(a, copy=True)
numpy.ma.copy has the order keyword with "c" as default changing the default from "C" to "A" doesn't look like a minor API change. (I'm using numpy 1.5 help file, so maybe I'm outdated.) Josef
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Oct 25, 2012 at 8:39 PM,
On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley
wrote: On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg
wrote: On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing.
Ah, okay. I figured it was something like that.
I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that?
Oops, you're right. I assumed I was changing __copy__ only. Pull request updated.
Given that behaviour is documented it really ought to be tested. I'll add one.
Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess.
Agreed, but for now the .copy() method's default shouldn't change. I think the scikit-learn usecase I described is a good reason why the copy protocol methods should maintain data ordering, though.
I think this might be something that could wait for a big numpy version change.
At least I always assumed that a new array created by copying is the default numpy C order (unless otherwise requested). I never rely on it except sometimes when I think about speed of operation in fortran versus c order.
np.copy says: This is equivalent to
np.array(a, copy=True)
numpy.ma.copy has the order keyword with "c" as default
changing the default from "C" to "A" doesn't look like a minor API change.
Hi Josef, Just to be clear: this pull request ( https://github.com/numpy/numpy/pull/2699 ) affects only the behaviour when arrays are copied with the standard library "copy" module. The use of the .copy() method of ndarrays, or the numpy.copy library function, remains the same, with all defaults intact. An oversight on my part had briefly changed it, but Sebastian pointed this out and I've updated the PR. The main application is, as I said, with objects that have ndarray members and interface with external code that relies on Fortran ordering of those ndarray members (of which there are several in scikit-learn, as one example). An object once deepcopy'd should ideally "just work" in any situation where the original worked, but with the current implementation of the copy protocol methods this wasn't possible. Frequent users of .copy()/np.copy() who are familiar with *NumPy's* API behaviour should be largely unaffected. David
On Thu, Oct 25, 2012 at 11:48 PM, David Warde-Farley
On Thu, Oct 25, 2012 at 8:39 PM,
wrote: On Thu, Oct 25, 2012 at 6:58 PM, David Warde-Farley
wrote: On Thu, Oct 25, 2012 at 6:15 PM, Sebastian Berg
wrote: On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing.
Ah, okay. I figured it was something like that.
I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that?
Oops, you're right. I assumed I was changing __copy__ only. Pull request updated.
Given that behaviour is documented it really ought to be tested. I'll add one.
Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess.
Agreed, but for now the .copy() method's default shouldn't change. I think the scikit-learn usecase I described is a good reason why the copy protocol methods should maintain data ordering, though.
I think this might be something that could wait for a big numpy version change.
At least I always assumed that a new array created by copying is the default numpy C order (unless otherwise requested). I never rely on it except sometimes when I think about speed of operation in fortran versus c order.
np.copy says: This is equivalent to
np.array(a, copy=True)
numpy.ma.copy has the order keyword with "c" as default
changing the default from "C" to "A" doesn't look like a minor API change.
Hi Josef,
Just to be clear: this pull request ( https://github.com/numpy/numpy/pull/2699 ) affects only the behaviour when arrays are copied with the standard library "copy" module. The use of the .copy() method of ndarrays, or the numpy.copy library function, remains the same, with all defaults intact. An oversight on my part had briefly changed it, but Sebastian pointed this out and I've updated the PR.
Fine, I didn't understand that part correctly. I have no opinion in that case. (In statsmodels we only use copy the array method and through np.array().) Thanks for the clarification Josef
The main application is, as I said, with objects that have ndarray members and interface with external code that relies on Fortran ordering of those ndarray members (of which there are several in scikit-learn, as one example). An object once deepcopy'd should ideally "just work" in any situation where the original worked, but with the current implementation of the copy protocol methods this wasn't possible.
Frequent users of .copy()/np.copy() who are familiar with *NumPy's* API behaviour should be largely unaffected.
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Oct 26, 2012 at 1:04 AM,
Fine, I didn't understand that part correctly.
I have no opinion in that case. (In statsmodels we only use copy the array method and through np.array().)
Do you implement __copy__ or __deepcopy__ on your objects? If not, client code using statsmodels might try to use the copy module and run into the same problem. David
On Fri, Oct 26, 2012 at 1:27 AM, David Warde-Farley
On Fri, Oct 26, 2012 at 1:04 AM,
wrote: Fine, I didn't understand that part correctly.
I have no opinion in that case. (In statsmodels we only use copy the array method and through np.array().)
Do you implement __copy__ or __deepcopy__ on your objects? If not, client code using statsmodels might try to use the copy module and run into the same problem.
We don't implement either. I haven't heard yet of a use case for copying a model or results instance in statsmodels (that doesn't have a more memory efficient alternative). I make a note about this if we need it in future. (asides: We had to work around pickling problems. We try to use views instead of copies. Nothing depends on fortran versus c order of arrays except speed, and I think linalg will make copies if users provide data arrays with the wrong properties. We never systematically investigated differences in speed depending on array order or views, and some differences will have changed with recent numpy versions. ) Josef
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi Sebastian,
On Thu, Oct 25, 2012 at 4:15 PM, Sebastian Berg
On Thu, 2012-10-25 at 17:48 -0400, David Warde-Farley wrote:
I submitted a pull request and one of the Travis builds is failing:
Don't worry about that failure on Travis... It happens randomly on at the moment and its unrelated to anything you are doing. I am not sure though you can change behavior like that since you also change the default behavior of the `.copy()` method and someone might rely on that? Maybe making it specific to the copy model would make it unlikely that anyone relies on the default, it would seem sensible that copy.copy(array) does basically the same as np.copy(array) and not as the method .copy, though ideally maybe the differences could be removed in the long run I guess.
You seem to becoming more involved in the code maintenance. Would you be interested in gaining commit rights at some point? Chuck
Hey On Thu, 2012-10-25 at 19:27 -0600, Charles R Harris wrote:
Hi Sebastian,
<snip>
You seem to becoming more involved in the code maintenance. Would you be interested in gaining commit rights at some point?
Maybe, but honestly I am not sure if I will keep following numpy very closely in the future. Regards, Sebastian
Chuck
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Oct 26, 2012 at 3:10 PM, Sebastian Berg
Hey
On Thu, 2012-10-25 at 19:27 -0600, Charles R Harris wrote:
Hi Sebastian,
<snip>
You seem to becoming more involved in the code maintenance. Would you be interested in gaining commit rights at some point?
Maybe, but honestly I am not sure if I will keep following numpy very closely in the future.
Fair enough. We need to get new people to get involved, so if you find yourself tending to more involvement it is a definite option. Chuck
participants (5)
-
Charles R Harris
-
David Warde-Farley
-
David Warde-Farley
-
josef.pktd@gmail.com
-
Sebastian Berg