subclassing ndarray, was Memap in numpypy
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/numpy/user/basics.subclassing.html describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__ Failing tests and patches are welcome Matti
Matti That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought. On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next. Mike On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus <matti.picus@gmail.com> wrote:
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html<http://docs.scipy.org/doc/numpy/user/basics.subclassing.html> describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__
Failing tests and patches are welcome
Matti
You correctly point out, in other words, that it was an ugly hack. It is on the way to being removed by adding a space argument to calls to from_space...(). Let us know how your tests progress. Matti On 07/04/2013 09:57 PM, Mike Beller wrote:
Matti
That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought.
On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next.
Mike
On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus <matti.picus@gmail.com <mailto:matti.picus@gmail.com>> wrote:
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/numpy/user/basics.subclassing.html describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__
Failing tests and patches are welcome
Matti
Hi Matti, I think you still have to implement the __array_finalize__ protocol. Otherwise methods like .transpose() won't work on a memmap object. 2013/7/4 Matti Picus <matti.picus@gmail.com>
You correctly point out, in other words, that it was an ugly hack. It is on the way to being removed by adding a space argument to calls to from_space...(). Let us know how your tests progress. Matti
On 07/04/2013 09:57 PM, Mike Beller wrote:
Matti
That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought.
On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next.
Mike
On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus <matti.picus@gmail.com<mailto: matti.picus@gmail.com>**> wrote:
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html<http://docs.scipy.org/doc/numpy/user/basics.subclassing.html> describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__
Failing tests and patches are welcome
Matti
______________________________**_________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/**mailman/listinfo/pypy-dev<http://mail.python.org/mailman/listinfo/pypy-dev>
-- Amaury Forgeot d'Arc
True. changeset fd71a983d387 includes the start of tests for __array_finalize__ and a "few" failing tests Help is welcome :) Matti On 07/04/2013 10:30 PM, Amaury Forgeot d'Arc wrote:
Hi Matti,
I think you still have to implement the __array_finalize__ protocol. Otherwise methods like .transpose() won't work on a memmap object.
2013/7/4 Matti Picus <matti.picus@gmail.com <mailto:matti.picus@gmail.com>>
You correctly point out, in other words, that it was an ugly hack. It is on the way to being removed by adding a space argument to calls to from_space...(). Let us know how your tests progress. Matti
On 07/04/2013 09:57 PM, Mike Beller wrote:
Matti
That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought.
On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next.
Mike
On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus <matti.picus@gmail.com <mailto:matti.picus@gmail.com> <mailto:matti.picus@gmail.com <mailto:matti.picus@gmail.com>>> wrote:
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/numpy/user/basics.subclassing.html describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__
Failing tests and patches are welcome
Matti
_______________________________________________ pypy-dev mailing list pypy-dev@python.org <mailto:pypy-dev@python.org> http://mail.python.org/mailman/listinfo/pypy-dev
-- Amaury Forgeot d'Arc
Matti/Amaury Well -- that's excellent. The basic functionality of memmap.py seems to work fine now, against the ndarray-subtype branch. I had copied the tests from the original modules doctest into a pytest module. Had to modify because numpypy doesn't support .flags attribute, and also because offset is not supported yet. I am adding some additional tests to see what happens when sizes of things are wrong, and when one tries to write to read-only mmaps -- don't want to allow segfaults presumably. Other than that -- how do I go about making my code available for review / merging? Should I wait until Matti's branch is merged, then make a new branch called numpypy-memmap? Or should memmap just be part of the ndarray-subtype branch (since it is a user of the subtype functionality)? Mike On Thu, Jul 4, 2013 at 3:27 PM, Matti Picus <matti.picus@gmail.com> wrote:
You correctly point out, in other words, that it was an ugly hack. It is on the way to being removed by adding a space argument to calls to from_space...(). Let us know how your tests progress. Matti
On 07/04/2013 09:57 PM, Mike Beller wrote:
Matti
That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought.
On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next.
Mike
On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus <matti.picus@gmail.com<mailto: matti.picus@gmail.com>**> wrote:
I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html<http://docs.scipy.org/doc/numpy/user/basics.subclassing.html> describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__
Failing tests and patches are welcome
Matti
participants (3)
-
Amaury Forgeot d'Arc -
Matti Picus -
Mike Beller