I downloaded the prebuilt binary pypy-5.1.1-osx64, dropped it into my homedir, and built a virtualenv using it. The files look like they were built Apr. 30th, I installed May 11th. I don't think that I've installed numpy-specific C libs for the cpython numpy I also have installed, but this system is old enough that I might have. If that becomes relevant, I can dig in more. First I installed numpy by following the instructions here: http://pypy.org/download.html#installing-numpy IIRC, I just used the virtualenv pip to do it. Then when I started to notice the difference in behavior between cpython and pypy, I ended up messing around a bit (wondering if the slight version difference was important, etc.) and ended up going into the pypy venv site-packages and doing rm -rf numpy* to get a clean slate, then doing a git clone of the pypy/numpy repo and doing pypy setup.py install, as directed here: https://bitbucket.org/pypy/numpy Then I run this: pypy -c "import numpy; numpy.test('doesntexist')" And get (among a lot of other noise): ====================================================================== ERROR: Failure: ImportError (No module named numpy.core.multiarray_tests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/elis/venv/droidblue-pypy/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/Users/elis/venv/droidblue-pypy/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/Users/elis/venv/droidblue-pypy/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Users/elis/venv/droidblue-pypy/site-packages/numpy/core/tests/test_multiarray.py", line 22, in <module> from numpy.core.multiarray_tests import ( ImportError: No module named numpy.core.multiarray_tests Commenting out those imports results in the datetime64 issue I gave earlier. That's when I decided it was getting too hacky, and it made sense to reach out and make sure that I was running the tests the right way, etc. Of course, if you'd like more detail on anything, please let me know. :) Thanks! Eli On Wed, May 18, 2016 at 8:42 AM, Matti Picus <matti.picus@gmail.com> wrote:

On 18/05/16 08:43, Eli Stevens (Gmail) wrote:

...

The following works on cpython, but fails on pypy:

import numpy as np def test_writeable(): a = np.zeros((2,2), np.int8)

...

a.flags.writeable = False

E TypeError: readonly attribute

So I went and installed pypy's fork of numpy from source, and tried to start poking around looking for tests, etc. to see what was going on.

It looks like numpy/core/tests/test_multiarray.py TestFlags.test_writeable is doing something similar, but when I attempt to run the tests like so:

pypy -c "import numpy; numpy.test()"

(Is that the right way to do it?)

...

Thanks for looking into this. It seems there are two seperate issues, the failure to set flag attributes and the failure to run tests. Could you give a few more details about how to reproduce the second problem: which pypy and how did you obtain it, and how you built/installed numpy into it? What imports did you have to change? You shouldn't have to change imports, but it is entirely possible that we have overlooked something. Matti

Great, thanks for the pointers. I'll hopefully be able to dig in after work tonight. Is there a from-scratch guide to getting to the point where I can run those micronumpy tests? NBD if not, I'm sure I can figure it out. Do you think it would make sense to start off by copying test_multiarray.py:TestFlags there? Cheers, Eli On Wed, May 18, 2016 at 9:59 AM, Matti Picus <matti.picus@gmail.com> wrote:

On 18/05/16 19:35, Eli Stevens (Gmail) wrote:

I downloaded the prebuilt binary pypy-5.1.1-osx64, dropped it into my homedir, and built a virtualenv using it. The files look like they were built Apr. 30th, I installed May 11th. I don't think that I've installed numpy-specific C libs for the cpython numpy I also have installed, but this system is old enough that I might have. If that becomes relevant, I can dig in more.

First I installed numpy by following the instructions here: http://pypy.org/download.html#installing-numpy IIRC, I just used the virtualenv pip to do it.

Then when I started to notice the difference in behavior between cpython and pypy, I ended up messing around a bit (wondering if the slight version difference was important, etc.) and ended up going into the pypy venv site-packages and doing rm -rf numpy* to get a clean slate, then doing a git clone of the pypy/numpy repo and doing pypy setup.py install, as directed here: https://bitbucket.org/pypy/numpy

Then I run this:

pypy -c "import numpy; numpy.test('doesntexist')"

And get (among a lot of other noise):

====================================================================== ERROR: Failure: ImportError (No module named numpy.core.multiarray_tests) ---------------------------------------------------------------------- ...

Commenting out those imports results in the datetime64 issue I gave earlier. That's when I decided it was getting too hacky, and it made sense to reach out and make sure that I was running the tests the right way, etc.

Of course, if you'd like more detail on anything, please let me know. :)

Thanks! Eli

It seems you are doing everything correctly. multiarray_tests comes from numpy/core/src/multiarray/multiarray.c.src which is compiled to a C-API module. We skip building it as we have quite a way to go before we can support that level of C-API compatibility. The issue with readonly flag attributes actually lies with micronumpy, in the pypy interpreter itself. If you wish to work on this, you should add a test (in the pypy repo) to pypy/module/micronumpy/test/test_flagsobj.py and continue from there. Matti

I've got a pypy clone and checkout, and have added TestFlags. When I run it, I see:

a.flags.writeable = False

E TypeError: readonly attribute But nothing that looks like it should raise a TypeError in either of: pypy/pypy/module/micronumpy/flagsobj.py pypy/pypy/module/micronumpy/ndarray.py Still trying to get oriented with the code. Any suggestions? Thanks, Eli On Wed, May 18, 2016 at 10:55 AM, Matti Picus <matti.picus@gmail.com> wrote:

On 18/05/16 20:19, Eli Stevens (Gmail) wrote:

Great, thanks for the pointers. I'll hopefully be able to dig in after work tonight. Is there a from-scratch guide to getting to the point where I can run those micronumpy tests? NBD if not, I'm sure I can figure it out.

Do you think it would make sense to start off by copying test_multiarray.py:TestFlags there?

Cheers, Eli

On Wed, May 18, 2016 at 9:59 AM, Matti Picus <matti.picus@gmail.com> wrote:

It seems you are doing everything correctly. multiarray_tests comes from numpy/core/src/multiarray/multiarray.c.src which is compiled to a C-API module. We skip building it as we have quite a way to go before we can support that level of C-API compatibility. The issue with readonly flag attributes actually lies with micronumpy, in the pypy interpreter itself. If you wish to work on this, you should add a test (in the pypy repo) to pypy/module/micronumpy/test/test_flagsobj.py and continue from there. Matti

There is an explanation of running tests here http://doc.pypy.org/en/latest/getting-started-dev.html#running-pypy-s-unit-t... The tests in TestFlags require refactoring for our simpler 'assert' style - no fancy assert_equal() or assert_() functions Matti

Looks like I need to do something along the lines of: def descr_set_writeable(self, space, w_value): if space.is_true(w_value) != bool(self.flags & NPY.ARRAY_WRITEABLE): self.flags ^= NPY.ARRAY_WRITEABLE (Though I probably need more robust checking to see if the flag *can* be turned off) def descr_setitem(self, space, w_item, w_value): # This function already exists, but just contains the last line with the raise key = space.str_w(w_item) value = space.bool_w(w_value) if key == "W" or key == "WRITEABLE": return self.descr_set_writeable(space, value) raise oefmt(space.w_KeyError, "Unknown flag") ... writeable = GetSetProperty(W_FlagsObject.descr_get_writeable, W_FlagsObject.descr_set_writeable), However I'm not entirely confident about things like space.bool_w, etc. I've read http://doc.pypy.org/en/latest/objspace.html but am still working on internalizing it. Setting the GetSetProperty still results in the TypeError, which makes me wonder how to tell if I'm getting the right flagsobj.py. I don't think that I am. The results of the tests should be the same no matter what python interpreter I'm using, correct? Would running the tests with a virtualenv that has a stock pypy/numpy installed cause issues? What if the virtualenv is cpython? When I run py.test, I see: pytest-2.5.2 from /Users/elis/edit/play/pypy/pytest.pyc Which looks correct (.../play/pypy is my source checkout). But I get the same thing when using cpython to run test_all.py, and there the test passes, so I don't think it's indicative. When I print out np.__file__ inside the test, I get /Users/elis/venv/droidblue-pypy/site-packages/numpy/__init__.pyc Which is the pypy venv I am using to run the tests in the first place, but I'm not sure what the on-disk relationship between numpy and micronumpy actually is. Is there a way from the test_flagobjs.py file to determine what the on-disk location of micronumpy is? I strongly suspect I've got something basic wrong. I also think that the information at http://doc.pypy.org/en/latest/getting-started-dev.html#running-pypy-s-unit-t... and http://doc.pypy.org/en/latest/coding-guide.html#command-line-tool-test-all conflict somewhat, or are at least unclear as to which approach is the right way in what situation. I'll attempt to clarify whatever it is that's tripping me up once I've got it sorted out. Some other questions I have, looking at micornumpy/concrete.py line 37: class BaseConcreteArray(object): _immutable_fields_ = ['dtype?', 'storage', 'start', 'size', 'shape[*]', 'strides[*]', 'backstrides[*]', 'order', 'gcstruct', 'flags'] start = 0 parent = None flags = 0 Does that immutable status cascade down into the objects, or is that saying only that myInstance.flags cannot be reassigned (but myInstance.flags.foo = 3 is fine)? interpreter/typedef.py 221: @specialize.arg(0) def make_objclass_getter(tag, func, cls): if func and hasattr(func, 'im_func'): assert not cls or cls is func.im_class cls = func.im_class return _make_objclass_getter(cls) What's the purpose of the tag argument? It doesn't seem to be used here or in _make_descr_typecheck_wrapper, both of which are called from GetSetProperty init. Based on docstrings on _Specialize, it seems like they might be JIT hints. Is that correct? Matti: If it's okay, I'd like to keep the discussion on the list, as I've actively searched through discussions here to avoid asking questions a second time. Hopefully this thread can help the next person. Sorry for the mega-post; thanks for reading. Eli On Thu, May 19, 2016 at 8:23 AM, Armin Rigo <arigo@tunes.org> wrote:

Hi Eli,

On 19 May 2016 at 08:58, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

...

I've got a pypy clone and checkout, and have added TestFlags. When I run it, I see:

...

a.flags.writeable = False

E TypeError: readonly attribute

But nothing that looks like it should raise a TypeError in either of:

Grep for 'writable'. You'll see that it is defined as a GetSetProperty() with a getter but no setter so far.

A bientôt,

Armin.

the option is --withmod-micronumpy or --allworkingmodules but the tests are in the test directory and *that's* how you should run tests (not by playing with interactive) On Fri, May 20, 2016 at 7:44 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

More questions! :)

When I run

pypy> /usr/bin/python bin/pyinteractive.py

I get to a (presumably interpreted, given the startup time) pypy prompt, but I cannot import numpy. Is the intent that I somehow install numpy into the source checkout's site-packages directory (the one listed in sys.path from that interpreted pypy prompt)?

Also, it's pretty clear that when running the tests that "import numpy" just gets the numpy from the base interpreter, not from the micronumpy included in the pypy source. Is it possible to run the numpy tests without doing a full translation?

Thanks, Eli

On Thu, May 19, 2016 at 1:36 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

...

Looks like I need to do something along the lines of:

def descr_set_writeable(self, space, w_value): if space.is_true(w_value) != bool(self.flags & NPY.ARRAY_WRITEABLE): self.flags ^= NPY.ARRAY_WRITEABLE

(Though I probably need more robust checking to see if the flag *can* be turned off)

def descr_setitem(self, space, w_item, w_value): # This function already exists, but just contains the last line with the raise key = space.str_w(w_item) value = space.bool_w(w_value) if key == "W" or key == "WRITEABLE": return self.descr_set_writeable(space, value) raise oefmt(space.w_KeyError, "Unknown flag")

... writeable = GetSetProperty(W_FlagsObject.descr_get_writeable, W_FlagsObject.descr_set_writeable),

However I'm not entirely confident about things like space.bool_w, etc. I've read http://doc.pypy.org/en/latest/objspace.html but am still working on internalizing it.

Setting the GetSetProperty still results in the TypeError, which makes me wonder how to tell if I'm getting the right flagsobj.py. I don't think that I am. The results of the tests should be the same no matter what python interpreter I'm using, correct? Would running the tests with a virtualenv that has a stock pypy/numpy installed cause issues? What if the virtualenv is cpython?

When I run py.test, I see:

pytest-2.5.2 from /Users/elis/edit/play/pypy/pytest.pyc

Which looks correct (.../play/pypy is my source checkout). But I get the same thing when using cpython to run test_all.py, and there the test passes, so I don't think it's indicative. When I print out np.__file__ inside the test, I get

/Users/elis/venv/droidblue-pypy/site-packages/numpy/__init__.pyc

Which is the pypy venv I am using to run the tests in the first place, but I'm not sure what the on-disk relationship between numpy and micronumpy actually is. Is there a way from the test_flagobjs.py file to determine what the on-disk location of micronumpy is?

I strongly suspect I've got something basic wrong. I also think that the information at http://doc.pypy.org/en/latest/getting-started-dev.html#running-pypy-s-unit-t... and http://doc.pypy.org/en/latest/coding-guide.html#command-line-tool-test-all conflict somewhat, or are at least unclear as to which approach is the right way in what situation. I'll attempt to clarify whatever it is that's tripping me up once I've got it sorted out.

Some other questions I have, looking at micornumpy/concrete.py line 37:

class BaseConcreteArray(object): _immutable_fields_ = ['dtype?', 'storage', 'start', 'size', 'shape[*]', 'strides[*]', 'backstrides[*]', 'order', 'gcstruct', 'flags'] start = 0 parent = None flags = 0

Does that immutable status cascade down into the objects, or is that saying only that myInstance.flags cannot be reassigned (but myInstance.flags.foo = 3 is fine)?

interpreter/typedef.py 221:

@specialize.arg(0) def make_objclass_getter(tag, func, cls): if func and hasattr(func, 'im_func'): assert not cls or cls is func.im_class cls = func.im_class return _make_objclass_getter(cls)

What's the purpose of the tag argument? It doesn't seem to be used here or in _make_descr_typecheck_wrapper, both of which are called from GetSetProperty init. Based on docstrings on _Specialize, it seems like they might be JIT hints. Is that correct?

Matti: If it's okay, I'd like to keep the discussion on the list, as I've actively searched through discussions here to avoid asking questions a second time. Hopefully this thread can help the next person.

Sorry for the mega-post; thanks for reading. Eli

On Thu, May 19, 2016 at 8:23 AM, Armin Rigo <arigo@tunes.org> wrote:

...

Hi Eli,

On 19 May 2016 at 08:58, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

...

I've got a pypy clone and checkout, and have added TestFlags. When I run it, I see:

...

a.flags.writeable = False

E TypeError: readonly attribute

But nothing that looks like it should raise a TypeError in either of:

Grep for 'writable'. You'll see that it is defined as a GetSetProperty() with a getter but no setter so far.

A bientôt,

Armin.

---

pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev

You should commit your changes to a branch and push to a bitbucket repo so we can see your changes. Our test runner compiles part of PyPy and calls the tests using that partial interpreter (unless run with -A), when you call import numpy inside a test you are using micronumpy. You should not try to stop in a debugger in the test code itself (app level), rather print or set_trace inside micronumpy code (interpreter level). Matti

On 20 May 2016, at 10:18 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

I understand that the tests are in the test directory, but the issue I'm currently trying to figure out is that when I invoke either of:

/usr/bin/python test_all.py test_all.py module/micronumpy/test/test_flagsobj.py

/usr/bin/python pytest.py pypy/module/micronumpy/test/test_flagsobj.py (a directory level up)

With any of system cpython, venv cpython, venv pypy, etc. then the changes that I've made locally to micronumpy aren't used, since inside of the tests "import numpy" grabs the numpy from whatever interpreter the tests were invoked with.

I'm sure there's something simple that I'm missing about the environment that's needed to make this work, but I haven't figured it out yet. Do I need to be doing something with the PYTHONPATH prior to running the tests?

Cheers, Eli

...

Le 20/05/16 21:21, Eli Stevens (Gmail) a écrit :

Here you go: https://bitbucket.org/elistevens/pypy/commits/branch/numpy_flags_writeable

The name of your class needs to start with 'AppTest', so that our test runner knows it needs to use the black magic that enables application-level tests (i.e. tests that run on top of our PyPy interpreter itself running on top of your regular Python).

Anyone have any pointers for understanding the relationship between the application-level myarray.flags, and self.flags and/or orig_arr.w_flags in BaseConcreteArray.descr_setitem? I'm even sure where to start doing code dives from. Thanks, Eli On Sat, May 21, 2016 at 10:51 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:

Ah-ha! Renaming the class seemed to do the trick. Is that documented anywhere? Similarly, is there a full list of the things that are available to AppTestXXX classes? It took quite a bit of trial and error to figure out that "with py.test.raises" wasn't available, but "raises" was (I spent a lot of time trying to figure out how to import py.test, though now I know what to look for, I see that there are already examples of raises use in test_flagsobj.py. Sigh.).

In any case, I'm able to run the tests and see my code executing now, so that's good.

What does the method name prefix 'descr_' imply?

I'm now stuck trying to understand the relationship between the application level myarray.flags and the flag instances on BaseConcreteArray subclasses.

I now have this situation:

In the AppTest my array has id 0x110d648d0. The writeable flag is False, according to printing a.flags (I set it in the test, having added that code to W_FlagsObject.descr_set_writeable, which also seems to be behaving).

In BaseConcreteArray.descr_setitem the orig_arr id is 0x110d648d0, orig_arr.w_flags is None, and self.flags is 0x507, with writeable being 0x400. Any ideas? I've pushed up what I have so far.

Thanks, Eli

On Fri, May 20, 2016 at 1:52 PM, Ronan Lamy <ronan.lamy@gmail.com> wrote:

...

Le 20/05/16 21:21, Eli Stevens (Gmail) a écrit :

...

Here you go: https://bitbucket.org/elistevens/pypy/commits/branch/numpy_flags_writeable

The name of your class needs to start with 'AppTest', so that our test runner knows it needs to use the black magic that enables application-level tests (i.e. tests that run on top of our PyPy interpreter itself running on top of your regular Python).

_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev