unaligned raw_load (align_float_cast branch)
I would like to ask for help with this branch, as I have not been able to move it forward fast enough. micronumpy tests crash on ARM, due to issues accessing raw memory in struct arrays. It turn out that casting unaligned memory to a float or double causes a segfault on ARM, I started to fix this on the align_float_cast branch. I wrote some tests, added some XXX in places that seem to call raw_load, but got stuck trying to fix the test in test_rawstorage that uses RawStorageGetitemEntry in rlib/rawstorage. It is not clear to me how to add the "allocate temporary aligned memory, copy from unaligned memory, cast" trick using HighLevelOps in the RawStorageGetitemEntry class. Should this use a piece of C instead? Please help by either committing some code or pointing out how to move forward. Matti
Hi Matti, On 18 February 2014 14:05, matti picus <matti.picus@gmail.com> wrote:
I would like to ask for help with this branch, as I have not been able to move it forward fast enough.
I moved it forward a little bit, but now I'm wondering if it actually makes sense. What is the goal? Is it to fix a problem in ctypes only, by fixing pypy/module/_rawffi? If that's the case, then we might as well write the hack locally in _rawffi, e.g. by using different versions of the helpers there depending on whether we get aligned or misaligned pointers. This certainly looks cleaner than adding raw_load_unaligned and raw_store_unaligned *everywhere* including in the JIT. A bientôt, Armin.
Thanks for taking a look. My immediate goal is to get micronumpy tests passing on ARM. micronumpy uses raw_storage_getitem, which needs to be fixed. I thought nonaligned access would be a problem in other parts of pypy, but do not have the perspective to judge that, so I tried to stick xxx in all raw_load places in the code base, whether needed or not. In retrospect, that may be overkill, and it may be enough to fix for a few cases in micronumpy (reading from a user-supplied buffer or from a record array) and, as you point out, ctypes. What about struct.unpack() ? Matti On 02/22/2014 10:30 AM, Armin Rigo wrote:
Hi Matti,
On 18 February 2014 14:05, matti picus <matti.picus@gmail.com> wrote:
I would like to ask for help with this branch, as I have not been able to move it forward fast enough. I moved it forward a little bit, but now I'm wondering if it actually makes sense. What is the goal? Is it to fix a problem in ctypes only, by fixing pypy/module/_rawffi? If that's the case, then we might as well write the hack locally in _rawffi, e.g. by using different versions of the helpers there depending on whether we get aligned or misaligned pointers. This certainly looks cleaner than adding raw_load_unaligned and raw_store_unaligned *everywhere* including in the JIT.
A bientôt,
Armin.
Hi Matti, I've checked in some new code in rawstorage.py on default: raw_storage_{get,set}item_unaligned(). It's regular RPython code that checks if the argument is aligned or not, and if not, it falls back to a memcpy(). It should be relatively jit-friendly. Note that I also changed the non-unaligned functions so that, when untranslated, they complain if the address is unaligned, whatever the platform. Can you try to change some or all of the raw_storage_{get,set}item() calls in micronumpy? I don't really know which ones are at a risk of seeing unaligned accesses, and which ones not. It seems that all of them are at risk: see the failure in test_numarray.AppTestRecordDtype.test_string_record. A bientôt, Armin.
I believe all of them in micronumpy are at risk -- any type could be a subtype of a record type consisting of something like a single char followed by whatever type. On Sun, Feb 23, 2014 at 1:43 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Matti,
I've checked in some new code in rawstorage.py on default: raw_storage_{get,set}item_unaligned(). It's regular RPython code that checks if the argument is aligned or not, and if not, it falls back to a memcpy(). It should be relatively jit-friendly. Note that I also changed the non-unaligned functions so that, when untranslated, they complain if the address is unaligned, whatever the platform.
Can you try to change some or all of the raw_storage_{get,set}item() calls in micronumpy? I don't really know which ones are at a risk of seeing unaligned accesses, and which ones not. It seems that all of them are at risk: see the failure in test_numarray.AppTestRecordDtype.test_string_record.
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Brian, On 23 February 2014 20:14, Brian Kearns <bdkearns@gmail.com> wrote:
I believe all of them in micronumpy are at risk -- any type could be a subtype of a record type consisting of something like a single char followed by whatever type.
I agree about the ones in the types.py module, but there are more in the arrayimpl/ subdirectory. Can each and every one also access a record? It's the kind of question I'm not sure about. A bientôt, Armin.
Thanks to Arimin and Brian Kearns. They fixed unaligned memory access, now ARM micronumpy no longer crashes during testing. Matti On 02/23/2014 11:43 AM, Armin Rigo wrote:
I've checked in some new code in rawstorage.py on default: raw_storage_{get,set}item_unaligned(). It's regular RPython code that checks if the argument is aligned or not, and if not, it falls back to a memcpy(). It should be relatively jit-friendly. Note that I also changed the non-unaligned functions so that, when untranslated, they complain if the address is unaligned, whatever the platform.
Can you try to change some or all of the raw_storage_{get,set}item() calls in micronumpy? I don't really know which ones are at a risk of seeing unaligned accesses, and which ones not. It seems that all of them are at risk: see the failure in test_numarray.AppTestRecordDtype.test_string_record.
A bientôt,
Armin.
participants (4)
-
Armin Rigo -
Brian Kearns -
matti picus -
Matti Picus