[pypy-dev] pickle numpy array from pypy to cpython?

[Eli Stevens (Gmail)]

Any thoughts on if this approach is acceptable? Happy to incorporate feedback.

I wouldn't be surprised if there are more functions than just
_reconstruct that will need to be special cased, but without a
concrete use case I wasn't going to complicate things.

Thanks,
Eli

On Sat, Jun 25, 2016 at 8:19 PM, Eli Stevens (Gmail)
<wickedgrey at gmail.com> wrote:
> That did the trick.
>
> Pull request here:
> https://bitbucket.org/pypy/pypy/pull-requests/460/changes-reported-location-of-_reconstruct/diff
>
> Please let me know if there are changes that should be made. As noted,
> I'm not super happy with the tests, but am unsure what direction I
> should go with them.
>
> Cheers,
> Eli
>
> On Sat, Jun 25, 2016 at 10:26 AM, Matti Picus <matti.picus at gmail.com> wrote:
>> You need to add the modules to those that the class-local space is built
>> with using a spaceconfig, so something like
>>
>> class AppTestPicklingNumpy(BaseNumpyAppTest):
>>     spaceconfig = dict(usemodules=["micronumpy", "struct", "binascii"])
>>
>>     def setup_class(cls):
>>         if option.runappdirect and '__pypy__' not in
>> sys.builtin_module_names:
>>             py.test.skip("pypy only test")
>>         BaseNumpyAppTest.setup_class.im_func(cls)
>>
>>
>>     def test_pickle_module(self):
>>         import pickle
>>
>>
>>
>>
>> On 25/06/16 09:01, Eli Stevens (Gmail) wrote:
>>>
>>> I was thinking about doing it on import of the micronumpy module
>>> (pypy/module/micronumpy/app_numpy.py).
>>>
>>> Right now, when I try and import pickle during the tests:
>>>
>>> $ cat pypy/module/micronumpy/test/test_pickling_app.py
>>> import sys
>>> import py
>>>
>>> from pypy.module.micronumpy.test.test_base import BaseNumpyAppTest
>>> from pypy.conftest import option
>>>
>>> class AppTestPicklingNumpy(BaseNumpyAppTest):
>>>      def setup_class(cls):
>>>          if option.runappdirect and '__pypy__' not in
>>> sys.builtin_module_names:
>>>              py.test.skip("pypy only test")
>>>          BaseNumpyAppTest.setup_class.im_func(cls)
>>>
>>>      def test_pickle_module(self):
>>>          import pickle
>>>          ... # more code
>>>
>>>
>>>   I get this error:
>>>
>>>>    import struct
>>>
>>> lib-python/2.7/pickle.py:34:
>>> _ _ _ _ _ _
>>>
>>>>    from _struct import *
>>>
>>> E   (application-level) ImportError: No module named _struct
>>>
>>> lib-python/2.7/struct.py:1: ImportError
>>>
>>>
>>> But everything seems fine with struct:
>>>
>>> $ ./pytest.py pypy/module/struct/test/test_struct.py
>>> ==== test session starts ====
>>> platform linux2 -- Python 2.7.11 -- py-1.4.20 -- pytest-2.5.2
>>> pytest-2.5.2 from /home/elis/edit/play/pypy/pytest.pyc
>>> collected 30 items
>>>
>>> pypy/module/struct/test/test_struct.py ..............................
>>>
>>> ==== 30 passed in 11.95 seconds ====
>>>
>>> Any idea what's going on here?
>>>
>>> Thanks,
>>> Eli
>>>
>>>
>>> On Fri, Jun 24, 2016 at 9:19 PM, matti picus <matti.picus at gmail.com>
>>> wrote:
>>>>
>>>> Sounds reasonable. You might want to generalize it a bit by trying to
>>>> import
>>>> _numpypy /numpy, and setting up the replacement by whichever fails to
>>>> import.
>>>> Matti
>>>>
>>>> On Saturday, 25 June 2016, Eli Stevens (Gmail) <wickedgrey at gmail.com>
>>>> wrote:
>>>>>
>>>>> Heh, interestingly, if I add the following to the local dir and files
>>>>> when trying to unpickle under cpython, it works (note that cpython to
>>>>> pypy actually works out of the box, which I hadn't realized):
>>>>>
>>>>> $ cat _numpypy/__init__.py
>>>>> from numpy.core import *
>>>>>
>>>>> $ cat _numpypy/multiarray.py
>>>>> from numpy.core.multiarray import *
>>>>> import numpy.core.multiarray as _ncm
>>>>> _reconstruct = _ncm._reconstruct
>>>>>
>>>>> This is obviously a total hack, and not one I'm comfortable with
>>>>> (since I need to use this codebase from both cpython and pypy), but it
>>>>> demonstrates that it's just bookkeeping that needs to change to get
>>>>> things to work.
>>>>>
>>>>> My first approach would be to add a wrapper around save_global here
>>>>>
>>>>>
>>>>> https://bitbucket.org/pypy/pypy/src/a0105e0d00dbd0f73d06fc704db704868a6c6ed2/lib-python/2.7/pickle.py?at=default&fileviewer=file-view-default#pickle.py-814
>>>>> that special-cases the global '_numpypy.multiarray' to instead be
>>>>> 'numpy.core.multiarray'. That seem like a reasonable thing to do?
>>>>>
>>>>> Cheers,
>>>>> Eli
>>>>>
>>