[pypy-dev] pickle numpy array from pypy to cpython?

[Eli Stevens (Gmail)]

That did the trick.

Pull request here:
https://bitbucket.org/pypy/pypy/pull-requests/460/changes-reported-location-of-_reconstruct/diff

Please let me know if there are changes that should be made. As noted,
I'm not super happy with the tests, but am unsure what direction I
should go with them.

Cheers,
Eli

On Sat, Jun 25, 2016 at 10:26 AM, Matti Picus <matti.picus at gmail.com> wrote:
> You need to add the modules to those that the class-local space is built
> with using a spaceconfig, so something like
>
> class AppTestPicklingNumpy(BaseNumpyAppTest):
>     spaceconfig = dict(usemodules=["micronumpy", "struct", "binascii"])
>
>     def setup_class(cls):
>         if option.runappdirect and '__pypy__' not in
> sys.builtin_module_names:
>             py.test.skip("pypy only test")
>         BaseNumpyAppTest.setup_class.im_func(cls)
>
>
>     def test_pickle_module(self):
>         import pickle
>
>
>
>
> On 25/06/16 09:01, Eli Stevens (Gmail) wrote:
>>
>> I was thinking about doing it on import of the micronumpy module
>> (pypy/module/micronumpy/app_numpy.py).
>>
>> Right now, when I try and import pickle during the tests:
>>
>> $ cat pypy/module/micronumpy/test/test_pickling_app.py
>> import sys
>> import py
>>
>> from pypy.module.micronumpy.test.test_base import BaseNumpyAppTest
>> from pypy.conftest import option
>>
>> class AppTestPicklingNumpy(BaseNumpyAppTest):
>>      def setup_class(cls):
>>          if option.runappdirect and '__pypy__' not in
>> sys.builtin_module_names:
>>              py.test.skip("pypy only test")
>>          BaseNumpyAppTest.setup_class.im_func(cls)
>>
>>      def test_pickle_module(self):
>>          import pickle
>>          ... # more code
>>
>>
>>   I get this error:
>>
>>>    import struct
>>
>> lib-python/2.7/pickle.py:34:
>> _ _ _ _ _ _
>>
>>>    from _struct import *
>>
>> E   (application-level) ImportError: No module named _struct
>>
>> lib-python/2.7/struct.py:1: ImportError
>>
>>
>> But everything seems fine with struct:
>>
>> $ ./pytest.py pypy/module/struct/test/test_struct.py
>> ==== test session starts ====
>> platform linux2 -- Python 2.7.11 -- py-1.4.20 -- pytest-2.5.2
>> pytest-2.5.2 from /home/elis/edit/play/pypy/pytest.pyc
>> collected 30 items
>>
>> pypy/module/struct/test/test_struct.py ..............................
>>
>> ==== 30 passed in 11.95 seconds ====
>>
>> Any idea what's going on here?
>>
>> Thanks,
>> Eli
>>
>>
>> On Fri, Jun 24, 2016 at 9:19 PM, matti picus <matti.picus at gmail.com>
>> wrote:
>>>
>>> Sounds reasonable. You might want to generalize it a bit by trying to
>>> import
>>> _numpypy /numpy, and setting up the replacement by whichever fails to
>>> import.
>>> Matti
>>>
>>> On Saturday, 25 June 2016, Eli Stevens (Gmail) <wickedgrey at gmail.com>
>>> wrote:
>>>>
>>>> Heh, interestingly, if I add the following to the local dir and files
>>>> when trying to unpickle under cpython, it works (note that cpython to
>>>> pypy actually works out of the box, which I hadn't realized):
>>>>
>>>> $ cat _numpypy/__init__.py
>>>> from numpy.core import *
>>>>
>>>> $ cat _numpypy/multiarray.py
>>>> from numpy.core.multiarray import *
>>>> import numpy.core.multiarray as _ncm
>>>> _reconstruct = _ncm._reconstruct
>>>>
>>>> This is obviously a total hack, and not one I'm comfortable with
>>>> (since I need to use this codebase from both cpython and pypy), but it
>>>> demonstrates that it's just bookkeeping that needs to change to get
>>>> things to work.
>>>>
>>>> My first approach would be to add a wrapper around save_global here
>>>>
>>>>
>>>> https://bitbucket.org/pypy/pypy/src/a0105e0d00dbd0f73d06fc704db704868a6c6ed2/lib-python/2.7/pickle.py?at=default&fileviewer=file-view-default#pickle.py-814
>>>> that special-cases the global '_numpypy.multiarray' to instead be
>>>> 'numpy.core.multiarray'. That seem like a reasonable thing to do?
>>>>
>>>> Cheers,
>>>> Eli
>>>>
>