PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++
Hi, I’ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I’ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime. The code I’m trying to build is: https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datet... <https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datet...> (older branch, better luck in building it) Which is basically: PyDateTime_CAPI* datetime_api = nullptr; void InitDatetime() { PyAcquireGIL lock; datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); if (datetime_api == nullptr) { Py_FatalError("Could not import datetime C API"); } } I’ve tried about a million different ways, but I’m way outside my comfort zone :) I can get it to build by doing: datetime_api = PyDateTimeAPI; And also: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime", 0)); And: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime.datetime_CAPI", 0)); But both of these trigger the fatal error in the code after (“could not import date time C API” or "PyCapsule_Import "datetime" is not valid” or module 'datetime' has no attribute 'datetime_CAPI') I will be posting reproducible builds once I get them working. I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run: Import pandas d = pandas.read_parquet(‘file.parq’) Obviously that’s not enough money to cover things but at least it’s something :) obviously all results and builds will be public Regards, Niklas
Have you tried the documented interface https://docs.python.org/3.6/c-api/datetime.html which is to dispense with all that code and use #import <datetime.h> |PyDateTime_IMPORT| which does the right thing on each implementation (CPython - does the PyCapsule_Import; PyPy - calls _PyDateTime_Import() ) A donation to https://opencollective.com/pypy is always appreciated. Matti On 9/9/20 12:12 AM, Niklas B wrote:
Thank you, I managed to get it built using an older version (instructions and whl file for those who are interested is available at https://github.com/bivald/pyarrow-on-pypy3 <https://github.com/bivald/pyarrow-on-pypy3>) I have setup a monthly 50USD recurring donation (under “Enplore”)
On 9/9/20 3:08 PM, Niklas B wrote:
Cool. It would be nice to backport the datetime changes, I think you could put in a #ifdef PYPY_VERSION macro and add the code I pasted. I'll se what I can do about PyMemoryView_GetContiguous. Are there more missing C-API functions? Thanks for the donation(s)! Matti
I will try, I tried with the 2020-09-10 (https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2 <https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2>) but your changes might have not made it into that one. The errors I get is: https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... <https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python...> In file included from /arrow/cpp/src/arrow/python/common.cc:18: /arrow/cpp/src/arrow/python/common.h: In member function ‘arrow::Status arrow::py::PyBytesView::FromBinary(PyObject*, const char*)’: /arrow/cpp/src/arrow/python/common.h:256:31: error: ‘PyMemoryView_GetContiguous’ was not declared in this scope PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/common.h:256:31: note: suggested alternative: ‘PyMemoryView_FromMemory’ PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ PyMemoryView_FromMemory Or I might need to patch something :) Odd thing is that running it on the latest code I don’t get any errors on date time, only on above. That said, it might simply be that the errors are “below” the above errors I did a new branch for latest: https://github.com/bivald/pyarrow-on-pypy3/blob/feature/latest-pypy-latest-p...
The PyMemoryView_GetContiguous errors are all gone - good work! It didn’t really like my butchering of datetime.cc https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... Added: #include <datetime.h> And PyDateTime_IMPORT Then changed: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); To: datetime_api = PyDateTimeAPI; They do some PyDateTimeAPI voodoo at https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... <https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python...> which might be a culprit as well. Gives the following outputs: /arrow/cpp/src/arrow/python/datetime.h:34: warning: "PyDateTimeAPI" redefined #define PyDateTimeAPI ::arrow::py::internal::datetime_api In file included from /opt/pypy/include/Python.h:144, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/pypy_decl.h:1121: note: this is the location of the previous definition #define PyDateTimeAPI PyPyDateTimeAPI In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:4:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DateType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:5:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DateTimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:6:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *TimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:7:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DeltaType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:8:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *TZInfoType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:11:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:12:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int, ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:14:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:15:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:18:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:19:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Date_FromTimestamp)(PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:24:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:24:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:35:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:35:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:37:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:42:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:42:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:44:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:49:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:49:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:54:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:54:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/datetime.h:9:30: error: expected constructor, destructor, or type conversion before ‘PyDateTimeAPI’ PyAPI_DATA(PyDateTime_CAPI*) PyDateTimeAPI; ^~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:37:1: error: expected ‘)’ before ‘=’ token PyDateTime_IMPORT ^~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘Time_FromTime’ *out = PyTime_FromTime(static_cast<int32_t>(hour), static_cast<int32_t>(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘TimeType’ *out = PyTime_FromTime(static_cast<int32_t>(hour), static_cast<int32_t>(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyDate_from_int(int64_t, arrow::DateUnit, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘Date_FromDate’ *out = PyDate_FromDate(static_cast<int32_t>(year), static_cast<int32_t>(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateType’ *out = PyDate_FromDate(static_cast<int32_t>(year), static_cast<int32_t>(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyDateTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateTime_FromDateAndTime’ *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateTimeType’ *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: *** [src/arrow/python/CMakeFiles/arrow_python_objlib.dir/build.make:121: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/datetime.cc.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:1628: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/all] Error
Thse changes seems to compile for me, but I did not run the tests. https://gist.github.com/mattip/c9c8398b58721ae5893dc8134c353f28 Matti On 9/11/20 1:01 PM, Niklas B wrote:
Yep, I can confirm your patch builds and works for parquet the very least (https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-p... <https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-p...>) I would say it works surprisingly well for parquet: ======================================================================================= test session starts ======================================================================================== platform linux -- Python 3.6.9[pypy-7.3.3-alpha], pytest-6.0.2, py-1.9.0, pluggy-0.13.1 rootdir: /arrow/python, configfile: setup.cfg plugins: hypothesis-5.35.1 collected 286 items pyarrow2/tests/test_parquet.py .s.s.s.s..s.s.s.s.ss.s.s.s.s.s.s..s.s.s.s.s.s.s.s.s.s..s.s..s.s.................................s.s...........s..s.s.sxs.s.s.sxs.sxsss....s.s.s.s....s.s.s.s. [ 54%] s.sxsxs.s...s.s.s.s.s.s..s.ssssssss.s.s.s.s.s.s.s.sx.s.s.s....s..s.s..s.s.s..s.s.s.s..s.s...ss..s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.sss [100%] =========================================================================== 164 passed, 116 skipped, 6 xfailed in 43.69s =========================================================================== Overall the entire test suite: ====================================================================================================== short test summary info ====================================================================================================== FAILED pyarrow2/tests/test_array.py::test_to_pandas_zero_copy - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_slice - SystemError: Function returned an error result without setting an exception FAILED pyarrow2/tests/test_array.py::test_array_ref_to_ndarray_base - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_conversions_no_sentinel_values - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_nbytes_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_cffi.py::test_export_import_array - assert 1528 == 896 FAILED pyarrow2/tests/test_cffi.py::test_export_import_batch - assert 1048 == 128 FAILED pyarrow2/tests/test_convert_builtin.py::test_garbage_collection - assert 128 == 766912 FAILED pyarrow2/tests/test_convert_builtin.py::test_sequence_bytes - NotImplementedError: creating contiguous readonly buffer from non-contiguous not implemented yet FAILED pyarrow2/tests/test_convert_builtin.py::test_map_from_dicts - AssertionError: Regex pattern 'integer is required' does not match 'expected integer, got str object'. FAILED pyarrow2/tests/test_csv.py::test_read_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::test_parse_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::test_convert_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::TestSerialStreamingCSVRead::test_batch_lifetime - AssertionError: assert 1464704 == 1464576 FAILED pyarrow2/tests/test_cython.py::test_cython_api - subprocess.CalledProcessError: Command '['/pyarrow/bin/pypy3', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1. FAILED pyarrow2/tests/test_extension_type.py::test_ext_type__lifetime - AssertionError: assert UuidType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_extension_type.py::test_uuid_type_pickle - AssertionError: assert UuidType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_extension_type.py::test_ext_array_lifetime - AssertionError: assert ParamExtType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_fs.py::test_py_filesystem_lifetime - AssertionError: assert <pyarrow2.tests.test_fs.DummyHandler object at 0x0000000003e696a8> is None FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_deduplicate_integers_as_objects - assert 100 == 991 FAILED pyarrow2/tests/test_pandas.py::test_array_uses_memory_pool - assert 103552 == 465152 FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_self_destruct - assert 6112064 == 4112064 FAILED pyarrow2/tests/test_pandas.py::test_table_uses_memory_pool - assert 6249408 == 6112064 FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_numpy_array - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_dataframe - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_schema.py::test_schema_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_coo_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csr_matrix_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csf_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_table.py::test_chunked_array_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_recordbatch_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_table_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_tensor.py::test_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' ========================================================================= 33 failed, 2620 passed, 532 skipped, 13 xfailed, 10 warnings in 104.02s (0:01:44) ========================================================================= Segfaults on a couple of the tests, but a lot passes as well
Have you tried the documented interface https://docs.python.org/3.6/c-api/datetime.html which is to dispense with all that code and use #import <datetime.h> |PyDateTime_IMPORT| which does the right thing on each implementation (CPython - does the PyCapsule_Import; PyPy - calls _PyDateTime_Import() ) A donation to https://opencollective.com/pypy is always appreciated. Matti On 9/9/20 12:12 AM, Niklas B wrote:
Thank you, I managed to get it built using an older version (instructions and whl file for those who are interested is available at https://github.com/bivald/pyarrow-on-pypy3 <https://github.com/bivald/pyarrow-on-pypy3>) I have setup a monthly 50USD recurring donation (under “Enplore”)
On 9/9/20 3:08 PM, Niklas B wrote:
Cool. It would be nice to backport the datetime changes, I think you could put in a #ifdef PYPY_VERSION macro and add the code I pasted. I'll se what I can do about PyMemoryView_GetContiguous. Are there more missing C-API functions? Thanks for the donation(s)! Matti
I will try, I tried with the 2020-09-10 (https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2 <https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2>) but your changes might have not made it into that one. The errors I get is: https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... <https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python...> In file included from /arrow/cpp/src/arrow/python/common.cc:18: /arrow/cpp/src/arrow/python/common.h: In member function ‘arrow::Status arrow::py::PyBytesView::FromBinary(PyObject*, const char*)’: /arrow/cpp/src/arrow/python/common.h:256:31: error: ‘PyMemoryView_GetContiguous’ was not declared in this scope PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/common.h:256:31: note: suggested alternative: ‘PyMemoryView_FromMemory’ PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ PyMemoryView_FromMemory Or I might need to patch something :) Odd thing is that running it on the latest code I don’t get any errors on date time, only on above. That said, it might simply be that the errors are “below” the above errors I did a new branch for latest: https://github.com/bivald/pyarrow-on-pypy3/blob/feature/latest-pypy-latest-p...
The PyMemoryView_GetContiguous errors are all gone - good work! It didn’t really like my butchering of datetime.cc https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... Added: #include <datetime.h> And PyDateTime_IMPORT Then changed: datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); To: datetime_api = PyDateTimeAPI; They do some PyDateTimeAPI voodoo at https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python... <https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python...> which might be a culprit as well. Gives the following outputs: /arrow/cpp/src/arrow/python/datetime.h:34: warning: "PyDateTimeAPI" redefined #define PyDateTimeAPI ::arrow::py::internal::datetime_api In file included from /opt/pypy/include/Python.h:144, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/pypy_decl.h:1121: note: this is the location of the previous definition #define PyDateTimeAPI PyPyDateTimeAPI In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:4:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DateType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:5:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DateTimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:6:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *TimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:7:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *DeltaType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:8:5: error: ‘PyTypeObject’ does not name a type; did you mean ‘PyType_Check’? PyTypeObject *TZInfoType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:11:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:12:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int, ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:14:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:15:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:18:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:19:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *(*Date_FromTimestamp)(PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:24:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:24:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:35:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:35:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:37:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:42:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:42:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:44:5: error: ‘PyObject’ does not name a type; did you mean ‘PyObject_New’? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:49:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:49:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:54:5: error: ‘PyObject_HEAD’ does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:54:5: note: the macro ‘PyObject_HEAD’ had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/datetime.h:9:30: error: expected constructor, destructor, or type conversion before ‘PyDateTimeAPI’ PyAPI_DATA(PyDateTime_CAPI*) PyDateTimeAPI; ^~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:37:1: error: expected ‘)’ before ‘=’ token PyDateTime_IMPORT ^~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘Time_FromTime’ *out = PyTime_FromTime(static_cast<int32_t>(hour), static_cast<int32_t>(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘TimeType’ *out = PyTime_FromTime(static_cast<int32_t>(hour), static_cast<int32_t>(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyDate_from_int(int64_t, arrow::DateUnit, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘Date_FromDate’ *out = PyDate_FromDate(static_cast<int32_t>(year), static_cast<int32_t>(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateType’ *out = PyDate_FromDate(static_cast<int32_t>(year), static_cast<int32_t>(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ‘arrow::Status arrow::py::internal::PyDateTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)’: /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateTime_FromDateAndTime’ *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ‘struct PyDateTime_CAPI’ has no member named ‘DateTimeType’ *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: *** [src/arrow/python/CMakeFiles/arrow_python_objlib.dir/build.make:121: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/datetime.cc.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:1628: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/all] Error
Thse changes seems to compile for me, but I did not run the tests. https://gist.github.com/mattip/c9c8398b58721ae5893dc8134c353f28 Matti On 9/11/20 1:01 PM, Niklas B wrote:
Yep, I can confirm your patch builds and works for parquet the very least (https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-p... <https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-p...>) I would say it works surprisingly well for parquet: ======================================================================================= test session starts ======================================================================================== platform linux -- Python 3.6.9[pypy-7.3.3-alpha], pytest-6.0.2, py-1.9.0, pluggy-0.13.1 rootdir: /arrow/python, configfile: setup.cfg plugins: hypothesis-5.35.1 collected 286 items pyarrow2/tests/test_parquet.py .s.s.s.s..s.s.s.s.ss.s.s.s.s.s.s..s.s.s.s.s.s.s.s.s.s..s.s..s.s.................................s.s...........s..s.s.sxs.s.s.sxs.sxsss....s.s.s.s....s.s.s.s. [ 54%] s.sxsxs.s...s.s.s.s.s.s..s.ssssssss.s.s.s.s.s.s.s.sx.s.s.s....s..s.s..s.s.s..s.s.s.s..s.s...ss..s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.sss [100%] =========================================================================== 164 passed, 116 skipped, 6 xfailed in 43.69s =========================================================================== Overall the entire test suite: ====================================================================================================== short test summary info ====================================================================================================== FAILED pyarrow2/tests/test_array.py::test_to_pandas_zero_copy - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_slice - SystemError: Function returned an error result without setting an exception FAILED pyarrow2/tests/test_array.py::test_array_ref_to_ndarray_base - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_conversions_no_sentinel_values - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_nbytes_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_cffi.py::test_export_import_array - assert 1528 == 896 FAILED pyarrow2/tests/test_cffi.py::test_export_import_batch - assert 1048 == 128 FAILED pyarrow2/tests/test_convert_builtin.py::test_garbage_collection - assert 128 == 766912 FAILED pyarrow2/tests/test_convert_builtin.py::test_sequence_bytes - NotImplementedError: creating contiguous readonly buffer from non-contiguous not implemented yet FAILED pyarrow2/tests/test_convert_builtin.py::test_map_from_dicts - AssertionError: Regex pattern 'integer is required' does not match 'expected integer, got str object'. FAILED pyarrow2/tests/test_csv.py::test_read_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::test_parse_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::test_convert_options - Failed: DID NOT RAISE <class 'AttributeError'> FAILED pyarrow2/tests/test_csv.py::TestSerialStreamingCSVRead::test_batch_lifetime - AssertionError: assert 1464704 == 1464576 FAILED pyarrow2/tests/test_cython.py::test_cython_api - subprocess.CalledProcessError: Command '['/pyarrow/bin/pypy3', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1. FAILED pyarrow2/tests/test_extension_type.py::test_ext_type__lifetime - AssertionError: assert UuidType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_extension_type.py::test_uuid_type_pickle - AssertionError: assert UuidType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_extension_type.py::test_ext_array_lifetime - AssertionError: assert ParamExtType(extension<arrow.py_extension_type>) is None FAILED pyarrow2/tests/test_fs.py::test_py_filesystem_lifetime - AssertionError: assert <pyarrow2.tests.test_fs.DummyHandler object at 0x0000000003e696a8> is None FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_deduplicate_integers_as_objects - assert 100 == 991 FAILED pyarrow2/tests/test_pandas.py::test_array_uses_memory_pool - assert 103552 == 465152 FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_self_destruct - assert 6112064 == 4112064 FAILED pyarrow2/tests/test_pandas.py::test_table_uses_memory_pool - assert 6249408 == 6112064 FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_numpy_array - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_dataframe - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_schema.py::test_schema_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_coo_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csr_matrix_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csf_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_table.py::test_chunked_array_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_recordbatch_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_table_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_tensor.py::test_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' ========================================================================= 33 failed, 2620 passed, 532 skipped, 13 xfailed, 10 warnings in 104.02s (0:01:44) ========================================================================= Segfaults on a couple of the tests, but a lot passes as well
participants (2)
-
Matti Picus -
Niklas B