[pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++

[Niklas B]

Hi,

I’ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I’ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime.

The code I’m trying to build is:

https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37 <https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37> (older branch, better luck in building it)

Which is basically:

PyDateTime_CAPI* datetime_api = nullptr;

void InitDatetime() {
  PyAcquireGIL lock;
  datetime_api =
      reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0));
  if (datetime_api == nullptr) {
    Py_FatalError("Could not import datetime C API");
  }
}

I’ve tried about a million different ways, but I’m way outside my comfort zone :) I can get it to build by doing:

datetime_api = PyDateTimeAPI;

And also:

datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime", 0));

And:

datetime_api = reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import("datetime.datetime_CAPI", 0));

But both of these trigger the fatal error in the code after (“could not import date time C API” or "PyCapsule_Import "datetime" is not valid” or module 'datetime' has no attribute 'datetime_CAPI')

I will be posting reproducible builds once I get them working. 

I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run:

Import pandas
d = pandas.read_parquet(‘file.parq’)

Obviously that’s not enough money to cover things but at least it’s something :) obviously all results and builds will be public

Regards,
Niklas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20200908/af4d912c/attachment.html>