Re: [Cython] Bringing Cython and PyPy closer together
[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32:
The current state of the discussion seems to be that PyPy provides ways to talk to C code, but nothing as complete as CPython's C-API in the sense that it allows efficient two-way communication between C code and Python objects. Thus, we need to either improve this or look for alternatives.
In order to get us more focussed on what can be done and what the implications are, so that we may eventually be able to decide what should be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement Proposal).
The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan
Stefan Behnel, 18.02.2012 09:54:
Stefan Behnel, 15.02.2012 12:32:
The current state of the discussion seems to be that PyPy provides ways to talk to C code, but nothing as complete as CPython's C-API in the sense that it allows efficient two-way communication between C code and Python objects. Thus, we need to either improve this or look for alternatives.
In order to get us more focussed on what can be done and what the implications are, so that we may eventually be able to decide what should be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement Proposal).
The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not.
Update: Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run (although with some clear bugs in the exception handling code). There is a Jenkins job now to (try to) run the test suite of my own branch in the latest PyPy nightly build: https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scod... It currently crashes rather badly at some point, but at least it looks like it's actually getting somewhere. Stefan
On 18 February 2012 at 19:11, Stefan Behnel <stefan_ml@behnel.de> wrote:
Stefan Behnel, 18.02.2012 09:54:
Stefan Behnel, 15.02.2012 12:32:
The current state of the discussion seems to be that PyPy provides ways to talk to C code, but nothing as complete as CPython's C-API in the sense that it allows efficient two-way communication between C code and Python objects. Thus, we need to either improve this or look for alternatives.
In order to get us more focussed on what can be done and what the implications are, so that we may eventually be able to decide what should be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement Proposal).
The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not.
Update:
Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run (although with some clear bugs in the exception handling code). There is a Jenkins job now to (try to) run the test suite of my own branch in the latest PyPy nightly build:
https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scod...
It currently crashes rather badly at some point, but at least it looks like it's actually getting somewhere.
One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime. This issue will be eventually fixed, I hope. Unce that happens, how can we know it is save to use the call for that pypy version and upwards? I mean, Cython should still support previous PyPy releases... -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Lisandro Dalcin schrieb am 29.03.2015 um 12:23:
One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime. This issue will be eventually fixed, I hope. Unce that happens, how can we know it is save to use the call for that pypy version and upwards? I mean, Cython should still support previous PyPy releases...
Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX should still change sometimes over PyPy releases, so that might at least help a bit. In general, however, you shouldn't expect any CPython internals to work in PyPy. Avoiding multiple C-API calls when a generic one exists is really the best of all strategies. For example, why read just the sign, when you can read the whole value? Everything else just risks aggregating slowness and running into buggy corners. That being said, if you find anything that can be optimised, I'm happy to learn about your changes. I think the general problem is that the PyPy developers loudly push for cffi adoption, and improving anything in cpyext appears to undermine that goal for them. So the current state of cpyext is a result of political decisions, general dislikes and the usual OSS project lack of workpower. Given all that, it actually works quite well. :) Stefan
On 29 March 2015 at 14:16, Stefan Behnel <stefan_ml@behnel.de> wrote:
Lisandro Dalcin schrieb am 29.03.2015 um 12:23:
One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime. This issue will be eventually fixed, I hope. Unce that happens, how can we know it is save to use the call for that pypy version and upwards? I mean, Cython should still support previous PyPy releases...
Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX should still change sometimes over PyPy releases, so that might at least help a bit.
Well, you know that's not enough ;-)
In general, however, you shouldn't expect any CPython internals to work in PyPy. Avoiding multiple C-API calls when a generic one exists is really the best of all strategies. For example, why read just the sign, when you can read the whole value? Everything else just risks aggregating slowness and running into buggy corners.
Well, I agree, however I do require _PyLong_Sign to work if I want to report correct overflow error messages when converting PyLong values to a C integral type. In CPython, we are using Py_SIZE for that, not only for speed, but also to figure out the right overflow kind when converting to unsigned integers.
That being said, if you find anything that can be optimised, I'm happy to learn about your changes.
Again, this is not about optimization, but proper error reporting. diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c index 4cd8229..fbac649 100644 --- a/Cython/Utility/TypeConversion.c +++ b/Cython/Utility/TypeConversion.c @@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}} {{FROM_PY_FUNCTION}}(PyObject *x) { if (unlikely(Py_SIZE(x) < 0)) { goto raise_neg_overflow; } +#elif CYTHON_COMPILING_IN_PYPY + if (unlikely(_PyLong_Sign(x) < 0)) { + goto raise_neg_overflow; + } #endif if (sizeof({{TYPE}}) <= sizeof(unsigned long)) { __PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long, PyLong_AsUnsignedLong(x)) -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Lisandro Dalcin schrieb am 29.03.2015 um 14:52:
Lisandro Dalcin schrieb am 29.03.2015 um 12:23:
One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime.
however I do require _PyLong_Sign to work if I want to report correct overflow error messages when converting PyLong values to a C integral type. In CPython, we are using Py_SIZE for that, not only for speed, but also to figure out the right overflow kind when converting to unsigned integers.
diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c index 4cd8229..fbac649 100644 --- a/Cython/Utility/TypeConversion.c +++ b/Cython/Utility/TypeConversion.c @@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}} {{FROM_PY_FUNCTION}}(PyObject *x) { if (unlikely(Py_SIZE(x) < 0)) { goto raise_neg_overflow; } +#elif CYTHON_COMPILING_IN_PYPY + if (unlikely(_PyLong_Sign(x) < 0)) { + goto raise_neg_overflow; + } #endif if (sizeof({{TYPE}}) <= sizeof(unsigned long)) { __PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long, PyLong_AsUnsignedLong(x))
Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0? Stefan
On 29 March 2015 at 16:10, Stefan Behnel <stefan_ml@behnel.de> wrote:
Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0?
That should definitely work. Let me try to put a patch together. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
On 29 March 2015 at 20:30, Lisandro Dalcin <dalcinl@gmail.com> wrote:
On 29 March 2015 at 16:10, Stefan Behnel <stefan_ml@behnel.de> wrote:
Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0?
That should definitely work. Let me try to put a patch together.
Stefan, how to you feel about abusing of Py_False in the call to RichCompareBool ? -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Lisandro Dalcin schrieb am 29.03.2015 um 19:40:
On 29 March 2015 at 20:30, Lisandro Dalcin wrote:
On 29 March 2015 at 16:10, Stefan Behnel wrote:
Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0?
That should definitely work. Let me try to put a patch together.
Stefan, how to you feel about abusing of Py_False in the call to RichCompareBool ?
Wow - evil idea! :) But then, it's only for cpyext - as long as it works there and comes with a comment, it's certainly as simple as it gets. bool is clearly defined as an int subtype in Python. Just remember to check the error code. Py_SIZE() can't fail in CPython, but comparisons can, sadly. Stefan
On 29 March 2015 at 21:02, Stefan Behnel <stefan_ml@behnel.de> wrote:
Lisandro Dalcin schrieb am 29.03.2015 um 19:40:
On 29 March 2015 at 20:30, Lisandro Dalcin wrote:
On 29 March 2015 at 16:10, Stefan Behnel wrote:
Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0?
That should definitely work. Let me try to put a patch together.
Stefan, how to you feel about abusing of Py_False in the call to RichCompareBool ?
Wow - evil idea! :)
But then, it's only for cpyext - as long as it works there and comes with a comment, it's certainly as simple as it gets. bool is clearly defined as an int subtype in Python.
Just remember to check the error code. Py_SIZE() can't fail in CPython, but comparisons can, sadly.
Please review: https://github.com/cython/cython/pull/377 ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
participants (2)
-
Lisandro Dalcin -
Stefan Behnel