From kai.huuhko at gmail.com Fri Mar 6 06:55:45 2015 From: kai.huuhko at gmail.com (Kai Huuhko) Date: Fri, 6 Mar 2015 07:55:45 +0200 Subject: [Cython] PyMethod_New compatibility Message-ID: This change: https://github.com/cython/cython/commit/499d98628a8a929ed065bf38ee4ec6405145e092 removed compatibility code for PyMethod_New to have the same signature with both py2 and py3. Is there an alternative way - that is py2/py3 compatible - within a code block to bind functions to an instance? For reference, we use it here: https://git.enlightenment.org/bindings/python/python-efl.git/tree/efl/evas/efl.evas_object_smart.pxi#n431 From stefan_ml at behnel.de Fri Mar 6 09:54:17 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Mar 2015 09:54:17 +0100 Subject: [Cython] PyMethod_New compatibility In-Reply-To: References: Message-ID: <54F96B39.8000303@behnel.de> [moving to cython-users mailing list] Kai Huuhko schrieb am 06.03.2015 um 06:55: > This change: > https://github.com/cython/cython/commit/499d98628a8a929ed065bf38ee4ec6405145e092 > > removed compatibility code for PyMethod_New to have the same signature > with both py2 and py3. Is there an alternative way - that is py2/py3 > compatible - within a code block to bind functions to an instance? > > For reference, we use it here: > https://git.enlightenment.org/bindings/python/python-efl.git/tree/efl/evas/efl.evas_object_smart.pxi#n431 You can set the compiler directive "binding=True", that will allow Cython functions to bind as methods. Stefan From kai.huuhko at gmail.com Fri Mar 6 12:02:29 2015 From: kai.huuhko at gmail.com (Kai Huuhko) Date: Fri, 6 Mar 2015 03:02:29 -0800 (PST) Subject: [Cython] PyMethod_New compatibility In-Reply-To: <54F96B39.8000303@behnel.de> References: <54F96B39.8000303@behnel.de> Message-ID: <8665fc8a-53fd-45b0-8678-8a466062aa46@googlegroups.com> perjantai 6. maaliskuuta 2015 10.54.23 UTC+2 Stefan Behnel kirjoitti: > > [moving to cython-users mailing list] > > Kai Huuhko schrieb am 06.03.2015 um 06:55: > > This change: > > > https://github.com/cython/cython/commit/499d98628a8a929ed065bf38ee4ec6405145e092 > > > > removed compatibility code for PyMethod_New to have the same signature > > with both py2 and py3. Is there an alternative way - that is py2/py3 > > compatible - within a code block to bind functions to an instance? > > > > For reference, we use it here: > > > https://git.enlightenment.org/bindings/python/python-efl.git/tree/efl/evas/efl.evas_object_smart.pxi#n431 > > You can set the compiler directive "binding=True", that will allow Cython > functions to bind as methods. > > Stefan > > The code is from before my time and there are no code comments so I can only guess what the original author(s) were thinking. The __cinit__ there is binding methods from other extension classes, which doesn't work using this new Cython function binding mechanism. I think it's doing the binding there needlessly though, since the methods are from the parent class and the class itself(!). Why it's not doing simple class inheritance, is beyond me. Time to dig through the revision control history for clues. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Sun Mar 8 20:25:01 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 8 Mar 2015 22:25:01 +0300 Subject: [Cython] PyPy3 fixes Message-ID: 1) Stefan, I had to push this fix, just to be sure, please double check. https://github.com/cython/cython/commit/6bd3f7b9e494d1259082aecfc0366da15fc105ec 2) PyPy3 defines some legacy (Py2) Py_TPFLAGS_XXX values. Do you like the following fix? IOW, I check if the value is #define'd, if not I #define it to 0 (zero): diff --git a/Cython/Utility/ModuleSetupCode.c b/Cython/Utility/ModuleSetupCode.c index 6477fb2..3ae8a0c 100644 --- a/Cython/Utility/ModuleSetupCode.c +++ b/Cython/Utility/ModuleSetupCode.c @@ -59,13 +59,16 @@ #define __Pyx_DefaultClassType PyType_Type #endif -#if PY_MAJOR_VERSION >= 3 +#if !defined(Py_TPFLAGS_CHECKTYPES) #define Py_TPFLAGS_CHECKTYPES 0 +#endif +#if !defined(Py_TPFLAGS_HAVE_INDEX) #define Py_TPFLAGS_HAVE_INDEX 0 +#endif +#if !defined(Py_TPFLAGS_HAVE_NEWBUFFER) #define Py_TPFLAGS_HAVE_NEWBUFFER 0 #endif - -#if PY_VERSION_HEX < 0x030400a1 && !defined(Py_TPFLAGS_HAVE_FINALIZE) +#if !defined(Py_TPFLAGS_HAVE_FINALIZE) #define Py_TPFLAGS_HAVE_FINALIZE 0 #endif -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Tue Mar 10 15:28:36 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2015 17:28:36 +0300 Subject: [Cython] Py3, TypeConversion.c, PyLong internals Message-ID: Robert, please look at this line: https://github.com/cython/cython/blob/0.22.x/Cython/Utility/TypeConversion.c#L559 You are still using a PyLong internal detail to test for signedness. I'm getting a strange error with PyPy3 that might be related to this, see this traceback: https://travis-ci.org/mpi4py/mpi4py/jobs/53793609#L5650 The code generating this traceback is linked below, the__setitem__method of the variable "m" is coded to only accept "usigned char" values. https://bitbucket.org/mpi4py/mpi4py/src/86e4235ee2ec31832031ce020421810c51d4910b/test/test_rma.py?at=master#cl-13 Any suggestions about how to check signedness in Python 3 (CPython and PyPy) without using Py_SIZE() ? -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From njs at vorpus.org Sat Mar 14 09:17:56 2015 From: njs at vorpus.org (Nathaniel Smith) Date: Sat, 14 Mar 2015 01:17:56 -0700 Subject: [Cython] Cython inserting unqualified module name into sys.module on python 3? Message-ID: Hi all, Can anyone shed any light on this? https://github.com/numpy/numpy/issues/5680 -n -- Nathaniel J. Smith -- http://vorpus.org From robertwb at gmail.com Sat Mar 14 22:03:01 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 14 Mar 2015 14:03:01 -0700 Subject: [Cython] Cython inserting unqualified module name into sys.module on python 3? In-Reply-To: References: Message-ID: That is strange, looks like it was an attempt to support relative imports? https://github.com/cython/cython/blob/384cc660f5c7958524b8839ba24099fdbc6eaffd/Cython/Compiler/ModuleNode.py#L2271 On Sat, Mar 14, 2015 at 1:17 AM, Nathaniel Smith wrote: > Hi all, > > Can anyone shed any light on this? > > https://github.com/numpy/numpy/issues/5680 > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From njs at vorpus.org Sat Mar 14 23:48:53 2015 From: njs at vorpus.org (Nathaniel Smith) Date: Sat, 14 Mar 2015 15:48:53 -0700 Subject: [Cython] Cython inserting unqualified module name into sys.module on python 3? In-Reply-To: References: Message-ID: On Mar 14, 2015 2:03 PM, "Robert Bradshaw" wrote: > > That is strange, looks like it was an attempt to support relative imports? > > https://github.com/cython/cython/blob/384cc660f5c7958524b8839ba24099fdbc6eaffd/Cython/Compiler/ModuleNode.py#L2271 Ah, I see. I don't see how this could affect relative imports, because if foo.bar does 'import .baz', this doesn't actually trigger any access to sys.modules["foo.bar"]. (Exception: if you have foo/__init__.pyx. Is that actually supported?) The critical thing for relative imports is having correct __name__ and/or __package__ module-level global variables, and AFAICT cython is not currently doing anything to set these up. But it probably should, because relative imports are a thing. OTOH, putting the module into sys.modules *is* crucial to handle recursive imports, i.e. where foo.pyx's module init function imports bar.py, and bar.py imports foo. For regular python modules or for python 2 extension modules, this works because even while foo is still initializing, you can already get its (incomplete) module object from sys.modules; for python 3 extension modules this won't work unless we do it by hand. So the code that Cython is generating seems to be correct and necessary, it just has the wrong idea about what the fully-qualified module name is, and this breaks things. So I'm convinced that Cython has to know the fully-qualified module name for correct operation, and if it's wrong then weird real bugs will happen. Next question: how am I supposed to make this work? Maybe I'm just missing it, but I can't find anything in the docs about how I should tell cython that mtrand.pyx is really numpy.random.mtrand...? -n > On Sat, Mar 14, 2015 at 1:17 AM, Nathaniel Smith wrote: > > Hi all, > > > > Can anyone shed any light on this? > > > > https://github.com/numpy/numpy/issues/5680 > > > > -n > > > > -- > > Nathaniel J. Smith -- http://vorpus.org > > _______________________________________________ > > cython-devel mailing list > > cython-devel at python.org > > https://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From dalcinl at gmail.com Tue Mar 17 21:10:34 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 17 Mar 2015 23:10:34 +0300 Subject: [Cython] Sorry: 0.22.x and master broken Message-ID: Folks, I'm very sorry. In an attempt to fix issues with PyPy3, I pushed a fix to 0.22.x (https://github.com/cython/cython/commit/8573e7898a27ef607a3264909cc3bf24d4e13959) and merged to master, but now one test is failing with CPython 3 (https://travis-ci.org/cython/cython/jobs/54762494). I did not run the full test suite with python3, my fault. I'll look at this issue tomorrow and push a fix ASAP. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From jdemeyer at cage.ugent.be Sat Mar 7 09:58:06 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Sat, 07 Mar 2015 09:58:06 +0100 Subject: [Cython] Proposal: extern implementations of functions Message-ID: <54FABD9E.4030501@cage.ugent.be> Hello Cython developers, I would like to propose a new mechanism to implement functions in an external .c file. The goal is to solve the problem I was having in https://groups.google.com/forum/#!topic/cython-users/TsbBNyvwZn4 With the attached patch, you can do the following: * in foo.pxd, declare the function as usual: cdef void myfunc() * in foo.c, actually implement the function: static void myfunc() { /* ... */ } * in foo.pyx, declare the function extern: cdef extern from "foo.c": void myfunc() With my patch, the extern declaration in foo.pyx will be interpreted as implementing the function. Therefore, myfunc() is now available in the module foo, and it can also be cimported by other modules. What do you think of this proposed feature? Jeroen. -------------- next part -------------- A non-text attachment was scrubbed... Name: extern_implementation.patch Type: text/x-patch Size: 1221 bytes Desc: not available URL: From nathan12343 at gmail.com Wed Mar 11 06:27:18 2015 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Tue, 10 Mar 2015 22:27:18 -0700 Subject: [Cython] Cython magic annotate option is broken under IPython 3.0 Message-ID: Hi all, To reproduce this issue, be on cython 0.22 and IPython 3.0 and run the following notebook: http://nbviewer.ipython.org/gist/ngoldbaum/855a629d997aa7959254 Googling the error returns some several year old discussions in IPython related to the autoreload magic and I'm not sure if that is a red herring here: http://mail.scipy.org/pipermail/ipython-dev/2012-May/009288.html For how I'm working around this by calling cython at the command line in my notebook and then loading the annotated HTML with an IFrame. Thanks for your help! -------------- next part -------------- An HTML attachment was scrubbed... URL: From grunwald at axivion.com Tue Mar 17 18:24:26 2015 From: grunwald at axivion.com (Daniel Grunwald) Date: Tue, 17 Mar 2015 18:24:26 +0100 Subject: [Cython] Generated function has invalid name when converting from python object to C struct defined in C++ namespace Message-ID: <5508634A.4050201@axivion.com> Hello, I have Cython code like this: cdef extern from "cpp_library.h" namespace "CppLibrary": struct SomeStruct: int i void do_with_struct(SomeStruct s) def run(): do_with_struct(object()) With cython 0.21.1, invalid C++ code is generated: struct CppLibrary::SomeStruct; static struct CppLibrary::SomeStruct __pyx_convert__from_py_CppLibrary::SomeStruct(PyObject *); With cython master, a compiler error occurs: @cname("__pyx_convert__from_py_CppLibrary::SomeStruct") cdef SomeStruct __pyx_convert__from_py_CppLibrary::SomeStruct(obj) except *: ^ FromPyStructUtility:11:50: Expected an identifier or literal Cython should mangle the struct name to ensure the helper function has a valid name. Regards, -- Daniel Grunwald Axivion GmbH Nobelstr. 15 70569 Stuttgart Deutschland Tel: +49 711 6204378-33 Fax: +49 711 6204378-99 Geschaeftsfuehrung: Stefan Bellon, Thomas Eisenbarth, Sebastian Rummler Sitz der Gesellschaft: Stuttgart Registergericht: Amtsgericht Stuttgart, HRB 720590 From stefan_ml at behnel.de Wed Mar 18 15:07:03 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2015 15:07:03 +0100 Subject: [Cython] PyPy3 fixes In-Reply-To: References: Message-ID: <55098687.7010900@behnel.de> Lisandro Dalcin schrieb am 08.03.2015 um 20:25: > 1) Stefan, I had to push this fix, just to be sure, please double check. > > https://github.com/cython/cython/commit/6bd3f7b9e494d1259082aecfc0366da15fc105ec Thanks - sorry for the bug. > 2) PyPy3 defines some legacy (Py2) Py_TPFLAGS_XXX values. Do you like > the following fix? IOW, I check if the value is #define'd, if not I > #define it to 0 (zero): > > diff --git a/Cython/Utility/ModuleSetupCode.c b/Cython/Utility/ModuleSetupCode.c > index 6477fb2..3ae8a0c 100644 > --- a/Cython/Utility/ModuleSetupCode.c > +++ b/Cython/Utility/ModuleSetupCode.c > @@ -59,13 +59,16 @@ > #define __Pyx_DefaultClassType PyType_Type > #endif > > -#if PY_MAJOR_VERSION >= 3 > +#if !defined(Py_TPFLAGS_CHECKTYPES) > #define Py_TPFLAGS_CHECKTYPES 0 > +#endif > +#if !defined(Py_TPFLAGS_HAVE_INDEX) > #define Py_TPFLAGS_HAVE_INDEX 0 > +#endif > +#if !defined(Py_TPFLAGS_HAVE_NEWBUFFER) > #define Py_TPFLAGS_HAVE_NEWBUFFER 0 > #endif > - > -#if PY_VERSION_HEX < 0x030400a1 && !defined(Py_TPFLAGS_HAVE_FINALIZE) > +#if !defined(Py_TPFLAGS_HAVE_FINALIZE) > #define Py_TPFLAGS_HAVE_FINALIZE 0 > #endif LGTM. I tend to use the shorter "#ifndef" instead of "#if !defined()", though. Stefan From dalcinl at gmail.com Thu Mar 19 16:13:45 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 19 Mar 2015 18:13:45 +0300 Subject: [Cython] PyPy3 fixes In-Reply-To: <55098687.7010900@behnel.de> References: <55098687.7010900@behnel.de> Message-ID: On 18 March 2015 at 17:07, Stefan Behnel wrote: > LGTM. I tend to use the shorter "#ifndef" instead of "#if !defined()", though. Do you want me to change it in master? No problem, just ask for it. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From robertwb at gmail.com Thu Mar 19 16:50:54 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 19 Mar 2015 08:50:54 -0700 Subject: [Cython] PyPy3 fixes In-Reply-To: References: <55098687.7010900@behnel.de> Message-ID: On Thu, Mar 19, 2015 at 8:13 AM, Lisandro Dalcin wrote: > On 18 March 2015 at 17:07, Stefan Behnel wrote: >> LGTM. I tend to use the shorter "#ifndef" instead of "#if !defined()", though. > > Do you want me to change it in master? No problem, just ask for it. Go for it. From dalcinl at gmail.com Sun Mar 29 12:17:27 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 13:17:27 +0300 Subject: [Cython] "long long" versus "PY_LONG_LONG" Message-ID: At some point Cython lost the ability of using PY_LONG_LONG instead of "long long" in generated C code. The big offenders are the following two files: Cython/Utility/Overflow.c Cython/Utility/TypeConversion.c Do we want to continue using PY_LONG_LONG? Otherwise, what about Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite easy right now, let me know and I'll contribute the patch for the upcoming 0.22.1 release (if that ever happens). -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Sun Mar 29 12:23:59 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 13:23:59 +0300 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: <4F3FCDA3.7090102@behnel.de> References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> Message-ID: On 18 February 2012 at 19:11, Stefan Behnel wrote: > Stefan Behnel, 18.02.2012 09:54: >> Stefan Behnel, 15.02.2012 12:32: >>> The current state of the discussion seems to be that PyPy provides ways to >>> talk to C code, but nothing as complete as CPython's C-API in the sense >>> that it allows efficient two-way communication between C code and Python >>> objects. Thus, we need to either improve this or look for alternatives. >>> >>> In order to get us more focussed on what can be done and what the >>> implications are, so that we may eventually be able to decide what should >>> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement >>> Proposal). >>> >>> http://wiki.cython.org/enhancements/pypy >> >> The discussion so far makes me rather certain that the most promising >> short-term solution is to make Cython generate C code that PyPy's cpyext >> can handle. This should get us a rather broad set of running code somewhat >> quickly, while requiring the least design-from-scratch type of work in a >> direction that does not yet allow us to see if it will really make existing >> code work or not. > > Update: > > Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run > (although with some clear bugs in the exception handling code). There is a > Jenkins job now to (try to) run the test suite of my own branch in the > latest PyPy nightly build: > > https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/ > > It currently crashes rather badly at some point, but at least it looks like > it's actually getting somewhere. > One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime. This issue will be eventually fixed, I hope. Unce that happens, how can we know it is save to use the call for that pypy version and upwards? I mean, Cython should still support previous PyPy releases... -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sun Mar 29 12:33:06 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Mar 2015 12:33:06 +0200 Subject: [Cython] "long long" versus "PY_LONG_LONG" In-Reply-To: References: Message-ID: <5517D4E2.2070407@behnel.de> Lisandro Dalcin schrieb am 29.03.2015 um 12:17: > At some point Cython lost the ability of using PY_LONG_LONG instead of > "long long" in generated C code. The big offenders are the following > two files: > > Cython/Utility/Overflow.c > Cython/Utility/TypeConversion.c > > Do we want to continue using PY_LONG_LONG? Otherwise, what about > Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite > easy right now, let me know and I'll contribute the patch for the > upcoming 0.22.1 release (if that ever happens). If PyLL helps in any way, then, by all means, we should use it. And yes, there should be a 0.22.1 release. I selected changes for it already, and saw that you did, too. We should give the current branch another review to make sure we didn't forget anything else, and then release it. Stefan From dalcinl at gmail.com Sun Mar 29 13:19:03 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 14:19:03 +0300 Subject: [Cython] "long long" versus "PY_LONG_LONG" In-Reply-To: <5517D4E2.2070407@behnel.de> References: <5517D4E2.2070407@behnel.de> Message-ID: On 29 March 2015 at 13:33, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 29.03.2015 um 12:17: >> At some point Cython lost the ability of using PY_LONG_LONG instead of >> "long long" in generated C code. The big offenders are the following >> two files: >> >> Cython/Utility/Overflow.c >> Cython/Utility/TypeConversion.c >> >> Do we want to continue using PY_LONG_LONG? Otherwise, what about >> Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite >> easy right now, let me know and I'll contribute the patch for the >> upcoming 0.22.1 release (if that ever happens). > > If PyLL helps in any way, then, by all means, we should use it. > > And yes, there should be a 0.22.1 release. I selected changes for it > already, and saw that you did, too. We should give the current branch > another review to make sure we didn't forget anything else, and then > release it. > OK, please give me a day to push the PyLL patch. Thank you! -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sun Mar 29 13:16:07 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Mar 2015 13:16:07 +0200 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> Message-ID: <5517DEF7.9010306@behnel.de> Lisandro Dalcin schrieb am 29.03.2015 um 12:23: > One thing that Cython developers really need is PyPy defining a macro > such as PYPY_VERSION_HEX in such a way us we can properly use > conditional compilation. For example, a few days ago I was pushing > PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the > interpreter broke at runtime. This issue will be eventually fixed, I > hope. Unce that happens, how can we know it is save to use the call > for that pypy version and upwards? I mean, Cython should still support > previous PyPy releases... Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX should still change sometimes over PyPy releases, so that might at least help a bit. In general, however, you shouldn't expect any CPython internals to work in PyPy. Avoiding multiple C-API calls when a generic one exists is really the best of all strategies. For example, why read just the sign, when you can read the whole value? Everything else just risks aggregating slowness and running into buggy corners. That being said, if you find anything that can be optimised, I'm happy to learn about your changes. I think the general problem is that the PyPy developers loudly push for cffi adoption, and improving anything in cpyext appears to undermine that goal for them. So the current state of cpyext is a result of political decisions, general dislikes and the usual OSS project lack of workpower. Given all that, it actually works quite well. :) Stefan From dalcinl at gmail.com Sun Mar 29 14:52:26 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 15:52:26 +0300 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: <5517DEF7.9010306@behnel.de> References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> Message-ID: On 29 March 2015 at 14:16, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 29.03.2015 um 12:23: >> One thing that Cython developers really need is PyPy defining a macro >> such as PYPY_VERSION_HEX in such a way us we can properly use >> conditional compilation. For example, a few days ago I was pushing >> PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the >> interpreter broke at runtime. This issue will be eventually fixed, I >> hope. Unce that happens, how can we know it is save to use the call >> for that pypy version and upwards? I mean, Cython should still support >> previous PyPy releases... > > Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX > should still change sometimes over PyPy releases, so that might at least > help a bit. > Well, you know that's not enough ;-) > In general, however, you shouldn't expect any CPython internals to work in > PyPy. Avoiding multiple C-API calls when a generic one exists is really the > best of all strategies. For example, why read just the sign, when you can > read the whole value? Everything else just risks aggregating slowness and > running into buggy corners. > Well, I agree, however I do require _PyLong_Sign to work if I want to report correct overflow error messages when converting PyLong values to a C integral type. In CPython, we are using Py_SIZE for that, not only for speed, but also to figure out the right overflow kind when converting to unsigned integers. > That being said, if you find anything that can be optimised, I'm happy to > learn about your changes. > Again, this is not about optimization, but proper error reporting. diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c index 4cd8229..fbac649 100644 --- a/Cython/Utility/TypeConversion.c +++ b/Cython/Utility/TypeConversion.c @@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}} {{FROM_PY_FUNCTION}}(PyObject *x) { if (unlikely(Py_SIZE(x) < 0)) { goto raise_neg_overflow; } +#elif CYTHON_COMPILING_IN_PYPY + if (unlikely(_PyLong_Sign(x) < 0)) { + goto raise_neg_overflow; + } #endif if (sizeof({{TYPE}}) <= sizeof(unsigned long)) { __PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long, PyLong_AsUnsignedLong(x)) -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sun Mar 29 15:10:49 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Mar 2015 15:10:49 +0200 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> Message-ID: <5517F9D9.5070407@behnel.de> Lisandro Dalcin schrieb am 29.03.2015 um 14:52: >> Lisandro Dalcin schrieb am 29.03.2015 um 12:23: >>> One thing that Cython developers really need is PyPy defining a macro >>> such as PYPY_VERSION_HEX in such a way us we can properly use >>> conditional compilation. For example, a few days ago I was pushing >>> PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the >>> interpreter broke at runtime. > > however I do require _PyLong_Sign to work if I want to > report correct overflow error messages when converting PyLong values > to a C integral type. In CPython, we are using Py_SIZE for that, not > only for speed, but also to figure out the right overflow kind when > converting to unsigned integers. > > diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c > index 4cd8229..fbac649 100644 > --- a/Cython/Utility/TypeConversion.c > +++ b/Cython/Utility/TypeConversion.c > @@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}} > {{FROM_PY_FUNCTION}}(PyObject *x) { > if (unlikely(Py_SIZE(x) < 0)) { > goto raise_neg_overflow; > } > +#elif CYTHON_COMPILING_IN_PYPY > + if (unlikely(_PyLong_Sign(x) < 0)) { > + goto raise_neg_overflow; > + } > #endif > if (sizeof({{TYPE}}) <= sizeof(unsigned long)) { > __PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long, > PyLong_AsUnsignedLong(x)) Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0? Stefan From dalcinl at gmail.com Sun Mar 29 19:29:20 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 20:29:20 +0300 Subject: [Cython] "long long" versus "PY_LONG_LONG" In-Reply-To: References: <5517D4E2.2070407@behnel.de> Message-ID: On 29 March 2015 at 14:19, Lisandro Dalcin wrote: > On 29 March 2015 at 13:33, Stefan Behnel wrote: >> Lisandro Dalcin schrieb am 29.03.2015 um 12:17: >>> At some point Cython lost the ability of using PY_LONG_LONG instead of >>> "long long" in generated C code. The big offenders are the following >>> two files: >>> >>> Cython/Utility/Overflow.c >>> Cython/Utility/TypeConversion.c >>> >>> Do we want to continue using PY_LONG_LONG? Otherwise, what about >>> Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite >>> easy right now, let me know and I'll contribute the patch for the >>> upcoming 0.22.1 release (if that ever happens). >> >> If PyLL helps in any way, then, by all means, we should use it. >> >> And yes, there should be a 0.22.1 release. I selected changes for it >> already, and saw that you did, too. We should give the current branch >> another review to make sure we didn't forget anything else, and then >> release it. >> > > OK, please give me a day to push the PyLL patch. Thank you! > Done: [0.22.x] https://github.com/cython/cython/commit/60f0e2f99c3e587bc6d267608ad13e0f384b527c [master] https://github.com/cython/cython/commit/2e245fc1e3881ed66ce9e8a5e07c44071b1a499e -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Sun Mar 29 19:30:02 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 20:30:02 +0300 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: <5517F9D9.5070407@behnel.de> References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> <5517F9D9.5070407@behnel.de> Message-ID: On 29 March 2015 at 16:10, Stefan Behnel wrote: > Why not call PyObject_RichCompareBool() to make cpyext itself compare the > value to 0? That should definitely work. Let me try to put a patch together. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Sun Mar 29 19:40:41 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 20:40:41 +0300 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> <5517F9D9.5070407@behnel.de> Message-ID: On 29 March 2015 at 20:30, Lisandro Dalcin wrote: > On 29 March 2015 at 16:10, Stefan Behnel wrote: >> Why not call PyObject_RichCompareBool() to make cpyext itself compare the >> value to 0? > > That should definitely work. Let me try to put a patch together. > Stefan, how to you feel about abusing of Py_False in the call to RichCompareBool ? -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sun Mar 29 20:02:31 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Mar 2015 20:02:31 +0200 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> <5517F9D9.5070407@behnel.de> Message-ID: <55183E37.7090505@behnel.de> Lisandro Dalcin schrieb am 29.03.2015 um 19:40: > On 29 March 2015 at 20:30, Lisandro Dalcin wrote: >> On 29 March 2015 at 16:10, Stefan Behnel wrote: >>> Why not call PyObject_RichCompareBool() to make cpyext itself compare the >>> value to 0? >> >> That should definitely work. Let me try to put a patch together. > > Stefan, how to you feel about abusing of Py_False in the call to > RichCompareBool ? Wow - evil idea! :) But then, it's only for cpyext - as long as it works there and comes with a comment, it's certainly as simple as it gets. bool is clearly defined as an int subtype in Python. Just remember to check the error code. Py_SIZE() can't fail in CPython, but comparisons can, sadly. Stefan From dalcinl at gmail.com Sun Mar 29 21:51:19 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Mar 2015 22:51:19 +0300 Subject: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together In-Reply-To: <55183E37.7090505@behnel.de> References: <4F3F6733.3070609@behnel.de> <4F3FCDA3.7090102@behnel.de> <5517DEF7.9010306@behnel.de> <5517F9D9.5070407@behnel.de> <55183E37.7090505@behnel.de> Message-ID: On 29 March 2015 at 21:02, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 29.03.2015 um 19:40: >> On 29 March 2015 at 20:30, Lisandro Dalcin wrote: >>> On 29 March 2015 at 16:10, Stefan Behnel wrote: >>>> Why not call PyObject_RichCompareBool() to make cpyext itself compare the >>>> value to 0? >>> >>> That should definitely work. Let me try to put a patch together. >> >> Stefan, how to you feel about abusing of Py_False in the call to >> RichCompareBool ? > > Wow - evil idea! :) > > But then, it's only for cpyext - as long as it works there and comes with a > comment, it's certainly as simple as it gets. bool is clearly defined as an > int subtype in Python. > > Just remember to check the error code. Py_SIZE() can't fail in CPython, but > comparisons can, sadly. > Please review: https://github.com/cython/cython/pull/377 ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459