From Martin.Fiers at intec.ugent.be Mon Apr 1 10:19:52 2013 From: Martin.Fiers at intec.ugent.be (Martin Fiers) Date: Mon, 01 Apr 2013 10:19:52 +0200 Subject: [Cython] Bug: Returning real value crashes the code, complex value does not In-Reply-To: References: <51516FC2.1090802@intec.ugent.be> <5152395D.4000408@intec.ugent.be> <5152BF75.5030708@intec.ugent.be> Message-ID: <51594328.4020803@intec.ugent.be> On 03/27/2013 10:36 PM, Robert Bradshaw wrote: > On Wed, Mar 27, 2013 at 2:44 AM, Martin Fiers > wrote: >> On 3/27/2013 3:54 AM, Robert Bradshaw wrote: >>> On Tue, Mar 26, 2013 at 5:12 PM, Martin Fiers >>> wrote: >>>> On 3/26/2013 6:48 PM, Robert Bradshaw wrote: >>>>> On Tue, Mar 26, 2013 at 2:52 AM, Martin Fiers >>>>> wrote: >>>>>> Dear Cython developers, >>>>>> >>>>>> I stumbled upon a strange error when using Cython. I made a minimal >>>>>> working >>>>>> example, see attachment for the two necessary files. (btw I didn't find >>>>>> the >>>>>> e-mail address of Robert Bradshaw so I could not request him for an >>>>>> account >>>>>> on the issue tracker. Is it possible to put the bug on there?) >>>>> Sure. You should have my email now. >>>> Thank you! I just sent a mail. >>>> >>>> Also, thanks for replying so quickly. Replies follow inline. >>>> >>>>>> To reproduce the bug: >>>>>> 1) Reboot to Windows :) (the bug only appears on Windows) >>>>>> 2) Run compile_bug.py to generate the Cython extension >>>>>> 3) Try to run the my_func_exposed function: >>>>>> >>>>>> python >>>>>>>>> import complex_double >>>>>> (does not crash) >>>>>>>>> complex_double.my_func_exposed(1,1j) >>>>>> (crashes) >>>>>>>>> complex_double.my_func_exposed(1,1) >>>>>> If I put a breakpoint in the code with gdb, jump in the code, and leave >>>>>> the >>>>>> function again, it does not crash! Also, it is no problem on Linux. >>>>>> >>>>>> It has to do with the fact that in the first case, a real value was >>>>>> used. >>>>>> In >>>>>> the complex-value case, it does not crash. I went through the generated >>>>>> cpp >>>>>> file and I don't see any issues there (the reason I use cpp is because >>>>>> it's >>>>>> in a big project that needs cpp enabled; it is further linked and so >>>>>> on). >>>>>> >>>>>> gcc version used: 4.6.2 (mingw) >>>>>> cython version used: 0.18 (I did pip install Cython) >>>>>> python version used: python 2.7.3 (MSC v.1500 32 bit). >>>>> Very strange. Does calling PyComplex_AsCComplex directly produce the >>>>> same crash? What about >>>> I'm not sure how to call this directly. Do you mean by modifying the >>>> generated cpp file and then manually building an extension module? >>>> >>>>> cdef complex double x = 1.0 >>>> This one works. >>>> >>>>> or >>>>> >>>>> cdef object py_x = 1.0 >>>>> cdef complex double x = py_x >>>> This one crashes! >>> Ah. Try >>> >>> from cpython.complex cimport Py_complex, PyComplex_AsCComplex >>> cdef Py_complex x = PyComplex_AsCComplex(py_x) >>> print x.real, x.imag >> Ok. I tried this, and it also crashes. Here's the modification: >> >> from cpython.complex cimport Py_complex >> from cpython.complex cimport PyComplex_AsCComplex >> >> @cython.cdivision(True) >> cdef public double complex my_func(int a, b): >> >> cdef object py_x = 1.0 >> >> #cdef double complex x = 1.0 # Does not crash >> #cdef double complex x2 = py_x # Crashes for py_x = 1, >> not for py_x=1j. >> #cdef Py_complex x = PyComplex_AsCComplex(py_x) # Crashes, even for >> py_x=1j >> #print x.real, x.imag >> >> And as you can see, the PyComplex_AsCComplex also crashes (SIGSEGV). >> I tried to compile with debug information, as in the instructions in >> http://docs.cython.org/src/userguide/debugging.html >> But I cannot get the line numbers. Probably I need a debug-python version, >> but that seems to be very nontrivial on Windows. >> >> Not sure if I can think of other options to test it and/or track down the >> bug... >> >> Now it even crashes when py_x = 1j. So maybe there's something else going >> wrong here too. > I wonder if it's a compiler miss-match or something like that. In this case, this causes a significant problem for us. All other code works as expected, and I'm compiling a very large GCC project which uses Cython for certain functionality. Switching to the MSC tools would be quite a pain. I attached the latest version that isolates the bug. Can you give me directions on how to put this on the tracker as an issue: - type: defect - milestone: ? - keywords: complex, cpp - priority: for me it is critical (there's a workaround -although it requires the user to manually add some code) - component: C++ ? Is there anything else I could try to resolve/debug this issue? Are there a debug build binaries of Python available? Apparently, it's not that easy to find. Thanks for the help so far! Martin Fiers >> Regards, >> Martin >> >> P.S. I only replied to you because you didn't put the >> cython-devel at python.org in the previous mail. > Oops. Un-intentional oversight. -------------- next part -------------- A non-text attachment was scrubbed... Name: compile_bug.py Type: text/x-python Size: 1133 bytes Desc: not available URL: -------------- next part -------------- import cython from cpython.complex cimport Py_complex from cpython.complex cimport PyComplex_AsCComplex @cython.cdivision(True) cdef public double complex my_func(int a, b): cdef object py_x = 1j #cdef double complex x = 1.0 # Does not crash #cdef double complex x2 = py_x # Crashes for py_x = 1, not for py_x=1j. #cdef Py_complex x = PyComplex_AsCComplex(py_x) # Crashes, even for py_x=1j #print x.real, x.imag if(a==0): return 1; else: return b; # Crashes if b is real. def my_func_exposed(int a, b): return my_func(a,b) -------------- next part -------------- A non-text attachment was scrubbed... Name: trigger_bug.py Type: text/x-python Size: 199 bytes Desc: not available URL: From stefan_ml at behnel.de Mon Apr 1 10:29:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 01 Apr 2013 10:29:28 +0200 Subject: [Cython] Bug: Returning real value crashes the code, complex value does not In-Reply-To: <51594328.4020803@intec.ugent.be> References: <51516FC2.1090802@intec.ugent.be> <5152395D.4000408@intec.ugent.be> <5152BF75.5030708@intec.ugent.be> <51594328.4020803@intec.ugent.be> Message-ID: <51594568.3080004@behnel.de> Martin Fiers, 01.04.2013 10:19: > Is there anything else I could try to resolve/debug this issue? Could you try to come up with a stack trace that shows where the crash happens? Or better, run it in a debugger and investigate it post-mortem to figure out what actually goes wrong here? None of the Cython core developers uses Windows (or even just has it available AFAICT), so it won't be easy for us to debug this. > Are there a > debug build binaries of Python available? Apparently, it's not that easy to > find. CPython has a debug build feature that does lots of safety checks, but you have to build it yourself for that. Pass the "--with-pydebug" option to the build. However, if you're unlucky and it's really a compiler issue, building it yourself might make the crash go away... Stefan From nikita at nemkin.ru Tue Apr 2 10:07:32 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Tue, 02 Apr 2013 14:07:32 +0600 Subject: [Cython] Bug: Returning real value crashes the code, complex value does not In-Reply-To: <51516FC2.1090802@intec.ugent.be> References: <51516FC2.1090802@intec.ugent.be> Message-ID: Hi, For your information, I was not able to reproduce the crash in the following environment: * Python 2.7.3 (x86, download from python.org) * gcc (GCC) 4.7.2 (download from mingw.org) * Windows 7 x64 The immediate fix for your problem seems to be upgrading gcc. Note: using gcc 4.7 on Windows requires a patch to distutils, see http://stackoverflow.com/a/6035864/204882 for details. (You can probably monkey-patch it for production use.) Best regards, Nikita Nemkin On Tue, 26 Mar 2013 15:52:02 +0600, Martin Fiers wrote: > Dear Cython developers, > > I stumbled upon a strange error when using Cython. I made a minimal > working example, see attachment for the two necessary files. (btw I > didn't find the e-mail address of Robert Bradshaw so I could not request > him for an account on the issue tracker. Is it possible to put the bug > on there?) > > To reproduce the bug: > 1) Reboot to Windows :) (the bug only appears on Windows) > 2) Run compile_bug.py to generate the Cython extension > 3) Try to run the my_func_exposed function: > > python > >>> import complex_double > (does not crash) > >>> complex_double.my_func_exposed(1,1j) > (crashes) > >>> complex_double.my_func_exposed(1,1) From volker.mische at gmail.com Tue Apr 2 19:20:46 2013 From: volker.mische at gmail.com (Volker Mische) Date: Tue, 02 Apr 2013 19:20:46 +0200 Subject: [Cython] Constant pointers not working In-Reply-To: <514C6108.5040709@gmail.com> References: <514C6108.5040709@gmail.com> Message-ID: <515B136E.7000901@gmail.com> On 03/22/2013 02:47 PM, Volker Mische wrote: > Hi all, > > I was excited to see that 'const' is finally supported, but constant > pointers are not. Here's an example with the corresponding error: > > Error compiling Cython file: > ------------------------------------------------------------ > ... > cdef extern int foo(const int *const bar) > ^ > ------------------------------------------------------------ > > const.pxd:1:37: Expected ')', found 'bar' I tried to implement it myself, please let me know if that's the way to go: https://github.com/cython/cython/pull/203 Cheers, Volker From d.s.seljebotn at astro.uio.no Tue Apr 2 21:17:21 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Tue, 02 Apr 2013 21:17:21 +0200 Subject: [Cython] [cython-users] ANN: XDress v0.1 -- Automatic Code Generator and C/C++ Wrapper In-Reply-To: References: Message-ID: <515B2EC1.90501@astro.uio.no> [Crossing over to cython-devel to discuss further development] Wow, my first impression is this looks great! It's a shame that clang wasn't able to give a good enough parse tree for templates. Is gcc-xml still a fork of a really old gcc? Did you look into the code using the gcc plugin architecture that Phillip Heron worked on in the GSoC last year for this? Also, Cython devs, where's the wiki page listing all the wrapper generators in existence? I tried to find it but couldn't.. Dag Sverre On 04/02/2013 08:26 PM, Anthony Scopatz wrote: > Hello All, > > I am spamming the lists which may be interested in a C/C++ automatic > API wrapper / code generator / type system / thing I wrote. I'll keep > future updates more discrete. I'd love to help folks get started with this > and more participation is always welcome! Release notes are below. > > Please visit the docs: http://bit.ly/xdress-code > > Or just grab the repo: http://github.com/scopatz/xdress > > Be Well > Anthony > > ======================== > XDress 0.1 Release Notes > ======================== > XDress is an automatic wrapper generator for C/C++ written in pure > Python. Currently, > xdress may generate Python bindings (via Cython) for C++ classes & functions > and in-memory wrappers for C++ standard library containers (sets, > vectors, maps). > In the future, other tools and bindings will be supported. > > The main enabling feature of xdress is a dynamic type system that was > designed with > the purpose of API generation in mind. > > Release highlights: > > - Dynamic system for specifying types > - Automatically describes C/C++ APIs from source code with no > modifications. > - Python extension module generation (via Cython) from C++ API > descriptions > - Python views into C++ STL containers. Vectors are NumPy arrays > while maps > and sets have custom collections.MutableMapping and > collections.MutableSet > subclasses. > - Command line interface to the above tools. > > Please visit the website for more information: http://bit.ly/xdress-code > > Or grab the code from GitHub: http://github.com/scopatz/xdress > > XDress is free & open source (BSD 2-clause license) and requires Python 2.7, > NumPy 1.5+, PyTables 2.1+, Cython 0.18+, GCC-XML, and lxml. > > New Features > ============ > > Type System > ----------- > This module provides a suite of tools for denoting, describing, and > converting > between various data types and the types coming from various systems. > This is > achieved by providing canonical abstractions of various kinds of types: > > * Base types (int, str, float, non-templated classes) > * Refined types (even or odd ints, strings containing the letter 'a') > * Dependent types (templates such arrays, maps, sets, vectors) > > All types are known by their name (a string identifier) and may be > aliased with > other names. However, the string id of a type is not sufficient to > fully describe > most types. The system here implements a canonical form for all kinds > of types. > This canonical form is itself hashable, being comprised only of strings, > ints, > and tuples. > > Descriptions > ------------ > A key component of API wrapper generation is having a a top-level, abstract > representation of the software that is being wrapped. In C++ there are > three > basic constructs which may be wrapped: variables, functions, and classes. > Here we restrict ourselves to wrapping classes and functions, though > variables > may be added in the future. > > The abstract representation of a C++ class is known as a description > (abbr. desc). > This description is simply a Python dictionary with a specific structure. > This structure makes heavy use of the type system to declare the types > of all needed > parameters. > > Mini-FAQ > ======== > * Why not use an existing solution (eg, SWIG)? > > Their type systems don't support run-time, user provided refinement > types, > and thus are unsuited for verification & validation use cases that > often > arise in computational science. > > Furthermore, they tend to not handle C++ dependent types well (i.e. > vector > does not come back as a np.view(..., dtype=T)). > > * Why GCC-XML and not Clang's AST? > > I tried using Clang's AST (and the remnants of a broken visitor > class remain > in the code base). However, the official Clang AST Python bindings > lack > support for template argument types. This is a really big deal. > Other C++ ASTs > may be supported in the future -- including Clang's. > > * I run xdress and it creates these files, now what?! > > It is your job to integrate the files created by xdress into your > build system. > > Join in the Fun! > ================ > If you are interested in using xdress on your project (and need help), > contributing > back to xdress, starting up a development team, or writing your own code > generation > front end tool on top of the type system and autodescriber, please let > me know. > Participation is very welcome! > > Authors > ======= > XDress was written by `Anthony Scopatz `_, who had many > type system discussions with John Bachan over coffee at the Div school, > and was > polished up and released under the encouragement of Christopher > Jordan-Squire at > `PyCon 2013 `_. > > -- > > --- > You received this message because you are subscribed to the Google > Groups "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to cython-users+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > From scopatz at gmail.com Tue Apr 2 21:52:39 2013 From: scopatz at gmail.com (Anthony Scopatz) Date: Tue, 2 Apr 2013 14:52:39 -0500 Subject: [Cython] [cython-users] ANN: XDress v0.1 -- Automatic Code Generator and C/C++ Wrapper In-Reply-To: <515B2EC1.90501@astro.uio.no> References: <515B2EC1.90501@astro.uio.no> Message-ID: On Tue, Apr 2, 2013 at 2:17 PM, Dag Sverre Seljebotn < d.s.seljebotn at astro.uio.no> wrote: > [Crossing over to cython-devel to discuss further development] > > Wow, my first impression is this looks great! > Thanks Dag! > It's a shame that clang wasn't able to give a good enough parse tree for > templates Yeah, this was pretty infuriating since it is the last part of a visitor pattern that you write. To be clear though, this wouldn't be super hard for them to expose it would just take some work and then some effort to get the Clang people to merge these changes back into their mainline. They are open to this kind of activity and Eli has a great blog post describing the process in brutal detail [1]. I just personally don't have time to go down this road. > Is gcc-xml still a fork of a really old gcc? Hmm, now that you mention it, I am not sure. The website gives conflicting information on how much of an extension / plugin it is. However, gccxml does support gcc 4.7 so it can't be that old. > Did you look into the code using the gcc plugin architecture that Phillip > Heron worked on in the GSoC last year for this? > No, I didn't even know about it! Though pxd generation is only part of what xdress does, it will be good to look at. Be Well Anthony 1. http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/ > > Also, Cython devs, where's the wiki page listing all the wrapper > generators in existence? I tried to find it but couldn't.. > > Dag Sverre > > > On 04/02/2013 08:26 PM, Anthony Scopatz wrote: > >> Hello All, >> >> I am spamming the lists which may be interested in a C/C++ automatic >> API wrapper / code generator / type system / thing I wrote. I'll keep >> future updates more discrete. I'd love to help folks get started with >> this >> and more participation is always welcome! Release notes are below. >> >> Please visit the docs: http://bit.ly/xdress-code >> >> Or just grab the repo: http://github.com/scopatz/**xdress >> >> Be Well >> Anthony >> >> ======================== >> XDress 0.1 Release Notes >> ======================== >> XDress is an automatic wrapper generator for C/C++ written in pure >> Python. Currently, >> xdress may generate Python bindings (via Cython) for C++ classes & >> functions >> and in-memory wrappers for C++ standard library containers (sets, >> vectors, maps). >> In the future, other tools and bindings will be supported. >> >> The main enabling feature of xdress is a dynamic type system that was >> designed with >> the purpose of API generation in mind. >> >> Release highlights: >> >> - Dynamic system for specifying types >> - Automatically describes C/C++ APIs from source code with no >> modifications. >> - Python extension module generation (via Cython) from C++ API >> descriptions >> - Python views into C++ STL containers. Vectors are NumPy arrays >> while maps >> and sets have custom collections.MutableMapping and >> collections.MutableSet >> subclasses. >> - Command line interface to the above tools. >> >> Please visit the website for more information: http://bit.ly/xdress-code >> >> Or grab the code from GitHub: http://github.com/scopatz/**xdress >> >> XDress is free & open source (BSD 2-clause license) and requires Python >> 2.7, >> NumPy 1.5+, PyTables 2.1+, Cython 0.18+, GCC-XML, and lxml. >> >> New Features >> ============ >> >> Type System >> ----------- >> This module provides a suite of tools for denoting, describing, and >> converting >> between various data types and the types coming from various systems. >> This is >> achieved by providing canonical abstractions of various kinds of types: >> >> * Base types (int, str, float, non-templated classes) >> * Refined types (even or odd ints, strings containing the letter 'a') >> * Dependent types (templates such arrays, maps, sets, vectors) >> >> All types are known by their name (a string identifier) and may be >> aliased with >> other names. However, the string id of a type is not sufficient to >> fully describe >> most types. The system here implements a canonical form for all kinds >> of types. >> This canonical form is itself hashable, being comprised only of strings, >> ints, >> and tuples. >> >> Descriptions >> ------------ >> A key component of API wrapper generation is having a a top-level, >> abstract >> representation of the software that is being wrapped. In C++ there are >> three >> basic constructs which may be wrapped: variables, functions, and classes. >> Here we restrict ourselves to wrapping classes and functions, though >> variables >> may be added in the future. >> >> The abstract representation of a C++ class is known as a description >> (abbr. desc). >> This description is simply a Python dictionary with a specific structure. >> This structure makes heavy use of the type system to declare the types >> of all needed >> parameters. >> >> Mini-FAQ >> ======== >> * Why not use an existing solution (eg, SWIG)? >> >> Their type systems don't support run-time, user provided refinement >> types, >> and thus are unsuited for verification & validation use cases that >> often >> arise in computational science. >> >> Furthermore, they tend to not handle C++ dependent types well (i.e. >> vector >> does not come back as a np.view(..., dtype=T)). >> >> * Why GCC-XML and not Clang's AST? >> >> I tried using Clang's AST (and the remnants of a broken visitor >> class remain >> in the code base). However, the official Clang AST Python bindings >> lack >> support for template argument types. This is a really big deal. >> Other C++ ASTs >> may be supported in the future -- including Clang's. >> >> * I run xdress and it creates these files, now what?! >> >> It is your job to integrate the files created by xdress into your >> build system. >> >> Join in the Fun! >> ================ >> If you are interested in using xdress on your project (and need help), >> contributing >> back to xdress, starting up a development team, or writing your own code >> generation >> front end tool on top of the type system and autodescriber, please let >> me know. >> Participation is very welcome! >> >> Authors >> ======= >> XDress was written by `Anthony Scopatz `_, who had >> many >> type system discussions with John Bachan over coffee at the Div school, >> and was >> polished up and released under the encouragement of Christopher >> Jordan-Squire at >> `PyCon 2013 `_**. >> >> -- >> >> --- >> You received this message because you are subscribed to the Google >> Groups "cython-users" group. >> To unsubscribe from this group and stop receiving emails from it, send >> an email to cython-users+unsubscribe@**googlegroups.com >> . >> For more options, visit https://groups.google.com/**groups/opt_out >> . >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Wed Apr 3 08:18:55 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 2 Apr 2013 23:18:55 -0700 Subject: [Cython] Constant pointers not working In-Reply-To: <515B136E.7000901@gmail.com> References: <514C6108.5040709@gmail.com> <515B136E.7000901@gmail.com> Message-ID: Looks good to me. On Tue, Apr 2, 2013 at 10:20 AM, Volker Mische wrote: > On 03/22/2013 02:47 PM, Volker Mische wrote: >> Hi all, >> >> I was excited to see that 'const' is finally supported, but constant >> pointers are not. Here's an example with the corresponding error: >> >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> cdef extern int foo(const int *const bar) >> ^ >> ------------------------------------------------------------ >> >> const.pxd:1:37: Expected ')', found 'bar' > > I tried to implement it myself, please let me know if that's the way to > go: https://github.com/cython/cython/pull/203 > > Cheers, > Volker > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From linkmauve at linkmauve.fr Wed Apr 3 10:55:32 2013 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Wed, 3 Apr 2013 10:55:32 +0200 Subject: [Cython] Add support for the offsetof() C macro Message-ID: <20130403085532.GB7126@yuyuko> Hi, offsetof() is not supported by current Cython, and I have not found any workaround (except hardcoding offsets for a specific architecture and compiler, but this is obviously wrong). offsetof(struct_type, member) is a C macro that expands to the offset of the member in the struct, in bytes. Could you add support for it, perhaps like sizeof as a special case in the parser? And many thanks for the whole project! ? -- Emmanuel Gil Peyrot XMPP: OpenPGP: 24B1D609 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From nikita at nemkin.ru Wed Apr 3 12:53:50 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 03 Apr 2013 16:53:50 +0600 Subject: [Cython] Add support for the offsetof() C macro In-Reply-To: <20130403085532.GB7126@yuyuko> References: <20130403085532.GB7126@yuyuko> Message-ID: Hi, > offsetof() is not supported by current Cython, and I have not found any > workaround (except hardcoding offsets for a specific architecture and > compiler, but this is obviously wrong). offsetof() would certainly be very useful, but in the meantime offsetof(Struct, field) can be replaced with: &(NULL).field It's not ANSI C, but is portable enough. Another option (for extern or public structs only) is to abuse renaming: enum: Struct_offsetof_field1 "offsetof(Struct, field1)" This way a symbolic name can be given to any expression to be literally pasted into the generated code. Best regards, Nikita Nemkin From scopatz at gmail.com Thu Apr 4 20:18:37 2013 From: scopatz at gmail.com (Anthony Scopatz) Date: Thu, 4 Apr 2013 13:18:37 -0500 Subject: [Cython] [cython-users] ANN: XDress v0.1 -- Automatic Code Generator and C/C++ Wrapper In-Reply-To: References: <515B2EC1.90501@astro.uio.no> Message-ID: On Tue, Apr 2, 2013 at 2:52 PM, Anthony Scopatz wrote: > On Tue, Apr 2, 2013 at 2:17 PM, Dag Sverre Seljebotn < > d.s.seljebotn at astro.uio.no> wrote: > >> >> Also, Cython devs, where's the wiki page listing all the wrapper >> generators in existence? I tried to find it but couldn't.. >> > Dag, Did you mean this page -- http://wiki.cython.org/AutoPxd . I would really appreciate it if xdress could go on here. Be Well Anthony > >> Dag Sverre >> >> >> On 04/02/2013 08:26 PM, Anthony Scopatz wrote: >> >>> Hello All, >>> >>> I am spamming the lists which may be interested in a C/C++ automatic >>> API wrapper / code generator / type system / thing I wrote. I'll keep >>> future updates more discrete. I'd love to help folks get started with >>> this >>> and more participation is always welcome! Release notes are below. >>> >>> Please visit the docs: http://bit.ly/xdress-code >>> >>> Or just grab the repo: http://github.com/scopatz/**xdress >>> >>> Be Well >>> Anthony >>> >>> ======================== >>> XDress 0.1 Release Notes >>> ======================== >>> XDress is an automatic wrapper generator for C/C++ written in pure >>> Python. Currently, >>> xdress may generate Python bindings (via Cython) for C++ classes & >>> functions >>> and in-memory wrappers for C++ standard library containers (sets, >>> vectors, maps). >>> In the future, other tools and bindings will be supported. >>> >>> The main enabling feature of xdress is a dynamic type system that was >>> designed with >>> the purpose of API generation in mind. >>> >>> Release highlights: >>> >>> - Dynamic system for specifying types >>> - Automatically describes C/C++ APIs from source code with no >>> modifications. >>> - Python extension module generation (via Cython) from C++ API >>> descriptions >>> - Python views into C++ STL containers. Vectors are NumPy arrays >>> while maps >>> and sets have custom collections.MutableMapping and >>> collections.MutableSet >>> subclasses. >>> - Command line interface to the above tools. >>> >>> Please visit the website for more information: http://bit.ly/xdress-code >>> >>> Or grab the code from GitHub: http://github.com/scopatz/**xdress >>> >>> XDress is free & open source (BSD 2-clause license) and requires Python >>> 2.7, >>> NumPy 1.5+, PyTables 2.1+, Cython 0.18+, GCC-XML, and lxml. >>> >>> New Features >>> ============ >>> >>> Type System >>> ----------- >>> This module provides a suite of tools for denoting, describing, and >>> converting >>> between various data types and the types coming from various systems. >>> This is >>> achieved by providing canonical abstractions of various kinds of types: >>> >>> * Base types (int, str, float, non-templated classes) >>> * Refined types (even or odd ints, strings containing the letter 'a') >>> * Dependent types (templates such arrays, maps, sets, vectors) >>> >>> All types are known by their name (a string identifier) and may be >>> aliased with >>> other names. However, the string id of a type is not sufficient to >>> fully describe >>> most types. The system here implements a canonical form for all kinds >>> of types. >>> This canonical form is itself hashable, being comprised only of strings, >>> ints, >>> and tuples. >>> >>> Descriptions >>> ------------ >>> A key component of API wrapper generation is having a a top-level, >>> abstract >>> representation of the software that is being wrapped. In C++ there are >>> three >>> basic constructs which may be wrapped: variables, functions, and classes. >>> Here we restrict ourselves to wrapping classes and functions, though >>> variables >>> may be added in the future. >>> >>> The abstract representation of a C++ class is known as a description >>> (abbr. desc). >>> This description is simply a Python dictionary with a specific structure. >>> This structure makes heavy use of the type system to declare the types >>> of all needed >>> parameters. >>> >>> Mini-FAQ >>> ======== >>> * Why not use an existing solution (eg, SWIG)? >>> >>> Their type systems don't support run-time, user provided refinement >>> types, >>> and thus are unsuited for verification & validation use cases that >>> often >>> arise in computational science. >>> >>> Furthermore, they tend to not handle C++ dependent types well (i.e. >>> vector >>> does not come back as a np.view(..., dtype=T)). >>> >>> * Why GCC-XML and not Clang's AST? >>> >>> I tried using Clang's AST (and the remnants of a broken visitor >>> class remain >>> in the code base). However, the official Clang AST Python bindings >>> lack >>> support for template argument types. This is a really big deal. >>> Other C++ ASTs >>> may be supported in the future -- including Clang's. >>> >>> * I run xdress and it creates these files, now what?! >>> >>> It is your job to integrate the files created by xdress into your >>> build system. >>> >>> Join in the Fun! >>> ================ >>> If you are interested in using xdress on your project (and need help), >>> contributing >>> back to xdress, starting up a development team, or writing your own code >>> generation >>> front end tool on top of the type system and autodescriber, please let >>> me know. >>> Participation is very welcome! >>> >>> Authors >>> ======= >>> XDress was written by `Anthony Scopatz `_, who had >>> many >>> type system discussions with John Bachan over coffee at the Div school, >>> and was >>> polished up and released under the encouragement of Christopher >>> Jordan-Squire at >>> `PyCon 2013 `_**. >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "cython-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to cython-users+unsubscribe@**googlegroups.com >>> . >>> For more options, visit https://groups.google.com/**groups/opt_out >>> . >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Fri Apr 5 19:31:44 2013 From: sturla at molden.no (Sturla Molden) Date: Fri, 05 Apr 2013 19:31:44 +0200 Subject: [Cython] Add support for the offsetof() C macro In-Reply-To: References: <20130403085532.GB7126@yuyuko> Message-ID: <515F0A80.8000501@molden.no> On 03.04.2013 12:53, Nikita Nemkin wrote: >> offsetof() is not supported by current Cython, and I have not found any >> workaround (except hardcoding offsets for a specific architecture and >> compiler, but this is obviously wrong). > > offsetof() would certainly be very useful, but in the meantime > offsetof(Struct, field) can be replaced with: > > &(NULL).field > > It's not ANSI C, but is portable enough. This will dereference a NULL pointer. Also, Py_ssize_t is not guaranteed to be long enough to store a pointer (but Py_intptr_t is). Use Py_intptr_t when you cast pointers to integers. > Another option (for extern or public structs only) is to abuse > renaming: > > enum: > Struct_offsetof_field1 "offsetof(Struct, field1)" This will fail if "Struct" is name mangled by Cython. Basically it requires that it is defined outside of the Cython code, e.g. in a header file. To be valid C, offsets must be computed: cdef Struct tmp cdef Py_ssize_t offset offset = (&(tmp.field) - &(tmp)) For that reason, most C compilers defines offsetof as a builtin function. Sturla From nikita at nemkin.ru Fri Apr 5 20:26:09 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sat, 06 Apr 2013 00:26:09 +0600 Subject: [Cython] Add support for the offsetof() C macro In-Reply-To: <515F0A80.8000501@molden.no> References: <20130403085532.GB7126@yuyuko> <515F0A80.8000501@molden.no> Message-ID: On Fri, 05 Apr 2013 23:31:44 +0600, Sturla Molden wrote: >>> offsetof() is not supported by current Cython, and I have not found any >>> workaround (except hardcoding offsets for a specific architecture and >>> compiler, but this is obviously wrong). >> >> offsetof() would certainly be very useful, but in the meantime >> offsetof(Struct, field) can be replaced with: >> >> &(NULL).field >> >> It's not ANSI C, but is portable enough. > > This will dereference a NULL pointer. Nope it won't. It is purely an address calculation. Works fine in both msvc and gcc. offsetof is a builtin in gcc, but msvc implements it as a macro similar to the above. > Also, Py_ssize_t is not guaranteed to be long enough to store a pointer > (but Py_intptr_t is). Use Py_intptr_t when you cast pointers to integers. Thanks, Py_intptr_t is indeed more appropriate. I didn't know it existed. >> Another option (for extern or public structs only) is to abuse >> renaming: >> >> enum: >> Struct_offsetof_field1 "offsetof(Struct, field1)" > > This will fail if "Struct" is name mangled by Cython. Basically it > requires that it is defined outside of the Cython code, e.g. in a header > file. Please note "(for extern or public structs only)". These are not mangled. Best regards, Nikita Nemkin From njs at pobox.com Sat Apr 6 16:19:48 2013 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 6 Apr 2013 15:19:48 +0100 Subject: [Cython] Upcoming cython/numpy breakage with stride checking Message-ID: Hi all, If you build current numpy master with NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install then Cython code using ndarrays starts blowing up, e.g.: # foo.pyx def add_one(array): cdef double[::1] a = array a[0] += 1. return array >>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) Traceback (most recent call last): File "", line 1, in File "foo.pyx", line 2, in foo.add_one (foo.c:1210) cdef double[::1] a = array ValueError: Buffer and memoryview are not contiguous in the same dimension. The problem (as discussed before) is that Cython has an unnecessarily strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 pretty much breaks all existing compiled Cython modules. Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default sooner or later, and Cython is a major blocker on this plan. It may become the default as soon as the 1.8 pre-releases (with the expectation that we'll probably have to switch back again before the actual release, but still). References: Previous thread: http://thread.gmane.org/gmane.comp.python.cython.devel/14634 Detailed discussion of the difference between numpy/cython's current definition of "contiguity", and the correct definition: http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640 The PR implementing NPY_RELAXED_STRIDES_CHECKING: https://github.com/numpy/numpy/pull/3162 Another test case: https://github.com/numpy/numpy/issues/2956 We're hoping that Cython will also switch soon to the more accurate check for contiguity. This shouldn't cause any backwards compatibility problems -- it just means Cython code would make strictly fewer copies, and error out due to lack of contiguity strictly less often, even with older numpys. And it seems like a necessary step for getting this untangled and minimizing user pain. What do you think? -n From stefan_ml at behnel.de Sat Apr 6 21:36:20 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Apr 2013 21:36:20 +0200 Subject: [Cython] Fwd: Re: JIT compilers for Python, what is the latest news? In-Reply-To: References: Message-ID: <51607934.9090207@behnel.de> Maybe we should update the docs here? -------- Original-Message -------- Subject: Re: JIT compilers for Python, what is the latest news? Date: Sat, 6 Apr 2013 11:23:43 +0100 From: Joshua Landau To: gmane.comp.python.general On 5 April 2013 19:37, Devin Jeanpierre wrote: > On Fri, Apr 5, 2013 at 4:34 AM, John Ladasky wrote: >> On Thursday, April 4, 2013 7:39:16 PM UTC-7, MRAB wrote: >>> Have you looked at Cython? Not quite the same, but still... >> >> I'm already using Numpy, compiled with what is supposed to be a fast >> LAPACK. I don't think I want to attempt to improve on all the work that >> has gone into Numpy. > > There's no reason you can't use both cython and numpy. See: > http://docs.cython.org/src/tutorial/numpy.html Don't use this. Use memoryviews: http://docs.cython.org/src/userguide/memoryviews.html. I have no idea why that doc page isn't headed "DEPRICATED" by now. From stefan_ml at behnel.de Sat Apr 6 21:58:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Apr 2013 21:58:12 +0200 Subject: [Cython] [cython-users] Re: Cython classes and attribute cache In-Reply-To: <7c92e46b-11ed-42f2-9c02-57f8ff46eefa@b20g2000yqo.googlegroups.com> References: <7c92e46b-11ed-42f2-9c02-57f8ff46eefa@b20g2000yqo.googlegroups.com> Message-ID: <51607E54.5090904@behnel.de> Nils Bruin, 06.04.2013 21:37: > I can confirm that changing Compiler/TypeSlots.py, line 380 from: > > value = "Py_TPFLAGS_DEFAULT|Py_TPFLAGS_CHECKTYPES| > Py_TPFLAGS_HAVE_NEWBUFFER" > > to: > > value = "Py_TPFLAGS_DEFAULT|Py_TPFLAGS_CHECKTYPES| > Py_TPFLAGS_HAVE_NEWBUFFER|Py_TPFLAGS_HAVE_VERSION_TAG" > > indeed makes the difference in timing disappear. For extension classes > that don't mess directly with their tp_dict having this flag should be > entirely safe. > > See http://bugs.python.org/issue1700288 for background on the > introduction of the method cache in python and http://bugs.python.org/issue1878 > for the reason why the flag is not included in Py_TPFLAGS_DEFAULT on > 2.7, but is in 3.*. I wouldn't mind making it an "on by default" compiler directive. That would mean that you could switch it off with a class decorator at need. Any objections? Stefan From stefan_ml at behnel.de Sat Apr 6 22:33:22 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Apr 2013 22:33:22 +0200 Subject: [Cython] [cython-users] Re: Cython classes and attribute cache In-Reply-To: <51607E54.5090904@behnel.de> References: <7c92e46b-11ed-42f2-9c02-57f8ff46eefa@b20g2000yqo.googlegroups.com> <51607E54.5090904@behnel.de> Message-ID: <51608692.50002@behnel.de> Stefan Behnel, 06.04.2013 21:58: > Nils Bruin, 06.04.2013 21:37: >> I can confirm that changing Compiler/TypeSlots.py, line 380 from: >> >> value = "Py_TPFLAGS_DEFAULT|Py_TPFLAGS_CHECKTYPES| >> Py_TPFLAGS_HAVE_NEWBUFFER" >> >> to: >> >> value = "Py_TPFLAGS_DEFAULT|Py_TPFLAGS_CHECKTYPES| >> Py_TPFLAGS_HAVE_NEWBUFFER|Py_TPFLAGS_HAVE_VERSION_TAG" >> >> indeed makes the difference in timing disappear. For extension classes >> that don't mess directly with their tp_dict having this flag should be >> entirely safe. >> >> See http://bugs.python.org/issue1700288 for background on the >> introduction of the method cache in python and http://bugs.python.org/issue1878 >> for the reason why the flag is not included in Py_TPFLAGS_DEFAULT on >> 2.7, but is in 3.*. > > I wouldn't mind making it an "on by default" compiler directive. That would > mean that you could switch it off with a class decorator at need. Proposed patch: https://github.com/scoder/cython/commit/b1f10fa2c1bbb0a076807a9f391204c6a997ba9a > Any objections? Stefan From robertwb at gmail.com Sun Apr 7 09:13:08 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Sun, 7 Apr 2013 00:13:08 -0700 Subject: [Cython] [cython-users] ANN: XDress v0.1 -- Automatic Code Generator and C/C++ Wrapper In-Reply-To: References: <515B2EC1.90501@astro.uio.no> Message-ID: First off, let me say that this is very cool! On Tue, Apr 2, 2013 at 12:52 PM, Anthony Scopatz wrote: > On Tue, Apr 2, 2013 at 2:17 PM, Dag Sverre Seljebotn > wrote: >> >> [Crossing over to cython-devel to discuss further development] >> >> Wow, my first impression is this looks great! > > > Thanks Dag! > >> >> It's a shame that clang wasn't able to give a good enough parse tree for >> templates > > > Yeah, this was pretty infuriating since it is the last part of a visitor > pattern that you write. To be clear though, this wouldn't be super hard for > them to expose it would just take some work and then some effort to get the > Clang people to merge these changes back into their mainline. They are open > to this kind of activity and Eli has a great blog post describing the > process in brutal detail [1]. I just personally don't have time to go down > this road. In the long run, I think this will be a good option, but don't have the time to do it myself either... >> Is gcc-xml still a fork of a really old gcc? > > Hmm, now that you mention it, I am not sure. The website gives conflicting > information on how much of an extension / plugin it is. However, gccxml > does support gcc 4.7 so it can't be that old. > >> >> Did you look into the code using the gcc plugin architecture that Phillip >> Heron worked on in the GSoC last year for this? > > > No, I didn't even know about it! Though pxd generation is only part of what > xdress does, it will be good to look at. This is much more complete than that GSoC project ever got. Probably the biggest takeaway there was that gcc's plugin architecture, let alone the Python bindings, simply weren't stable or complete enough to seriously use (at that point at least). - Robert From stefan_ml at behnel.de Sun Apr 7 09:32:29 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Apr 2013 09:32:29 +0200 Subject: [Cython] [cython-users] ANN: XDress v0.1 -- Automatic Code Generator and C/C++ Wrapper In-Reply-To: References: <515B2EC1.90501@astro.uio.no> Message-ID: <5161210D.6040801@behnel.de> Robert Bradshaw, 07.04.2013 09:13: > This is much more complete than that GSoC project ever got. Probably > the biggest takeaway there was that gcc's plugin architecture, let > alone the Python bindings, simply weren't stable or complete enough to > seriously use (at that point at least). I would guess that it got better in the meantime. David Malcolm has been using it for static analysis of CPython extension modules and gave a talk about it at this year's PyCon-US. http://pyvideo.org/video/1698/death-by-a-thousand-leaks-what-statically-analys At the end of the talk, he was asked what bugs his tools found not only in manually written code but in generated code, specifically C code generated by Cython. He answered that it was rather the other way round: he had used Cython generated code to prune false positives from his analysis tool. :) Stefan From stefan_ml at behnel.de Sun Apr 7 10:18:34 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Apr 2013 10:18:34 +0200 Subject: [Cython] Cython 0.19 beta 1 released Message-ID: <51612BDA.9040403@behnel.de> Hi, I uploaded a first beta release for Cython 0.19. Please give it a try with your code. http://cython.org/release/Cython-0.19b1.tar.gz http://cython.org/release/Cython-0.19b1.zip There were many enhancements and changes regarding performance, usability and Python compatibility, including some very long standing bugs and issues. The changelog is on github: https://github.com/cython/cython/blob/b1f10fa2c1bbb0a076807a9f391204c6a997ba9a/CHANGES.rst The latest documentation is here: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ Have fun, Stefan From d.s.seljebotn at astro.uio.no Mon Apr 8 08:42:19 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 08 Apr 2013 08:42:19 +0200 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: References: Message-ID: <516266CB.5030607@astro.uio.no> On 04/06/2013 04:19 PM, Nathaniel Smith wrote: > Hi all, > > If you build current numpy master with > NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install > then Cython code using ndarrays starts blowing up, e.g.: > > # foo.pyx > def add_one(array): > cdef double[::1] a = array > a[0] += 1. > return array > >>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) > Traceback (most recent call last): > File "", line 1, in > File "foo.pyx", line 2, in foo.add_one (foo.c:1210) > cdef double[::1] a = array > ValueError: Buffer and memoryview are not contiguous in the same dimension. > > The problem (as discussed before) is that Cython has an unnecessarily > strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 > pretty much breaks all existing compiled Cython modules. > > Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default > sooner or later, and Cython is a major blocker on this plan. It may > become the default as soon as the 1.8 pre-releases (with the > expectation that we'll probably have to switch back again before the > actual release, but still). > > References: > > Previous thread: > http://thread.gmane.org/gmane.comp.python.cython.devel/14634 > Detailed discussion of the difference between numpy/cython's current > definition of "contiguity", and the correct definition: > http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640 > The PR implementing NPY_RELAXED_STRIDES_CHECKING: > https://github.com/numpy/numpy/pull/3162 > Another test case: > https://github.com/numpy/numpy/issues/2956 > > We're hoping that Cython will also switch soon to the more accurate > check for contiguity. This shouldn't cause any backwards compatibility > problems -- it just means Cython code would make strictly fewer > copies, and error out due to lack of contiguity strictly less often, > even with older numpys. And it seems like a necessary step for getting > this untangled and minimizing user pain. What do you think? I agree that we should follow NumPy here, but can't see myself having time to do the change in near future. I don't know about Mark? I think it is a fairly simple change though if any NumPyers would like to do it, look at in Cython/Utility/MemoryView_C.c in the function _pyx_memviewslice_is_contig looks like it should just be to add a check for shape too. I guess you have changed your implementation of PEP 3118 too slightly on the NumPy side? Is this undefined in the PEP or are you now not in strict adherence to it? Dag Sverre From sebastian at sipsolutions.net Mon Apr 8 09:59:23 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 08 Apr 2013 09:59:23 +0200 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: <516266CB.5030607@astro.uio.no> References: <516266CB.5030607@astro.uio.no> Message-ID: <1365407963.21672.10.camel@sebastian-laptop> On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote: > On 04/06/2013 04:19 PM, Nathaniel Smith wrote: > > Hi all, > > > > If you build current numpy master with > > NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install > > then Cython code using ndarrays starts blowing up, e.g.: > > > > # foo.pyx > > def add_one(array): > > cdef double[::1] a = array > > a[0] += 1. > > return array > > > >>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) > > Traceback (most recent call last): > > File "", line 1, in > > File "foo.pyx", line 2, in foo.add_one (foo.c:1210) > > cdef double[::1] a = array > > ValueError: Buffer and memoryview are not contiguous in the same dimension. > > > > The problem (as discussed before) is that Cython has an unnecessarily > > strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 > > pretty much breaks all existing compiled Cython modules. > > > > Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default > > sooner or later, and Cython is a major blocker on this plan. It may > > become the default as soon as the 1.8 pre-releases (with the > > expectation that we'll probably have to switch back again before the > > actual release, but still). > > > > References: > > > > Previous thread: > > http://thread.gmane.org/gmane.comp.python.cython.devel/14634 > > Detailed discussion of the difference between numpy/cython's current > > definition of "contiguity", and the correct definition: > > http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640 > > The PR implementing NPY_RELAXED_STRIDES_CHECKING: > > https://github.com/numpy/numpy/pull/3162 > > Another test case: > > https://github.com/numpy/numpy/issues/2956 > > > > We're hoping that Cython will also switch soon to the more accurate > > check for contiguity. This shouldn't cause any backwards compatibility > > problems -- it just means Cython code would make strictly fewer > > copies, and error out due to lack of contiguity strictly less often, > > even with older numpys. And it seems like a necessary step for getting > > this untangled and minimizing user pain. What do you think? > > I agree that we should follow NumPy here, but can't see myself having > time to do the change in near future. I don't know about Mark? > > I think it is a fairly simple change though if any NumPyers would like > to do it, look at in Cython/Utility/MemoryView_C.c in the function > > _pyx_memviewslice_is_contig > > looks like it should just be to add a check for shape too. > > I guess you have changed your implementation of PEP 3118 too slightly on > the NumPy side? Is this undefined in the PEP or are you now not in > strict adherence to it? > Hi, maybe I will have a look at that, but not sure if I will manage. The PEP 3118 is a point, but it does not seem to cover this (probably we should try to get a clarification into 3118). I am still wondering whether for buffers that are requested contiguous numpy should set the strides again, since it cannot hurt. Would that make a difference for Cython? I expected Cython just got any buffer and then checked the strides instead of requesting the buffer contiguous and then double checking. Regards, Sebastian > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From matej at laitl.cz Mon Apr 8 12:29:23 2013 From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl) Date: Mon, 08 Apr 2013 12:29:23 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class Message-ID: <2179674.3aDoa4uPCt@edgy> Hi cython-devel and Mark, I was getting > Fatal Python error: Acquisition count is 0 (line XYZ) when I was doing > cdef class MemViewContainer: > cdef double[:, :] A > > cdef a_method(self): > self.A = np.eye(2) > some_function(self.A.T) > some_function(self.A.T) I have found out that it is caused by the self.A.T expression - Cython emits __PYX_XDEC_MEMVIEW() after the call, but no __PYX_INC_MEMVIEW() before the call. This doesn't happen if the memoryview is a function-local variable. Proper test case is in the pull request [1] along with an ad-hoc patch that fixes the problem here, but needs review whether it is actually correct. I'd be very grateful if a fix for this problem could get it into Cython 0.19. Cython version: 0.19b1 3a6b9856187d7e490e08 [1] https://github.com/cython/cython/pull/201 Regards, Mat?j From d.s.seljebotn at astro.uio.no Mon Apr 8 12:31:32 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 08 Apr 2013 12:31:32 +0200 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: <1365407963.21672.10.camel@sebastian-laptop> References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> Message-ID: <51629C84.9090008@astro.uio.no> On 04/08/2013 09:59 AM, Sebastian Berg wrote: > On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote: >> On 04/06/2013 04:19 PM, Nathaniel Smith wrote: >>> Hi all, >>> >>> If you build current numpy master with >>> NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install >>> then Cython code using ndarrays starts blowing up, e.g.: >>> >>> # foo.pyx >>> def add_one(array): >>> cdef double[::1] a = array >>> a[0] += 1. >>> return array >>> >>>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "foo.pyx", line 2, in foo.add_one (foo.c:1210) >>> cdef double[::1] a = array >>> ValueError: Buffer and memoryview are not contiguous in the same dimension. >>> >>> The problem (as discussed before) is that Cython has an unnecessarily >>> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 >>> pretty much breaks all existing compiled Cython modules. >>> >>> Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default >>> sooner or later, and Cython is a major blocker on this plan. It may >>> become the default as soon as the 1.8 pre-releases (with the >>> expectation that we'll probably have to switch back again before the >>> actual release, but still). >>> >>> References: >>> >>> Previous thread: >>> http://thread.gmane.org/gmane.comp.python.cython.devel/14634 >>> Detailed discussion of the difference between numpy/cython's current >>> definition of "contiguity", and the correct definition: >>> http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640 >>> The PR implementing NPY_RELAXED_STRIDES_CHECKING: >>> https://github.com/numpy/numpy/pull/3162 >>> Another test case: >>> https://github.com/numpy/numpy/issues/2956 >>> >>> We're hoping that Cython will also switch soon to the more accurate >>> check for contiguity. This shouldn't cause any backwards compatibility >>> problems -- it just means Cython code would make strictly fewer >>> copies, and error out due to lack of contiguity strictly less often, >>> even with older numpys. And it seems like a necessary step for getting >>> this untangled and minimizing user pain. What do you think? >> >> I agree that we should follow NumPy here, but can't see myself having >> time to do the change in near future. I don't know about Mark? >> >> I think it is a fairly simple change though if any NumPyers would like >> to do it, look at in Cython/Utility/MemoryView_C.c in the function >> >> _pyx_memviewslice_is_contig >> >> looks like it should just be to add a check for shape too. >> >> I guess you have changed your implementation of PEP 3118 too slightly on >> the NumPy side? Is this undefined in the PEP or are you now not in >> strict adherence to it? >> > > Hi, > > maybe I will have a look at that, but not sure if I will manage. The > PEP 3118 is a point, but it does not seem to cover this (probably we > should try to get a clarification into 3118). I am still wondering > whether for buffers that are requested contiguous numpy should set the > strides again, since it cannot hurt. Would that make a difference for > Cython? I expected Cython just got any buffer and then checked the > strides instead of requesting the buffer contiguous and then double > checking. At least when I implemented cdef np.ndarray[double, mode='fortran'] arr that relies on PEP 3118 contiguous-flags and I did no checking myself. Lots of Cython code does this instead of memoryviews (I still write my own code that way). The memory views OTOH does their own checking, but I also see plenty of references to PyBUF_C_CONTIGUOUS etc. inside Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have the definitive answer here. Dag Sverre From volker.mische at gmail.com Mon Apr 8 13:43:31 2013 From: volker.mische at gmail.com (Volker Mische) Date: Mon, 08 Apr 2013 13:43:31 +0200 Subject: [Cython] Constant pointers not working In-Reply-To: <515B136E.7000901@gmail.com> References: <514C6108.5040709@gmail.com> <515B136E.7000901@gmail.com> Message-ID: <5162AD63.5090802@gmail.com> On 04/02/2013 07:20 PM, Volker Mische wrote: > On 03/22/2013 02:47 PM, Volker Mische wrote: >> Hi all, >> >> I was excited to see that 'const' is finally supported, but constant >> pointers are not. Here's an example with the corresponding error: >> >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> cdef extern int foo(const int *const bar) >> ^ >> ------------------------------------------------------------ >> >> const.pxd:1:37: Expected ')', found 'bar' > > I tried to implement it myself, please let me know if that's the way to > go: https://github.com/cython/cython/pull/203 I revive that this thread as I did another change, that hasn't seen any review. I thought about waiting a few days, but then I read that the 0.19b1 was tagged and I would like to see this change merged before the final 0.19. Though I don't know if that's possible with Cython's release procedure. The pull request is here: https://github.com/cython/cython/pull/204 With that one merged I got rid of all warnings I got when using a library that makes heavy use of consts. Cheers, Volker From stefan_ml at behnel.de Mon Apr 8 13:56:07 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 08 Apr 2013 13:56:07 +0200 Subject: [Cython] Constant pointers not working In-Reply-To: <5162AD63.5090802@gmail.com> References: <514C6108.5040709@gmail.com> <515B136E.7000901@gmail.com> <5162AD63.5090802@gmail.com> Message-ID: <5162B057.4030109@behnel.de> Volker Mische, 08.04.2013 13:43: > On 04/02/2013 07:20 PM, Volker Mische wrote: >> On 03/22/2013 02:47 PM, Volker Mische wrote: >>> Hi all, >>> >>> I was excited to see that 'const' is finally supported, but constant >>> pointers are not. Here's an example with the corresponding error: >>> >>> Error compiling Cython file: >>> ------------------------------------------------------------ >>> ... >>> cdef extern int foo(const int *const bar) >>> ^ >>> ------------------------------------------------------------ >>> >>> const.pxd:1:37: Expected ')', found 'bar' >> >> I tried to implement it myself, please let me know if that's the way to >> go: https://github.com/cython/cython/pull/203 > > I revive that this thread as I did another change, that hasn't seen any > review. I thought about waiting a few days, but then I read that the > 0.19b1 was tagged and I would like to see this change merged before the > final 0.19. Though I don't know if that's possible with Cython's release > procedure. The beta-cycle is there to get rid of bugs and find problems before the release. > The pull request is here: https://github.com/cython/cython/pull/204 > > With that one merged I got rid of all warnings I got when using a > library that makes heavy use of consts. Your change looks like a fix to me (i.e. should go in), but I'll leave it to Robert to decide as he's written the 'const' support code. Stefan From sebastian at sipsolutions.net Mon Apr 8 14:04:43 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 08 Apr 2013 14:04:43 +0200 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: <51629C84.9090008@astro.uio.no> References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> <51629C84.9090008@astro.uio.no> Message-ID: <1365422683.32378.22.camel@sebastian-laptop> On Mon, 2013-04-08 at 12:31 +0200, Dag Sverre Seljebotn wrote: > On 04/08/2013 09:59 AM, Sebastian Berg wrote: > > On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote: > >> On 04/06/2013 04:19 PM, Nathaniel Smith wrote: > >>> Hi all, > >>> > >>> If you build current numpy master with > >>> NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install > >>> then Cython code using ndarrays starts blowing up, e.g.: > >>> > >>> # foo.pyx > >>> def add_one(array): > >>> cdef double[::1] a = array > >>> a[0] += 1. > >>> return array > >>> > >>>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) > >>> Traceback (most recent call last): > >>> File "", line 1, in > >>> File "foo.pyx", line 2, in foo.add_one (foo.c:1210) > >>> cdef double[::1] a = array > >>> ValueError: Buffer and memoryview are not contiguous in the same dimension. > >>> > >>> The problem (as discussed before) is that Cython has an unnecessarily > >>> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 > >>> pretty much breaks all existing compiled Cython modules. > >>> > >> > >> I guess you have changed your implementation of PEP 3118 too slightly on > >> the NumPy side? Is this undefined in the PEP or are you now not in > >> strict adherence to it? > >> > > > > Hi, > > > > maybe I will have a look at that, but not sure if I will manage. The > > PEP 3118 is a point, but it does not seem to cover this (probably we > > should try to get a clarification into 3118). I am still wondering > > whether for buffers that are requested contiguous numpy should set the > > strides again, since it cannot hurt. Would that make a difference for > > Cython? I expected Cython just got any buffer and then checked the > > strides instead of requesting the buffer contiguous and then double > > checking. > > At least when I implemented > > cdef np.ndarray[double, mode='fortran'] arr > > that relies on PEP 3118 contiguous-flags and I did no checking myself. > Lots of Cython code does this instead of memoryviews (I still write my > own code that way). Yeah, though even if numpy "fixes" the buffer strides, `arr.strides` points at the original array I believe, leaving it to the user. But that just means users have to be a bit more careful with strides. I think I will have numpy "fix" the buffers stride anyway (when requested contiguous through the buffer protocol). I cannot think of any reason not to do it and it may work around bugs in extensions which may not even be aware of numpy's existence. > > The memory views OTOH does their own checking, but I also see plenty of > references to PyBUF_C_CONTIGUOUS etc. inside > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have > the definitive answer here. > After a quick check, it seems to me that those are there for exposing the memoryviews buffer and not for getting a buffer from another object. - Sebastian > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From njs at pobox.com Mon Apr 8 20:10:27 2013 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 8 Apr 2013 19:10:27 +0100 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: <516266CB.5030607@astro.uio.no> References: <516266CB.5030607@astro.uio.no> Message-ID: On Mon, Apr 8, 2013 at 7:42 AM, Dag Sverre Seljebotn wrote: > I guess you have changed your implementation of PEP 3118 too slightly on the NumPy side? Is this undefined in the PEP or are you now not in strict adherence to it? I just checked, and PEP 3118 just says that if a {C,F,ANY}_CONTIGUOUS buffer is requested, then that should be provided, and the strides array should contain something valid (not be NULL). They don't define any algorithm for matching {C,F,ANY}_CONTIGUOUS to particular strides. My interpretation would be, we're satisfying this just fine so long as our memory layout actually does match what was requested (i.e., the strides are not needed). Obviously Cython goes above and beyond in checking the strides -- if you just *trusted* us then it would all work out ;-). If there's other code that's similarly picky then it might be useful for us to take some special effort of "canonicalize" the strides when exporting a PEP 3118 buffer, like Sebastian says, but eh. This is arguably just papering over buggy code, and such code may or may not exist... (Or is it only the PEP 3118 buffer strides that Cython actually checks? If there's a simple hack we can do that will let us avoid forcing everyone to rebuild their Cython modules then it will make this transition significantly easier...) -n From dave.hirschfeld at gmail.com Tue Apr 9 03:04:14 2013 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Tue, 9 Apr 2013 01:04:14 +0000 (UTC) Subject: [Cython] Upcoming cython/numpy breakage with stride checking References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> <51629C84.9090008@astro.uio.no> Message-ID: Dag Sverre Seljebotn writes: > > cdef np.ndarray[double, mode='fortran'] arr > > that relies on PEP 3118 contiguous-flags and I did no checking myself. > Lots of Cython code does this instead of memoryviews (I still write my > own code that way). > > The memory views OTOH does their own checking, but I also see plenty of > references to PyBUF_C_CONTIGUOUS etc. inside > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have > the definitive answer here. > > Dag Sverre > Is it the case that MemoryViews are doing some redundant checking? If so, is that the cause of the order of mangnitude performance difference between the MemoryView and ndarray syntax that I observed in the following thread? http://thread.gmane.org/gmane.comp.python.cython.devel/14626 Sorry for hijacking the thread, but it would be fantastic if the performance issues could be addressed at the same time if they are indeed linked Thanks, Dave From markflorisson88 at gmail.com Tue Apr 9 14:05:20 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 13:05:20 +0100 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> <51629C84.9090008@astro.uio.no> Message-ID: On 9 April 2013 02:04, Dave Hirschfeld wrote: > Dag Sverre Seljebotn writes: > > > > > cdef np.ndarray[double, mode='fortran'] arr > > > > that relies on PEP 3118 contiguous-flags and I did no checking myself. > > Lots of Cython code does this instead of memoryviews (I still write my > > own code that way). > > > > The memory views OTOH does their own checking, but I also see plenty of > > references to PyBUF_C_CONTIGUOUS etc. inside > > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have > > the definitive answer here. > > > > Dag Sverre > > > > Is it the case that MemoryViews are doing some redundant checking? > > If so, is that the cause of the order of mangnitude performance difference > between the MemoryView and ndarray syntax that I observed in the following > thread? > > http://thread.gmane.org/gmane.comp.python.cython.devel/14626 > > Sorry for hijacking the thread, but it would be fantastic if the > performance > issues could be addressed at the same time if they are indeed linked > > > Thanks, > Dave > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > Hey Dave, I can't see what 'x' is in that thread, but I assume it's a numpy array. The difference with the memoryviews and the buffer syntax is that memoryviews require conversion to a different format. The validation part is more or less the same. But this conversion then means that if you go back to object, Cython will return a cython.memoryview. Going then back to a numpy array means you again go through the buffer interface and buffer format parsing, in addition to chaining views together (doing this often, e.g. in a loop, will even make you run out of memory). There are advantages and disadvantages to memoryviews. For instance the advantage is that we can pass them around without the GIL, and slice them quickly compared to NumPy (the buffer syntax). However, the conversion cost can be prohibitive. We actually have a check that avoids creating a new memoryview object and avoids the buffer format re-parse altogether. However, the memoryview class is duplicated in every cython module, which means a memoryview object from another module will fail this check. This is a general problem in Cython that could be worked around for memoryviews, but in general the lack of a Cython runtime is a blessing for distribution purposes and a wart for most other purposes. In any case, this is only a small part of the problem. I think memoryviews could perhaps better be implemented by keeping an ndarray alive and by adding quicker NumPy view construction if it's been sliced by simply throwing in pre-constructed dtypes. Conversion back from objects could be fast if we could all agree on a standard that allows us to verify the type in a few cycles, with a fallback to the buffer interface. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Apr 9 14:09:39 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 13:09:39 +0100 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: <1365422683.32378.22.camel@sebastian-laptop> References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> <51629C84.9090008@astro.uio.no> <1365422683.32378.22.camel@sebastian-laptop> Message-ID: On 8 April 2013 13:04, Sebastian Berg wrote: > On Mon, 2013-04-08 at 12:31 +0200, Dag Sverre Seljebotn wrote: > > On 04/08/2013 09:59 AM, Sebastian Berg wrote: > > > On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote: > > >> On 04/06/2013 04:19 PM, Nathaniel Smith wrote: > > >>> Hi all, > > >>> > > >>> If you build current numpy master with > > >>> NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install > > >>> then Cython code using ndarrays starts blowing up, e.g.: > > >>> > > >>> # foo.pyx > > >>> def add_one(array): > > >>> cdef double[::1] a = array > > >>> a[0] += 1. > > >>> return array > > >>> > > >>>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) > > >>> Traceback (most recent call last): > > >>> File "", line 1, in > > >>> File "foo.pyx", line 2, in foo.add_one (foo.c:1210) > > >>> cdef double[::1] a = array > > >>> ValueError: Buffer and memoryview are not contiguous in the same > dimension. > > >>> > > >>> The problem (as discussed before) is that Cython has an unnecessarily > > >>> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 > > >>> pretty much breaks all existing compiled Cython modules. > > >>> > > > >> > > >> I guess you have changed your implementation of PEP 3118 too slightly > on > > >> the NumPy side? Is this undefined in the PEP or are you now not in > > >> strict adherence to it? > > >> > > > > > > Hi, > > > > > > maybe I will have a look at that, but not sure if I will manage. The > > > PEP 3118 is a point, but it does not seem to cover this (probably we > > > should try to get a clarification into 3118). I am still wondering > > > whether for buffers that are requested contiguous numpy should set the > > > strides again, since it cannot hurt. Would that make a difference for > > > Cython? I expected Cython just got any buffer and then checked the > > > strides instead of requesting the buffer contiguous and then double > > > checking. > > > > At least when I implemented > > > > cdef np.ndarray[double, mode='fortran'] arr > > > > that relies on PEP 3118 contiguous-flags and I did no checking myself. > > Lots of Cython code does this instead of memoryviews (I still write my > > own code that way). > > Yeah, though even if numpy "fixes" the buffer strides, `arr.strides` > points at the original array I believe, leaving it to the user. But that > just means users have to be a bit more careful with strides. > > I think I will have numpy "fix" the buffers stride anyway (when > requested contiguous through the buffer protocol). I cannot think of > any reason not to do it and it may work around bugs in extensions which > may not even be aware of numpy's existence. > > > > > The memory views OTOH does their own checking, but I also see plenty of > > references to PyBUF_C_CONTIGUOUS etc. inside > > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have > > the definitive answer here. > > > > After a quick check, it seems to me that those are there for exposing > the memoryviews buffer and not for getting a buffer from another object. Yeah, I think most of the checking is in MemoryView_C.c, when obtaining the memoryview. I don't know how well buffer producers check these flags, but maybe we can simply remove some of those checks, or amend them otherwise. I'll try to look into it one of these days. > > - Sebastian > > > Dag Sverre > > _______________________________________________ > > cython-devel mailing list > > cython-devel at python.org > > http://mail.python.org/mailman/listinfo/cython-devel > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Apr 9 14:10:38 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 13:10:38 +0100 Subject: [Cython] Upcoming cython/numpy breakage with stride checking In-Reply-To: References: <516266CB.5030607@astro.uio.no> <1365407963.21672.10.camel@sebastian-laptop> <51629C84.9090008@astro.uio.no> <1365422683.32378.22.camel@sebastian-laptop> Message-ID: On 9 April 2013 13:09, mark florisson wrote: > > > > On 8 April 2013 13:04, Sebastian Berg wrote: > >> On Mon, 2013-04-08 at 12:31 +0200, Dag Sverre Seljebotn wrote: >> > On 04/08/2013 09:59 AM, Sebastian Berg wrote: >> > > On Mon, 2013-04-08 at 08:42 +0200, Dag Sverre Seljebotn wrote: >> > >> On 04/06/2013 04:19 PM, Nathaniel Smith wrote: >> > >>> Hi all, >> > >>> >> > >>> If you build current numpy master with >> > >>> NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install >> > >>> then Cython code using ndarrays starts blowing up, e.g.: >> > >>> >> > >>> # foo.pyx >> > >>> def add_one(array): >> > >>> cdef double[::1] a = array >> > >>> a[0] += 1. >> > >>> return array >> > >>> >> > >>>>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100])) >> > >>> Traceback (most recent call last): >> > >>> File "", line 1, in >> > >>> File "foo.pyx", line 2, in foo.add_one (foo.c:1210) >> > >>> cdef double[::1] a = array >> > >>> ValueError: Buffer and memoryview are not contiguous in the same >> dimension. >> > >>> >> > >>> The problem (as discussed before) is that Cython has an >> unnecessarily >> > >>> strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1 >> > >>> pretty much breaks all existing compiled Cython modules. >> > >>> >> >> > >> >> > >> I guess you have changed your implementation of PEP 3118 too >> slightly on >> > >> the NumPy side? Is this undefined in the PEP or are you now not in >> > >> strict adherence to it? >> > >> >> > > >> > > Hi, >> > > >> > > maybe I will have a look at that, but not sure if I will manage. The >> > > PEP 3118 is a point, but it does not seem to cover this (probably we >> > > should try to get a clarification into 3118). I am still wondering >> > > whether for buffers that are requested contiguous numpy should set the >> > > strides again, since it cannot hurt. Would that make a difference for >> > > Cython? I expected Cython just got any buffer and then checked the >> > > strides instead of requesting the buffer contiguous and then double >> > > checking. >> > >> > At least when I implemented >> > >> > cdef np.ndarray[double, mode='fortran'] arr >> > >> > that relies on PEP 3118 contiguous-flags and I did no checking myself. >> > Lots of Cython code does this instead of memoryviews (I still write my >> > own code that way). >> >> Yeah, though even if numpy "fixes" the buffer strides, `arr.strides` >> points at the original array I believe, leaving it to the user. But that >> just means users have to be a bit more careful with strides. >> >> I think I will have numpy "fix" the buffers stride anyway (when >> requested contiguous through the buffer protocol). I cannot think of >> any reason not to do it and it may work around bugs in extensions which >> may not even be aware of numpy's existence. >> >> > >> > The memory views OTOH does their own checking, but I also see plenty of >> > references to PyBUF_C_CONTIGUOUS etc. inside >> > Cython/Utility/MemoryView.pyx, so perhaps it does both. Mark would have >> > the definitive answer here. >> > >> >> After a quick check, it seems to me that those are there for exposing >> the memoryviews buffer and not for getting a buffer from another object. > > > Yeah, I think most of the checking is in MemoryView_C.c, when obtaining > the memoryview. I don't know how well buffer producers check these flags, > but maybe we can simply remove some of those checks, or amend them > otherwise. I'll try to look into it one of these days. > OTOH if someone else wants to take a crack at it, I'll be happy to review patches as well. > >> - Sebastian >> >> > Dag Sverre >> > _______________________________________________ >> > cython-devel mailing list >> > cython-devel at python.org >> > http://mail.python.org/mailman/listinfo/cython-devel >> > >> >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 9 14:25:02 2013 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 9 Apr 2013 13:25:02 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) Message-ID: On 9 Apr 2013 13:05, "mark florisson" wrote: > However, the memoryview class is duplicated in every cython module, which means a memoryview object from another module will fail this check. This is a general problem in Cython that could be worked around for memoryviews, but in general the lack of a Cython runtime is a blessing for distribution purposes and a wart for most other purposes. This seems easy to fix - a Cython runtime would be a python module like cython.runtime that Cython modules imported and used, right? You could emulate this perfectly by just checking for such a module in sys.modules at import time, and injecting a copy if none was found. (With some versioning in the name so modules compiled with different versions of Cython could coexist.) This would be much easier than coming up with elaborate object model hacks, since it just uses existing standard python mechanisms... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Apr 9 14:48:11 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 14:48:11 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: Message-ID: <51640E0B.90306@behnel.de> Nathaniel Smith, 09.04.2013 14:25: > On 9 Apr 2013 13:05, "mark florisson" wrote: >> However, the memoryview class is duplicated in every cython module, which >> means a memoryview object from another module will fail this check. This is >> a general problem in Cython that could be worked around for memoryviews, >> but in general the lack of a Cython runtime is a blessing for distribution >> purposes and a wart for most other purposes. > > This seems easy to fix - a Cython runtime would be a python module like > cython.runtime that Cython modules imported and used, right? You could > emulate this perfectly by just checking for such a module in sys.modules at > import time, and injecting a copy if none was found. (With some versioning > in the name so modules compiled with different versions of Cython could > coexist.)> This would be much easier than coming up with elaborate object > model hacks, since it just uses existing standard python mechanisms... "easy" may not be the optimal way of putting this - otherwise, it would have been done long ago, back when we decided that it should eventually work that way. The main problems is that this would move code out of the module, and thus out of the module's control and out of the sight of the C compiler, so we'd need to decide rather carefully what to externalise, based on criteria like API stability, performance, actual doability, ... There's also the problem of dependency hell and getting rid of old modules once they are no longer used on the user side. And also, how to get them there in the first place. Having one package overwrite the files of another during its installation is just asking for trouble. "easy" seems rather the right word for the current situation, given all of the open questions that are linked to this can of worms. Oh, and not to forget that someone actually has to do it. Stefan From njs at pobox.com Tue Apr 9 15:00:16 2013 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 9 Apr 2013 14:00:16 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <51640E0B.90306@behnel.de> References: <51640E0B.90306@behnel.de> Message-ID: On 9 Apr 2013 13:50, "Stefan Behnel" wrote: > > Nathaniel Smith, 09.04.2013 14:25: > > On 9 Apr 2013 13:05, "mark florisson" wrote: > >> However, the memoryview class is duplicated in every cython module, which > >> means a memoryview object from another module will fail this check. This is > >> a general problem in Cython that could be worked around for memoryviews, > >> but in general the lack of a Cython runtime is a blessing for distribution > >> purposes and a wart for most other purposes. > > > > This seems easy to fix - a Cython runtime would be a python module like > > cython.runtime that Cython modules imported and used, right? You could > > emulate this perfectly by just checking for such a module in sys.modules at > > import time, and injecting a copy if none was found. (With some versioning > > in the name so modules compiled with different versions of Cython could > > coexist.)> This would be much easier than coming up with elaborate object > > model hacks, since it just uses existing standard python mechanisms... > > "easy" may not be the optimal way of putting this - otherwise, it would > have been done long ago, back when we decided that it should eventually > work that way. Of course. But by this definition there are only two kinds of code: already written, and hard. Might be true, but it's easier to discuss things if we have more gradations than that ;-) > The main problems is that this would move code out of the module, and thus > out of the module's control and out of the sight of the C compiler, so we'd > need to decide rather carefully what to externalise, based on criteria like > API stability, performance, actual doability, ... Sure, but obviously the memoryview type object would be a good candidate, from what Mark said. Inlineable functions would remain in each compilation unit, no real point in trying to share those. > There's also the problem of dependency hell and getting rid of old modules > once they are no longer used on the user side. And also, how to get them > there in the first place. Having one package overwrite the files of another > during its installation is just asking for trouble. The system I described does not involve the addition of any new files to any package. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Apr 9 15:11:02 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 15:11:02 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> Message-ID: <51641366.2090202@behnel.de> Nathaniel Smith, 09.04.2013 15:00: > On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >> Nathaniel Smith, 09.04.2013 14:25: >> There's also the problem of dependency hell and getting rid of old modules >> once they are no longer used on the user side. And also, how to get them >> there in the first place. Having one package overwrite the files of >> another during its installation is just asking for trouble. > > The system I described does not involve the addition of any new files to > any package. I take it then that you were envisaging a separate "cython-runtime" package on PyPI that Cython compiled modules would have to depend on? As long as people install their stuff using pip, that could work for them mostly ok, although with the regrettable Cython user impact of having to set that dependency for their packages in the first place. If people want to install stuff manually, dependency hell gets close. Or did you see any other ways of getting these things installed automatically, with a smaller user impact? Stefan From markflorisson88 at gmail.com Tue Apr 9 15:13:22 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 14:13:22 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <51641366.2090202@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> Message-ID: On 9 April 2013 14:11, Stefan Behnel wrote: > Nathaniel Smith, 09.04.2013 15:00: > > On 9 Apr 2013 13:50, "Stefan Behnel" wrote: > >> Nathaniel Smith, 09.04.2013 14:25: > >> There's also the problem of dependency hell and getting rid of old > modules > >> once they are no longer used on the user side. And also, how to get them > >> there in the first place. Having one package overwrite the files of > >> another during its installation is just asking for trouble. > > > > The system I described does not involve the addition of any new files to > > any package. > > I take it then that you were envisaging a separate "cython-runtime" package > on PyPI that Cython compiled modules would have to depend on? > > As long as people install their stuff using pip, that could work for them > mostly ok, although with the regrettable Cython user impact of having to > set that dependency for their packages in the first place. > > If people want to install stuff manually, dependency hell gets close. > > Or did you see any other ways of getting these things installed > automatically, with a smaller user impact? > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > For reference, here's a CEP about this written last year: http://wiki.cython.org/enhancements/libcython In numba we want to create a numba loader, which allows users to compile their code to some intermediate form, making the dependency smaller but most importantly allowing library authors to use a suitable version without burdening users. This would also include a runtime which is substitutable (which includes CPython API calls, exception control, etc). This would always be needed to run numba code. Cython doesn't necessarily need a runtime dependency, since each module could create it, but I imagine it may reduce the memory footprint of projects like sage? Maybe Cython could allow a runtime dependency as well as allowing you to compile it into a module (e.g. cython --include-runtime). Maybe this could be a single module that the user imports from a package's __init__.py before any other cython module (and this module will create and inject a 'cython.runtime' module into sys.modules if not already there). That way there duplication is reduced to different projects. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Apr 9 15:24:48 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 15:24:48 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> Message-ID: <516416A0.9070203@behnel.de> mark florisson, 09.04.2013 15:13: > On 9 April 2013 14:11, Stefan Behnel wrote: >> Nathaniel Smith, 09.04.2013 15:00: >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >>>> There's also the problem of dependency hell and getting rid of old >>>> modules >>>> once they are no longer used on the user side. And also, how to get them >>>> there in the first place. Having one package overwrite the files of >>>> another during its installation is just asking for trouble. >>> >>> The system I described does not involve the addition of any new files to >>> any package. >> >> I take it then that you were envisaging a separate "cython-runtime" package >> on PyPI that Cython compiled modules would have to depend on? >> >> As long as people install their stuff using pip, that could work for them >> mostly ok, although with the regrettable Cython user impact of having to >> set that dependency for their packages in the first place. >> >> If people want to install stuff manually, dependency hell gets close. >> >> Or did you see any other ways of getting these things installed >> automatically, with a smaller user impact? > > For reference, here's a CEP about this written last year: > http://wiki.cython.org/enhancements/libcython Ok, but that CEP excludes the rather vital problem of distribution and installation. I also fail to see a reference to the problem of how multiple modules will interact that use different Cython runtime versions. That's a substantially bigger problem once symbols start becoming externally visible. Stefan From markflorisson88 at gmail.com Tue Apr 9 15:33:20 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 14:33:20 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <516416A0.9070203@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On 9 April 2013 14:24, Stefan Behnel wrote: > mark florisson, 09.04.2013 15:13: > > On 9 April 2013 14:11, Stefan Behnel wrote: > >> Nathaniel Smith, 09.04.2013 15:00: > >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: > >>>> There's also the problem of dependency hell and getting rid of old > >>>> modules > >>>> once they are no longer used on the user side. And also, how to get > them > >>>> there in the first place. Having one package overwrite the files of > >>>> another during its installation is just asking for trouble. > >>> > >>> The system I described does not involve the addition of any new files > to > >>> any package. > >> > >> I take it then that you were envisaging a separate "cython-runtime" > package > >> on PyPI that Cython compiled modules would have to depend on? > >> > >> As long as people install their stuff using pip, that could work for > them > >> mostly ok, although with the regrettable Cython user impact of having to > >> set that dependency for their packages in the first place. > >> > >> If people want to install stuff manually, dependency hell gets close. > >> > >> Or did you see any other ways of getting these things installed > >> automatically, with a smaller user impact? > > > > For reference, here's a CEP about this written last year: > > http://wiki.cython.org/enhancements/libcython > > Ok, but that CEP excludes the rather vital problem of distribution and > installation. I also fail to see a reference to the problem of how multiple > modules will interact that use different Cython runtime versions. That's a > substantially bigger problem once symbols start becoming externally > visible. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > I didn't say it was complete :) But the way I see it is basically what Nathaniel said, i.e. Cython modules dependent on the runtime import it at import time. It simply imports 'cython.runtime', which has been made available by the first module to initialize the runtime (compiled with --include-runtime), or otherwise must be present on the filesystem. So user packages can depend on a cython-runtime-x.y package (where each x.y is a different package), so pip will install all the runtime versions users need (or maybe we can otherwise improve upon this scheme). -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Apr 9 15:35:17 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 14:35:17 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On 9 April 2013 14:33, mark florisson wrote: > > > On 9 April 2013 14:24, Stefan Behnel wrote: > >> mark florisson, 09.04.2013 15:13: >> > On 9 April 2013 14:11, Stefan Behnel wrote: >> >> Nathaniel Smith, 09.04.2013 15:00: >> >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >> >>>> There's also the problem of dependency hell and getting rid of old >> >>>> modules >> >>>> once they are no longer used on the user side. And also, how to get >> them >> >>>> there in the first place. Having one package overwrite the files of >> >>>> another during its installation is just asking for trouble. >> >>> >> >>> The system I described does not involve the addition of any new files >> to >> >>> any package. >> >> >> >> I take it then that you were envisaging a separate "cython-runtime" >> package >> >> on PyPI that Cython compiled modules would have to depend on? >> >> >> >> As long as people install their stuff using pip, that could work for >> them >> >> mostly ok, although with the regrettable Cython user impact of having >> to >> >> set that dependency for their packages in the first place. >> >> >> >> If people want to install stuff manually, dependency hell gets close. >> >> >> >> Or did you see any other ways of getting these things installed >> >> automatically, with a smaller user impact? >> > >> > For reference, here's a CEP about this written last year: >> > http://wiki.cython.org/enhancements/libcython >> >> Ok, but that CEP excludes the rather vital problem of distribution and >> installation. I also fail to see a reference to the problem of how >> multiple >> modules will interact that use different Cython runtime versions. That's a >> substantially bigger problem once symbols start becoming externally >> visible. >> >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > I didn't say it was complete :) But the way I see it is basically what > Nathaniel said, i.e. Cython modules dependent on the runtime import it at > import time. It simply imports 'cython.runtime', which has been made > available by the first module to initialize the runtime (compiled with > --include-runtime), or otherwise must be present on the filesystem. So user > packages can depend on a cython-runtime-x.y package (where each x.y is a > different package), so pip will install all the runtime versions users need > (or maybe we can otherwise improve upon this scheme). > Symbols needn't be linked, but can be imported through capsules. Maybe that adds some call overhead for functions, but then we should do it only for bigger functions which are also not likely to be called often. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Apr 9 15:38:20 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 14:38:20 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On 9 April 2013 14:35, mark florisson wrote: > > > On 9 April 2013 14:33, mark florisson wrote: > >> >> >> On 9 April 2013 14:24, Stefan Behnel wrote: >> >>> mark florisson, 09.04.2013 15:13: >>> > On 9 April 2013 14:11, Stefan Behnel wrote: >>> >> Nathaniel Smith, 09.04.2013 15:00: >>> >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >>> >>>> There's also the problem of dependency hell and getting rid of old >>> >>>> modules >>> >>>> once they are no longer used on the user side. And also, how to get >>> them >>> >>>> there in the first place. Having one package overwrite the files of >>> >>>> another during its installation is just asking for trouble. >>> >>> >>> >>> The system I described does not involve the addition of any new >>> files to >>> >>> any package. >>> >> >>> >> I take it then that you were envisaging a separate "cython-runtime" >>> package >>> >> on PyPI that Cython compiled modules would have to depend on? >>> >> >>> >> As long as people install their stuff using pip, that could work for >>> them >>> >> mostly ok, although with the regrettable Cython user impact of having >>> to >>> >> set that dependency for their packages in the first place. >>> >> >>> >> If people want to install stuff manually, dependency hell gets close. >>> >> >>> >> Or did you see any other ways of getting these things installed >>> >> automatically, with a smaller user impact? >>> > >>> > For reference, here's a CEP about this written last year: >>> > http://wiki.cython.org/enhancements/libcython >>> >>> Ok, but that CEP excludes the rather vital problem of distribution and >>> installation. I also fail to see a reference to the problem of how >>> multiple >>> modules will interact that use different Cython runtime versions. That's >>> a >>> substantially bigger problem once symbols start becoming externally >>> visible. >>> >>> Stefan >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >> >> I didn't say it was complete :) But the way I see it is basically what >> Nathaniel said, i.e. Cython modules dependent on the runtime import it at >> import time. It simply imports 'cython.runtime', which has been made >> available by the first module to initialize the runtime (compiled with >> --include-runtime), or otherwise must be present on the filesystem. So user >> packages can depend on a cython-runtime-x.y package (where each x.y is a >> different package), so pip will install all the runtime versions users need >> (or maybe we can otherwise improve upon this scheme). >> > > Symbols needn't be linked, but can be imported through capsules. Maybe > that adds some call overhead for functions, but then we should do it only > for bigger functions which are also not likely to be called often. > More favourably though, it'd be nice to a sufficiently stable runtime that avoids the need to have multiple versions available at a time. But that may not be practical. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 9 15:46:33 2013 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 9 Apr 2013 14:46:33 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <51641366.2090202@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 2:11 PM, Stefan Behnel wrote: > Nathaniel Smith, 09.04.2013 15:00: >> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >>> Nathaniel Smith, 09.04.2013 14:25: >>> There's also the problem of dependency hell and getting rid of old modules >>> once they are no longer used on the user side. And also, how to get them >>> there in the first place. Having one package overwrite the files of >>> another during its installation is just asking for trouble. >> >> The system I described does not involve the addition of any new files to >> any package. > > I take it then that you were envisaging a separate "cython-runtime" package > on PyPI that Cython compiled modules would have to depend on? No, I must have been unclear -- I envisage that each cython-compiled module continues to contain all the code it depends on (just like now). But, the first cython-compiled module that is imported tweaks sys.modules *as if* there were a separate cython-runtime package, and sticks the necessary pieces (like the memoryview TypeObject) into this module, and later modules import it from there. There's no additional dependencies anywhere. You have to be careful to make sure you version the runtime module appropriately so that you don't end up with a module compiled with cython version X using the runtime injected by a module compiled with the incompatible cython version Y, but this is no harder than any other versioning problem. Just make sure to change the runtime module name whenever you break compatibility. -n From nikita at nemkin.ru Tue Apr 9 15:55:19 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Tue, 09 Apr 2013 19:55:19 +0600 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On Tue, 09 Apr 2013 19:33:20 +0600, mark florisson wrote: > On 9 April 2013 14:24, Stefan Behnel wrote: > >> mark florisson, 09.04.2013 15:13: >> > On 9 April 2013 14:11, Stefan Behnel wrote: >> >> Nathaniel Smith, 09.04.2013 15:00: >> >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >> >>>> There's also the problem of dependency hell and getting rid of old >> >>>> modules >> >>>> once they are no longer used on the user side. And also, how to get >> them >> >>>> there in the first place. Having one package overwrite the files of >> >>>> another during its installation is just asking for trouble. >> >>> >> >>> The system I described does not involve the addition of any new >> files >> to >> >>> any package. >> >> >> >> I take it then that you were envisaging a separate "cython-runtime" >> package >> >> on PyPI that Cython compiled modules would have to depend on? >> >> >> >> As long as people install their stuff using pip, that could work for >> them >> >> mostly ok, although with the regrettable Cython user impact of >> having to >> >> set that dependency for their packages in the first place. >> >> >> >> If people want to install stuff manually, dependency hell gets close. >> >> >> >> Or did you see any other ways of getting these things installed >> >> automatically, with a smaller user impact? >> > >> > For reference, here's a CEP about this written last year: >> > http://wiki.cython.org/enhancements/libcython >> >> Ok, but that CEP excludes the rather vital problem of distribution and >> installation. I also fail to see a reference to the problem of how >> multiple >> modules will interact that use different Cython runtime versions. >> That's a >> substantially bigger problem once symbols start becoming externally >> visible. >> >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > I didn't say it was complete :) But the way I see it is basically what > Nathaniel said, i.e. Cython modules dependent on the runtime import it at > import time. It simply imports 'cython.runtime', which has been made > available by the first module to initialize the runtime (compiled with > --include-runtime), or otherwise must be present on the filesystem. So > user > packages can depend on a cython-runtime-x.y package (where each x.y is a > different package), so pip will install all the runtime versions users > need > (or maybe we can otherwise improve upon this scheme). Extra dependency may not matter much for scientific code, but there are other users of Cython: library bindings, speedups for general purpose modules etc. Another (versioned!) binary dependency will become a liability to them. If memoryviews rely on autogenerated (module-specific) code, how is common runtime supposed to help? The bulk of Cython utility code is also either inline or module-specific. One alternative for code reuse in large Cython projects could be packaging multiple modules into one shared library. Best regards, Nikita Nemkin From markflorisson88 at gmail.com Tue Apr 9 15:56:52 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 14:56:52 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> Message-ID: On 9 April 2013 14:46, Nathaniel Smith wrote: > On Tue, Apr 9, 2013 at 2:11 PM, Stefan Behnel wrote: > > Nathaniel Smith, 09.04.2013 15:00: > >> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: > >>> Nathaniel Smith, 09.04.2013 14:25: > >>> There's also the problem of dependency hell and getting rid of old > modules > >>> once they are no longer used on the user side. And also, how to get > them > >>> there in the first place. Having one package overwrite the files of > >>> another during its installation is just asking for trouble. > >> > >> The system I described does not involve the addition of any new files to > >> any package. > > > > I take it then that you were envisaging a separate "cython-runtime" > package > > on PyPI that Cython compiled modules would have to depend on? > > No, I must have been unclear -- I envisage that each cython-compiled > module continues to contain all the code it depends on (just like > now). But, the first cython-compiled module that is imported tweaks > sys.modules *as if* there were a separate cython-runtime package, and > sticks the necessary pieces (like the memoryview TypeObject) into this > module, and later modules import it from there. > > There's no additional dependencies anywhere. You have to be careful to > make sure you version the runtime module appropriately so that you > don't end up with a module compiled with cython version X using the > runtime injected by a module compiled with the incompatible cython > version Y, but this is no harder than any other versioning problem. > Just make sure to change the runtime module name whenever you break > compatibility. > > -n > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > I'd actually really like it if I could compile a trivial module in under a hundred lines of generated code :) I.e. allow both a cython header and a seperately installable runtime, each of which is optional. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Apr 9 16:15:57 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 16:15:57 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> Message-ID: <5164229D.5080200@behnel.de> Nathaniel Smith, 09.04.2013 15:46: > On Tue, Apr 9, 2013 at 2:11 PM, Stefan Behnel wrote: >> Nathaniel Smith, 09.04.2013 15:00: >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >>>> Nathaniel Smith, 09.04.2013 14:25: >>>> There's also the problem of dependency hell and getting rid of old modules >>>> once they are no longer used on the user side. And also, how to get them >>>> there in the first place. Having one package overwrite the files of >>>> another during its installation is just asking for trouble. >>> >>> The system I described does not involve the addition of any new files to >>> any package. >> >> I take it then that you were envisaging a separate "cython-runtime" package >> on PyPI that Cython compiled modules would have to depend on? > > No, I must have been unclear -- I envisage that each cython-compiled > module continues to contain all the code it depends on (just like > now). But, the first cython-compiled module that is imported tweaks > sys.modules *as if* there were a separate cython-runtime package, and > sticks the necessary pieces (like the memoryview TypeObject) into this > module, and later modules import it from there. > > There's no additional dependencies anywhere. You have to be careful to > make sure you version the runtime module appropriately so that you > don't end up with a module compiled with cython version X using the > runtime injected by a module compiled with the incompatible cython > version Y, but this is no harder than any other versioning problem. > Just make sure to change the runtime module name whenever you break > compatibility. Ok, got it now. That solves the distribution problem, assuming that all installed runtimes with a given version are equivalent. Basically, we'd move the code out and then cimport the stuff back that we need, which would then let the first import of a given runtime version insert it into sys.modules. I so can't wait seeing the surprise in the eyes of our users when they let their IDE reorganise their import order and their code starts crashing. :] Stefan From njs at pobox.com Tue Apr 9 16:22:02 2013 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 9 Apr 2013 15:22:02 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <5164229D.5080200@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <5164229D.5080200@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 3:15 PM, Stefan Behnel wrote: > Ok, got it now. That solves the distribution problem, assuming that all > installed runtimes with a given version are equivalent. Basically, we'd > move the code out and then cimport the stuff back that we need, which would > then let the first import of a given runtime version insert it into > sys.modules. Yeah, that's the definition of "given version" :-), and any kind of shared runtime does require versioning. If we wanted to be extra careful we could put the shared code into its own block of boilerplate that gets injected into each generated .c file, and then have the "version" be the sha1 of that block of boilerplate... the trade-offs depend on what exactly is getting shared. > I so can't wait seeing the surprise in the eyes of our users when they let > their IDE reorganise their import order and their code starts crashing. :] That's easy to fix, just don't create such bugs ;-). -n From markflorisson88 at gmail.com Tue Apr 9 16:32:17 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 15:32:17 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On 9 April 2013 14:55, Nikita Nemkin wrote: > On Tue, 09 Apr 2013 19:33:20 +0600, mark florisson < > markflorisson88 at gmail.com> wrote: > >> On 9 April 2013 14:24, Stefan Behnel wrote: >> >> mark florisson, 09.04.2013 15:13: >>> > On 9 April 2013 14:11, Stefan Behnel wrote: >>> >> Nathaniel Smith, 09.04.2013 15:00: >>> >>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote: >>> >>>> There's also the problem of dependency hell and getting rid of old >>> >>>> modules >>> >>>> once they are no longer used on the user side. And also, how to get >>> them >>> >>>> there in the first place. Having one package overwrite the files of >>> >>>> another during its installation is just asking for trouble. >>> >>> >>> >>> The system I described does not involve the addition of any new files >>> to >>> >>> any package. >>> >> >>> >> I take it then that you were envisaging a separate "cython-runtime" >>> package >>> >> on PyPI that Cython compiled modules would have to depend on? >>> >> >>> >> As long as people install their stuff using pip, that could work for >>> them >>> >> mostly ok, although with the regrettable Cython user impact of having >>> to >>> >> set that dependency for their packages in the first place. >>> >> >>> >> If people want to install stuff manually, dependency hell gets close. >>> >> >>> >> Or did you see any other ways of getting these things installed >>> >> automatically, with a smaller user impact? >>> > >>> > For reference, here's a CEP about this written last year: >>> > http://wiki.cython.org/**enhancements/libcython >>> >>> Ok, but that CEP excludes the rather vital problem of distribution and >>> installation. I also fail to see a reference to the problem of how >>> multiple >>> modules will interact that use different Cython runtime versions. That's >>> a >>> substantially bigger problem once symbols start becoming externally >>> visible. >>> >>> Stefan >>> >>> ______________________________**_________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/**mailman/listinfo/cython-devel >>> >>> >> I didn't say it was complete :) But the way I see it is basically what >> Nathaniel said, i.e. Cython modules dependent on the runtime import it at >> import time. It simply imports 'cython.runtime', which has been made >> available by the first module to initialize the runtime (compiled with >> --include-runtime), or otherwise must be present on the filesystem. So >> user >> packages can depend on a cython-runtime-x.y package (where each x.y is a >> different package), so pip will install all the runtime versions users >> need >> (or maybe we can otherwise improve upon this scheme). >> > > Extra dependency may not matter much for scientific code, > but there are other users of Cython: library bindings, speedups > for general purpose modules etc. Another (versioned!) binary dependency > will become a liability to them. > Which is why it's optional: include the runtime separately, in a single module, or in every module. > If memoryviews rely on autogenerated (module-specific) code, > how is common runtime supposed to help? > The bulk of Cython utility code is also either inline or module-specific. > A fair amount is, though there is certainly room for externalizing (in the form of a runtime library as well as a header file). And as mentioned, it'd be useful to share Cython extension classes that cython inserts in each module. And sure, a header file is really arguing over a few bytes on disk and over the network. But it really does make it easier to inspect generated C code. > One alternative for code reuse in large Cython projects > could be packaging multiple modules into one shared library. > We have 'include'! :) Seriously though, that wouldn't work well with the import mechanism, and probably not for C compile time either. > > Best regards, > Nikita Nemkin > > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 9 16:45:51 2013 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 9 Apr 2013 15:45:51 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 3:32 PM, mark florisson wrote: > On 9 April 2013 14:55, Nikita Nemkin wrote: >> One alternative for code reuse in large Cython projects >> could be packaging multiple modules into one shared library. > > We have 'include'! :) Seriously though, that wouldn't work well with the > import mechanism, and probably not for C compile time either. You can link multiple .c files into a single shared library. This is off-topic again, but I've often thought in the past it would be nice if one could easily build a single module out of a combination of multiple .c and .pyx files. Specifically it'd be nice to be able to port bits of numpy/core/multiarray.so to Cython, but it's already 50,000 lines of C code spread out over 38 files and defining a single module; no real way to move just part of it to Cython so far as I know... -n From stefan_ml at behnel.de Tue Apr 9 16:47:30 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 16:47:30 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: <51642A02.7060404@behnel.de> mark florisson, 09.04.2013 16:32: > On 9 April 2013 14:55, Nikita Nemkin wrote: >> One alternative for code reuse in large Cython projects >> could be packaging multiple modules into one shared library. > > We have 'include'! :) Seriously though, that wouldn't work well with the > import mechanism, and probably not for C compile time either. CPython has support for it. It's actually quite easy, you basically just register all module init functions explicitly at init time and let CPython do the rest. So, compiling a package into one big Cython module with shared internal types and utility code should be quite simple and would IMHO be a very cool feature. Stefan From markflorisson88 at gmail.com Tue Apr 9 16:51:17 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 15:51:17 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: <51642A02.7060404@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> Message-ID: On 9 April 2013 15:47, Stefan Behnel wrote: > mark florisson, 09.04.2013 16:32: > > On 9 April 2013 14:55, Nikita Nemkin wrote: > >> One alternative for code reuse in large Cython projects > >> could be packaging multiple modules into one shared library. > > > > We have 'include'! :) Seriously though, that wouldn't work well with the > > import mechanism, and probably not for C compile time either. > > CPython has support for it. It's actually quite easy, you basically just > register all module init functions explicitly at init time and let CPython > do the rest. > > So, compiling a package into one big Cython module with shared internal > types and utility code should be quite simple and would IMHO be a very cool > feature. > > Stefan > I was unaware of that, it'd be a nice feature then (though still not imho the right solution to a runtime system). How does this work? Feel free to point me at some documentation. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Apr 9 16:54:39 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 9 Apr 2013 15:54:39 +0100 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> Message-ID: On 9 April 2013 15:45, Nathaniel Smith wrote: > On Tue, Apr 9, 2013 at 3:32 PM, mark florisson > wrote: > > On 9 April 2013 14:55, Nikita Nemkin wrote: > >> One alternative for code reuse in large Cython projects > >> could be packaging multiple modules into one shared library. > > > > We have 'include'! :) Seriously though, that wouldn't work well with the > > import mechanism, and probably not for C compile time either. > > You can link multiple .c files into a single shared library. > I did mean the Cython include statement, not the C one :) Include actually works out of the box with distutils. Though that may not be a very compelling argument for anything :) > This is off-topic again, but I've often thought in the past it would > be nice if one could easily build a single module out of a combination > of multiple .c and .pyx files. Specifically it'd be nice to be able to > port bits of numpy/core/multiarray.so to Cython, but it's already > 50,000 lines of C code spread out over 38 files and defining a single > module; no real way to move just part of it to Cython so far as I > know... I don't know if it's what you want, but you can simply list .c files in the 'sources' list and call 'cdef public' Cython functions from C. > -n > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Apr 9 17:12:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 17:12:28 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> Message-ID: <51642FDC.3030500@behnel.de> mark florisson, 09.04.2013 16:51: > On 9 April 2013 15:47, Stefan Behnel wrote: > >> mark florisson, 09.04.2013 16:32: >>> On 9 April 2013 14:55, Nikita Nemkin wrote: >>>> One alternative for code reuse in large Cython projects >>>> could be packaging multiple modules into one shared library. >>> >>> We have 'include'! :) Seriously though, that wouldn't work well with the >>> import mechanism, and probably not for C compile time either. >> >> CPython has support for it. It's actually quite easy, you basically just >> register all module init functions explicitly at init time and let CPython >> do the rest. >> >> So, compiling a package into one big Cython module with shared internal >> types and utility code should be quite simple and would IMHO be a very cool >> feature. > > I was unaware of that, it'd be a nice feature then (though still not imho > the right solution to a runtime system). How does this work? Feel free to > point me at some documentation. It's the inittab mechanism, as described starting here: http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab Stefan From stefan_ml at behnel.de Tue Apr 9 17:21:25 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 17:21:25 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: <51642FDC.3030500@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> Message-ID: <516431F5.6090706@behnel.de> Stefan Behnel, 09.04.2013 17:12: > mark florisson, 09.04.2013 16:51: >> On 9 April 2013 15:47, Stefan Behnel wrote: >> >>> mark florisson, 09.04.2013 16:32: >>>> On 9 April 2013 14:55, Nikita Nemkin wrote: >>>>> One alternative for code reuse in large Cython projects >>>>> could be packaging multiple modules into one shared library. >>>> >>>> We have 'include'! :) Seriously though, that wouldn't work well with the >>>> import mechanism, and probably not for C compile time either. >>> >>> CPython has support for it. It's actually quite easy, you basically just >>> register all module init functions explicitly at init time and let CPython >>> do the rest. >>> >>> So, compiling a package into one big Cython module with shared internal >>> types and utility code should be quite simple and would IMHO be a very cool >>> feature. >> >> I was unaware of that, it'd be a nice feature then (though still not imho >> the right solution to a runtime system). How does this work? Feel free to >> point me at some documentation. > > It's the inittab mechanism, as described starting here: > > http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab Oh, and even simpler, you can just stick multiple Cython modules with their module init functions into one big shared library and have the "main" init function of that module call the others. That will stick them into sys.modules as a side effect, so it's not really different from going through Python's import machinery for each module separately (just way more efficient - at least if you really need all of the modules...). There may be glitches with stuff like "__file__" and friends, but that should not be any worse than the current way of workings. Stefan From nikita at nemkin.ru Tue Apr 9 18:30:53 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Tue, 09 Apr 2013 22:30:53 +0600 Subject: [Cython] Shared Cython runtime In-Reply-To: <516431F5.6090706@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> Message-ID: On Tue, 09 Apr 2013 21:21:25 +0600, Stefan Behnel wrote: > Oh, and even simpler, you can just stick multiple Cython modules with > their > module init functions into one big shared library and have the "main" > init > function of that module call the others. That will stick them into > sys.modules as a side effect, so it's not really different from going > through Python's import machinery for each module separately (just way > more > efficient - at least if you really need all of the modules...). That's exactly what I meant. > There may be glitches with stuff like "__file__" and friends, but that > should not be any worse than the current way of workings. I know of two "glitches": 1) Py_InitModule4 expects qualified module name to be provided via the _Py_PackageContext global, in order to initialize new module's __name__. Of course __name__ can also be set manually afterwards. 2) "Top level" init function does not have access to it's own __file__, but it has to initialize submodules' __file__ somehow. My solution is to query the current shared library name directly from the OS (GetModuleFileName() on Windows, dladdr() on everything else). I'm interested in implementing this feature someday. For now, doing it manually is good enough. Best regards, Nikita Nemkin From stefan_ml at behnel.de Tue Apr 9 19:01:20 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 19:01:20 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> Message-ID: <51644960.8060607@behnel.de> Nikita Nemkin, 09.04.2013 18:30: > On Tue, 09 Apr 2013 21:21:25 +0600, Stefan Behnel wrote: >> Oh, and even simpler, you can just stick multiple Cython modules with their >> module init functions into one big shared library and have the "main" init >> function of that module call the others. That will stick them into >> sys.modules as a side effect, so it's not really different from going >> through Python's import machinery for each module separately (just way more >> efficient - at least if you really need all of the modules...). > > That's exactly what I meant. > >> There may be glitches with stuff like "__file__" and friends, but that >> should not be any worse than the current way of workings. > > I know of two "glitches": > > 1) Py_InitModule4 expects qualified module name to be provided via > the _Py_PackageContext global, in order to initialize new module's > __name__. > Of course __name__ can also be set manually afterwards. Cython could easily do that. It needs to know the FQMN at compile time anyway, so adding a C compile time switch that sets _Py_PackageContext before calling that function is trivial. (Would also have to store and restore the old value to support recursive imports, obviously) > 2) "Top level" init function does not have access to it's own __file__, > but it has to initialize submodules' __file__ somehow. > My solution is to query the current shared library name directly > from the OS (GetModuleFileName() on Windows, dladdr() on everything else). Yep: http://bugs.python.org/issue13429 Sadly, the new module init mechanism in Py3 wasn't designed to address these issues. > I'm interested in implementing this feature someday. For now, doing it > manually is good enough. I assume you're aware of the cython_freeze tool? That currently only works for embedding Python in a Cython compiled executable. Supporting this also (automatically) for a build where you have multiple .pyx files in a single distutils Extension() would be great. So, please do. And don't wait too long. Anything that can be automated should be. Stefan From robertwb at gmail.com Tue Apr 9 20:16:17 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 9 Apr 2013 11:16:17 -0700 Subject: [Cython] Shared Cython runtime In-Reply-To: <516431F5.6090706@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel wrote: > Stefan Behnel, 09.04.2013 17:12: >> mark florisson, 09.04.2013 16:51: >>> On 9 April 2013 15:47, Stefan Behnel wrote: >>> >>>> mark florisson, 09.04.2013 16:32: >>>>> On 9 April 2013 14:55, Nikita Nemkin wrote: >>>>>> One alternative for code reuse in large Cython projects >>>>>> could be packaging multiple modules into one shared library. >>>>> >>>>> We have 'include'! :) Seriously though, that wouldn't work well with the >>>>> import mechanism, and probably not for C compile time either. >>>> >>>> CPython has support for it. It's actually quite easy, you basically just >>>> register all module init functions explicitly at init time and let CPython >>>> do the rest. >>>> >>>> So, compiling a package into one big Cython module with shared internal >>>> types and utility code should be quite simple and would IMHO be a very cool >>>> feature. >>> >>> I was unaware of that, it'd be a nice feature then (though still not imho >>> the right solution to a runtime system). How does this work? Feel free to >>> point me at some documentation. >> >> It's the inittab mechanism, as described starting here: >> >> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab > > Oh, and even simpler, you can just stick multiple Cython modules with their > module init functions into one big shared library and have the "main" init > function of that module call the others. That will stick them into > sys.modules as a side effect, so it's not really different from going > through Python's import machinery for each module separately (just way more > efficient - at least if you really need all of the modules...). Yep. We could even create stub .so files for the included ones that did nothing but import the big one to avoid having to worry about importing things in the right order. The big question is what API to specify that things should be linked together. An option to cythonize(...) could be used, but for some projects (e.g. Sage) one would want more control over which modules get bundled together rather than all-or-nothing. > There may be glitches with stuff like "__file__" and friends, but that > should not be any worse than the current way of workings. This can probably be manipulated by Cython, though it's unclear what the best value would be. - Robert From stefan_ml at behnel.de Tue Apr 9 20:35:55 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 20:35:55 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> Message-ID: <51645F8B.9050100@behnel.de> Robert Bradshaw, 09.04.2013 20:16: > On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel wrote: >> Stefan Behnel, 09.04.2013 17:12: >>> mark florisson, 09.04.2013 16:51: >>>> On 9 April 2013 15:47, Stefan Behnel wrote: >>>> >>>>> mark florisson, 09.04.2013 16:32: >>>>>> On 9 April 2013 14:55, Nikita Nemkin wrote: >>>>>>> One alternative for code reuse in large Cython projects >>>>>>> could be packaging multiple modules into one shared library. >>>>>> >>>>>> We have 'include'! :) Seriously though, that wouldn't work well with the >>>>>> import mechanism, and probably not for C compile time either. >>>>> >>>>> CPython has support for it. It's actually quite easy, you basically just >>>>> register all module init functions explicitly at init time and let CPython >>>>> do the rest. >>>>> >>>>> So, compiling a package into one big Cython module with shared internal >>>>> types and utility code should be quite simple and would IMHO be a very cool >>>>> feature. >>>> >>>> I was unaware of that, it'd be a nice feature then (though still not imho >>>> the right solution to a runtime system). How does this work? Feel free to >>>> point me at some documentation. >>> >>> It's the inittab mechanism, as described starting here: >>> >>> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab >> >> Oh, and even simpler, you can just stick multiple Cython modules with their >> module init functions into one big shared library and have the "main" init >> function of that module call the others. That will stick them into >> sys.modules as a side effect, so it's not really different from going >> through Python's import machinery for each module separately (just way more >> efficient - at least if you really need all of the modules...). > > Yep. We could even create stub .so files for the included ones that > did nothing but import the big one to avoid having to worry about > importing things in the right order. > > The big question is what API to specify that things should be linked > together. An option to cythonize(...) could be used, but for some > projects (e.g. Sage) one would want more control over which modules > get bundled together rather than all-or-nothing. No-one forces you to use a plain cythonize("**/*.pyx"). Just be more specific about what files to include in each pattern that you pass to in. And you can always call cythonize() more than once if you need to. Once for each meta-module should usually be acceptable, given that the bundled source modules would tend to have a common configuration anyway. Also, I'd really encourage users to merge things at a package level, rather than compiling half of a package into one module and then having separate Python or Cython modules lying around next to it. It keeps projects easier to understand when you see their source directory layout. Shouldn't be the only option, though. >> There may be glitches with stuff like "__file__" and friends, but that >> should not be any worse than the current way of workings. > > This can probably be manipulated by Cython, though it's unclear what > the best value would be. I added an option to pyximport that simply uses the file path of the source file. It makes sense there, but less so in other contexts. I actually think that it's impossible to generally fix in the current infrastructure. http://bugs.python.org/issue13429 Stefan From robertwb at gmail.com Tue Apr 9 20:43:17 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 9 Apr 2013 11:43:17 -0700 Subject: [Cython] Constant pointers not working In-Reply-To: <5162B057.4030109@behnel.de> References: <514C6108.5040709@gmail.com> <515B136E.7000901@gmail.com> <5162AD63.5090802@gmail.com> <5162B057.4030109@behnel.de> Message-ID: On Mon, Apr 8, 2013 at 4:56 AM, Stefan Behnel wrote: > Volker Mische, 08.04.2013 13:43: >> On 04/02/2013 07:20 PM, Volker Mische wrote: >>> On 03/22/2013 02:47 PM, Volker Mische wrote: >>>> Hi all, >>>> >>>> I was excited to see that 'const' is finally supported, but constant >>>> pointers are not. Here's an example with the corresponding error: >>>> >>>> Error compiling Cython file: >>>> ------------------------------------------------------------ >>>> ... >>>> cdef extern int foo(const int *const bar) >>>> ^ >>>> ------------------------------------------------------------ >>>> >>>> const.pxd:1:37: Expected ')', found 'bar' >>> >>> I tried to implement it myself, please let me know if that's the way to >>> go: https://github.com/cython/cython/pull/203 >> >> I revive that this thread as I did another change, that hasn't seen any >> review. I thought about waiting a few days, but then I read that the >> 0.19b1 was tagged and I would like to see this change merged before the >> final 0.19. Though I don't know if that's possible with Cython's release >> procedure. > > The beta-cycle is there to get rid of bugs and find problems before the > release. > > >> The pull request is here: https://github.com/cython/cython/pull/204 >> >> With that one merged I got rid of all warnings I got when using a >> library that makes heavy use of consts. > > Your change looks like a fix to me (i.e. should go in), but I'll leave it > to Robert to decide as he's written the 'const' support code.\ Merged. - Robert From robertwb at gmail.com Tue Apr 9 20:47:52 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 9 Apr 2013 11:47:52 -0700 Subject: [Cython] Shared Cython runtime In-Reply-To: <51645F8B.9050100@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 11:35 AM, Stefan Behnel wrote: > Robert Bradshaw, 09.04.2013 20:16: >> On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel wrote: >>> Stefan Behnel, 09.04.2013 17:12: >>>> mark florisson, 09.04.2013 16:51: >>>>> On 9 April 2013 15:47, Stefan Behnel wrote: >>>>> >>>>>> mark florisson, 09.04.2013 16:32: >>>>>>> On 9 April 2013 14:55, Nikita Nemkin wrote: >>>>>>>> One alternative for code reuse in large Cython projects >>>>>>>> could be packaging multiple modules into one shared library. >>>>>>> >>>>>>> We have 'include'! :) Seriously though, that wouldn't work well with the >>>>>>> import mechanism, and probably not for C compile time either. >>>>>> >>>>>> CPython has support for it. It's actually quite easy, you basically just >>>>>> register all module init functions explicitly at init time and let CPython >>>>>> do the rest. >>>>>> >>>>>> So, compiling a package into one big Cython module with shared internal >>>>>> types and utility code should be quite simple and would IMHO be a very cool >>>>>> feature. >>>>> >>>>> I was unaware of that, it'd be a nice feature then (though still not imho >>>>> the right solution to a runtime system). How does this work? Feel free to >>>>> point me at some documentation. >>>> >>>> It's the inittab mechanism, as described starting here: >>>> >>>> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab >>> >>> Oh, and even simpler, you can just stick multiple Cython modules with their >>> module init functions into one big shared library and have the "main" init >>> function of that module call the others. That will stick them into >>> sys.modules as a side effect, so it's not really different from going >>> through Python's import machinery for each module separately (just way more >>> efficient - at least if you really need all of the modules...). >> >> Yep. We could even create stub .so files for the included ones that >> did nothing but import the big one to avoid having to worry about >> importing things in the right order. >> >> The big question is what API to specify that things should be linked >> together. An option to cythonize(...) could be used, but for some >> projects (e.g. Sage) one would want more control over which modules >> get bundled together rather than all-or-nothing. > > No-one forces you to use a plain cythonize("**/*.pyx"). Just be more > specific about what files to include in each pattern that you pass to in. > And you can always call cythonize() more than once if you need to. Once for > each meta-module should usually be acceptable, given that the bundled > source modules would tend to have a common configuration anyway. That would still be painful. Ideally, users should never have to modify the setup.py file. > Also, I'd really encourage users to merge things at a package level, rather > than compiling half of a package into one module and then having separate > Python or Cython modules lying around next to it. It keeps projects easier > to understand when you see their source directory layout. Shouldn't be the > only option, though. Yes. Actually, this could be a reasonable option for cythonize("**/*.pyx"), but still wouldn't make sense for Sage (e.g we have sage/rings and while it would make sense to bundle the common rings (integers, rationals, reals, etc.) there are others (e.g. the whole suite of p-adics) that shouldn't be imported by default. Maybe every exception should really be packages, but that's hard to change...) >>> There may be glitches with stuff like "__file__" and friends, but that >>> should not be any worse than the current way of workings. >> >> This can probably be manipulated by Cython, though it's unclear what >> the best value would be. > > I added an option to pyximport that simply uses the file path of the source > file. It makes sense there, but less so in other contexts. I actually think > that it's impossible to generally fix in the current infrastructure. > > http://bugs.python.org/issue13429 > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From robertwb at gmail.com Tue Apr 9 21:00:57 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 9 Apr 2013 12:00:57 -0700 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <5164229D.5080200@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 7:22 AM, Nathaniel Smith wrote: > On Tue, Apr 9, 2013 at 3:15 PM, Stefan Behnel wrote: >> Ok, got it now. That solves the distribution problem, assuming that all >> installed runtimes with a given version are equivalent. Basically, we'd >> move the code out and then cimport the stuff back that we need, which would >> then let the first import of a given runtime version insert it into >> sys.modules. > > Yeah, that's the definition of "given version" :-), and any kind of > shared runtime does require versioning. If we wanted to be extra > careful we could put the shared code into its own block of boilerplate > that gets injected into each generated .c file, and then have the > "version" be the sha1 of that block of boilerplate... the trade-offs > depend on what exactly is getting shared. I actually thought a lot about this way back, and one of the tricky bits is that identical C code can have entirely different (and incompatible) meanings depending on typedefs, macros etc. We'd have to use the preparsed c. Also, the runtime should be the union of utility code (some of it dynamically generated) which is different for each individual module. That being said, there's probably a huge chunk of common boilerplate (generators, bound functions, memory views) that could get shared without too much effort. We could also share .h files for much of this (which would cut down on disk useage and compilation time). Also, as a first pass, we could have a single, dynamically generated (and hash-named) runtime file for a single call to cythonize(...) that would allow much sharing withing a project without worrying about cross-project versioning (and it'd be easier logistically to create this as a new, external module rather than package it up in each .so file). I am curious what percentage of the final .so file is boilerplate. A huge chunk of the .c code often is, but that consists largely of specialize macros and inline functions that (ideally) optimize away to nearly nothing. >> I so can't wait seeing the surprise in the eyes of our users when they let >> their IDE reorganise their import order and their code starts crashing. :] > > That's easy to fix, just don't create such bugs ;-). > > -n > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From pav at iki.fi Tue Apr 9 21:11:11 2013 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 09 Apr 2013 22:11:11 +0300 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <5164229D.5080200@behnel.de> Message-ID: 09.04.2013 22:00, Robert Bradshaw kirjoitti: [clip] > I am curious what percentage of the final .so file is boilerplate. A > huge chunk of the .c code often is, but that consists largely of > specialize macros and inline functions that (ideally) optimize away to > nearly nothing. FWIW, switching from np.ndarray+templating to fused types+memoryviews increased the stripped size of scipy.interpolate.interpnd [1] from 368k to 633k. [1] https://github.com/scipy/scipy/pull/211 -- Pauli Virtanen From stefan_ml at behnel.de Tue Apr 9 21:28:11 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Apr 2013 21:28:11 +0200 Subject: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking) In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <5164229D.5080200@behnel.de> Message-ID: <51646BCB.60806@behnel.de> Robert Bradshaw, 09.04.2013 21:00: > On Tue, Apr 9, 2013 at 7:22 AM, Nathaniel Smith wrote: >> On Tue, Apr 9, 2013 at 3:15 PM, Stefan Behnel wrote: >>> Ok, got it now. That solves the distribution problem, assuming that all >>> installed runtimes with a given version are equivalent. Basically, we'd >>> move the code out and then cimport the stuff back that we need, which would >>> then let the first import of a given runtime version insert it into >>> sys.modules. >> >> Yeah, that's the definition of "given version" :-), and any kind of >> shared runtime does require versioning. If we wanted to be extra >> careful we could put the shared code into its own block of boilerplate >> that gets injected into each generated .c file, and then have the >> "version" be the sha1 of that block of boilerplate... the trade-offs >> depend on what exactly is getting shared. > > I actually thought a lot about this way back, and one of the tricky > bits is that identical C code can have entirely different (and > incompatible) meanings depending on typedefs, macros etc. We'd have to > use the preparsed c. Also, the runtime should be the union of utility > code (some of it dynamically generated) which is different for each > individual module. > > That being said, there's probably a huge chunk of common boilerplate > (generators, bound functions, memory views) that could get shared > without too much effort. We could also share .h files for much of this > (which would cut down on disk useage and compilation time). Also, as a > first pass, we could have a single, dynamically generated (and > hash-named) runtime file for a single call to cythonize(...) that > would allow much sharing withing a project without worrying about > cross-project versioning (and it'd be easier logistically to create > this as a new, external module rather than package it up in each .so > file). Right, and that gives us two safe options already: users can build a single meta-module from multiple Cython modules, or they can build one shared Cython runtime library for an entire PyPI package. In both cases, no versioning is required, as the complete thing would always be expected to be built (and installed) in one go. > I am curious what percentage of the final .so file is boilerplate. A > huge chunk of the .c code often is, but that consists largely of > specialize macros and inline functions that (ideally) optimize away to > nearly nothing. I think the advantage lies more in the ability to use exactly the same type for functions, generators and memory views across a larger code base, rather than having a separate one for each individual module. I don't think the average utility function would make any difference, especially because most of them are utility functions exactly because we want to inline them. There's a possible exception for really high-level code like import, print, exec or things like that, but I don't think that weighs in to any real amount, even when you sum it up. So, sharing types is good, sharing functions is IMHO mostly useless. Stefan From nikita at nemkin.ru Wed Apr 10 06:22:29 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 10 Apr 2013 10:22:29 +0600 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> Message-ID: Robert Bradshaw, 09.04.2013 20:16: >>> Yep. We could even create stub .so files for the included ones that >>> did nothing but import the big one to avoid having to worry about >>> importing things in the right order. The stubs don't have to be shared libraries, pure python would work just fine (and reduce the complexity). >>> The big question is what API to specify that things should be linked >>> together. An option to cythonize(...) could be used, but for some >>> projects (e.g. Sage) one would want more control over which modules >>> get bundled together rather than all-or-nothing. >> >> No-one forces you to use a plain cythonize("**/*.pyx"). Just be more >> specific about what files to include in each pattern that you pass to >> in. >> And you can always call cythonize() more than once if you need to. Once >> for >> each meta-module should usually be acceptable, given that the bundled >> source modules would tend to have a common configuration anyway. > > That would still be painful. Ideally, users should never have to > modify the setup.py file. Users have to plan for what to bundle, what the package structure will be, etc. It is not an "enable and forget" type of thing. (Unless you have an all-Cython package tree.) I prefer explicitly creating "bundle" extensions with distutil.core.Extension, passing multiple pyx files as sources, then passing the result to cythonize. If you really want to push this into cythonize (it already does more than it should IMO), one option is to add a new comment directive, for example: # cython: bundle = mymodule._speedups similar to how distutils options are passed. All .pyx files with the same "bundle" value will be put in that bundle. >>>> There may be glitches with stuff like "__file__" and friends, but that >>>> should not be any worse than the current way of workings. >>> >>> This can probably be manipulated by Cython, though it's unclear what >>> the best value would be. The best value is the shared library pathname. All extension modules have it like that. The fact that multiple modules share the same .so is (luckily) irrelevant to the Python import system. Best regards, Nikita Nemkin From stefan_ml at behnel.de Wed Apr 10 07:24:33 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Apr 2013 07:24:33 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> Message-ID: <5164F791.2070704@behnel.de> Nikita Nemkin, 10.04.2013 06:22: > Robert Bradshaw, 09.04.2013 20:16: >>>> Yep. We could even create stub .so files for the included ones that >>>> did nothing but import the big one to avoid having to worry about >>>> importing things in the right order. > > The stubs don't have to be shared libraries, pure python would work > just fine (and reduce the complexity). Sure. That's actually the most common way to do it. Most C modules in CPython's stdlib have a Python wrapper script, for example. It's also not uncommon to add a certain amount of functionality in that script. That makes even just "having it there" a feature for future development. >>>> The big question is what API to specify that things should be linked >>>> together. An option to cythonize(...) could be used, but for some >>>> projects (e.g. Sage) one would want more control over which modules >>>> get bundled together rather than all-or-nothing. >>> >>> No-one forces you to use a plain cythonize("**/*.pyx"). Just be more >>> specific about what files to include in each pattern that you pass to in. >>> And you can always call cythonize() more than once if you need to. Once for >>> each meta-module should usually be acceptable, given that the bundled >>> source modules would tend to have a common configuration anyway. >> >> That would still be painful. Ideally, users should never have to >> modify the setup.py file. > > Users have to plan for what to bundle, what the package structure > will be, etc. It is not an "enable and forget" type of thing. > (Unless you have an all-Cython package tree.) Right. And even in that case, you'd want to make sure that importing one little module doesn't instantiate loads of useless other modules. That would just kill your startup time and waste memory. So this is really about careful layout and planning. Maybe compiling whole packages isn't all that a good idea after all, unless at least most of the modules in that package are actually required for the core functionality. > I prefer explicitly creating "bundle" extensions with > distutil.core.Extension, passing multiple pyx files as sources, > then passing the result to cythonize. And I don't see why we should not require users to do it exactly like this. As you said, it's an explicit thing anyway, so making it explicit in the setup.py script seems like a good way to express it to me. Sage as a huge project might be an exception, but I'm sure there are not many projects of similar size out there. And even in Sage, ISTM that a single point of explicit packaging configuration would be better than implicit meta-package building inside of cythonize. In the worst case, it can always be externalised into some kind of .ini or JSON file and loaded from there in a generic way. > If you really want to push this into cythonize (it already does more > than it should IMO), one option is to add a new comment directive, > for example: > # cython: bundle = mymodule._speedups > similar to how distutils options are passed. > All .pyx files with the same "bundle" value will be put in that bundle. I don't consider it a source level thing. It's a build time packaging thing, so it should be a build time decision, which makes setup.py the right place to express it. >>>>> There may be glitches with stuff like "__file__" and friends, but that >>>>> should not be any worse than the current way of workings. >>>> >>>> This can probably be manipulated by Cython, though it's unclear what >>>> the best value would be. > > The best value is the shared library pathname. All extension modules > have it like that. The fact that multiple modules share the same .so > is (luckily) irrelevant to the Python import system. The problem is not which value to use. The problem is that the shared library pathname is not known at init time, except inside of the loader, and it doesn't tell us about it. In the specific case of a meta-module, it won't even be set before *all* init functions of the submodules were called and the main init function terminates. Which is a really bad situation, because it means that even though the main module will eventually have its __file__ set by the loader, it has no way to propagate it to the submodules anymore. So they won't have their __file__ property set at all. Just in case someone missed it, here's the bug URL once again: http://bugs.python.org/issue13429 Cython can fix up stuff related to the FQMN and sys.modules (and it does that already), but can't do anything about the file path. That makes things like resource loading rather annoying. BTW, here's also the python-dev discussion on the topic: http://thread.gmane.org/gmane.comp.python.devel/135764 Maybe that discussion should be revived and this use case added to it. Stefan From robertwb at gmail.com Wed Apr 10 08:41:36 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 9 Apr 2013 23:41:36 -0700 Subject: [Cython] Shared Cython runtime In-Reply-To: <5164F791.2070704@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> <5164F791.2070704@behnel.de> Message-ID: On Tue, Apr 9, 2013 at 10:24 PM, Stefan Behnel wrote: > Nikita Nemkin, 10.04.2013 06:22: >> Robert Bradshaw, 09.04.2013 20:16: >>>>> Yep. We could even create stub .so files for the included ones that >>>>> did nothing but import the big one to avoid having to worry about >>>>> importing things in the right order. >> >> The stubs don't have to be shared libraries, pure python would work >> just fine (and reduce the complexity). > > Sure. That's actually the most common way to do it. Most C modules in > CPython's stdlib have a Python wrapper script, for example. It's also not > uncommon to add a certain amount of functionality in that script. That > makes even just "having it there" a feature for future development. > > >>>>> The big question is what API to specify that things should be linked >>>>> together. An option to cythonize(...) could be used, but for some >>>>> projects (e.g. Sage) one would want more control over which modules >>>>> get bundled together rather than all-or-nothing. >>>> >>>> No-one forces you to use a plain cythonize("**/*.pyx"). Just be more >>>> specific about what files to include in each pattern that you pass to in. >>>> And you can always call cythonize() more than once if you need to. Once for >>>> each meta-module should usually be acceptable, given that the bundled >>>> source modules would tend to have a common configuration anyway. >>> >>> That would still be painful. Ideally, users should never have to >>> modify the setup.py file. >> >> Users have to plan for what to bundle, what the package structure >> will be, etc. It is not an "enable and forget" type of thing. >> (Unless you have an all-Cython package tree.) > > Right. And even in that case, you'd want to make sure that importing one > little module doesn't instantiate loads of useless other modules. That > would just kill your startup time and waste memory. So this is really about > careful layout and planning. > > Maybe compiling whole packages isn't all that a good idea after all, unless > at least most of the modules in that package are actually required for the > core functionality. > > >> I prefer explicitly creating "bundle" extensions with >> distutil.core.Extension, passing multiple pyx files as sources, >> then passing the result to cythonize. > > And I don't see why we should not require users to do it exactly like this. > As you said, it's an explicit thing anyway, so making it explicit in the > setup.py script seems like a good way to express it to me. > > Sage as a huge project might be an exception, but I'm sure there are not > many projects of similar size out there. And even in Sage, ISTM that a > single point of explicit packaging configuration would be better than > implicit meta-package building inside of cythonize. In the worst case, it > can always be externalised into some kind of .ini or JSON file and loaded > from there in a generic way. > > >> If you really want to push this into cythonize (it already does more >> than it should IMO), one option is to add a new comment directive, >> for example: >> # cython: bundle = mymodule._speedups >> similar to how distutils options are passed. >> All .pyx files with the same "bundle" value will be put in that bundle. Yeah, something like this might make sense. Though it's less natural from a build perspective, it might be nicer to specify the dependency the other way. > I don't consider it a source level thing. It's a build time packaging > thing, so it should be a build time decision, which makes setup.py the > right place to express it. With Sage it's a major pain to keep the explicit extension module list in sync with the source tree, especially the (transitive) requirements like C++ness and referenced libraries. - Robert From nikita at nemkin.ru Wed Apr 10 08:44:29 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 10 Apr 2013 12:44:29 +0600 Subject: [Cython] Shared Cython runtime In-Reply-To: <5164F791.2070704@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> <5164F791.2070704@behnel.de> Message-ID: On Wed, 10 Apr 2013 11:24:33 +0600, Stefan Behnel wrote: > Nikita Nemkin, 10.04.2013 06:22: >> The stubs don't have to be shared libraries, pure python would work >> just fine (and reduce the complexity). > > Sure. That's actually the most common way to do it. Most C modules in > CPython's stdlib have a Python wrapper script, for example. It's also not > uncommon to add a certain amount of functionality in that script. That > makes even just "having it there" a feature for future development. Cython's ability to create self-contained binaries is an important feature. For bundles of unrelated modules, stubs seem to be the only option, but whole package bundles can do without them. > Maybe compiling whole packages isn't all that a good idea after all, > unless at least most of the modules in that package are actually required > for the core functionality. In-binary import is quite fast. No extra file access, no parsing. Capsule indirection can be eliminated, some other things optimized... As an option, PEP 302 (Import hooks) provides an elegant way to implement delayed imports for whole package bundles. > The problem is not which value to use. The problem is that the shared > library pathname is not known at init time, except inside of the loader, > and it doesn't tell us about it. In the specific case of a meta-module, > it won't even be set before *all* init functions of the submodules were > called and the main init function terminates. > > Which is a really bad situation, because it means that even though the > main > module will eventually have its __file__ set by the loader, it has no way > to propagate it to the submodules anymore. So they won't have their > __file__ property set at all. > > Just in case someone missed it, here's the bug URL once again: > > http://bugs.python.org/issue13429 > > Cython can fix up stuff related to the FQMN and sys.modules (and it does > that already), but can't do anything about the file path. That makes > things like resource loading rather annoying. > > BTW, here's also the python-dev discussion on the topic: > > http://thread.gmane.org/gmane.comp.python.devel/135764 > > Maybe that discussion should be revived and this use case added to it. Well, here is the workaround Cython can use: https://gist.github.com/nnemkin/5352088 (Warning: I haven't fully tested it yet). Best regards, Nikita Nemkin From stefan_ml at behnel.de Wed Apr 10 08:57:48 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Apr 2013 08:57:48 +0200 Subject: [Cython] Shared Cython runtime In-Reply-To: References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> <5164F791.2070704@behnel.de> Message-ID: <51650D6C.7040006@behnel.de> Nikita Nemkin, 10.04.2013 08:44: > On Wed, 10 Apr 2013 11:24:33 +0600, Stefan Behnel wrote: >> The problem is not which value to use. The problem is that the shared >> library pathname is not known at init time, except inside of the loader, >> and it doesn't tell us about it. In the specific case of a meta-module, >> it won't even be set before *all* init functions of the submodules were >> called and the main init function terminates. >> >> Which is a really bad situation, because it means that even though the main >> module will eventually have its __file__ set by the loader, it has no way >> to propagate it to the submodules anymore. So they won't have their >> __file__ property set at all. >> >> Just in case someone missed it, here's the bug URL once again: >> >> http://bugs.python.org/issue13429 >> >> Cython can fix up stuff related to the FQMN and sys.modules (and it does >> that already), but can't do anything about the file path. That makes >> things like resource loading rather annoying. >> >> BTW, here's also the python-dev discussion on the topic: >> >> http://thread.gmane.org/gmane.comp.python.devel/135764 >> >> Maybe that discussion should be revived and this use case added to it. > > Well, here is the workaround Cython can use: > https://gist.github.com/nnemkin/5352088 > (Warning: I haven't fully tested it yet). Interesting. Looks like that could help here, yes. Do you know if it also works for statically linked binaries? (i.e. would it return the name of the executable in that case?) Stefan From nikita at nemkin.ru Wed Apr 10 09:05:49 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 10 Apr 2013 13:05:49 +0600 Subject: [Cython] Shared Cython runtime In-Reply-To: <51650D6C.7040006@behnel.de> References: <51640E0B.90306@behnel.de> <51641366.2090202@behnel.de> <516416A0.9070203@behnel.de> <51642A02.7060404@behnel.de> <51642FDC.3030500@behnel.de> <516431F5.6090706@behnel.de> <51645F8B.9050100@behnel.de> <5164F791.2070704@behnel.de> <51650D6C.7040006@behnel.de> Message-ID: On Wed, 10 Apr 2013 12:57:48 +0600, Stefan Behnel wrote: > Nikita Nemkin, 10.04.2013 08:44: >> Well, here is the workaround Cython can use: >> https://gist.github.com/nnemkin/5352088 >> (Warning: I haven't fully tested it yet). > > Interesting. Looks like that could help here, yes. Do you know if it also > works for statically linked binaries? (i.e. would it return the name of > the > executable in that case?) It will return the executable name, but correct behavior for statically linked modules is to not have __file__ attribute at all. There is probably a #define that tells when the module is compiled statically, I just haven't looked into it yet. Best regards, Nikita Nemkin From gandalf013 at gmail.com Wed Apr 10 22:36:27 2013 From: gandalf013 at gmail.com (Alok Singhal) Date: Wed, 10 Apr 2013 13:36:27 -0700 Subject: [Cython] Bug(?): chained assignments and tuple unpacking Message-ID: Hi, The following code (also attached as a .pyx file) fails in the current Cython : cdef class test: pass def main(): cdef test a, b, c, d (a, b) = (c, d) = (None, None) When run, I get the following traceback: In [1]: import assign_test In [2]: assign_test.main() Traceback (most recent call last): File "", line 1, in assign_test.main() File "assign_test.pyx", line 6, in assign_test.main (assign_test.c:669) (a, b) = (c, d) = (None, None) TypeError: Cannot convert NoneType to assign_test.test I am using the latest Cython development version (from github). Also tested with Cython 0.18. $ git rev-parse HEAD e8bd1789905a58d8224e4be5019be401f360aa54 I don't get an error when I change the assignments to either: (a, b) = (None, None) or a = b = None Is this a bug in Cython? I tried doing a git bisect and it seems like commit ea6a71acb5c79afb080855be1cb6ca30d283ec25 is when the above code started failing (It works with the previous commit). Thanks, Alok -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: assign_test.pyx Type: application/octet-stream Size: 137 bytes Desc: not available URL: From robertwb at gmail.com Wed Apr 10 22:55:55 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 10 Apr 2013 13:55:55 -0700 Subject: [Cython] Bug(?): chained assignments and tuple unpacking In-Reply-To: References: Message-ID: Thanks for the detailed report. This does look like a bug. On Wed, Apr 10, 2013 at 1:36 PM, Alok Singhal wrote: > Hi, > > The following code (also attached as a .pyx file) fails in the current > Cython : > > cdef class test: > pass > > def main(): > cdef test a, b, c, d > (a, b) = (c, d) = (None, None) > > When run, I get the following traceback: > > In [1]: import assign_test > In [2]: assign_test.main() > Traceback (most recent call last): > File "", line 1, in > assign_test.main() > File "assign_test.pyx", line 6, in assign_test.main (assign_test.c:669) > (a, b) = (c, d) = (None, None) > TypeError: Cannot convert NoneType to assign_test.test > > I am using the latest Cython development version (from github). Also tested > with Cython 0.18. > > $ git rev-parse HEAD > e8bd1789905a58d8224e4be5019be401f360aa54 > > I don't get an error when I change the assignments to either: > > (a, b) = (None, None) > > or > > a = b = None > > Is this a bug in Cython? I tried doing a git bisect and it seems like > commit ea6a71acb5c79afb080855be1cb6ca30d283ec25 is when the above code > started failing (It works with the previous commit). > > Thanks, > Alok > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Apr 12 11:29:18 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 12 Apr 2013 11:29:18 +0200 Subject: [Cython] Bug(?): chained assignments and tuple unpacking In-Reply-To: References: Message-ID: <5167D3EE.4060002@behnel.de> Alok Singhal, 10.04.2013 22:36: > The following code (also attached as a .pyx file) fails in the current > Cython : > > cdef class test: > pass > > def main(): > cdef test a, b, c, d > (a, b) = (c, d) = (None, None) > > When run, I get the following traceback: > > In [1]: import assign_test > In [2]: assign_test.main() > Traceback (most recent call last): > File "", line 1, in > assign_test.main() > File "assign_test.pyx", line 6, in assign_test.main (assign_test.c:669) > (a, b) = (c, d) = (None, None) > TypeError: Cannot convert NoneType to assign_test.test > > I am using the latest Cython development version (from github). Also > tested with Cython 0.18. > > $ git rev-parse HEAD > e8bd1789905a58d8224e4be5019be401f360aa54 > > I don't get an error when I change the assignments to either: > > (a, b) = (None, None) > > or > > a = b = None > > Is this a bug in Cython? I tried doing a git bisect and it seems like > commit ea6a71acb5c79afb080855be1cb6ca30d283ec25 is when the above code > started failing (It works with the previous commit). Thanks for the report and the excellent analysis. Here is a fix: https://github.com/cython/cython/commit/eb1e7173133a5dcc266203e8928c60b2e3b8446a Plus, the generated code was horribly redundant. I cleaned it up. https://github.com/cython/cython/commit/2592064271ea1c1b9ad1844232c5890e06861eb3 Stefan From stefan_ml at behnel.de Fri Apr 12 14:16:52 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 12 Apr 2013 14:16:52 +0200 Subject: [Cython] Cython 0.19 beta 2 released Message-ID: <5167FB34.60709@behnel.de> Hi, I uploaded a second (and hopefully last) beta release for Cython 0.19. Please give it a try with your code. http://cython.org/release/Cython-0.19b2.tar.gz http://cython.org/release/Cython-0.19b2.zip Cython 0.19 features many enhancements and changes regarding performance, usability and Python compatibility, including some very long standing bugs and issues. The changelog is on github: https://github.com/cython/cython/blob/58131b68dc033fc7ca269d875a2aab2b4e9646a2/CHANGES.rst The latest documentation is here: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ Have fun, Stefan From matej at laitl.cz Fri Apr 12 14:20:42 2013 From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl) Date: Fri, 12 Apr 2013 14:20:42 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: <2179674.3aDoa4uPCt@edgy> References: <2179674.3aDoa4uPCt@edgy> Message-ID: <13797561.fK1teVm5Zm@edgy> On 8.?4.?2013 Mat?j Laitl wrote: > Hi cython-devel and Mark, > I was getting > > > Fatal Python error: Acquisition count is 0 (line XYZ) > > when I was doing > > > cdef class MemViewContainer: > > cdef double[:, :] A > > > > cdef a_method(self): > > self.A = np.eye(2) > > some_function(self.A.T) > > some_function(self.A.T) > > I have found out that it is caused by the self.A.T expression - Cython emits > __PYX_XDEC_MEMVIEW() after the call, but no __PYX_INC_MEMVIEW() before the > call. This doesn't happen if the memoryview is a function-local variable. > > Proper test case is in the pull request [1] along with an ad-hoc patch that > fixes the problem here, but needs review whether it is actually correct. I'd > be very grateful if a fix for this problem could get it into Cython 0.19. > > Cython version: 0.19b1 3a6b9856187d7e490e08 > > [1] https://github.com/cython/cython/pull/201 Bump. Anything else I can provide to help with getting this fixed in 0.19? Ragards, Mat?j From hroest_nospam2333 at quantentunnel.de Fri Apr 12 17:04:01 2013 From: hroest_nospam2333 at quantentunnel.de (=?UTF-8?B?SGFubmVzIFLDtnN0?=) Date: Fri, 12 Apr 2013 17:04:01 +0200 Subject: [Cython] Generated C++ code incompatible with Clang Message-ID: Dear mailing list In the process of developing a rather large Cython project where we wrap the scientific OpenMS library (http://sourceforge.net/projects/open-ms/) using Cython, we have come across an issue with the C++ code that is generated by Cython. The issue is that Cython generates C++ that is not compatible with the Clang compiler. I hope this is the correct mailing list to report this issue since it seems to be a bug in Cython and not in Clang (after some discussion we have concluded that Clang is most likely correct in rejecting the generated code). Since we do not have access to the issue tracker, we hoped posting our issue here would clear up matters. The issue occurs with the following minimal testcase: from libcpp.vector cimport vector as libcpp_vector from cython.operator cimport dereference as deref, preincrement as inc cdef class TestClass: cdef libcpp_vector[float] inst def __iter__(self): it = self.inst.begin() while it != self.inst.end(): yield deref(it) inc(it) When compiled with Cython to C++, it generates C++ that cannot be compiled with Clang (however it seems that gcc and MSVS accept the code). It seems that the same issue with Clang was already discussed here: https://bugzilla.mozilla.org/show_bug.cgi?id=623303 and the conclusion was that Clang is correct in reporting the code as erroneous. In the above case, the invalid C++ code that gets generated is: p->__pyx_v_it.std::vector::iterator::~iterator(); The correct code for the above case would probably be (this is just a suggestion, it compiles with gcc and clang on our machines and our tests run through with it): typedef std::vector::iterator _it; p->__pyx_v_it.~_it(); We have tested it with the 0.18 release as well as the newest 0.19b2 build from github (58131b68dc033fc7ca269d875a2aab2b4e9646a2) and the results were the same. Thanks, Hannes R?st -- Hannes R?st Institute of Molecular Systems Biology ETH Z?rich H?nggerberg HPT C75 Wolfgang-Pauli Strasse 16 CH-8093 Z?rich (Switzerland) phone: 0041 44 633 3945 fax: 0041 44 633 10 51 e-mail: roest at imsb.biol.ethz.ch From markflorisson88 at gmail.com Fri Apr 12 18:37:05 2013 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 12 Apr 2013 17:37:05 +0100 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: <13797561.fK1teVm5Zm@edgy> References: <2179674.3aDoa4uPCt@edgy> <13797561.fK1teVm5Zm@edgy> Message-ID: On 12 April 2013 13:20, Mat?j Laitl wrote: > On 8.?4.?2013 Mat?j Laitl wrote: > > Hi cython-devel and Mark, > > I was getting > > > > > Fatal Python error: Acquisition count is 0 (line XYZ) > > > > when I was doing > > > > > cdef class MemViewContainer: > > > cdef double[:, :] A > > > > > > cdef a_method(self): > > > self.A = np.eye(2) > > > some_function(self.A.T) > > > some_function(self.A.T) > > > > I have found out that it is caused by the self.A.T expression - Cython > emits > > __PYX_XDEC_MEMVIEW() after the call, but no __PYX_INC_MEMVIEW() before > the > > call. This doesn't happen if the memoryview is a function-local variable. > > > > Proper test case is in the pull request [1] along with an ad-hoc patch > that > > fixes the problem here, but needs review whether it is actually correct. > I'd > > be very grateful if a fix for this problem could get it into Cython 0.19. > > > > Cython version: 0.19b1 3a6b9856187d7e490e08 > > > > [1] https://github.com/cython/cython/pull/201 > > Bump. Anything else I can provide to help with getting this fixed in 0.19? > > Ragards, > Mat?j > Thanks for the bump, I merged it. Stefan, do you want to wait with 0.19 for the strides fix? We don't know when 0.20 would be, I'd like to give it a go this weekend. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Apr 12 18:44:40 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 12 Apr 2013 18:44:40 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: References: <2179674.3aDoa4uPCt@edgy> <13797561.fK1teVm5Zm@edgy> Message-ID: <516839F8.1030907@behnel.de> mark florisson, 12.04.2013 18:37: > Stefan, do you want to wait with 0.19 for the strides fix? We don't know > when 0.20 would be, I'd like to give it a go this weekend. No problem, I can wait. Let's plan the RC for early next week, if you can get a fix ready by then. Stefan From bos at serpentine.com Sat Apr 13 05:02:23 2013 From: bos at serpentine.com (Bryan O'Sullivan) Date: Fri, 12 Apr 2013 20:02:23 -0700 Subject: [Cython] Cython mishandles None in slices Message-ID: Here are two lines of valid Python: >>> 'abc'[None:2] 'ab' >>> 'abc'[1:None] 'bc' If I try this in code that I compile with Cython, it throws an exception: TypeError: 'NoneType' object cannot be interpreted as an index -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Apr 13 08:19:26 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 13 Apr 2013 08:19:26 +0200 Subject: [Cython] Cython mishandles None in slices In-Reply-To: References: Message-ID: <5168F8EE.8080709@behnel.de> Bryan O'Sullivan, 13.04.2013 05:02: > Here are two lines of valid Python: > > >>> 'abc'[None:2] > 'ab' > >>> 'abc'[1:None] > 'bc' > > If I try this in code that I compile with Cython, it throws an exception: > > TypeError: 'NoneType' object cannot be interpreted as an index Have you tried it with the latest beta? http://cython.org/release/Cython-0.19b2.tar.gz Stefan From stefan_ml at behnel.de Sun Apr 14 08:43:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 14 Apr 2013 08:43:28 +0200 Subject: [Cython] [cython-users] Property + C-level attribute with the same name In-Reply-To: References: Message-ID: <516A5010.9010004@behnel.de> Nikita Nemkin, 13.04.2013 10:07: > The following code exposes a C-level pointer as > an integer property _with the same name_. > ("public" won't work here because type conversion is required.) > > cdef class A: > cdef void* attr > > property attr: > def __get__(self): > return self.attr > > I have expected a naming error, but it works perfectly. > Can I rely on this behavior or is it accidental? I don't think it's intentional, and it certainly feels like a bug. Although I agree that there might be use in it, it's hard to tell if this is a special case that is useful enough to break the rules. A hugely more general feature would be user defined type conversions, for example, which would make "public" work here and this case could become the compile error that I think it should be. Other opinions? Stefan From stefan_ml at behnel.de Sun Apr 14 09:40:31 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 14 Apr 2013 09:40:31 +0200 Subject: [Cython] nogil declaration on C++ classes (ticket 805) Message-ID: <516A5D6F.5060209@behnel.de> Hi, this patch looks ok to me: http://trac.cython.org/cython_trac/ticket/805 Any objections from the C++ users against including it in 0.19? Stefan From matej at laitl.cz Sun Apr 14 11:03:06 2013 From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl) Date: Sun, 14 Apr 2013 11:03:06 +0200 Subject: [Cython] nogil declaration on C++ classes (ticket 805) In-Reply-To: <516A5D6F.5060209@behnel.de> References: <516A5D6F.5060209@behnel.de> Message-ID: <2551755.qiMoLR7t9Q@edgy> On 14.?4.?2013 Stefan Behnel wrote: > Hi, > this patch looks ok to me: > > http://trac.cython.org/cython_trac/ticket/805 > Any objections from the C++ users against including it in 0.19? Quite the opposite, certainly an improvement. Mat?j Laitl From nikita at nemkin.ru Sun Apr 14 11:33:14 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sun, 14 Apr 2013 15:33:14 +0600 Subject: [Cython] bint and autodoc signatures Message-ID: Hi, Arguments, return values and properties of type 'bint' appear as 'int' in autogenerated signatures. This is very confusing for the end user and logically wrong. (For one thing, bint and int handle input arguments differently: bint accepts pretty much any Python object, while int requries something numeric.) Would it be acceptable to change bint display presentation to 'bool' ? Note: other primitive types (short, int, float, long long etc) don't have this problem, because they are all numeric and coerce to/from Python numerics in an obvious way. Best regards, Nikita Nemkin From stefan_ml at behnel.de Sun Apr 14 11:49:22 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 14 Apr 2013 11:49:22 +0200 Subject: [Cython] bint and autodoc signatures In-Reply-To: References: Message-ID: <516A7BA2.1050109@behnel.de> Nikita Nemkin, 14.04.2013 11:33: > Arguments, return values and properties of type 'bint' > appear as 'int' in autogenerated signatures. > > This is very confusing for the end user and logically wrong. > (For one thing, bint and int handle input arguments differently: > bint accepts pretty much any Python object, while int requries > something numeric.) +1 > Would it be acceptable to change bint display presentation > to 'bool' ? > > Note: other primitive types (short, int, float, long long etc) > don't have this problem, because they are all numeric > and coerce to/from Python numerics in an obvious way. The problem I see with "bool" is that it would still be different from, say, "list" or "MyExtType". Where the latter only accept that type and reject anything else with a TypeError, "bool" would accept any object and do a conversion instead. So "bool x" isn't correct either. I'm also not sure if "bint" is a good idea as normal Python users won't know it, so it's just as confusing as "int". That being said, I guess I would find it less confusing to read "def func(x : bool)" than "def func(bool x)" as signature description in a docstring. So maybe that's the way to go eventually. Stefan From matej at laitl.cz Mon Apr 15 17:47:43 2013 From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl) Date: Mon, 15 Apr 2013 17:47:43 +0200 Subject: [Cython] [PATCH] Re: view[0].methodcall() produces invalid C when view is memoryview of extension type In-Reply-To: <1401112.QzbjNdWC9X@edgy> References: <1401112.QzbjNdWC9X@edgy> Message-ID: <1673988.Nlpse4j0qW@edgy> Hi again, especially Mark, I have some new observations and a patch regarding this problem, see below. On 23.?3.?2013 Mat?j Laitl wrote: > following test code produces C code that fails to compile: > > cdef class ExtensionType(object): > > cdef public int dummy > > > > def __init__(self, n): > > self.dummy = n > > > > cdef cfoo(self): > > print self.dummy > > > > items = [ExtensionType(1), ExtensionType(2)] > > cdef ExtensionType[:] view = np.array(items, dtype=ExtensionType) > > view[0].cfoo() > > with gcc error and relevant C file lines: > extension_type_memoryview.c:2604:94: error: ?PyObject? has no member named > ?__pyx_vtab? > > 2570: PyObject *__pyx_t_1 = NULL; > (...) > 2601: __pyx_t_1 = (PyObject *) *((struct > > :__pyx_obj_25extension_type_memoryview_ExtensionType * *) ( /* dim=0 */ > : > : (__pyx_v_25extension_type_memoryview_view.data + __pyx_t_2 * > : __pyx_v_25extension_type_memoryview_view.strides[0]) )); > > 2602: __Pyx_INCREF((PyObject*)__pyx_t_1); > 2603: /* __pyx_t_4 allocated */ > 2604: __pyx_t_4 = ((struct > > : __pyx_vtabstruct_25extension_type_memoryview_ExtensionType *)__pyx_t_1 > : > :->__pyx_vtab)->cfoo(__pyx_t_1); if (unlikely(!__pyx_t_4)) > :{__pyx_filename > : > : = __pyx_f[0]; __pyx_lineno = 69; > > It seems that generic PyObject* temporary for __pyx_t_1 is used here while > typed ExtensionType* temporary should have been used instead (as suggested > by excess casting on line 2601). I've found this is caused by following check in ExprNode.allocate_temp_result(): > if type.is_pyobject: > type = PyrexTypes.py_object_type Removing the check breaks everything else, so I suppose it is intentional (what's its purpose?) and I've resorted to overriding allocate_temp_result() in IndexNode and skipping the check in the very special case of memoryview direct indexing. This fixes the test-case and doesn't break other tests, but sounds a bit hacky and may be actually completely incorrect. The patch and test-case is here: https://github.com/cython/cython/pull/213 P.S.: Is this too late for 0.19? Cheers, Mat?j From stefan_ml at behnel.de Mon Apr 15 19:38:32 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 15 Apr 2013 19:38:32 +0200 Subject: [Cython] [PATCH] Re: view[0].methodcall() produces invalid C when view is memoryview of extension type In-Reply-To: <1673988.Nlpse4j0qW@edgy> References: <1401112.QzbjNdWC9X@edgy> <1673988.Nlpse4j0qW@edgy> Message-ID: <516C3B18.50101@behnel.de> Mat?j Laitl, 15.04.2013 17:47: > P.S.: Is this too late for 0.19? Unless one of the other core developers signs that this is totally correct and safe and must be merged, I'd say yes, too late, sorry. Stefan From robertwb at gmail.com Tue Apr 16 23:57:51 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 16 Apr 2013 14:57:51 -0700 Subject: [Cython] bint and autodoc signatures In-Reply-To: <516A7BA2.1050109@behnel.de> References: <516A7BA2.1050109@behnel.de> Message-ID: On Sun, Apr 14, 2013 at 2:49 AM, Stefan Behnel wrote: > Nikita Nemkin, 14.04.2013 11:33: >> Arguments, return values and properties of type 'bint' >> appear as 'int' in autogenerated signatures. >> >> This is very confusing for the end user and logically wrong. >> (For one thing, bint and int handle input arguments differently: >> bint accepts pretty much any Python object, while int requries >> something numeric.) > > +1 > > >> Would it be acceptable to change bint display presentation >> to 'bool' ? >> >> Note: other primitive types (short, int, float, long long etc) >> don't have this problem, because they are all numeric >> and coerce to/from Python numerics in an obvious way. > > The problem I see with "bool" is that it would still be different from, > say, "list" or "MyExtType". Where the latter only accept that type and > reject anything else with a TypeError, "bool" would accept any object and > do a conversion instead. So "bool x" isn't correct either. > > I'm also not sure if "bint" is a good idea as normal Python users won't > know it, so it's just as confusing as "int". But it would encourage users to look it; they wouldn't assume they know what it means. > That being said, I guess I would find it less confusing to read > > "def func(x : bool)" > > than > > "def func(bool x)" > > as signature description in a docstring. So maybe that's the way to go > eventually. Much better than int. It's not that bad if the documented signature is a bit stricter than reality, especially as the coercion is obvious. - Robert From robertwb at gmail.com Wed Apr 17 07:40:01 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 16 Apr 2013 22:40:01 -0700 Subject: [Cython] [cython-users] Cython 0.19 beta 2 released In-Reply-To: <5167FB34.60709@behnel.de> References: <5167FB34.60709@behnel.de> Message-ID: It'd be really good to get in two last-minute changes uncovered by Sage https://github.com/cython/cython/pull/214 https://github.com/cython/cython/pull/215 On Fri, Apr 12, 2013 at 5:16 AM, Stefan Behnel wrote: > Hi, > > I uploaded a second (and hopefully last) beta release for Cython 0.19. > Please give it a try with your code. > > http://cython.org/release/Cython-0.19b2.tar.gz > > http://cython.org/release/Cython-0.19b2.zip > > > Cython 0.19 features many enhancements and changes regarding performance, > usability and Python compatibility, including some very long standing bugs > and issues. The changelog is on github: > > https://github.com/cython/cython/blob/58131b68dc033fc7ca269d875a2aab2b4e9646a2/CHANGES.rst > > > The latest documentation is here: > > https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ > > > Have fun, > > Stefan > > -- > > --- > You received this message because you are subscribed to the Google Groups "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > From sebastian at sipsolutions.net Wed Apr 17 12:32:01 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 17 Apr 2013 12:32:01 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: References: <2179674.3aDoa4uPCt@edgy> <13797561.fK1teVm5Zm@edgy> Message-ID: <1366194721.9563.34.camel@sebastian-laptop> On Fri, 2013-04-12 at 17:37 +0100, mark florisson wrote: > > > On 12 April 2013 13:20, Mat?j Laitl wrote: > On 8.?4.?2013 Mat?j Laitl wrote: > > Hi cython-devel and Mark, > > I was getting > > > > > Fatal Python error: Acquisition count is 0 (line XYZ) > > > > when I was doing > > > > > cdef class MemViewContainer: > > > cdef double[:, :] A > > > > > > cdef a_method(self): > > > self.A = np.eye(2) > > > some_function(self.A.T) > > > some_function(self.A.T) > > > > I have found out that it is caused by the self.A.T > expression - Cython emits > > __PYX_XDEC_MEMVIEW() after the call, but no > __PYX_INC_MEMVIEW() before the > > call. This doesn't happen if the memoryview is a > function-local variable. > > > > Proper test case is in the pull request [1] along with an > ad-hoc patch that > > fixes the problem here, but needs review whether it is > actually correct. I'd > > be very grateful if a fix for this problem could get it into > Cython 0.19. > > > > Cython version: 0.19b1 3a6b9856187d7e490e08 > > > > [1] https://github.com/cython/cython/pull/201 > > > Bump. Anything else I can provide to help with getting this > fixed in 0.19? > > Ragards, > Mat?j > > Thanks for the bump, I merged it. > > > Stefan, do you want to wait with 0.19 for the strides fix? We don't > know when 0.20 would be, I'd like to give it a go this weekend. Hey, if you have started on the strides fix or a PR, give me a ping and I will have a look at it if you like. If you still need tests, checking that these arrays work as both C and F-contiguous should be sufficient (of course numpy does not know about that yet, unless NPY_RELAXED_STRIDES_CHECKING is set): max_intp = np.iinfo(np.intp).max # obviously weird stride a = np.ones((1, 10, 1)) a.strides = (max_intp, a.itemsize, max_intp) and: a = np.ones((10, 0, 10)) a.strides = (max_intp,) * 3 Regards, Sebastian > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Wed Apr 17 13:39:09 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 17 Apr 2013 13:39:09 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: <1366194721.9563.34.camel@sebastian-laptop> References: <2179674.3aDoa4uPCt@edgy> <13797561.fK1teVm5Zm@edgy> <1366194721.9563.34.camel@sebastian-laptop> Message-ID: <516E89DD.6010709@behnel.de> Sebastian Berg, 17.04.2013 12:32: > On Fri, 2013-04-12 at 17:37 +0100, mark florisson wrote: >> Stefan, do you want to wait with 0.19 for the strides fix? We don't >> know when 0.20 would be, I'd like to give it a go this weekend. > > if you have started on the strides fix or a PR, give me a ping and I > will have a look at it if you like. It's already in the latest master, so please give it a try there. I was planning to release an RC shortly. Stefan From stefan_ml at behnel.de Wed Apr 17 13:47:25 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 17 Apr 2013 13:47:25 +0200 Subject: [Cython] [cython-users] Possible enumerate automatic conversion In-Reply-To: <516D4D7E.5070200@grinta.net> References: <516D4D7E.5070200@grinta.net> Message-ID: <516E8BCD.1050708@behnel.de> Daniele Nicolodi, 16.04.2013 15:09: > would it be desirable to have an enumerate() automatic conversion > similar to the range() conversion when the enumerated iterable is a C > array or a numpy array? > > http://docs.cython.org/src/userguide/pyrex_differences.html#automatic-range-conversion enumerate() is optimised since something like 0.13 or so, maybe longer. It certainly works for char* and maybe also other C arrays. > I imagine that detecting the opportunity for the optimization may be a > bit tricky, but it should not be much different than a combination of > what is done for iteration over a C array and for the range() function. Yes, the tricky bit is to get the iterable through the type analysis without coercion errors. This couldn't easily be done at the time but became possible with my recent analyse_types() phase changes. Essentially, for-loop optimisations can now be moved into the type analysis phase and take the type of the iterable into account without running into accidental coercion errors. I'm not sure yet if this is the right way to do it - maybe we still want an explicit transformation instead. In any case, the right place to discuss this is the cython-devel mailing list, not the users mailing list. Stefan From sebastian at sipsolutions.net Wed Apr 17 14:04:43 2013 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 17 Apr 2013 14:04:43 +0200 Subject: [Cython] [PATCH] Refcount error when transposing memoryview attribute of an extension class In-Reply-To: <516E89DD.6010709@behnel.de> References: <2179674.3aDoa4uPCt@edgy> <13797561.fK1teVm5Zm@edgy> <1366194721.9563.34.camel@sebastian-laptop> <516E89DD.6010709@behnel.de> Message-ID: <1366200283.9563.40.camel@sebastian-laptop> On Wed, 2013-04-17 at 13:39 +0200, Stefan Behnel wrote: > Sebastian Berg, 17.04.2013 12:32: > > On Fri, 2013-04-12 at 17:37 +0100, mark florisson wrote: > >> Stefan, do you want to wait with 0.19 for the strides fix? We don't > >> know when 0.20 would be, I'd like to give it a go this weekend. > > > > if you have started on the strides fix or a PR, give me a ping and I > > will have a look at it if you like. > > It's already in the latest master, so please give it a try there. I was > planning to release an RC shortly. > Ah, awesome. I only checked the pull requests sporadically and it slipped past my notice. Small point, I think it does not always work for higher dimensional arrays that have no elements, because then it would error out before finding out that the array is empty. This should be very rare though, so no need to rush to fix it I think, so thanks again! - Sebastian > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Apr 17 21:54:24 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 17 Apr 2013 21:54:24 +0200 Subject: [Cython] Cython 0.19 release candidate Message-ID: <516EFDF0.80303@behnel.de> Hi, I uploaded a release candidate for Cython 0.19. Note that there were changes to the buffer validation code for memory views since the last beta, so please give this release another try with your code, especially if you use memory views in it. http://cython.org/release/Cython-0.19rc1.tar.gz http://cython.org/release/Cython-0.19rc1.zip Cython 0.19 features many enhancements and changes regarding performance, usability and Python compatibility, including some very long standing bugs and issues. The changelog is on github: https://github.com/cython/cython/blob/0ee15b1829125b6844ac13f437c2d9bd61a4ceca/CHANGES.rst The latest documentation is here: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ Have fun, Stefan From stefan_ml at behnel.de Fri Apr 19 11:43:23 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Apr 2013 11:43:23 +0200 Subject: [Cython] [cython-users] Re: Cython Typed Memoryviews versus Cython+Numpy Speed ups In-Reply-To: References: <8804baca-9772-4805-a6e5-922aa03d4443@googlegroups.com> Message-ID: <517111BB.8060009@behnel.de> Robert Bradshaw, 17.04.2013 06:02: > On Tue, Apr 16, 2013 at 6:46 PM, pythonOmetrist >> Attached HTML seems mostly white, and not clear where to go from here. Nogil >> seems to be one option. > > Just a brief glance at the code, you have a lot of n_k_w[x][y] which > is vastly more expensive than n_k_w[x, y]. Is that always equivalent? If so, we might want to transform it internally. Stefan From stefan_ml at behnel.de Fri Apr 19 17:40:20 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Apr 2013 17:40:20 +0200 Subject: [Cython] Cython 0.19 released Message-ID: <51716564.8090500@behnel.de> Hi all, Cython 0.19 has cleared the dockyard. This is a feature release that adds some major usability features especially for code that needs to run in both Py2 and Py3, as well as better Python compatibility and optimisations. You can get it from here: http://cython.org/release/Cython-0.19.tar.gz http://cython.org/release/Cython-0.19.zip Release notes: https://github.com/cython/cython/blob/29bf3493fdc80cb3261a9dfb9f73e3ccd46fec66/CHANGES.rst Documentation: http://docs.cython.org/ The major new features include: * New directives ``c_string_type`` and ``c_string_encoding`` to automatically convert between C strings and the different Python string types. * Keyword arguments are supported for cdef functions, including external C/C++ functions. * New freelist decorator for extension types. * Fast extension type instantiation using the ``Type.__new__(Type)`` idiom has gained support for passing arguments. * Slicing was optimised for several builtin types and otherwise conforms better with Python semantics for user defined types. * The mapping of dict view/item methods in Python 3 was improved. Have fun, Stefan From stefan_ml at behnel.de Fri Apr 19 22:11:21 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Apr 2013 22:11:21 +0200 Subject: [Cython] [cython] Autodoc improvements (#216) In-Reply-To: References: Message-ID: <5171A4E9.10700@behnel.de> Nikita Nemkin, 18.04.2013 07:07: > Three improvements are included in this patch that I have found helpful when using Sphinx autodoc to document my project: > > 1) Default values preserve and display binary expressions and attribute access. For example, the following function will have its signature displayed as is: > > cpdef func(int flags=MyModule.FLAG1 | MyModule.FLAG2) > > This only affects expressions not reducible at compilation time, i.e. `2+2` will appear as `4`. > > 2) Public/readonly cdef attributes can have docstrings attached, using the [PEP 258 attribute docstring syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings). Example: > > cdef class X: > cdef public attr > """attr docstring""" > > When ``embedsignature=True`` Cython already adds a signature docstring to the generated property (assuming the example above). This patch allows to add an actual docstring too. > > Dosctrings on private cdef attributes will give a warning ("docstring ignored"). > Other forms and uses of attribute docstrings, as described in the PEP 258, are not supported at all (SyntaxError), because there is no standard introspection mechanism designed for them. ([PEP 224](http://www.python.org/dev/peps/pep-0224/) describing such mechanism had been rejected.) > > 3) `bint` appears as `bool` (instead of `int`) in autogenerated signatures, as I suggested on the mailing list. > > Each change is contained in its own commit in case you want to review them separately. > > https://github.com/cython/cython/pull/216 I like 1) and 3), but I'm not sure about 2). Worth noting that PEP 258 was rejected, although definitely for other reasons than this feature. I remember discussing this topic before on the mailing list, but don't remember if we reached a conclusion. I faintly recall that we preferred making the property explicit as soon as there's more to it than a plain 1:1 mapping to the cdef attribute. The above syntax is not wrong, but it also doesn't feel right to me. For example, it's impossible to draw a distinction between the above and something like cdef class X: cdef public attr """ quotes used for commenting out code here. """ And users are unlikely to notice that they just dropped some internals into the docstring of a property they didn't even know they were implementing. IMHO, the user intention is way too unclear in this syntactic construct. Stefan From stefan_ml at behnel.de Fri Apr 19 22:25:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Apr 2013 22:25:28 +0200 Subject: [Cython] gsoc-kurt-regraft branch and ticket #469 Message-ID: <5171A838.9010008@behnel.de> Hi, since the current code base can explicitly express dependencies between utility code sections, do we still need this branch? Or can we close it together with the ticket? ISTM that explicit dependencies are a more general feature than just honouring type hierarchies. Stefan From nikita at nemkin.ru Sat Apr 20 09:11:41 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sat, 20 Apr 2013 13:11:41 +0600 Subject: [Cython] [cython] Autodoc improvements (#216) In-Reply-To: <5171A4E9.10700@behnel.de> References: <5171A4E9.10700@behnel.de> Message-ID: On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: > Nikita Nemkin, 18.04.2013 07:07: >> 2) Public/readonly cdef attributes can have docstrings attached, using >> the [PEP 258 attribute docstring >> syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings). >> Example: >> >> cdef class X: >> cdef public attr >> """attr docstring""" >> >> When ``embedsignature=True`` Cython already adds a signature docstring >> to the generated property (assuming the example above). This patch >> allows to add an actual docstring too. >> >> Dosctrings on private cdef attributes will give a warning >> ("docstring ignored"). >> Other forms and uses of attribute docstrings, as described in the PEP >> 258, are not supported at all (SyntaxError), because there is no >> standard introspection mechanism designed for them. ([PEP >> 224](http://www.python.org/dev/peps/pep-0224/) describing such >> mechanism had been rejected.) > > Worth noting that PEP 258 was rejected, although definitely for other > reasons than this feature. I remember discussing this topic before on the > mailing list, but don't remember if we reached a conclusion. I faintly > recall that we preferred making the property explicit as soon as there's > more to it than a plain 1:1 mapping to the cdef attribute. Separating property from the attribute requires mass renaming of the attribute (as was recently descussed here) and bloating the code x4. All this just to attach a docstring? PEP 258 was rejected, but Sphinx supports it, an so does epydoc. The need to autodocument attributes remains. > The above syntax is not wrong, but it also doesn't feel right to me. For > example, it's impossible to draw a distinction between the above and > something like > > cdef class X: > cdef public attr > > """ > quotes used for commenting out code here. > """ > > And users are unlikely to notice that they just dropped some internals > into > the docstring of a property they didn't even know they were implementing. > > IMHO, the user intention is way too unclear in this syntactic construct. Attribute docstring must appear immediately after the attribute. The extra blank line in your example makes the string standalone, just as you intended. Also, internal comments in the middle of the class are not usually put into strings, actual comment syntax is more appropriate here. Documenting attributes manually, for example, with ".. attribute" directives in the class docstring or .rst file, creates the following problems: 1) In embedsignature mode (which is almost mandatory for a decent autodoc) Cython adds signature docstrings to public attributes, making them appear twice: one instance with a manually written doc, another instance with a signature docstring. 2) Manually documented attributes don't follow autodoc settings (ordering, formatting). They are also invisible to autosummary and other extensions. 3) (Obviously) non-uniformity between property, function and attribute documentation practice. Attribute docstrings are the cleanest solution to all of these problems. And the patch is just a few lines (if you discount the refactoring of the "XXX This should go to AutoDocTransforms" block.) Best regards, Nikita Nemkin From stefan_ml at behnel.de Sat Apr 20 10:27:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Apr 2013 10:27:12 +0200 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: References: <5171A4E9.10700@behnel.de> Message-ID: <51725160.6020008@behnel.de> Nikita Nemkin, 20.04.2013 09:11: > On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: >> Nikita Nemkin, 18.04.2013 07:07: >>> 2) Public/readonly cdef attributes can have docstrings attached, using >>> the [PEP 258 attribute docstring >>> syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings). >>> Example: >>> >>> cdef class X: >>> cdef public attr >>> """attr docstring""" >>> >>> When ``embedsignature=True`` Cython already adds a signature docstring >>> to the generated property (assuming the example above). This patch >>> allows to add an actual docstring too. >>> >>> Dosctrings on private cdef attributes will give a warning >>> ("docstring ignored"). >>> Other forms and uses of attribute docstrings, as described in the PEP >>> 258, are not supported at all (SyntaxError), because there is no >>> standard introspection mechanism designed for them. ([PEP >>> 224](http://www.python.org/dev/peps/pep-0224/) describing such mechanism >>> had been rejected.) >> >> Worth noting that PEP 258 was rejected, although definitely for other >> reasons than this feature. I remember discussing this topic before on the >> mailing list, but don't remember if we reached a conclusion. I faintly >> recall that we preferred making the property explicit as soon as there's >> more to it than a plain 1:1 mapping to the cdef attribute. > > Separating property from the attribute requires mass renaming of > the attribute (as was recently descussed here) and bloating the code x4. > All this just to attach a docstring? Well, once there is a docstring, that factor goes down to at most 3x. It's not entirely overkill, although I agree that it's definitely way more complexity than just adding a docstring. > PEP 258 was rejected, but Sphinx supports it, an so does epydoc. That makes it more reasonable then, but see below. >> The above syntax is not wrong, but it also doesn't feel right to me. For >> example, it's impossible to draw a distinction between the above and >> something like >> >> cdef class X: >> cdef public attr >> >> """ >> quotes used for commenting out code here. >> """ >> >> And users are unlikely to notice that they just dropped some internals into >> the docstring of a property they didn't even know they were implementing. >> >> IMHO, the user intention is way too unclear in this syntactic construct. > > Attribute docstring must appear immediately after the attribute. The extra > blank line in your example makes the string standalone, just as you intended. I can't see that from the tests you wrote. > Also, internal comments in the middle of the class are not usually > put into strings, actual comment syntax is more appropriate here. The less common it is, the more likely it is to be a source of confusion when it happens. And the proposed syntax is definitely prone to errors that go undetected. > Documenting attributes manually, for example, with ".. attribute" > directives in the class docstring or .rst file, creates the following > problems: > 1) In embedsignature mode (which is almost mandatory for a decent autodoc) It's not more than a clumsy work-around for the CPython quirk that C implemented functions couldn't expose their signature. And that problem is much better tackled by completing the support for PEP 362. http://www.python.org/dev/peps/pep-0362/ > 1) In embedsignature mode (which is almost mandatory for a decent > autodoc) Cython adds signature docstrings to public attributes, making > them appear twice: one instance with a manually written doc, another > instance with a signature docstring. > > 2) Manually documented attributes don't follow autodoc settings > (ordering, formatting). They are also invisible to autosummary and other > extensions. > > 3) (Obviously) non-uniformity between property, function and attribute > documentation practice. The way I see it, if the embedsignature feature actually died because of PEP 362, you could just document attributes yourself without having Cython interfere with it at all. And AFAIU, that would solve most of your above problems. > Attribute docstrings are the cleanest solution to all of these problems. > And the patch is just a few lines (if you discount the refactoring of the > "XXX This should go to AutoDocTransforms" block.) It's not about the implementation. The question is if this feature should become part of the language because it's "special enough to break the rules" of OOTDI and if so, how we should expose it. And yes, that's a wonderful field for bike shedding. Stefan From nikita at nemkin.ru Sat Apr 20 12:43:08 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sat, 20 Apr 2013 16:43:08 +0600 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: <51725160.6020008@behnel.de> References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> Message-ID: On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: > Nikita Nemkin, 20.04.2013 09:11: >> On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: >> Separating property from the attribute requires mass renaming of >> the attribute (as was recently descussed here) and bloating the code x4. >> All this just to attach a docstring? > > Well, once there is a docstring, that factor goes down to at most 3x. > It's > not entirely overkill, although I agree that it's definitely way more > complexity than just adding a docstring. > >> PEP 258 was rejected, but Sphinx supports it, an so does epydoc. > > That makes it more reasonable then, but see below. > >>> The above syntax is not wrong, but it also doesn't feel right to me. >>> For >>> example, it's impossible to draw a distinction between the above and >>> something like >>> >>> cdef class X: >>> cdef public attr >>> >>> """ >>> quotes used for commenting out code here. >>> """ >>> >>> And users are unlikely to notice that they just dropped some internals >>> into >>> the docstring of a property they didn't even know they were >>> implementing. >>> >>> IMHO, the user intention is way too unclear in this syntactic >>> construct. >> >> Attribute docstring must appear immediately after the attribute. The >> extra >> blank line in your example makes the string standalone, just as you >> intended. > > I can't see that from the tests you wrote. I honestly didn't expect random string literals in the middle of the class. Adding an extra test is never a problem. >> Also, internal comments in the middle of the class are not usually >> put into strings, actual comment syntax is more appropriate here. > > The less common it is, the more likely it is to be a source of confusion > when it happens. And the proposed syntax is definitely prone to errors > that go undetected. Very unlikely and very minor documentation(!) errors. Wrong docstring on a property descriptor, couldn't be any more harmless. I don't think "bad syntax leads to errors" argument holds here. >> Documenting attributes manually, for example, with ".. attribute" >> directives in the class docstring or .rst file, creates the following >> problems: >> 1) In embedsignature mode (which is almost mandatory for a decent >> autodoc) > > It's not more than a clumsy work-around for the CPython quirk that C > implemented functions couldn't expose their signature. And that problem > is > much better tackled by completing the support for PEP 362. > > http://www.python.org/dev/peps/pep-0362/ PEP362 is irrelevant for properties and attributes. Anyway, I was just listing technicals problems biting autodoc users. >> 1) In embedsignature mode (which is almost mandatory for a decent >> autodoc) Cython adds signature docstrings to public attributes, making >> them appear twice: one instance with a manually written doc, another >> instance with a signature docstring. >> >> 2) Manually documented attributes don't follow autodoc settings >> (ordering, formatting). They are also invisible to autosummary and other >> extensions. >> >> 3) (Obviously) non-uniformity between property, function and attribute >> documentation practice. > > The way I see it, if the embedsignature feature actually died because of > PEP 362, you could just document attributes yourself without having > Cython > interfere with it at all. And AFAIU, that would solve most of your above > problems. I did document them manually at first. It is ugly (function doc in one place, attribute doc in another) and it does not work well with autodoc and autosummary (no summaries for attributes, attrbute section appears in the wrong place and other "minor" problems that require heavy Sphinx hacks to get around.) All in all, it completely defeats the purpose of AUTOdoc. The practice of autodocumentation is widespread in Python (docstrings are a part of the language semantics) and I'd rather it support properties and attributes without messy workarounds. Sphinx has some way to go in this area, but putting strings into __doc__ slots is Cython's responsibility. >> Attribute docstrings are the cleanest solution to all of these problems. >> And the patch is just a few lines (if you discount the refactoring of >> the "XXX This should go to AutoDocTransforms" block.) > > It's not about the implementation. The question is if this feature should > become part of the language because it's "special enough to break the > rules" of OOTDI and if so, how we should expose it. And yes, that's a > wonderful field for bike shedding. It does not change the language syntax one bit, it just adds a semantic on top, like all docstrings do. Think of it this way: cdef public/readonly is a shorthand for a complete property declaration. Similarly, attribute docstring is a shortcut for providing a __doc__ for this property. PEP258/Sphinx/epydoc already have an established convention for such docstrings, we just put these pieces together. A mention that a docstring can be provided for a cdef public attribute is all "exposure" it needs. Those who don't need/don't know about it won't be affected. Best regards, Nikita Nemkin From stefan_ml at behnel.de Sat Apr 20 13:41:56 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Apr 2013 13:41:56 +0200 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> Message-ID: <51727F04.9080307@behnel.de> Nikita Nemkin, 20.04.2013 12:43: > On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >> Nikita Nemkin, 20.04.2013 09:11: >>> On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: >>> Separating property from the attribute requires mass renaming of >>> the attribute (as was recently descussed here) and bloating the code x4. >>> All this just to attach a docstring? >> >> Well, once there is a docstring, that factor goes down to at most 3x. It's >> not entirely overkill, although I agree that it's definitely way more >> complexity than just adding a docstring. >> >>> PEP 258 was rejected, but Sphinx supports it, an so does epydoc. >> >> That makes it more reasonable then, but see below. >> >>>> The above syntax is not wrong, but it also doesn't feel right to me. For >>>> example, it's impossible to draw a distinction between the above and >>>> something like >>>> >>>> cdef class X: >>>> cdef public attr >>>> >>>> """ >>>> quotes used for commenting out code here. >>>> """ >>>> >>>> And users are unlikely to notice that they just dropped some internals >>>> into >>>> the docstring of a property they didn't even know they were implementing. >>>> >>>> IMHO, the user intention is way too unclear in this syntactic construct. >>> >>> Attribute docstring must appear immediately after the attribute. The extra >>> blank line in your example makes the string standalone, just as you >>> intended. >> >> I can't see that from the tests you wrote. > > I honestly didn't expect random string literals in the middle of > the class. Adding an extra test is never a problem. I was just asking for a proof of your above statement. My experience with Cython's parser doesn't indicate per se that this is the case. >>> Also, internal comments in the middle of the class are not usually >>> put into strings, actual comment syntax is more appropriate here. >> >> The less common it is, the more likely it is to be a source of confusion >> when it happens. And the proposed syntax is definitely prone to errors >> that go undetected. > > Very unlikely and very minor documentation(!) errors. > Wrong docstring on a property descriptor, couldn't be any more harmless. > I don't think "bad syntax leads to errors" argument holds here. So let's agree to disagree on this. >>> Documenting attributes manually, for example, with ".. attribute" >>> directives in the class docstring or .rst file, creates the following >>> problems: >>> 1) In embedsignature mode (which is almost mandatory for a decent autodoc) >> >> It's not more than a clumsy work-around for the CPython quirk that C >> implemented functions couldn't expose their signature. And that problem is >> much better tackled by completing the support for PEP 362. >> >> http://www.python.org/dev/peps/pep-0362/ > > PEP362 is irrelevant for properties and attributes. Yep, different things. That was part of my point. >>> 1) In embedsignature mode (which is almost mandatory for a decent >>> autodoc) Cython adds signature docstrings to public attributes, making >>> them appear twice: one instance with a manually written doc, another >>> instance with a signature docstring. >>> >>> 2) Manually documented attributes don't follow autodoc settings >>> (ordering, formatting). They are also invisible to autosummary and other >>> extensions. >>> >>> 3) (Obviously) non-uniformity between property, function and attribute >>> documentation practice. >> >> The way I see it, if the embedsignature feature actually died because of >> PEP 362, you could just document attributes yourself without having Cython >> interfere with it at all. And AFAIU, that would solve most of your above >> problems. > > I did document them manually at first. It is ugly (function doc in one place, > attribute doc in another) As it should be. The attributes are part of the class documentation, whereas the function/method documentation belongs to the functions. I can't see why you consider this totally normal distinction "ugly". > and it does not work well with autodoc and > autosummary (no summaries for attributes, attrbute section appears in the > wrong place and other "minor" problems that require heavy Sphinx hacks to > get around.) All in all, it completely defeats the purpose of AUTOdoc. > > The practice of autodocumentation is widespread in Python > (docstrings are a part of the language semantics) and I'd rather it > support properties and attributes without messy workarounds. > Sphinx has some way to go in this area, but putting strings > into __doc__ slots is Cython's responsibility. All I'm saying is that docstrings for cdef attributes are a very special case. If you had a Python class in a Cython module, you couldn't add a docstring to its attributes at all. And there already is a dedicated syntax for non-trivial properties in cdef classes. So the question is: is a property with a docstring still a trivial property or not? And is the use case common enough to complicate the simple special case that already exists, even though the completely general case can already be handled? >>> Attribute docstrings are the cleanest solution to all of these problems. >>> And the patch is just a few lines (if you discount the refactoring of >>> the "XXX This should go to AutoDocTransforms" block.) >> >> It's not about the implementation. The question is if this feature should >> become part of the language because it's "special enough to break the >> rules" of OOTDI and if so, how we should expose it. And yes, that's a >> wonderful field for bike shedding. > > It does not change the language syntax one bit I didn't say it would change the syntax. It changes the semantics of a string that follows an attribute declaration (for whatever reason) and makes strings that appear after such declarations the official One Right Way to document cdef attributes, i.e. a Cython language feature. BTW, the pure Python mode doesn't currently have this feature either, it seems. (I'm not even sure it has public/readonly attributes...) > cdef public/readonly is a shorthand for a complete property declaration. Sure, for the trivial case. > Similarly, attribute docstring is a shortcut for providing a __doc__ > for this property. PEP258/Sphinx/epydoc already have an established > convention for such docstrings, we just put these pieces together. ISTM that there is more than one convention: http://sphinx-doc.org/ext/autodoc.html#directive-autoattribute But the newer one apparently uses the docstring syntax (which IMHO makes way more sense than special casing comments). > A mention that a docstring can be provided for a cdef public attribute > is all "exposure" it needs. Those who don't need/don't know about it > won't be affected. Except by accident, as I said. Users of Sphinx *may* be aware of this, others most likely won't be. Stefan From nikita at nemkin.ru Sat Apr 20 16:44:06 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sat, 20 Apr 2013 20:44:06 +0600 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: <51727F04.9080307@behnel.de> References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> <51727F04.9080307@behnel.de> Message-ID: On Sat, 20 Apr 2013 17:41:56 +0600, Stefan Behnel wrote: > Nikita Nemkin, 20.04.2013 12:43: >> On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >> I did document them manually at first. It is ugly (function doc in one >> place, >> attribute doc in another) > > As it should be. The attributes are part of the class documentation, > whereas the function/method documentation belongs to the functions. I > can't > see why you consider this totally normal distinction "ugly". Why should attributes and especially properties (class members) be treated differently from methods (class members) for documentation purposes? From the OOP standpoint they certainly should not. Smalltalk and Java are good examples. (Smalltalk even has a documentation convention extremely similar to Python docstrings.) > All I'm saying is that docstrings for cdef attributes are a very special > case. If you had a Python class in a Cython module, you couldn't add a > docstring to its attributes at all. And there already is a dedicated > syntax > for non-trivial properties in cdef classes. So the question is: is a > property with a docstring still a trivial property or not? And is the use > case common enough to complicate the simple special case that already > exists, even though the completely general case can already be handled? "cdef public" is one of the two forms of property declaration in Cython. (The other being "property: ...") Both use nonstandard Python syntax and neither seems a "special case" to me. The property is trivial iif its code consists of one __get__ function with "return self.value" body. What does docstring have to do with it? "property:" is not completely general. I wont rename every single public attribute for the privilege of attaching a docstring to it (and a ton of boilerplate code). It makes no sense from technical and maintenance standpoint. Considering use cases. Currently I have 83 cdef public/readonly attributes across 46 classes that represent ~20% of the source codebase. All of them have docstrings (since they are public and part of the API.) This feature is very low impact to worry about added complexity. >>>> Attribute docstrings are the cleanest solution to all of these >>>> problems. >>>> And the patch is just a few lines (if you discount the refactoring of >>>> the "XXX This should go to AutoDocTransforms" block.) >>> >>> It's not about the implementation. The question is if this feature >>> should >>> become part of the language because it's "special enough to break the >>> rules" of OOTDI and if so, how we should expose it. And yes, that's a >>> wonderful field for bike shedding. >> >> It does not change the language syntax one bit > > I didn't say it would change the syntax. It changes the semantics of a > string that follows an attribute declaration (for whatever reason) and > makes strings that appear after such declarations the official One Right > Way to document cdef attributes, i.e. a Cython language feature. Since there are currently Zero Right Ways to document these attributes, having at least one is a good thing. "property:" form is not diminished by that, in cases _when it is warranted_. > BTW, the pure Python mode doesn't currently have this feature either, it > seems. (I'm not even sure it has public/readonly attributes...) Pure python mode project can be processed by Sphinx like any ordinary Python project, with complete support for all autodoc features. Since Python level attributes have no __doc__ slots, there is nothing Cython can do with their docstrings anyway (except parsing them without syntax errors). >> Similarly, attribute docstring is a shortcut for providing a __doc__ >> for this property. PEP258/Sphinx/epydoc already have an established >> convention for such docstrings, we just put these pieces together. > > ISTM that there is more than one convention: > > http://sphinx-doc.org/ext/autodoc.html#directive-autoattribute Other conventions? Do you mean "#:" comments? These forms are not part of PEP258, can't be paralleled to the "property:" syntax and require too many changes to Cython lexer and parser for no extra benefit. Therefore I have not implemented them. > But the newer one apparently uses the docstring syntax (which IMHO makes > way more sense than special casing comments). Exactly. >> A mention that a docstring can be provided for a cdef public attribute >> is all "exposure" it needs. Those who don't need/don't know about it >> won't be affected. > > Except by accident, as I said. Users of Sphinx *may* be aware of this, > others most likely won't be. In the worst case, probability of which is near zero, the unsuspecting victim will waste a few bytes of memory (the size of the comment). This is a non-issue. Best regards, Nikita Nemkin From stefan_ml at behnel.de Sat Apr 20 18:42:33 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Apr 2013 18:42:33 +0200 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> <51727F04.9080307@behnel.de> Message-ID: <5172C579.5060800@behnel.de> Nikita Nemkin, 20.04.2013 16:44: > On Sat, 20 Apr 2013 17:41:56 +0600, Stefan Behnel wrote: >> Nikita Nemkin, 20.04.2013 12:43: >>> On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >>> I did document them manually at first. It is ugly (function doc in one >>> place, attribute doc in another) >> >> As it should be. The attributes are part of the class documentation, >> whereas the function/method documentation belongs to the functions. I can't >> see why you consider this totally normal distinction "ugly". > > Why should attributes and especially properties (class members) be treated > differently from methods (class members) for documentation purposes? > From the OOP standpoint they certainly should not. Smalltalk and Java are > good examples. (Smalltalk even has a documentation convention extremely > similar to Python docstrings.) Just because some languages don't have properties and always require you to implement getters and setters doesn't mean that attributes, methods and properties are the same thing. The question whether an attribute is implemented as a field or a property is an implementation detail that shouldn't have an impact on users. That makes attributes different from methods. Thus my comment that attributes should be part of the class documentation. That way, you can document them consistently for both extension types and Python classes. Because in the best case, users shouldn't have to care about that distinction either. And with PEP 362 in place, there'll no longer be a need for automatically embedded signatures etc. either. Attributes can be documented as part of the class and methods document themselves anyway. >> All I'm saying is that docstrings for cdef attributes are a very special >> case. If you had a Python class in a Cython module, you couldn't add a >> docstring to its attributes at all. And there already is a dedicated syntax >> for non-trivial properties in cdef classes. So the question is: is a >> property with a docstring still a trivial property or not? And is the use >> case common enough to complicate the simple special case that already >> exists, even though the completely general case can already be handled? > > "cdef public" is one of the two forms of property declaration in Cython. > (The other being "property: ...") Both use nonstandard Python syntax and > neither seems a "special case" to me. "cdef public" is the special case because it's pure syntactic sugar that is redundant with the most simple application of the property syntax. > The property is trivial iif its code consists of one __get__ > function with "return self.value" body. What does docstring have to do > with it? The general way of doing it (i.e. the property syntax) already has a way of specifying a docstring. That's a reason to be hesitant about adding this feature also to the (currently) simple short-hand special case syntax. > "property:" is not completely general. I wont rename every > single public attribute for the privilege of attaching a docstring > to it (and a ton of boilerplate code). It makes no sense from > technical and maintenance standpoint. I agree that it might seem overkill. > Considering use cases. Currently I have 83 cdef public/readonly attributes > across 46 classes that represent ~20% of the source codebase. > All of them have docstrings (since they are public and part of the API.) That seems a lot to me. I wonder how common this actually is. In the code I've written so far, the trivial case of 1:1 properties was pretty rare. Might be a domain thing. Stefan From torsten.landschoff at dynamore.de Sun Apr 21 23:57:15 2013 From: torsten.landschoff at dynamore.de (Torsten Landschoff) Date: Sun, 21 Apr 2013 23:57:15 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions Message-ID: Hi Cython-Developers, I am using Cython to generate a trivial wrapper for a small subset of the (already small) interface of libp11 (see https://github.com/OpenSC/libp11 for information about libp11). I know a bit about Python extension modules and just wanted to avoid writing all that boiler plate code and decided to given Cython a try. The wrapper was done in a day and no big deal, but lately I got random segmentation faults using it. After a day of debugging I found the cause in my use of the __dealloc__ special method. You may now call me stupid, because it is all in the docs: /You need to be careful what you do in a //__dealloc__()//method. By the time your //__dealloc__()//method is called, the object may already have been partially destroyed and may not be in a valid state as far as Python is concerned, so you should avoid invoking any Python operations which might touch the object. In particular, don't call any other methods of the object or do anything which might cause the object to be resurrected. It's best if you stick to just deallocating C data.// / http://docs.cython.org/src/userguide/special_methods.html?highlight=__dealloc__#finalization-method-dealloc But this did not give me the crucial hint: Currently, any references to Python objects *may* be invalid at the time dealloc is called. This is because Cython generates a tp_clear function for each extension type that calls Py_CLEAR for each object reference. What I would really like to have is a possibility to exempt object attributes from being cleared in tp_clear, by adding a modifier to the cdef: *Exempt all attributes:* cdef noclear class Slots: cdef Context context *Exempt a single attribute:* cdef class Slots: cdef noclear Context context This is probably enough of explanation for the Cython experts, but I would still like to explain what happened in my case and a work around for illustration purposes and in the hope, that this will help somebody else at some time. Background When using the libp11 API, the client must create a Context object which is used for most operations. To query the list of slots where a card (token) could be inserted, there is a PKCS11_enumerate_slots function which returns a newly allocated buffer which contains all slots. Unfortunately this means that the resulting Slot objects can not be deallocated individually, only the whole buffer can be released using PKCS11_release_all_slots. This is completely unpythonic which is why I created a special Slots extension type that manages the storage of the Slot instances. When all Slot instances drop their reference to the Slots instance, the refcount of the Slots instance drops to zero and Python will release it automatically. The Slots type then calls the PKCS11_release_all_slots function to release the storage. This is modelled inside the __dealloc__ method for Slots. Unfortunately, a pointer to the context is required for that call. Therefore Slots also keeps a reference to Context object to keep it alive long enough and to identify the underlying C object. But: At the time when __dealloc__ is called, the tp_clear function already has cleared the context reference. Interestingly, it does not use Py_CLEAR as mandated in the Python documentation (see http://docs.python.org/2/c-api/typeobj.html?highlight=py_clear#PyTypeObject.tp_clear and http://docs.python.org/3.3/c-api/typeobj.html?highlight=py_clear#PyTypeObject.tp_clear) but instead redirects all Python objects to Py_NONE, which partially hides the problem. Somehow, this is not deterministic and tp_clear may have been called or not before tp_dealloc, therefore the random crashes. Even more funny: tp_clear seems to get only called if the Slots instance is involved in a reference cycle. Example Code I attached an example Cython project. If you build and install it with Cython 0.19, the following Python code will work: from fakep11 import * def works(): ctx = Context() my_slots = ctx.slots() while my_slots: assert ctx.usage_counter == 1 my_slots.pop() assert ctx.usage_counter == 0 But the following code will crash when the garbage collector runs: def crashes(): ctx = Context() slots = ctx.slots() a = {"slot": slots[0]} b = {"slot": slots[1], "cycle": a} a["cycle"] = b This is because a and b refer to each other, creating a cycle, and both refer to a slot, pulling the Slots instance into the object graph of the cycle. For this reason, tp_clear will be called for slots before tp_dealloc is called, leaving tp_dealloc with invalid data. Conclusion 1. As somebody who wrote Python extension modules manually before, I fell into this trap because I never implemented tp_clear. Reference cycles were a non-issue to me, since the wrapped objects could not refer to Python objects. Interestingly, SWIG does not seem to have any support for calling tp_clear it seems. 2. IMHO tp_clear should really use Py_CLEAR which would at least make this case a solid segfault. 3. Generating tp_clear functions is a feature that can do some harm, there I propose to allow to disable it. Work around For my current project I worked around this problem by managing the Python references manually by declaring them as PyObject* as in this diff: diff -r f2afc4865bbc fakep11.pyx --- a/fakep11.pyx Sat Apr 20 23:43:00 2013 +0200 +++ b/fakep11.pyx Sun Apr 21 23:50:10 2013 +0200 @@ -1,5 +1,10 @@ cimport cython +cdef extern from "Python.h": + ctypedef struct PyObject + void Py_INCREF(PyObject *) + void Py_DECREF(PyObject *) + cdef extern from "fakep11c.h": ctypedef struct fakep11_context: @@ -35,17 +40,19 @@ @cython.internal cdef class Slots: - cdef Context context + cdef PyObject *context cdef unsigned int nslots cdef fakep11_slot *slots def __cinit__(self, Context context): - self.context = context + self.context = context + Py_INCREF(self.context) self.slots = fakep11_enumerate(context.imp, &self.nslots) def __dealloc__(self): print "Slots dealloc" - fakep11_release_all(self.context.imp, self.slots, self.nslots) + fakep11_release_all((self.context).imp, self.slots, self.nslots) + Py_DECREF(self.context) def as_list(self): l = [None] * self.nslots That way I have full control over the reference counting and can make sure that context is a valid reference when __dealloc__ is called. Please find my fake libp11 wrapper example code attached, as well as a small patch for the documentation documenting the current status. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff at dynamore.de http://www.dynamore.de DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cython-tpclear.zip Type: application/zip Size: 2225 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Python-attributes-may-point-to-None-in-__dealloc__.patch Type: text/x-patch Size: 1881 bytes Desc: not available URL: From stefan_ml at behnel.de Mon Apr 22 07:31:11 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 22 Apr 2013 07:31:11 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: References: Message-ID: <5174CB1F.50202@behnel.de> Hi, Torsten Landschoff, 21.04.2013 23:57: > I am using Cython to generate a trivial wrapper for a small subset of > the (already small) interface of libp11 (see > https://github.com/OpenSC/libp11 for information about libp11). I know a > bit about Python extension modules and just wanted to avoid writing all > that boiler plate code and decided to given Cython a try. > > The wrapper was done in a day and no big deal, but lately I got random > segmentation faults using it. > > After a day of debugging I found the cause in my use of the __dealloc__ > special method. You may now call me stupid, because it is all in the docs: > [...] > But this did not give me the crucial hint: Currently, any references to > Python objects *may* be invalid at the time dealloc is called. This is > because Cython generates a tp_clear function for each extension type > that calls Py_CLEAR for each object reference. Correct, together with your complete analysis (thanks for writing it up). For objects participating in a reference cycle, the cyclic garbage collector will try to break the cycle at an arbitrary object, which may or may not be yours. Thus the unpredictable segfaults. Thanks for bringing this up. Your use case is not uncommon, and it's worth making it easier to handle. I also agree that setting fields to None is probably worse for the innocent user than just setting them to NULL. IIRC, we chose to set everything to None because that's something users can handle in their code. You can test an attribute for None in your __dealloc__ code, but you can't test it for NULL. OTOH, I'm not sure there are many use cases for gracefully handling a cleared reference in __dealloc__(). If there really is something to clean up, then being able to test for None won't usually help much. It will prevent a crash, but also the proper cleanup. > What I would really like to have is a possibility to exempt object > attributes from being cleared in tp_clear, by adding a modifier to the cdef: > > *Exempt all attributes:* > > cdef noclear class Slots: > > cdef Context context That would be a class decorator. Totally makes sense to me. In fact, a decorator to disable GC for a type would also make sense. > *Exempt a single attribute:* > > cdef class Slots: > > cdef noclear Context context I would like to avoid adding a separate modifier here. This could also be handled in a class decorator that lists the excluded attributes. My intuition tells me that by far the most common use cases are to exclude either the entire type or exactly one attribute (as in your case, and the lxml package has a similar need). Cython could also adopt a policy of automatically excluding attributes referencing types that are known to be safe, e.g. basic builtin types. No big savings, but it may drop the need for a useless tp_clear() entirely in some cases, and that might have an impact on the GC cleanup time, as the GC might succeed in breaking the cycle more quickly. Reconsidering tp_traverse() would also be worth it in this case. In some cases, we could even drop the type's GC support completely. Think of a type that only has C typed fields, except for one byte string attribute. That would currently trigger the GC support for no good reason. Another possibility (which I think we considered back in the old days) would be to automatically exclude fields from tp_clear() that are being syntactically referenced in a corresponding __dealloc__() method. However, that's obviously error prone and won't catch all cases (e.g. usage in subtypes, calling cleanup functions from __dealloc__, "if DEBUG: print(self.attr)", ...), so I guess we'd rather not go that route. Explicit is definitely better than implicit here. Stefan From torsten.landschoff at dynamore.de Mon Apr 22 13:07:43 2013 From: torsten.landschoff at dynamore.de (Torsten Landschoff) Date: Mon, 22 Apr 2013 13:07:43 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: <5174CB1F.50202@behnel.de> References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> Message-ID: Hi Stefan, thank you for your extremely quick reply! On 04/22/2013 07:31 AM, Stefan Behnel wrote: >> But this did not give me the crucial hint: Currently, any references to >> Python objects *may* be invalid at the time dealloc is called. This is >> because Cython generates a tp_clear function for each extension type >> that calls Py_CLEAR for each object reference. > Correct, together with your complete analysis (thanks for writing it up). > For objects participating in a reference cycle, the cyclic garbage > collector will try to break the cycle at an arbitrary object, which may or > may not be yours. Thus the unpredictable segfaults. Perhaps I am misreading the CPython source code but it appears to me that each and every object in the reference cycle will get its tp_clear called. http://hg.python.org/cpython/file/9c0a677dbbc0/Modules/gcmodule.c#l782 Ah, okay, I get it. The loop will terminate when it reaches an object that actually breaks the cycle in tp_clear: The decref will cascade over all objects in the cycle. > Thanks for bringing this up. Your use case is not uncommon, and it's worth > making it easier to handle. I am glad that you see it this way. I also think that this will happen often, but of course I was not sure if you would agree. > I also agree that setting fields to None is probably worse for the innocent > user than just setting them to NULL. IIRC, we chose to set everything to > None because that's something users can handle in their code. You can test > an attribute for None in your __dealloc__ code, but you can't test it for > NULL. Hmm, okay, that's a good point that I have missed. Wouldn't it be possible to allow the special check "foo is NULL" for any Python object? Of course Cython usually shields the developer from the possibility that foo actually becomes NULL so why bother. :-) > OTOH, I'm not sure there are many use cases for gracefully handling a > cleared reference in __dealloc__(). If there really is something to clean > up, then being able to test for None won't usually help much. It will > prevent a crash, but also the proper cleanup. Agreed. > That would be a class decorator. Totally makes sense to me. In fact, a > decorator to disable GC for a type would also make sense. That would be a great feature. After all, traversing the Python objects in my example is of no use as they can not create a cycle. >> *Exempt a single attribute:* >> >> cdef class Slots: >> >> cdef noclear Context context > I would like to avoid adding a separate modifier here. This could also be > handled in a class decorator that lists the excluded attributes. My > intuition tells me that by far the most common use cases are to exclude > either the entire type or exactly one attribute (as in your case, and the > lxml package has a similar need). I believe your intuition matches reality. :-) A class decorator supporting a list of excluded attributes would be fine! > Cython could also adopt a policy of automatically excluding attributes > referencing types that are known to be safe, e.g. basic builtin types. No > big savings, but it may drop the need for a useless tp_clear() entirely in > some cases, and that might have an impact on the GC cleanup time, as the GC > might succeed in breaking the cycle more quickly. Reconsidering One could even think about building a graph of possible object relationships based on the type declaration for the Python attributes. In the example, Slot refers only to Slots and Slots only to Context, so these can't build a cycle. > Another possibility (which I think we considered back in the old days) > would be to automatically exclude fields from tp_clear() that are being > syntactically referenced in a corresponding __dealloc__() method. However, > that's obviously error prone and won't catch all cases (e.g. usage in > subtypes, calling cleanup functions from __dealloc__, "if DEBUG: > print(self.attr)", ...), so I guess we'd rather not go that route. Explicit > is definitely better than implicit here. But a good idea would be to warn if __dealloc__ actually references a Python attribute that tp_clear could have cleared, with a pointer to the class decorator that exempts the attribute/instance from tp_clear. Thanks and greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff at dynamore.de http://www.dynamore.de DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz From stefan_ml at behnel.de Mon Apr 22 13:50:31 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 22 Apr 2013 13:50:31 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> Message-ID: <51752407.3070405@behnel.de> Torsten Landschoff, 22.04.2013 13:07: > On 04/22/2013 07:31 AM, Stefan Behnel wrote: >>> But this did not give me the crucial hint: Currently, any references to >>> Python objects *may* be invalid at the time dealloc is called. This is >>> because Cython generates a tp_clear function for each extension type >>> that calls Py_CLEAR for each object reference. >> Correct, together with your complete analysis (thanks for writing it up). >> For objects participating in a reference cycle, the cyclic garbage >> collector will try to break the cycle at an arbitrary object, which may or >> may not be yours. Thus the unpredictable segfaults. > > Perhaps I am misreading the CPython source code but it appears to me > that each and every object in the reference cycle will get its tp_clear > called. > > http://hg.python.org/cpython/file/9c0a677dbbc0/Modules/gcmodule.c#l782 > > Ah, okay, I get it. The loop will terminate when it reaches an object > that actually breaks the cycle in tp_clear: The decref will cascade over > all objects in the cycle. In any case, there are no guarantees about which objects within a reference cycle will be cleared or not. >> I also agree that setting fields to None is probably worse for the innocent >> user than just setting them to NULL. IIRC, we chose to set everything to >> None because that's something users can handle in their code. You can test >> an attribute for None in your __dealloc__ code, but you can't test it for >> NULL. > > Hmm, okay, that's a good point that I have missed. Wouldn't it be > possible to allow the special check "foo is NULL" for any Python object? > Of course Cython usually shields the developer from the possibility that > foo actually becomes NULL so why bother. :-) Exactly - an extremely special case. It's an inherent property of the Cython language that user visible references are never NULL. (minus evil C code, obviously.) >> That would be a class decorator. Totally makes sense to me. In fact, a >> decorator to disable GC for a type would also make sense. > > That would be a great feature. After all, traversing the Python objects > in my example is of no use as they can not create a cycle. > >> Cython could also adopt a policy of automatically excluding attributes >> referencing types that are known to be safe, e.g. basic builtin types. No >> big savings, but it may drop the need for a useless tp_clear() entirely in >> some cases, and that might have an impact on the GC cleanup time, as the GC >> might succeed in breaking the cycle more quickly. Reconsidering > > One could even think about building a graph of possible object > relationships based on the type declaration for the Python attributes. > In the example, Slot refers only to Slots and Slots only to Context, so > these can't build a cycle. Interesting. Yes, that might work. And it's even easy to control when we allow users to override this decision explicitly using a class decorator. > a good idea would be to warn if __dealloc__ actually references a > Python attribute that tp_clear could have cleared, with a pointer to the > class decorator that exempts the attribute/instance from tp_clear. That's a good idea. Any help with this is appreciated. :) Stefan From torsten.landschoff at dynamore.de Mon Apr 22 13:56:16 2013 From: torsten.landschoff at dynamore.de (Torsten Landschoff) Date: Mon, 22 Apr 2013 13:56:16 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: <51752407.3070405@behnel.de> References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <517519FF.1010206@dynamore.de> <51752407.3070405@behnel.de> Message-ID: Hi Stefan, On 04/22/2013 01:50 PM, Stefan Behnel wrote: > Exactly - an extremely special case. It's an inherent property of the > Cython language that user visible references are never NULL. (minus > evil C code, obviously.) Suits me. >> a good idea would be to warn if __dealloc__ actually references a >> Python attribute that tp_clear could have cleared, with a pointer to the >> class decorator that exempts the attribute/instance from tp_clear. > That's a good idea. > > Any help with this is appreciated. :) How can I help? If you want, I can attempt to create a patch and ask you if I don't make any progress. I do not have a good grip at Cython sourcecode yet, so if you can give me a head start by pointing out the locations where I can inject the new behaviour this would be most welcome. About the object graph analysis I do not know if that is easily added. I thought it might be because Cython already has some type inference in place!? Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff at dynamore.de http://www.dynamore.de DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz From stefan_ml at behnel.de Mon Apr 22 14:28:01 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 22 Apr 2013 14:28:01 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <517519FF.1010206@dynamore.de> <51752407.3070405@behnel.de> Message-ID: <51752CD1.6070408@behnel.de> Torsten Landschoff, 22.04.2013 13:56: > On 04/22/2013 01:50 PM, Stefan Behnel wrote: >>> a good idea would be to warn if __dealloc__ actually references a >>> Python attribute that tp_clear could have cleared, with a pointer to the >>> class decorator that exempts the attribute/instance from tp_clear. >> >> That's a good idea. >> >> Any help with this is appreciated. :) > > How can I help? If you want, I can attempt to create a patch and ask you > if I don't make any progress. Please do. Just ask back on this list if there's anything that's not clear to you. > I do not have a good grip at Cython sourcecode yet, so if you can give > me a head start by pointing out the locations where I can inject the new > behaviour this would be most welcome. The implementations of tp_clear() etc. are in ModuleNode.py. For example, switching off the clearing of known Python builtin types can be done directly in generate_clear_function() by testing the type of the "entry" (a symbol table entry, specifically a name a type attribute in this case). See the end of Builtin.py (and its usage) for known builtins. You might want to add a list of safe builtin types to Builtin.py and use it in other places. There's precedence in "types_that_construct_their_instance". I'd definitely start by writing a test. Test suite and test runner are explained here: http://wiki.cython.org/HackerGuide#Thetestsuite Regarding the decision about GC participation, there's a method needs_gc() in the CClassScope class in Symtab.py. Making that smarter should be enough to disable GC support in safe cases. You can clone Cython on github and give us changes to review in your own repo there. > About the object graph analysis I do not know if that is easily added. I > thought it might be because Cython already has some type inference in > place!? No type inference needed. Object attributes are either explicitly typed as a specific extension (or builtin) type, or they are just plain objects, in which case we must assume that they can introduce refcycles. You can start by walking the attribute type graph in needs_gc(). (I don't think it's called all that often, but in the worst case, the result should be cachable.) Just let a debugger stop inside of it and take a close look at what you see around you. Basically, the "scope" of an extension type knows what is defined inside of that type. That's how you get at the reference graph. For a bonus, walking type graphs might be a generally usable feature, so it may (or may not) be a good idea to implement a general way of doing this, maybe as some kind of generator. Oh, and your code needs to be compatible with Py2.4, as well as 2to3-able to run in Py3.x. But these things are usually quite easily fixable after the fact. Stefan From greg.ewing at canterbury.ac.nz Tue Apr 23 01:16:23 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Apr 2013 11:16:23 +1200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: <51752407.3070405@behnel.de> References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <51752407.3070405@behnel.de> Message-ID: <5175C4C7.4020309@canterbury.ac.nz> Stefan Behnel wrote: > Torsten Landschoff, 22.04.2013 13:07: >>One could even think about building a graph of possible object >>relationships ... Slot refers only to Slots and Slots only to Context, so >>these can't build a cycle. > > Interesting. Yes, that might work. Only if subclassing of Slot and Context are forbidden. -- Greg From robertwb at gmail.com Tue Apr 23 06:19:15 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 22 Apr 2013 21:19:15 -0700 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> <51727F04.9080307@behnel.de> Message-ID: Jumping into this thread late, improvements (1) and (3) certainly seem beneficial. As far as documenting "attributes," I can't see much of a use, but the downsides seem are low enough (accidentally assigning a stray string to the previously declared attribute) and the implementation non-invasive enough that I'm +0.5 for this change. It doesn't introduce new syntax or behavior, just additional semantic meaning for those (otherwise unlikely to occur) strings and sure beats writing them out as properties (though the fact that cdef attributes are properties should be considered more of an implementation detail IMHO). - Robert On Sat, Apr 20, 2013 at 7:44 AM, Nikita Nemkin wrote: > On Sat, 20 Apr 2013 17:41:56 +0600, Stefan Behnel > wrote: > >> Nikita Nemkin, 20.04.2013 12:43: >>> >>> On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >>> I did document them manually at first. It is ugly (function doc in one >>> place, >>> attribute doc in another) >> >> >> As it should be. The attributes are part of the class documentation, >> whereas the function/method documentation belongs to the functions. I >> can't >> see why you consider this totally normal distinction "ugly". > > > Why should attributes and especially properties (class members) be treated > differently from methods (class members) for documentation purposes? > From the OOP standpoint they certainly should not. Smalltalk and Java are > good examples. (Smalltalk even has a documentation convention extremely > similar to Python docstrings.) > > >> All I'm saying is that docstrings for cdef attributes are a very special >> case. If you had a Python class in a Cython module, you couldn't add a >> docstring to its attributes at all. And there already is a dedicated >> syntax >> for non-trivial properties in cdef classes. So the question is: is a >> property with a docstring still a trivial property or not? And is the use >> case common enough to complicate the simple special case that already >> exists, even though the completely general case can already be handled? > > > "cdef public" is one of the two forms of property declaration in Cython. > (The other being "property: ...") Both use nonstandard Python syntax and > neither seems a "special case" to me. > > The property is trivial iif its code consists of one __get__ > function with "return self.value" body. What does docstring have to do > with it? > "property:" is not completely general. I wont rename every > single public attribute for the privilege of attaching a docstring > to it (and a ton of boilerplate code). It makes no sense from > technical and maintenance standpoint. > > Considering use cases. Currently I have 83 cdef public/readonly attributes > across 46 classes that represent ~20% of the source codebase. > All of them have docstrings (since they are public and part of the API.) > > This feature is very low impact to worry about added complexity. > > >>>>> Attribute docstrings are the cleanest solution to all of these >>>>> problems. >>>>> And the patch is just a few lines (if you discount the refactoring of >>>>> the "XXX This should go to AutoDocTransforms" block.) >>>> >>>> >>>> It's not about the implementation. The question is if this feature >>>> should >>>> become part of the language because it's "special enough to break the >>>> rules" of OOTDI and if so, how we should expose it. And yes, that's a >>>> wonderful field for bike shedding. >>> >>> >>> It does not change the language syntax one bit >> >> >> I didn't say it would change the syntax. It changes the semantics of a >> string that follows an attribute declaration (for whatever reason) and >> makes strings that appear after such declarations the official One Right >> Way to document cdef attributes, i.e. a Cython language feature. > > > Since there are currently Zero Right Ways to document these attributes, > having at least one is a good thing. > "property:" form is not diminished by that, in cases _when it is warranted_. > > >> BTW, the pure Python mode doesn't currently have this feature either, it >> seems. (I'm not even sure it has public/readonly attributes...) > > > Pure python mode project can be processed by Sphinx like any ordinary > Python project, with complete support for all autodoc features. > > Since Python level attributes have no __doc__ slots, there is nothing > Cython can do with their docstrings anyway (except parsing them without > syntax errors). > > >>> Similarly, attribute docstring is a shortcut for providing a __doc__ >>> for this property. PEP258/Sphinx/epydoc already have an established >>> convention for such docstrings, we just put these pieces together. >> >> >> ISTM that there is more than one convention: >> >> http://sphinx-doc.org/ext/autodoc.html#directive-autoattribute > > > Other conventions? Do you mean "#:" comments? These forms are not part of > PEP258, can't be paralleled to the "property:" syntax and require too many > changes to Cython lexer and parser for no extra benefit. > Therefore I have not implemented them. > > >> But the newer one apparently uses the docstring syntax (which IMHO makes >> way more sense than special casing comments). > > > Exactly. > > >>> A mention that a docstring can be provided for a cdef public attribute >>> is all "exposure" it needs. Those who don't need/don't know about it >>> won't be affected. >> >> >> Except by accident, as I said. Users of Sphinx *may* be aware of this, >> others most likely won't be. > > > In the worst case, probability of which is near zero, the unsuspecting > victim will waste a few bytes of memory (the size of the comment). > This is a non-issue. > > > Best regards, > Nikita Nemkin > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Tue Apr 23 08:01:19 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 23 Apr 2013 08:01:19 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: <5175C4C7.4020309@canterbury.ac.nz> References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <51752407.3070405@behnel.de> <5175C4C7.4020309@canterbury.ac.nz> Message-ID: <517623AF.4080000@behnel.de> Greg Ewing, 23.04.2013 01:16: > Stefan Behnel wrote: >> Torsten Landschoff, 22.04.2013 13:07: >>> One could even think about building a graph of possible object >>> relationships ... Slot refers only to Slots and Slots only to Context, so >>> these can't build a cycle. >> >> Interesting. Yes, that might work. > > Only if subclassing of Slot and Context are forbidden. Right. Subtypes of a non-GC type can happily add attributes and start supporting cyclic garbage collection, including Python subclasses. So this only applies to "final" types and builtins. Not entirely uncommon, especially in the "keep this private thing alive until all referrers have died" use case, but I's say this restriction drops the priority a bit. Stefan From torsten.landschoff at dynamore.de Tue Apr 23 10:06:40 2013 From: torsten.landschoff at dynamore.de (Torsten Landschoff) Date: Tue, 23 Apr 2013 10:06:40 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: <517623AF.4080000@behnel.de> References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <517519FF.1010206@dynamore.de> <51752407.3070405@behnel.de> <5175C4C7.4020309@canterbury.ac.nz> <517623AF.4080000@behnel.de> Message-ID: On 04/23/2013 08:01 AM, Stefan Behnel wrote: > Greg Ewing, 23.04.2013 01:16: >> Only if subclassing of Slot and Context are forbidden. > Right. Subtypes of a non-GC type can happily add attributes and start > supporting cyclic garbage collection, including Python subclasses. So this > only applies to "final" types and builtins. Not entirely uncommon, > especially in the "keep this private thing alive until all referrers have > died" use case, but I's say this restriction drops the priority a bit. > Does Cython have an equivalent of the "final class DoNoExtend { ... }" of the Java world? In any case, Greg has a good point. I seriously did not think about it just because I did not plan to derive from those classes. Greetings, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff at dynamore.de http://www.dynamore.de DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz From stefan_ml at behnel.de Tue Apr 23 10:09:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 23 Apr 2013 10:09:12 +0200 Subject: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions In-Reply-To: References: <517460BB.5040307@dynamore.de> <5174CB1F.50202@behnel.de> <517519FF.1010206@dynamore.de> <51752407.3070405@behnel.de> <5175C4C7.4020309@canterbury.ac.nz> <517623AF.4080000@behnel.de> Message-ID: <517641A8.4070705@behnel.de> Torsten Landschoff, 23.04.2013 10:06: > On 04/23/2013 08:01 AM, Stefan Behnel wrote: >> Greg Ewing, 23.04.2013 01:16: >>> Only if subclassing of Slot and Context are forbidden. >> Right. Subtypes of a non-GC type can happily add attributes and start >> supporting cyclic garbage collection, including Python subclasses. So this >> only applies to "final" types and builtins. Not entirely uncommon, >> especially in the "keep this private thing alive until all referrers have >> died" use case, but I's say this restriction drops the priority a bit. >> > Does Cython have an equivalent of the "final class DoNoExtend { ... }" > of the Java world? @cython.final cdef class ExtType: ... You can also declare a class @cython.internal to prevent it from showing up in the module dict. Stefan From dave.hirschfeld at gmail.com Tue Apr 23 16:56:25 2013 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Tue, 23 Apr 2013 14:56:25 +0000 (UTC) Subject: [Cython] libc.string msvc incompatibility Message-ID: When trying to use strcasecmp from libc.string I get an error compiling with msvc: error C3861: 'strcasecmp': identifier not found It seems MS have decided to call it by another name - _stricmp ...amongst others: http://botsikas.blogspot.co.uk/2011/12/strcasecmp-identifier-not-found- when.html I'm wondering if it's possible for cython to include the mentioned ifdef defines by default or otherwise solve the problem so that code is portable between compilers on Windows? I understand if it's not a high priority issue. The workaround is for me to make sure all our code compiles with mingw which was on the radar anyway. Thanks, Dave From robertwb at gmail.com Tue Apr 23 19:24:18 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 23 Apr 2013 10:24:18 -0700 Subject: [Cython] libc.string msvc incompatibility In-Reply-To: References: Message-ID: It would at the very least be worth putting a note on these functions, but I'm not sure how far we should go with aliasing the platform-specific alternatives. Certainly a .h file that you "cdef extern from" include with these defines would be the shortest path to getting things working on your side. On Tue, Apr 23, 2013 at 7:56 AM, Dave Hirschfeld wrote: > > When trying to use strcasecmp from libc.string I get an error compiling with > msvc: > > error C3861: 'strcasecmp': identifier not found > > It seems MS have decided to call it by another name - _stricmp > > ...amongst others: > > http://botsikas.blogspot.co.uk/2011/12/strcasecmp-identifier-not-found- > when.html > > > I'm wondering if it's possible for cython to include the mentioned ifdef > defines by default or otherwise solve the problem so that code is portable > between compilers on Windows? I understand if it's not a high priority issue. > > The workaround is for me to make sure all our code compiles with mingw which > was on the radar anyway. > > Thanks, > Dave > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From torsten.landschoff at dynamore.de Tue Apr 23 22:51:05 2013 From: torsten.landschoff at dynamore.de (Torsten Landschoff) Date: Tue, 23 Apr 2013 22:51:05 +0200 Subject: [Cython] Unit testing Cython extensions Message-ID: Hi again, I am just wondering how to do unit testing on extensions built using Cython with py.test. My problem: The extension module I am working on is installed in our global python environment (inside the build slaves as well as on local machines) already. Before installing a new version of the extension module, I'd like to run the unit tests. However, that way the original module is tested. I am trying to illustrate. Doing TDD I create a new module which will fail the test: torsten at sharokan:~/foo$ cat foo.pyx def it_works(): return False torsten at sharokan:~/foo$ cat tests/test_foo.py import pprint, foo def test_foo(): pprint.pprint(foo.__file__) assert foo.it_works() torsten at sharokan:~/foo$ py.test -v tests ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 -- /opt/dynasdk/loco2-precise/bin/python plugins: cov, capturelog collected 0 items / 1 errors ================================================================== ERRORS =================================================================== ____________________________________________________ ERROR collecting tests/test_foo.py _____________________________________________________ tests/test_foo.py:1: in > import sys, pprint, foo E ImportError: No module named foo ========================================================== 1 error in 0.01 seconds ========================================================== Sure, module foo does not exists. I could use pyximport, but I want to check that the extension itself is correctly built. Easily done: torsten at sharokan:~/foo$ python setup.py build_ext -i running build_ext cythoning foo.pyx to foo.c building 'foo' extension creating build creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o build/temp.linux-x86_64-2.7/foo.o gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so But alas, py.test still will not find foo.so, which is now installed into the current directory (due to using the "-i" flag to build_ext). Workaround: Overwrite PYTHONPATH: torsten at sharokan:~/foo$ PYTHONPATH=`pwd` py.test ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 plugins: cov, capturelog collected 1 items tests/test_foo.py F ================================================================= FAILURES ================================================================== _________________________________________________________________ test_foo __________________________________________________________________ def test_foo(): pprint.pprint(foo.__file__) > assert foo.it_works() E assert () E + where = foo.it_works tests/test_foo.py:5: AssertionError -------------------------------------------------------------- Captured stdout -------------------------------------------------------------- '/home/torsten/foo/foo.so' ========================================================= 1 failed in 0.02 seconds ========================================================== Now it really loads our extension module and the failure is actually genuine. Let's assume we have the old version installed in our current Python environment: torsten at sharokan:~/foo$ python setup.py install [...] creating /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg Extracting foo-0.0-py2.7-linux-x86_64.egg to /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages Adding foo 0.0 to easy-install.pth file Installed /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg [...] Great. Now let's implement the feature that our test checks: torsten at sharokan:~/foo$ vim foo.pyx torsten at sharokan:~/foo$ cat foo.pyx def it_works(): return True The unit tests should pass now, after we rebuilt the extension: torsten at sharokan:~/foo$ python setup.py build_ext -i running build_ext cythoning foo.pyx to foo.c building 'foo' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o build/temp.linux-x86_64-2.7/foo.o gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so torsten at sharokan:~/foo$ PYTHONPATH=`pwd` py.test ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 plugins: cov, capturelog collected 1 items tests/test_foo.py F ================================================================= FAILURES ================================================================== _________________________________________________________________ test_foo __________________________________________________________________ def test_foo(): pprint.pprint(foo.__file__) > assert foo.it_works() E assert () E + where = foo.it_works tests/test_foo.py:5: AssertionError -------------------------------------------------------------- Captured stdout -------------------------------------------------------------- '/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg/foo.so' ========================================================= 1 failed in 0.02 seconds ========================================================== Unfortunately, it doesn't. The unit tests actually uses the installed version of the library, which we only want to replace after our tests pass. Let's try it another way: By creating a virtualenv just for our tests, we should be fine: torsten at sharokan:~/foo$ virtualenv --system-site-packages fooenv New python executable in fooenv/bin/python Please make sure you remove any previous custom paths from your /home/torsten/.pydistutils.cfg file. Installing setuptools............done. Installing pip...............done. torsten at sharokan:~/foo$ . fooenv/bin/activate (fooenv)torsten at sharokan:~/foo$ pip install --upgrade . Unpacking /home/torsten/foo Running setup.py egg_info for package from file:///home/torsten/foo Downloading/unpacking Cython from http://pypi.python.org/packages/source/C/Cython/Cython-0.19.tar.gz#md5=76989337dee4cf7afdcb5cde514423f8 (from foo==0.0) Downloading Cython-0.19.tar.gz (1.4MB): 270kB downloaded ... Good start, but I don't want to recreate the whole thing inside the virtualenv fooenv. This would pull Cython, numpy, scipy, paramiko and more. Any hint how to ensure that we are testing the local version of that extension instead of the installed one? Thanks, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff at dynamore.de http://www.dynamore.de DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Tue Apr 23 23:07:14 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 23 Apr 2013 14:07:14 -0700 Subject: [Cython] Unit testing Cython extensions In-Reply-To: References: Message-ID: There is nothing special about Cython here, if you have "foo.py" instead of "foo.so" you would be seeing exactly the same results. You need to either set PYTHONPATH or have foo be in the same package layout as your tests. On Tue, Apr 23, 2013 at 1:51 PM, Torsten Landschoff wrote: > Hi again, > > I am just wondering how to do unit testing on extensions built using Cython > with py.test. > > My problem: The extension module I am working on is installed in our global > python environment (inside the build slaves as well as on local machines) > already. Before installing a new version of the extension module, I'd like > to run the unit tests. However, that way the original module is tested. > > I am trying to illustrate. > > Doing TDD I create a new module which will fail the test: > > > torsten at sharokan:~/foo$ cat foo.pyx > def it_works(): > return False > > torsten at sharokan:~/foo$ cat tests/test_foo.py > import pprint, foo > > def test_foo(): > pprint.pprint(foo.__file__) > assert foo.it_works() > > torsten at sharokan:~/foo$ py.test -v tests > ============================================================ test session > starts ============================================================ > platform linux2 -- Python 2.7.3 -- pytest-2.3.4 -- > /opt/dynasdk/loco2-precise/bin/python > plugins: cov, capturelog > collected 0 items / 1 errors > > ================================================================== ERRORS > =================================================================== > ____________________________________________________ ERROR collecting > tests/test_foo.py _____________________________________________________ > tests/test_foo.py:1: in >> import sys, pprint, foo > E ImportError: No module named foo > ========================================================== 1 error in 0.01 > seconds ========================================================== > > Sure, module foo does not exists. I could use pyximport, but I want to check > that the extension itself is correctly built. > Easily done: > > torsten at sharokan:~/foo$ python setup.py build_ext -i > running build_ext > cythoning foo.pyx to foo.c > building 'foo' extension > creating build > creating build/temp.linux-x86_64-2.7 > gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall > -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c > foo.c -o build/temp.linux-x86_64-2.7/foo.o > gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o > -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so > > But alas, py.test still will not find foo.so, which is now installed into > the current directory (due to using the "-i" flag to build_ext). Workaround: > Overwrite PYTHONPATH: > > torsten at sharokan:~/foo$ PYTHONPATH=`pwd` py.test > ============================================================ test session > starts ============================================================ > platform linux2 -- Python 2.7.3 -- pytest-2.3.4 > plugins: cov, capturelog > collected 1 items > > tests/test_foo.py F > > ================================================================= FAILURES > ================================================================== > _________________________________________________________________ test_foo > __________________________________________________________________ > > def test_foo(): > pprint.pprint(foo.__file__) >> assert foo.it_works() > E assert () > E + where = foo.it_works > > tests/test_foo.py:5: AssertionError > -------------------------------------------------------------- Captured > stdout -------------------------------------------------------------- > '/home/torsten/foo/foo.so' > ========================================================= 1 failed in 0.02 > seconds ========================================================== > > Now it really loads our extension module and the failure is actually > genuine. Let's assume we have the old version installed in our current > Python environment: > > torsten at sharokan:~/foo$ python setup.py install > [...] > creating > /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg > Extracting foo-0.0-py2.7-linux-x86_64.egg to > /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages > Adding foo 0.0 to easy-install.pth file > > Installed > /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg > [...] > > Great. Now let's implement the feature that our test checks: > > torsten at sharokan:~/foo$ vim foo.pyx > torsten at sharokan:~/foo$ cat foo.pyx > def it_works(): > return True > > The unit tests should pass now, after we rebuilt the extension: > > torsten at sharokan:~/foo$ python setup.py build_ext -i > running build_ext > cythoning foo.pyx to foo.c > building 'foo' extension > gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall > -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c > foo.c -o build/temp.linux-x86_64-2.7/foo.o > gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o > -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so > torsten at sharokan:~/foo$ PYTHONPATH=`pwd` py.test > ============================================================ test session > starts ============================================================ > platform linux2 -- Python 2.7.3 -- pytest-2.3.4 > plugins: cov, capturelog > collected 1 items > > tests/test_foo.py F > > ================================================================= FAILURES > ================================================================== > _________________________________________________________________ test_foo > __________________________________________________________________ > > def test_foo(): > pprint.pprint(foo.__file__) >> assert foo.it_works() > E assert () > E + where = foo.it_works > > tests/test_foo.py:5: AssertionError > -------------------------------------------------------------- Captured > stdout -------------------------------------------------------------- > '/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg/foo.so' > ========================================================= 1 failed in 0.02 > seconds ========================================================== > > Unfortunately, it doesn't. The unit tests actually uses the installed > version of the library, which we only want to replace after our tests pass. > > Let's try it another way: By creating a virtualenv just for our tests, we > should be fine: > > torsten at sharokan:~/foo$ virtualenv --system-site-packages fooenv > New python executable in fooenv/bin/python > Please make sure you remove any previous custom paths from your > /home/torsten/.pydistutils.cfg file. > Installing setuptools............done. > Installing pip...............done. > torsten at sharokan:~/foo$ . fooenv/bin/activate > (fooenv)torsten at sharokan:~/foo$ pip install --upgrade . > Unpacking /home/torsten/foo > Running setup.py egg_info for package from file:///home/torsten/foo > > Downloading/unpacking Cython from > http://pypi.python.org/packages/source/C/Cython/Cython-0.19.tar.gz#md5=76989337dee4cf7afdcb5cde514423f8 > (from foo==0.0) > Downloading Cython-0.19.tar.gz (1.4MB): 270kB downloaded > ... > > Good start, but I don't want to recreate the whole thing inside the > virtualenv fooenv. This would pull Cython, numpy, scipy, paramiko and more. > > Any hint how to ensure that we are testing the local version of that > extension instead of the installed one? > > Thanks, Torsten > > -- > DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH > Torsten Landschoff > > Office Dresden > Tel: +49-(0)351-4519587 > Fax: +49-(0)351-4519561 > > mailto:torsten.landschoff at dynamore.de > http://www.dynamore.de > > DYNAmore Gesellschaft f?r FEM Ingenieurdienstleistungen mbH > Registration court: Stuttgart, HRB 733694 > Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From silvertrumpet999 at gmail.com Wed Apr 24 08:06:36 2013 From: silvertrumpet999 at gmail.com (Josh Warner) Date: Wed, 24 Apr 2013 01:06:36 -0500 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 Message-ID: Hi Cython devs, Over in scikit-image we have traced an odd problem with a particular Cython file to the 0.19 update of Cython. From at least Cython 0.15.1 (probably earlier) through 0.18, `_mcp.pyx` in `skimage.graph` compiled and executed correctly, passing all package tests on both Python 2.7 and Python 3. After 0.19 was released and the Travis builds began using it, we began getting 100% repeatable errors from the previously clean master branch (example of an otherwise clean Python 2.7Travis build; the Python 3 build passedall tests). All of these errors/failures trace back to this Cython file. Oddly, the errors only happen on Python 2.7; our Python 3 Travis build passes. We are discussing this issue in scikit-image Github Issue #534; feel free to join the discussion there. The .pyx Cython file is located hereand it has an associated .pxd file here. It should be noted the file compiles and executes without errors, but its output is now incorrect in Python 2.x. In case the compiled results might be relevant, for your diffing pleasure here is the compiled .c file from Cython 0.18which passes all tests on both Python 2.7 and Python 3.x, while here is the compiled .c file from Cython 0.19which produces different, incorrect results in Python 2.7. In the short term we are temporarily forcing Travis to use the 0.18 release of Cython, but that isn't a viable long term solution. It's possible the error is on our end, but seeing as it worked with prior Cython releases we'd appreciate you taking a look. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Apr 24 08:19:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Apr 2013 08:19:36 +0200 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: References: Message-ID: <51777978.9030502@behnel.de> Hi, Josh Warner, 24.04.2013 08:06: > Over in scikit-image we have traced an odd problem with a particular Cython > file to the 0.19 update of Cython. From at least Cython 0.15.1 (probably > earlier) through 0.18, `_mcp.pyx` in `skimage.graph` compiled and executed > correctly, passing all package tests on both Python 2.7 and Python 3. > > After 0.19 was released and the Travis builds began using it, we began > getting 100% repeatable errors from the previously clean master branch > (example of an otherwise clean Python > 2.7Travis > build; the > Python 3 build passedall > tests). All of these errors/failures trace back to this Cython file. > Oddly, the errors only happen on Python 2.7; our Python 3 Travis build > passes. > > We are discussing this issue in scikit-image Github Issue > #534; > feel free to join the discussion there. > > The .pyx Cython file is located > hereand > it has > an associated .pxd file > here. > It should be noted the file compiles and executes without errors, but its > output is now incorrect in Python 2.x. > > In case the compiled results might be relevant, for your diffing pleasure here > is the compiled .c file from Cython > 0.18which > passes all tests on both Python 2.7 and Python 3.x, while here > is the compiled .c file from Cython > 0.19which > produces different, incorrect results in Python 2.7. > > In the short term we are temporarily forcing Travis to use the 0.18 release > of Cython, but that isn't a viable long term solution. > > It's possible the error is on our end, but seeing as it worked with prior > Cython releases we'd appreciate you taking a look. Thanks for bringing this up. You could make it a little easier for us by pointing us at the code that produces the incorrect results you are experiencing. The set of failing tests seems to be quite small, but before we start digging through your code, I'm sure you can provide pointers to the relevant code snippets for a couple of these tests (i.e. the test code itself and the major code parts that produce the results) much more quickly. Stefan From silvertrumpet999 at gmail.com Wed Apr 24 08:58:24 2013 From: silvertrumpet999 at gmail.com (Josh Warner) Date: Wed, 24 Apr 2013 01:58:24 -0500 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: References: Message-ID: Hi Stefan, Every quantitative test is failing for _mcp.pyx (in skimage.graph.tests.test_mcp.py), suggesting the problem is fairly global to the MCP class. The three errors in skimage.graph.tests.test_spath.py are due to incorrect MCP results which have no valid traceback path. No need to investigate _spath.pyx. The pure Python utility functions in _mcp.pyx appear to return correct output, and I do not believe MCP.traceback is the culprit, rather, the actual generated graph is incorrect. These errors appear in the MCP class, so MCP_Geometric (subclass of MCP) is not the source. Thus, I strongly suspect the problem lies within the MCP class, most likely in MCP.find_costs(). Every failing test directly or indirectly uses this method, and several of them directly test (and fail) on the returned cost array. Hopefully that helps, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikita at nemkin.ru Wed Apr 24 09:33:46 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 24 Apr 2013 13:33:46 +0600 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: <51777978.9030502@behnel.de> References: <51777978.9030502@behnel.de> Message-ID: On Wed, 24 Apr 2013 12:19:36 +0600, Stefan Behnel wrote: The bug is here (at least one of them): https://github.com/scikit-image/scikit-image/blob/master/skimage/graph/_mcp.pyx#L179 "shape[:-1]" returns incorrect result: input (8, 8), expected output (8,), actual output (). I guess that means something is wrong with __Pyx_PyObject_GetSlice utility, although I wasn't able to create a simple repro yet. Best regards, Nikita Nemkin > Hi, > > Josh Warner, 24.04.2013 08:06: >> Over in scikit-image we have traced an odd problem with a particular >> Cython >> file to the 0.19 update of Cython. From at least Cython 0.15.1 (probably >> earlier) through 0.18, `_mcp.pyx` in `skimage.graph` compiled and >> executed >> correctly, passing all package tests on both Python 2.7 and Python 3. >> >> After 0.19 was released and the Travis builds began using it, we began >> getting 100% repeatable errors from the previously clean master branch >> (example of an otherwise clean Python >> 2.7Travis >> build; the >> Python 3 build >> passedall >> tests). All of these errors/failures trace back to this Cython file. >> Oddly, the errors only happen on Python 2.7; our Python 3 Travis build >> passes. >> >> We are discussing this issue in scikit-image Github Issue >> #534; >> feel free to join the discussion there. >> >> The .pyx Cython file is located >> hereand >> it has >> an associated .pxd file >> here. >> It should be noted the file compiles and executes without errors, but >> its >> output is now incorrect in Python 2.x. >> >> In case the compiled results might be relevant, for your diffing >> pleasure here >> is the compiled .c file from Cython >> 0.18which >> passes all tests on both Python 2.7 and Python 3.x, while here >> is the compiled .c file from Cython >> 0.19which >> produces different, incorrect results in Python 2.7. >> >> In the short term we are temporarily forcing Travis to use the 0.18 >> release >> of Cython, but that isn't a viable long term solution. >> >> It's possible the error is on our end, but seeing as it worked with >> prior >> Cython releases we'd appreciate you taking a look. > > Thanks for bringing this up. You could make it a little easier for us by > pointing us at the code that produces the incorrect results you are > experiencing. The set of failing tests seems to be quite small, but > before > we start digging through your code, I'm sure you can provide pointers to > the relevant code snippets for a couple of these tests (i.e. the test > code > itself and the major code parts that produce the results) much more > quickly. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From nikita at nemkin.ru Wed Apr 24 09:48:13 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Wed, 24 Apr 2013 13:48:13 +0600 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: References: <51777978.9030502@behnel.de> Message-ID: On Wed, 24 Apr 2013 13:33:46 +0600, Nikita Nemkin wrote: Update: this is not a Cython bug. _mcp.pyx declares #cython: wraparound=False, any negative index is expected to fail. It worked previously because Cython was using PySequence_GetSlice which of course is not sensitive to Cython directives. The only "fix" would be to produce a warning for constant negative indexes in wraparound=False mode. Best regards, Nikita Nemkin > On Wed, 24 Apr 2013 12:19:36 +0600, Stefan Behnel > wrote: > > The bug is here (at least one of them): > https://github.com/scikit-image/scikit-image/blob/master/skimage/graph/_mcp.pyx#L179 > "shape[:-1]" returns incorrect result: input (8, 8), expected output > (8,), > actual output (). > > I guess that means something is wrong with __Pyx_PyObject_GetSlice > utility, > although I wasn't able to create a simple repro yet. > > > Best regards, > Nikita Nemkin > > >> Hi, >> >> Josh Warner, 24.04.2013 08:06: >>> Over in scikit-image we have traced an odd problem with a particular >>> Cython >>> file to the 0.19 update of Cython. From at least Cython 0.15.1 >>> (probably >>> earlier) through 0.18, `_mcp.pyx` in `skimage.graph` compiled and >>> executed >>> correctly, passing all package tests on both Python 2.7 and Python 3. >>> >>> After 0.19 was released and the Travis builds began using it, we began >>> getting 100% repeatable errors from the previously clean master branch >>> (example of an otherwise clean Python >>> 2.7Travis >>> build; the >>> Python 3 build >>> passedall >>> tests). All of these errors/failures trace back to this Cython file. >>> Oddly, the errors only happen on Python 2.7; our Python 3 Travis build >>> passes. >>> >>> We are discussing this issue in scikit-image Github Issue >>> #534; >>> feel free to join the discussion there. >>> >>> The .pyx Cython file is located >>> hereand >>> it has >>> an associated .pxd file >>> here. >>> It should be noted the file compiles and executes without errors, but >>> its >>> output is now incorrect in Python 2.x. >>> >>> In case the compiled results might be relevant, for your diffing >>> pleasure here >>> is the compiled .c file from Cython >>> 0.18which >>> passes all tests on both Python 2.7 and Python 3.x, while here >>> is the compiled .c file from Cython >>> 0.19which >>> produces different, incorrect results in Python 2.7. >>> >>> In the short term we are temporarily forcing Travis to use the 0.18 >>> release >>> of Cython, but that isn't a viable long term solution. >>> >>> It's possible the error is on our end, but seeing as it worked with >>> prior >>> Cython releases we'd appreciate you taking a look. >> >> Thanks for bringing this up. You could make it a little easier for us by >> pointing us at the code that produces the incorrect results you are >> experiencing. The set of failing tests seems to be quite small, but >> before >> we start digging through your code, I'm sure you can provide pointers to >> the relevant code snippets for a couple of these tests (i.e. the test >> code >> itself and the major code parts that produce the results) much more >> quickly. >> >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel From silvertrumpet999 at gmail.com Thu Apr 25 05:17:58 2013 From: silvertrumpet999 at gmail.com (Josh Warner) Date: Wed, 24 Apr 2013 22:17:58 -0500 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: References: Message-ID: Many thanks Nikita, with your advice we've fixed the problem. Regards, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Apr 25 15:52:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 25 Apr 2013 15:52:12 +0200 Subject: [Cython] Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19 In-Reply-To: References: <51777978.9030502@behnel.de> Message-ID: <5179350C.8040500@behnel.de> Nikita Nemkin, 24.04.2013 09:48: > _mcp.pyx declares #cython: wraparound=False, any negative index is expected > to fail. > It worked previously because Cython was using PySequence_GetSlice > which of course is not sensitive to Cython directives. > > The only "fix" would be to produce a warning for constant negative indexes in > wraparound=False mode. Good call. I added a warning: https://github.com/cython/cython/commit/f72eb8966ac1816279c81c01d721f0d4d795509b Let's see what that gives us. It most likely needs a bit more tuning to avoid false positives in known-to-be-safe cases, but since it's not clear to me what those are, it's just always emitting the warning for now. Stefan From stefan_ml at behnel.de Fri Apr 26 07:07:52 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Apr 2013 07:07:52 +0200 Subject: [Cython] regarding ticket #810 (segfault in Sage) Message-ID: <517A0BA8.3050003@behnel.de> Hi, looking at this ticket: http://trac.cython.org/cython_trac/ticket/810 and at the underlying Sage ticket: http://trac.sagemath.org/sage_trac/ticket/14452 ISTM that this crash is not actually related to cdef kwargs but simply to the fact that the C code is now calling directly through the C layer and no longer through Python. The major difference being that Python puts arguments into a tuple that is kept alive for the whole timespan of the call, whereas C just passes in the reference and forgets about it. Since the method in question replaces references internally, it's quite possible that it somehow deletes the last reference to the object that was passed in. I wouldn't really know why, as we specifically guard against these things since somewhere around 0.17 (and comment #7 in the Sage ticket shows an example of that), but maybe this is just a corner case that we failed to handle so far, or maybe the Sage code is doing something 'unexpected' here. Stefan From stefan_ml at behnel.de Fri Apr 26 07:18:07 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Apr 2013 07:18:07 +0200 Subject: [Cython] regarding ticket #810 (segfault in Sage) In-Reply-To: <517A0BA8.3050003@behnel.de> References: <517A0BA8.3050003@behnel.de> Message-ID: <517A0E0F.6000203@behnel.de> Stefan Behnel, 26.04.2013 07:07: > looking at this ticket: > > http://trac.cython.org/cython_trac/ticket/810 > > and at the underlying Sage ticket: > > http://trac.sagemath.org/sage_trac/ticket/14452 > > ISTM that this crash is not actually related to cdef kwargs but simply to > the fact that the C code is now calling directly through the C layer and no > longer through Python. The major difference being that Python puts > arguments into a tuple that is kept alive for the whole timespan of the > call, whereas C just passes in the reference and forgets about it. > > Since the method in question replaces references internally, it's quite > possible that it somehow deletes the last reference to the object that was > passed in. I wouldn't really know why, as we specifically guard against > these things since somewhere around 0.17 (and comment #7 in the Sage ticket > shows an example of that), but maybe this is just a corner case that we > failed to handle so far, or maybe the Sage code is doing something > 'unexpected' here. To debug this further, I suggest stepping through the _get_object() method with a watchpoint set on the reference count of the "owner" object (or maybe others) to observe when it changes and if it drops down to zero at some point during that call. Stefan From stefan_ml at behnel.de Fri Apr 26 08:34:31 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Apr 2013 08:34:31 +0200 Subject: [Cython] Version of Cython packages in Ubuntu PPA Message-ID: <517A1FF7.9050105@behnel.de> Hi, the packages in the Ubuntu PPA still have 0.18.0 in their name, although they actually build the current master, which is at 0.19. Is there a way to adapt that automatically? Stefan From stefan_ml at behnel.de Fri Apr 26 11:50:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 26 Apr 2013 11:50:12 +0200 Subject: [Cython] Fwd: Re: Version of Cython packages in Ubuntu PPA In-Reply-To: References: Message-ID: <517A4DD4.4060108@behnel.de> [forwarding] -------- Original-Message -------- Subject: Re: Version of Cython packages in Ubuntu PPA Date: Fri, 26 Apr 2013 07:24:48 +0000 From: Thomas Pietrowski Hey Stefan, I wished it would work. Talked about that with the maintainers of the launchpad.net site. There is a option using tags as version number, but as we are importing code from github this feature does not work :-( So as the version changes it needs to be changed in the recipes: https://code.launchpad.net/~thopiekar/+recipe/cython-py2+3-daily https://code.launchpad.net/~thopiekar/+recipe/cython-py2-daily You can also use the packaging files for doing that, but I decided to overwrite it in the recipes, which might me easier and faster for you changing it online. Fell free to ask me, if any questions are still left :-) Regards, Thomas Am 26.04.2013 08:34 schrieb Stefan Behnel: > Hi, > > the packages in the Ubuntu PPA still have 0.18.0 in their name, although > they actually build the current master, which is at 0.19. > > Is there a way to adapt that automatically? > > Stefan > From nikita at nemkin.ru Fri Apr 26 13:33:05 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Fri, 26 Apr 2013 17:33:05 +0600 Subject: [Cython] Autodoc improvements (#216) In-Reply-To: References: <5171A4E9.10700@behnel.de> <51725160.6020008@behnel.de> <51727F04.9080307@behnel.de> Message-ID: On Tue, 23 Apr 2013 10:19:15 +0600, Robert Bradshaw wrote: > Jumping into this thread late, improvements (1) and (3) certainly seem > beneficial. As far as documenting "attributes," I can't see much of a > use, but the downsides seem are low enough (accidentally assigning a > stray string to the previously declared attribute) and the > implementation non-invasive enough that I'm +0.5 for this change. It > doesn't introduce new syntax or behavior, just additional semantic > meaning for those (otherwise unlikely to occur) strings and sure beats > writing them out as properties (though the fact that cdef attributes > are properties should be considered more of an implementation detail > IMHO). I've just found a 4 year old ticket requesting the same feature (2): http://trac.cython.org/cython_trac/ticket/206 Best regards, Nikita Nemkin From nikita at nemkin.ru Fri Apr 26 17:53:52 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Fri, 26 Apr 2013 21:53:52 +0600 Subject: [Cython] A little bugtracker cleanup Message-ID: Hi, While browsing Cython's bugtracker I've found a few issues that should be closed as fixed: http://trac.cython.org/cython_trac/ticket/42 http://trac.cython.org/cython_trac/ticket/94 http://trac.cython.org/cython_trac/ticket/113 http://trac.cython.org/cython_trac/ticket/246 http://trac.cython.org/cython_trac/ticket/358 Best regards, Nikita Nemkin From robertwb at gmail.com Sat Apr 27 00:07:35 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 26 Apr 2013 15:07:35 -0700 Subject: [Cython] A little bugtracker cleanup In-Reply-To: References: Message-ID: Thanks. I closed some of these, for the rest I'd like to verify we at least have a regression test. On Fri, Apr 26, 2013 at 8:53 AM, Nikita Nemkin wrote: > Hi, > > While browsing Cython's bugtracker I've found a few > issues that should be closed as fixed: > > http://trac.cython.org/cython_trac/ticket/42 > http://trac.cython.org/cython_trac/ticket/94 > http://trac.cython.org/cython_trac/ticket/113 > http://trac.cython.org/cython_trac/ticket/246 > http://trac.cython.org/cython_trac/ticket/358 > > > Best regards, > Nikita Nemkin > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From nikita at nemkin.ru Sat Apr 27 05:44:52 2013 From: nikita at nemkin.ru (Nikita Nemkin) Date: Sat, 27 Apr 2013 09:44:52 +0600 Subject: [Cython] A little bugtracker cleanup In-Reply-To: References: Message-ID: On Sat, 27 Apr 2013 04:07:35 +0600, Robert Bradshaw wrote: > Thanks. I closed some of these, for the rest I'd like to verify we at > least have a regression test. http://trac.cython.org/cython_trac/ticket/42 is about const support. I'm not sure it's necessary to test it separately, but here is the test anyway https://github.com/cython/cython/pull/219 http://trac.cython.org/cython_trac/ticket/113 was fixed here https://github.com/cython/cython/pull/200, tests included. Best regards, Nikita Nemkin From robertwb at gmail.com Sun Apr 28 09:24:44 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Sun, 28 Apr 2013 00:24:44 -0700 Subject: [Cython] A little bugtracker cleanup In-Reply-To: References: Message-ID: Thanks! On Fri, Apr 26, 2013 at 8:44 PM, Nikita Nemkin wrote: > On Sat, 27 Apr 2013 04:07:35 +0600, Robert Bradshaw > wrote: > >> Thanks. I closed some of these, for the rest I'd like to verify we at >> least have a regression test. > > > http://trac.cython.org/cython_trac/ticket/42 is about const support. > I'm not sure it's necessary to test it separately, but > here is the test anyway https://github.com/cython/cython/pull/219 > > http://trac.cython.org/cython_trac/ticket/113 was fixed > here https://github.com/cython/cython/pull/200, tests included. > > > > Best regards, > Nikita Nemkin > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From dave.hirschfeld at gmail.com Mon Apr 29 11:58:02 2013 From: dave.hirschfeld at gmail.com (David Hirschfeld) Date: Mon, 29 Apr 2013 10:58:02 +0100 Subject: [Cython] -O3 causes MinGW to segfault python In-Reply-To: References: Message-ID: Forwarded because attachments were too large. The source files and generated .c file can now be viewed at: https://gist.github.com/dhirschfeld/5480711 Is this a bug or is the recommendation to not build with -O3? > > Build log, segfault and version info below. > Input files and generated .c file attached. > > C:\temp> python setup.py build_ext --inplace --force > running build_ext > cythoning test_o3.pyx to test_o3.c > building 'test_o3' extension > C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall > -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC > -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o > test_o3.c: In function '__Pyx_RaiseArgtupleInvalid': > test_o3.c:1002:18: warning: unknown conversion type character 'z' in > format [-Wformat] > test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *', > but argument 5 has type 'Py_ssize_t' [-Wformat] > test_o3.c:1002:18: warning: unknown conversion type character 'z' in > format [-Wformat] > test_o3.c:1002:18: warning: too many arguments for format > [-Wformat-extra-args] > writing build\temp.win32-2.7\Release\test_o3.def > C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s > build\temp.win32-2.7\Release\test_o3.o > build\temp.win32-2.7\Release\test_o3.def > -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 > -lmsvcr90 -o > C:\temp\test_o3.pyd > > C:\temp> python > Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] > on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import Cython; Cython.__version__ > '0.19' > >>> import faulthandler; faulthandler.enable() > >>> from test_o3 import f > >>> f(1., 2., 3., 4.) > Fatal Python error: Segmentation fault > > Current thread 0x00001c38: > File "", line 1 in > > C:\temp> gcc --version > gcc.exe (tdm-1) 4.7.1 > Copyright (C) 2012 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > Thanks, > Dave > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Apr 29 12:10:13 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 29 Apr 2013 12:10:13 +0200 Subject: [Cython] -O3 causes MinGW to segfault python In-Reply-To: References: Message-ID: <517E4705.2030801@behnel.de> David Hirschfeld, 29.04.2013 11:58: > Forwarded because attachments were too large. The source files and > generated .c file can now be viewed at: > https://gist.github.com/dhirschfeld/5480711 > > Is this a bug or is the recommendation to not build with -O3? > >> Build log, segfault and version info below. >> Input files and generated .c file attached. >> >> C:\temp> python setup.py build_ext --inplace --force >> running build_ext >> cythoning test_o3.pyx to test_o3.c >> building 'test_o3' extension >> C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall >> -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC >> -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o >> test_o3.c: In function '__Pyx_RaiseArgtupleInvalid': >> test_o3.c:1002:18: warning: unknown conversion type character 'z' in >> format [-Wformat] >> test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *', >> but argument 5 has type 'Py_ssize_t' [-Wformat] >> test_o3.c:1002:18: warning: unknown conversion type character 'z' in >> format [-Wformat] >> test_o3.c:1002:18: warning: too many arguments for format >> [-Wformat-extra-args] >> writing build\temp.win32-2.7\Release\test_o3.def >> C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s >> build\temp.win32-2.7\Release\test_o3.o >> build\temp.win32-2.7\Release\test_o3.def >> -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 >> -lmsvcr90 -o >> C:\temp\test_o3.pyd >> >> C:\temp> python >> Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] >> on win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import Cython; Cython.__version__ >> '0.19' >>>>> import faulthandler; faulthandler.enable() >>>>> from test_o3 import f >>>>> f(1., 2., 3., 4.) >> Fatal Python error: Segmentation fault >> >> Current thread 0x00001c38: >> File "", line 1 in >> >> C:\temp> gcc --version >> gcc.exe (tdm-1) 4.7.1 >> Copyright (C) 2012 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -O3 should generally be fine. Could you provide a stack trace of the crash? Stefan From dave.hirschfeld at gmail.com Mon Apr 29 12:47:13 2013 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Mon, 29 Apr 2013 10:47:13 +0000 (UTC) Subject: [Cython] -O3 causes MinGW to segfault python References: <517E4705.2030801@behnel.de> Message-ID: Stefan Behnel writes: > > David Hirschfeld, 29.04.2013 11:58: > > Forwarded because attachments were too large. The source files and > > generated .c file can now be viewed at: > > https://gist.github.com/dhirschfeld/5480711 > > > > Is this a bug or is the recommendation to not build with -O3? > > > >> Build log, segfault and version info below. > >> Input files and generated .c file attached. > >> > >> C:\temp> python setup.py build_ext --inplace --force > >> running build_ext > >> cythoning test_o3.pyx to test_o3.c > >> building 'test_o3' extension > >> C:\dev\bin\MinGW32\bin\gcc.exe -march=native -mdll -O3 -Wall > >> -IC:\dev\bin\Python27\include -IC:\dev\bin\Python27\PC > >> -c test_o3.c -o build\temp.win32-2.7\Release\test_o3.o > >> test_o3.c: In function '__Pyx_RaiseArgtupleInvalid': > >> test_o3.c:1002:18: warning: unknown conversion type character 'z' in > >> format [-Wformat] > >> test_o3.c:1002:18: warning: format '%s' expects argument of type 'char *', > >> but argument 5 has type 'Py_ssize_t' [- Wformat] > >> test_o3.c:1002:18: warning: unknown conversion type character 'z' in > >> format [-Wformat] > >> test_o3.c:1002:18: warning: too many arguments for format > >> [-Wformat-extra-args] > >> writing build\temp.win32-2.7\Release\test_o3.def > >> C:\dev\bin\MinGW32\bin\gcc.exe -march=native -shared -s > >> build\temp.win32-2.7\Release\test_o3.o > >> build\temp.win32-2.7\Release\test_o3.def > >> -LC:\dev\bin\Python27\libs -LC:\dev\bin\Python27\PCbuild -lpython27 > >> -lmsvcr90 -o > >> C:\temp\test_o3.pyd > >> > >> C:\temp> python > >> Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] > >> on win32 > >> Type "help", "copyright", "credits" or "license" for more information. > >>>>> import Cython; Cython.__version__ > >> '0.19' > >>>>> import faulthandler; faulthandler.enable() > >>>>> from test_o3 import f > >>>>> f(1., 2., 3., 4.) > >> Fatal Python error: Segmentation fault > >> > >> Current thread 0x00001c38: > >> File "", line 1 in > >> > >> C:\temp> gcc --version > >> gcc.exe (tdm-1) 4.7.1 > >> Copyright (C) 2012 Free Software Foundation, Inc. > >> This is free software; see the source for copying conditions. There is NO > >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > -O3 should generally be fine. Could you provide a stack trace of the crash? > > Stefan > > I don't have a debug version of Python or a gdb built with Python support on my Windows PC but I tried running it through gdb nonetheless with the following results: C:\temp\gsl>gdb python GNU gdb (GDB) 7.5 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-mingw32". For bug reporting instructions, please see: ... Reading symbols from C:\dev\bin\Python27\python.exe...(no debugging symbols found)...done. (gdb) run test.py Starting program: C:\dev\bin\Python27\python.exe test.py [New Thread 6208.0x18a0] Program received signal SIGSEGV, Segmentation fault. 0x6b2c160e in ?? () from C:\temp\gsl\test_o3.pyd (gdb) bt #0 0x6b2c160e in ?? () from C:\temp\gsl\test_o3.pyd #1 0x1e0bf781 in python27!PyEval_GetFuncDesc () from C:\dev\bin\Python27\python27.dll #2 0x0244f0a8 in ?? () #3 0x1e1d57f8 in python27!PySuper_Type () from C:\dev\bin\Python27\python27.dll #4 0x00000004 in ?? () #5 0x0203ff60 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) Let me know if there's something else I should be doing to debug on Windows or if there's anything else you need to track down the problem... Thanks, Dave From robertwb at gmail.com Tue Apr 30 07:48:25 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 29 Apr 2013 22:48:25 -0700 Subject: [Cython] Generated C++ code incompatible with Clang In-Reply-To: References: Message-ID: On Fri, Apr 12, 2013 at 8:04 AM, Hannes R?st wrote: > Dear mailing list > > In the process of developing a rather large Cython project where we wrap the > scientific OpenMS library (http://sourceforge.net/projects/open-ms/) using > Cython, we have come across an issue with the C++ code that is generated by > Cython. The issue is that Cython generates C++ that is not compatible with the > Clang compiler. > > I hope this is the correct mailing list to report this issue since it seems to > be a bug in Cython and not in Clang (after some discussion we have concluded > that Clang is most likely correct in rejecting the generated code). Since we do > not have access to the issue tracker, we hoped posting our issue here would > clear up matters. > > The issue occurs with the following minimal testcase: > > from libcpp.vector cimport vector as libcpp_vector > from cython.operator cimport dereference as deref, preincrement as inc > > cdef class TestClass: > > cdef libcpp_vector[float] inst > > def __iter__(self): > it = self.inst.begin() > while it != self.inst.end(): > yield deref(it) > inc(it) > > When compiled with Cython to C++, it generates C++ that cannot be compiled with > Clang (however it seems that gcc and MSVS accept the code). It seems that the > same issue with Clang was already discussed here: > https://bugzilla.mozilla.org/show_bug.cgi?id=623303 and the conclusion was that > Clang is correct in reporting the code as erroneous. > > In the above case, the invalid C++ code that gets generated is: > > p->__pyx_v_it.std::vector::iterator::~iterator(); > > The correct code for the above case would probably be (this is just a > suggestion, it compiles with gcc and clang on our machines and our tests run > through with it): > > typedef std::vector::iterator _it; > p->__pyx_v_it.~_it(); > > > We have tested it with the 0.18 release as well as the newest 0.19b2 build from > github (58131b68dc033fc7ca269d875a2aab2b4e9646a2) and the results were the > same. Thanks for the report. Can you think of a solution that doesn't require a special typedef declaration for each type you might want to call a destructor on?