Mailman 3 February 2011 - cython-devel

Re: [Cython] adding support for __dict__ in extension types
by Lisandro Dalcin Sept. 26, 2011

Sept. 26, 2011

Bringing up this old post... On 21 June 2010 15:41, Robert Bradshaw <robertwb(a)math.washington.edu> wrote: > On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: > >> If we special case a __dict__ attribute in extension types, i.e: >> >> cdef class Foo: >> cdef dict __dict__ >> >> and fill type->tp_dictoffset, then we can support __dict__ in >> extension types. >> >> What do you think? > > Sounds like a good idea to me. Note that we check tp_dictoffset for > fast dispatching for cpdef methods (which would be correct as a dict > lookup *would* be needed if __dict__ is available). > I still have this patch lying around in my disk. I remember Stefan had some objections. For example, when the user ask for __dict__, a new dict is unconditionally created (in CPython, type dict are allocated on-demand). I propose to get this patch pushed now, and optimize later (however, I really don't know how to safely implement this optimization). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 4

Re: [Cython] Control flow graph
by Vitja Makarov April 14, 2011

April 14, 2011

2011/2/23 Vitja Makarov <vitja.makarov(a)gmail.com>: > 2011/2/23 Vitja Makarov <vitja.makarov(a)gmail.com>: >> 2011/2/22 Stefan Behnel <stefan_ml(a)behnel.de>: >>> Vitja Makarov, 20.02.2011 18:23: >>>> >>>> 2011/2/16 Vitja Makarov: >>>>> >>>>> Hmm... both python and codespeaks in the thread >>> >>> Yes, we should keep it to cython-devel only. Sorry for mixing it up. >>> >>> >>>>> Here is my commit it's mostly broken now but anyway >>>>> >>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb1998… >>> >>> Flow control support is large enough to merit its own module. Not sure how >>> 'smart' git is here, but you can always keep the history by explicitly >>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated >>> sections from both files. >>> >> >> Ok. >> > > Oops there is no copy command in git. > > >>> You are duplicating some code from the type inferencer. We might want to >>> clean that up at some point. However, given that flow control analysis will >>> allow us to improve the type inferencer, I think it's best to keep this code >>> in the FCA part. >>> >> >> Yes, I think it could replace MarkAssignments transform later. >> Unreachable code could be delete there too. >> >>> >>>> I've update stuff: >>>> - algo for finding definitions >>>> - warnings for uninitialized and may be uninitialised use >>>> - few test cases >>> >>> That looks very nice so far. Any idea how well it scales? >>> >> >> "Usually iterative algorithm takes no more then 5 iterations" >> >> For ExprNodes.py max number is 15 while avg is about 3 >> >> About execution time: >> >> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325, >> ~10% slow down. >> -O flag could be introduced but I don't think that's a good idea. >> >> Should later try to execute cython compiled code. >> >>> >>>> Trying to compile ParseTreeTransforms.py I've found this for example: >>>> >>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable >>>> 'template' may be used uninitialized >>>> >>>> def create_Property(self, entry): >>>> if entry.visibility == 'public': >>>> if entry.type.is_pyobject: >>>> template = self.basic_pyobject_property >>>> else: >>>> template = self.basic_property >>>> elif entry.visibility == 'readonly': >>>> template = self.basic_property_ro >>>> property = template.substitute({ >>>> u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, >>>> >>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"), >>>> attribute=entry.name), >>>> }, pos=entry.pos).stats[0] >>> >>> Ok, I guess that code generally works, but it's better to get rid of the >>> code smell. >>> >> >> Might be used warning should be disabled by default, because algorithm >> isn't smart enough: >> >> a = 1 >> if (a): b = 1 >> if (a): print b >> >> See also: >> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized… >> I've updated things: - Now it passes all the tests (some tests are run with remove_unreachable=False) - Control flow graph support all the control statements (hope so) - Analyse PyClassDefNode (now only inner, need some more work) - Now we have the following optional warnings: 1. uninitialized variable use [default on] 2. possibly uninitialized use [-Wextra] 3. unreachable code [default on] (this needs rework, unreachable code detection could be better handled inside CreateControlFlowGraph) 4. unused variable [-Wextra] 5. unused argument [-X warn.unused_arg=True] 6. unused result [-X warn_unused_result=True] - Optional dot graph output: $ cython foo.pyx -X control_flow.dot_output=foo.dot $ dot -Tpng -o foo.png foo.dot -- vitja.

2 3

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
by Robert Bradshaw March 3, 2011

March 3, 2011

Forgot reply-all... didin't we have this discussion before about making that the default for this list as it is by-far the most common desired behavior? On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw <robertwb(a)math.washington.edu> wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King <wking(a)drexel.edu> wrote: >> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King <wking(a)drexel.edu> wrote: >>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King <wking(a)drexel.edu> wrote: >>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>> >> > name)` will work in expose.pyx. >>> >> >>> >> You have also hit into the thorny issue that .pxd files are used for >>> >> many things. They may be pure C library declarations with no Python >>> >> module backing, they may be declarations of (externally implemented) >>> >> Python modules (such as numpy.pxd), or they may be declarations for >>> >> Cython-implemented modules. >>> >> >>> >> Here's another idea, what if extern blocks could contain cpdef >>> >> declarations, which would automatically generate a Python-level >>> >> wrappers for the declared members (if possible, otherwise an error)? >>> > >>> > Ah, this sounds good! Of the three .pxd roles you list above, >>> > external Python modules (e.g. numpy) and Cython-implemented modules >>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>> > What's missing is a way to give (where possible) declarations of >>> > external C libraries a Python presence. cpdef fills this hole nicely, >>> > since its whole purpose is to expose Python interfaces to >>> > C-based elements. >>> >>> In the case of external Python modules, I'm not so sure we want to >>> monkey-patch our stuff in >> >> I don't think any of the changes we are suggesting would require >> changes to existing code, so .pxd-s with external implementations >> wouldn't be affected unless they brough the changes upon themselves. > > Say, in numpy.pxd, I have > > cdef extern from "...": > cpdef struct obscure_internal_struct: > ... > > Do we add an "obscure_internal_struct" onto the (global) numpy module? > What if it conflicts with a (runtime) name? This is the issue I'm > bringing up. > >>> (and where would we do it--on the first import of a cimporting >>> module?) >> >> Compilation is an issue. I think that .pxd files should be able to be >> cythoned directly, since then they Cython can build any wrappers they >> request. If the file has a matching .pyx file, cythoning either one >> should compile both together, since they'll produce a single Python >> .so module. > > In this case, I think it may make more sense to consider cimporting > stuff from .pyx files if no .pxd file is present. In any case, this is > a big change. I don't like the idea of always creating such a module > (it may be empty, or have name conflicts) nor the idea of > conditionally compiling it (does one have to look at the contents and > really understand Cython to see if a Python shadow will be created?) > >>> > A side effect of this cpdef change would be that now even bare .pxd >>> > files (no matching .pyx) would have a Python presence, >>> >>> Where would it live? Would we just create this module (in essence, >>> acting as if there was an empty .pyx file sitting there as well)? On >>> this note, it may be worth pursuing the idea of a "cython helper" >>> module where common code and objects could live. >> >> I'm not sure exactly what you mean by "cython helper", but this sounds >> like my 'bare .pyx can create a Python .so module idea above. > > I'm thinking of a place to put, e.g. the generator and bind-able > function classes, which are now re-implemented in every module that > uses them. I think there will be more cases like this in the future > rather than less. C-level code could be #included and linked from > "global" stores as well. However, that's somewhat tangential. > >>> > so You could do >>> > something like >>> > >>> > cimport mylib as mylib_c >>> > import mylib as mylib_py >>> > import sys >>> > >>> > # Access through Python >>> > for name in dir(mylib_py): >>> > setattr(sys.modules[__name__], name, getattr(mylib_py, name)) >>> >>> I think this smells worse than "import *" >> >> Aha, thanks ;). I was stuck in my old >> .pxd-files-don't-create-modules-by-themselves mindset. Obviously, >> once they do, any Python code can access the contents directly and I >> can throw out all this indirection. >> >>> > However, from Parsing.py:2369: >>> > >>> > error(pos, "C struct/union/enum cannot be declared cpdef") >>> > >>> > From pyrex_differences.rst: >>> > >>> > If a function is declared :keyword:`cpdef` it can be called from >>> > and overridden by both extension and normal python subclasses. >>> > >>> > I believe the reason that cpdef-ed enums and similar are currently >>> > illegal is confusion between "can be called from Python" and "can be >>> > overridden from Python". >>> >>> The reason that error statement is there is because it had no meaning, >>> so an error was better than just ignoring it. >> >> Why does it have no meaning? I understand that it's not implemented >> yet, but a cpdef-ed enum or struct seems just as valid an idea as a >> cpdef-ed method. > > We're only deciding what the meaning is now. I agree it's a valid > idea, its just that no one had even considered it yet. (I'll also > concede that it's a much more natural idea for someone who came to the > language once cpdef was already implemented.) > >>> > Unions don't really have a Python parallel, >>> >>> They can be a cdef class wrapping the union type. >> >> But I would think coercion would be difficult. Unions are usually (in >> my limited experience) for "don't worry about the type, just make sure >> it fits in X bytes". How would union->Python conversion work? > > There would be a wrapping type, e.g. > > cdef class MyUnion: > cdef union_type value > > with a bunch of setters/getters for the values, just like there are > for structs. (In fact the same code would handle structs and unions). > > This is getting into the wrapper-generator territory, but I'm starting > to think for simple things that might be worth it. > >>> > cpdef struct Foo: >>> > cpdef public int intA >>> > cpdef readonly int intB >>> > cdef void *ptr >>> > >>> > We would both declare the important members of the C struct (as we can >>> > already do in Cython) and also have Cython automatically generate a >>> > Python class wrapping the struct (because of `cpdef struct`). The >>> > Python class would have: >>> > >>> > * Cython-generated getter/setter for intA (because of `cpdef public`) >>> > using the standard Python<->int coercion. >>> > * Similar Cython-generated getter for int B (because of `cpdef >>> > readonly`). >>> > * No Python access to ptr (standard C-access still possible through >>> > Cython). >>> > >>> > Doing something crazy like `cdef public void *ptr` would raise a >>> > compile-time error. >>> >>> Yes, all of the above was exactly what I was proposing. >>> >>> > I'm definately willing to help out with this (if someone will point me >>> > in the right direction), >>> >>> That would be great. >> >> Ok, I think we're pretty much agreed ;). I think that the next step >> is to start working on implementations of: >> >> * Stand alone .pxd -> Python module > > I'm not sure we're agreed on this one. > >> * Extending class cdef/cdpef/public/readonly handling to cover enums, >> stucts, and possibly unions. > > This seems like the best first step. > >> Problems with me getting started now: >> >> * I don't know where I should start mucking about in the source. > > I would start by (1) creating/declaring cdef classes for cdef > structs/unions/enums (not textually, within, e.g. > CStructOrUnionDefNode.analyse_declarations(...) in Nodes.py) and then > making sure they get added to the module's namespace (look at > ModuleNode.py where the classes get added). The other files to look at > would be Parsing.py, PyrexTypes.py and Symtab.py. Could either try to > patch up parsing first, but I think it'd be more effective to make > this the default during development, as then all the tests would > exercise your code, and it's probably both easier to dive in there and > would give you a better idea of how best to handle the parsing. > > Also, the creating of cpdef functions is a bit hackish (for historical > reasons) and something we want to clean up, so I wouldn't go to great > lengths to try and emulate that. > >> * I don't know how to handle things like dummy enums (perhaps by >> requiring all cdef-ed enums to be named). > > All enums in C are named. > >> * I'm going to go watch a movie ;). > > :) > >> It's good to be moving forward! > > Yep. You might want to consider creating a github fork. > > - Robert >

5 45

lurker thanks ya'll for all the awesome new stuff you're tinkering on
by Thomas Keller March 2, 2011

March 2, 2011

I don't have the time or knowledge to contribute to Cython right now, but I just want to thank everybody for the hard work and time they've spent on the project. I do usually read the emails that come from this listserv, and it seems like things are progressing rapidly! I sincerely hope everyone has a great day. Cheers, Thomas

3 2

Expected errors of tests/errors/cdef_members_T517.pxd
by W. Trevor King Feb. 27, 2011

Feb. 27, 2011

I'm splitting Symtab visibilities into explicit C and Python visibilities, but am having trouble reproducing the expected error messages for cdef_members_T517: $ python runtests.py cdef_members_T517 Python 2.6.6 (r266:84292, Dec 8 2010, 09:53:33) [GCC 4.4.4] Running tests against Cython 0.14.1+ === Expected errors: === 5:24: C attribute of type 'VoidP' cannot be accessed from Python 5:24: Cannot convert 'VoidP' to Python object 6:24: C attribute of type 'VoidP' cannot be accessed from Python 6:24: Cannot convert 'VoidP' to Python object 6:24: Cannot convert Python object to 'VoidP' 14:22: C attribute of type 'Foo' cannot be accessed from Python 14:22: Cannot convert Python object to 'Foo' === Got errors: === 5:24: C attribute of type 'VoidP' cannot be accessed from Python 5:24: Cannot convert 'VoidP' to Python object 6:24: C attribute of type 'VoidP' cannot be accessed from Python 6:24: Cannot convert 'VoidP' to Python object 6:24: Cannot convert Python object to 'VoidP' 14:22: Cannot convert Python object to 'Foo' So my code is not raising: 14:22: C attribute of type 'Foo' cannot be accessed from Python The relevant lines from tests/errors/cdef_members_T517.pxd are: $ grep -n ^ tests/errors/cdef_members_T517.pyx ... 8:ctypedef struct Foo: 9: int i 10: 11:cdef class Bar: 12: cdef Foo foo0 13: cdef readonly Foo foo2 14: cdef public Foo foo1 15: pass ... I see two options: 1) Foo attributes are readable from Python, in which case the expected errors should be changed to match mine. 2) Foo attributes are not readable from Python, in which case the expected errors should be extended with 13:22: C attribute of type 'Foo' cannot be accessed from Python and probably also also 14:22: Cannot convert 'Foo' to Python object I think (2) is more reasonable, until I get my public Python enums, stucts, and unions branch working, after which it would switch to (1) if the Foo struct was declared with readonly (vs. private) Python visibility. You can't alter the struct definition from Python, so a public Python visibility doesn't really make sense for the struct itself. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt

2 3

Py_LIMITED_API
by Lisandro Dalcin Feb. 23, 2011

Feb. 23, 2011

Should we try to support Py_LIMITED_API for Py>=3.2? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 2

hasattr() swallows any exception (Py<3.2)
by Lisandro Dalcin Feb. 23, 2011

Feb. 23, 2011

Take a look here: http://bugs.python.org/issue9666 'hasattr' default behaviour should be changed to suppress only AttributeError exceptions. Other should pass through. Should we do something about this in Cython? We currently use PyObject_HasAttr(), but even in Python 3.2 this is not the same as builtins.hasattr(), it swallows any exception (do you think this is a bug in core Python?). I'm inclined to fix the behavior for ALL Python versions to suppress only AttributeError. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 4

Re: [Cython] Control flow graph
by Robert Bradshaw Feb. 23, 2011

Feb. 23, 2011

On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov <vitja.makarov(a)gmail.com> wrote: > 2011/2/15 Robert Bradshaw <robertwb(a)math.washington.edu>: >> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov <vitja.makarov(a)gmail.com> wrote: >>> Hi! >>> >>> In order to implement "reaching definitions" algorithm. >>> I'm now working on control-flow (or data-flow) graph. >>> >>> Here is funny picture made with graphviz ;) >>> >>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >> >> Cool. Any plans on handling exceptions? >> > > Sure, but I don't have much time for this :( > > Linear block inside try...except body should be split by assignments > and each subblock should point to exception handling entry point. Would every possible failing sub-expression have to point to the exception handling point(s)? I suppose it depends on whether you'll be handling more than assignment tracking. > As result I want to have set of possible assignments for each NameNode position. > > So handling of uninitialized variables, unused, unused results should be easy, > later it may help to implement local variable deletion. And guess that > could help type inference. Ye. - Robert

4 7

Gmane archive
by Stefan Behnel Feb. 20, 2011

Feb. 20, 2011

Hi, I still didn't get a response from Gmane, but this article doesn't look promising: http://article.gmane.org/gmane.discuss/13987 So I guess we'll have to request a new group. Very unfortunate. Stefan

1 1

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
by W. Trevor King Feb. 19, 2011

Feb. 19, 2011

On Thu, Feb 17, 2011 at 03:53:13PM -0800, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King <wking(a)drexel.edu> wrote: > > * Extending class cdef/cdpef/public/readonly handling to cover enums, > > stucts, and possibly unions. > > This seems like the best first step. > > > Problems with me getting started now: > > > > * I don't know where I should start mucking about in the source. > > ...The other files to look at would be Parsing.py,.. I've got the the Parsing pretty much working, but portions of the test suite are now failing with things like === Expected errors: === 1:5: function definition in pxd file must be declared 'cdef inline' 4:5: inline function definition in pxd file cannot be 'public' 7:5: inline function definition in pxd file cannot be 'api' === Got errors: === 1:5: function definition in pxd file must be declared 'cdef inline' 4:12: inline function definition in pxd file cannot be 'public' 7:5: inline function definition in pxd file cannot be 'api' while cythoning tests/errors/e_func_in_pxd_support.pxd: cdef foo(): return 1 cdef public inline foo2(): return 1 cdef api inline foo3(): return 1 The current Cython implementation does not consider 'cdef' to be part of the node code, and my current implementation does not consider any qualifiers to be part of the node code. It's unclear to me what the "right" position is for the start of a function (or variable, class, ...) should be, or how errors should be formatted. My initial impression is that Node.pos should point to the beginning of the def (here 1:1, 4:1, and 7:1), but that positions of the various qualifiers (cdef, public, inline, api, ...) should be cached so that the error points to the offending qualifier. Thoughs? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt

2 3