[Cython] [cython-users] Re: How to find out where an AttributeError is ignored

Thu Feb 2 22:10:49 CET 2012

On 1 February 2012 18:50, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Tue, Jan 31, 2012 at 8:30 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 31 January 2012 02:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>>>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>>>>> mark florisson, 27.01.2012 17:30:
>>>>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com>  wrote:
>>>>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de>  wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>>>>
>>>>>>>>> How can I patch Cython such that some more information on the location
>>>>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>>>>> message
>>>>>>>>>    Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>>>>> object has no attribute '_modulus' in  ignored
>>>>>>>>>
>>>>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>>>>> responsible for printing the warning?
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Simon
>>>>>>>>
>>>>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>>>>> typically two reasons for unraisable exceptions:
>>>>>>>>
>>>>>>>>     1) as Robert mentioned, a function that does not allow propagation
>>>>>>>> of exceptions, e.g.
>>>>>>>>
>>>>>>>>         cdef int func():
>>>>>>>>             raise Exception
>>>>>>>>
>>>>>>>>         Here there is no way to propagate the raised exception, so
>>>>>>>> instead one should write something like
>>>>>>>>
>>>>>>>>             cdef int func() except -1: ...
>>>>>>>>
>>>>>>>>         Alternatively one may use 'except *' in case there is no error
>>>>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>>>>> "-1 may or may not indicate an error".
>>>>>>>>
>>>>>>>>     2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>>>>
>>>>>>>> For functions the right thing is to add an except clause, for
>>>>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>>>>
>>>>>>>>     try:
>>>>>>>>         ...
>>>>>>>>     except:
>>>>>>>>         traceback.print_exc()
>>>>>>>>
>>>>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>>>>
>>>>>>> Actually, I don't see why the default is to write unraisable
>>>>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>>>>> and have callers do the check (i.e. make it implicitly "except *").
>>>>>
>>>>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>>>>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>>>>
>>>>> And for floating point, we could make our own NaN -- that's obscure enough
>>>>> that it could probably be made "except cython.cython_exception_nan" by
>>>>> default, not "except? cython.cython_exception_nan".
>>>>
>>>> The problem with that is that we can't be sure that Cython will be the only
>>>> caller. So exceptions may still not propagate in cases, and users will have
>>>> to know about these "obscure" values and that they must deal with them
>>>> manually then.
>>>>
>>>> You could add that we'd just have to disable this when user code takes a
>>>> pointer from a function, but then, how many rules are there that users will
>>>> have to learn and remember after such a change? And what's that for a
>>>> language that changes the calling semantics of a function because way down
>>>> in the code someone happens to take a pointer to it?
>>>>
>>>>
>>>>>>> Was this not implemented because Cython only knows whether functions
>>>>>>> may propagate exceptions at code generation time by looking at the
>>>>>>> presence of an error label?
>>>>>>> Maybe it could keep code insertion points around for every call to
>>>>>>> such a potential function and if the function uses the error label
>>>>>>> have the caller perform the check? Although I do forsee problems for
>>>>>>> external such functions... maybe Cython could have it's own
>>>>>>> threadstate regardless of the GIL which would indicate whether an
>>>>>>> error has occurred? e.g. CyErr_Occurred()?
>>>>>>
>>>>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>>>>> default.
>>>>>
>>>>> Still,
>>>>
>>>> I wasn't really advocating this behaviour, just indicating that it's hard
>>>> to do "better", because this "better" isn't all that clear. It's also not
>>>> "better" for all code, which means that we get from one trade-off to
>>>> another, while breaking existing code at the same time. Not exactly
>>>> paradise on either side of the tunnel.
>>>
>>> I still feel like we're stuck in the wrong default. I'd rather require
>>> more work to interact with C libraries than require more work to
>>> convert innocent-looking Python to Cython.
>>>
>>>> One example that keeps popping up in my mind is callback functions that
>>>> cannot propagate errors, at least not the CPython way. I have a couple of
>>>> those in lxml, even some returning void. So I wrap their code in a bare
>>>> try-except and when an exception strikes, I set a C error flag to tell the
>>>> C library that something went wrong and return normally. No Python code
>>>> outside of the try block. But Cython still generates code for unraisable
>>>> errors. Why? Because the internal code that handles the bare except clause
>>>> may fail and raise an exception. How about that?
>>>>
>>>>
>>>>> the need to explicitly declare "except *" keeps coming up again and
>>>>> again, and is really a blemish on the usability of Cython. When teaching
>>>>> people Cython, then it's really irritating to have to follow "all you need
>>>>> to do is add some 'cdef' and some types" with "and then you need to
>>>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>>>> looks very elegant until that point...
>>>>
>>>> I know what this feels like. The problem is that these things *are* complex.
>>>
>>> Yes. We've been wrestling with this issue almost since Cython's inception...
>>>
>>> I like Mark's two-function idea, with the caveat that f(bad_argument)
>>> now behaves quite differently than (&f)[0](bad_argument) for even more
>>> obscure reasons. But it may be the way to go.
>>>
>>> The other option is to embed the error behavior into the signature and
>>> require casts to explicitly go from one to the other. This would
>>> probably require a notation for never raising an exception (e.g.
>>> "except -"). Cdef public or api functions could require an except
>>> declaration (positive or negative), ordinary cdef functions would be
>>> "except *" by default, and cdef extern functions would be "except -"
>>> by default.
>>
>> Only except * and except ? have ever made some sense to me. Except +
>> is the most mysterious syntax ever, imho it should have been 'except
>> cpperror' or something. And when you try to search for "except +" or
>> "except *" etc on docs.cython.org it doesn't find anything, which
>> makes it hard for people reading the code and unfamiliar with the
>> syntax to figure out what it means. In general I also think decorators
>> would have been clearer when defining such functions. Let's please not
>> introduce more weird syntax.
>
> "except IDENTIFIER" already has a meaning, and it's nice to have as
> few keywords as possible (though I agree for searching). The problem
> with decorators is that they don't lend themselves being declarable
> part of a type declaration. What would the syntax be for a function
> pointer (or extern function) that does propagate Python exceptions?
> What about one that throws a C++ exception?

Right, that's why I said 'when defining the function' :) The except
clause would still be used for declarations (and may also be used in
definitions, if wanted).
But it's best to provide the best possible defaults, in which case you
only need except in rare cases.

>> In any event I don't see why we'd want 'except -', as we're trying to
>> get rid of the except clause.
>
> Is it possible to get rid of it entirely? I was thinking we could
> provide more natural defaults, but the user might still need to
> declare when things are different.
>
>> So you can still get your old behaviour
>> for function pointers by not using the except clause and having it
>> write unraisable exceptions in the function, but in Cython space you'd
>> simply get better semantics (that is, propagating exceptions).
>
> I agree with Dag that this is very similar to the GIL issue, and
> should probably be tackled similarly.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel