[Cython] [cython-users] Re: How to find out where an AttributeError is ignored

[Robert Bradshaw]

On Tue, Jan 31, 2012 at 8:30 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 31 January 2012 02:12, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Fri, Jan 27, 2012 at 1:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Dag Sverre Seljebotn, 27.01.2012 21:03:
>>>> On 01/27/2012 05:58 PM, Stefan Behnel wrote:
>>>>> mark florisson, 27.01.2012 17:30:
>>>>>> On 27 January 2012 16:22, mark florisson<markflorisson88 at gmail.com>  wrote:
>>>>>>> On 27 January 2012 15:47, Simon King<simon.king at uni-jena.de>  wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I am still *very* frustrated about the fact that Cython does not tell
>>>>>>>> where the error occurs. Since about one week, I am adding lots and
>>>>>>>> lots of lines into Sage that write a log into some file, so that I get
>>>>>>>> at least some idea where the error occurs. But still: Even these
>>>>>>>> extensive logs do not provide a hint on what exactly is happening.
>>>>>>>>
>>>>>>>> How can I patch Cython such that some more information on the location
>>>>>>>> of the error is printed? I unpacked Sage's Cython spkg, and did "grep -
>>>>>>>> R ignored .", but the code lines containing the word "ignored" did not
>>>>>>>> seem to be the lines that are responsible for printing the warning
>>>>>>>> message
>>>>>>>>    Exception AttributeError: 'PolynomialRing_field_with_category'
>>>>>>>> object has no attribute '_modulus' in  ignored
>>>>>>>>
>>>>>>>> Can you point me to the file in Sage's Cython spkg which is
>>>>>>>> responsible for printing the warning?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Simon
>>>>>>>
>>>>>>> These messages are written by PyErr_WriteUnraisable, which is a
>>>>>>> CPython C API function that writes unraisable exceptions. There are
>>>>>>> typically two reasons for unraisable exceptions:
>>>>>>>
>>>>>>>     1) as Robert mentioned, a function that does not allow propagation
>>>>>>> of exceptions, e.g.
>>>>>>>
>>>>>>>         cdef int func():
>>>>>>>             raise Exception
>>>>>>>
>>>>>>>         Here there is no way to propagate the raised exception, so
>>>>>>> instead one should write something like
>>>>>>>
>>>>>>>             cdef int func() except -1: ...
>>>>>>>
>>>>>>>         Alternatively one may use 'except *' in case there is no error
>>>>>>> indicator and Cython should always check, or "except ? -1" which means
>>>>>>> "-1 may or may not indicate an error".
>>>>>>>
>>>>>>>     2) in deallocators or finalizers (e.g. __dealloc__ or __del__)
>>>>>>>
>>>>>>> For functions the right thing is to add an except clause, for
>>>>>>> finalizers and destructors one could use the traceback module, e.g.
>>>>>>>
>>>>>>>     try:
>>>>>>>         ...
>>>>>>>     except:
>>>>>>>         traceback.print_exc()
>>>>>>>
>>>>>>> If this all still doesn't help, try setting a (deferred) breakpoint on
>>>>>>> __Pyx_WriteUnraisable or PyErr_WriteUnraisable.
>>>>>>
>>>>>> Actually, I don't see why the default is to write unraisable
>>>>>> exceptions. Instead Cython could detect that exceptions may propagate
>>>>>> and have callers do the check (i.e. make it implicitly "except *").
>>>>
>>>> As for speed, there's optimizations on this, e.g., "except? 32434623" if
>>>> the return type is int, "except? 0xfffff..." if the return type is a pointer.
>>>>
>>>> And for floating point, we could make our own NaN -- that's obscure enough
>>>> that it could probably be made "except cython.cython_exception_nan" by
>>>> default, not "except? cython.cython_exception_nan".
>>>
>>> The problem with that is that we can't be sure that Cython will be the only
>>> caller. So exceptions may still not propagate in cases, and users will have
>>> to know about these "obscure" values and that they must deal with them
>>> manually then.
>>>
>>> You could add that we'd just have to disable this when user code takes a
>>> pointer from a function, but then, how many rules are there that users will
>>> have to learn and remember after such a change? And what's that for a
>>> language that changes the calling semantics of a function because way down
>>> in the code someone happens to take a pointer to it?
>>>
>>>
>>>>>> Was this not implemented because Cython only knows whether functions
>>>>>> may propagate exceptions at code generation time by looking at the
>>>>>> presence of an error label?
>>>>>> Maybe it could keep code insertion points around for every call to
>>>>>> such a potential function and if the function uses the error label
>>>>>> have the caller perform the check? Although I do forsee problems for
>>>>>> external such functions... maybe Cython could have it's own
>>>>>> threadstate regardless of the GIL which would indicate whether an
>>>>>> error has occurred? e.g. CyErr_Occurred()?
>>>>>
>>>>> Yep, those are the kind of reasons why writing unraisable exceptions is the
>>>>> default.
>>>>
>>>> Still,
>>>
>>> I wasn't really advocating this behaviour, just indicating that it's hard
>>> to do "better", because this "better" isn't all that clear. It's also not
>>> "better" for all code, which means that we get from one trade-off to
>>> another, while breaking existing code at the same time. Not exactly
>>> paradise on either side of the tunnel.
>>
>> I still feel like we're stuck in the wrong default. I'd rather require
>> more work to interact with C libraries than require more work to
>> convert innocent-looking Python to Cython.
>>
>>> One example that keeps popping up in my mind is callback functions that
>>> cannot propagate errors, at least not the CPython way. I have a couple of
>>> those in lxml, even some returning void. So I wrap their code in a bare
>>> try-except and when an exception strikes, I set a C error flag to tell the
>>> C library that something went wrong and return normally. No Python code
>>> outside of the try block. But Cython still generates code for unraisable
>>> errors. Why? Because the internal code that handles the bare except clause
>>> may fail and raise an exception. How about that?
>>>
>>>
>>>> the need to explicitly declare "except *" keeps coming up again and
>>>> again, and is really a blemish on the usability of Cython. When teaching
>>>> people Cython, then it's really irritating to have to follow "all you need
>>>> to do is add some 'cdef' and some types" with "and then you need to
>>>> remember to say "except *", or you're in deep trouble". Cython sort of
>>>> looks very elegant until that point...
>>>
>>> I know what this feels like. The problem is that these things *are* complex.
>>
>> Yes. We've been wrestling with this issue almost since Cython's inception...
>>
>> I like Mark's two-function idea, with the caveat that f(bad_argument)
>> now behaves quite differently than (&f)[0](bad_argument) for even more
>> obscure reasons. But it may be the way to go.
>>
>> The other option is to embed the error behavior into the signature and
>> require casts to explicitly go from one to the other. This would
>> probably require a notation for never raising an exception (e.g.
>> "except -"). Cdef public or api functions could require an except
>> declaration (positive or negative), ordinary cdef functions would be
>> "except *" by default, and cdef extern functions would be "except -"
>> by default.
>
> Only except * and except ? have ever made some sense to me. Except +
> is the most mysterious syntax ever, imho it should have been 'except
> cpperror' or something. And when you try to search for "except +" or
> "except *" etc on docs.cython.org it doesn't find anything, which
> makes it hard for people reading the code and unfamiliar with the
> syntax to figure out what it means. In general I also think decorators
> would have been clearer when defining such functions. Let's please not
> introduce more weird syntax.

"except IDENTIFIER" already has a meaning, and it's nice to have as
few keywords as possible (though I agree for searching). The problem
with decorators is that they don't lend themselves being declarable
part of a type declaration. What would the syntax be for a function
pointer (or extern function) that does propagate Python exceptions?
What about one that throws a C++ exception?

> In any event I don't see why we'd want 'except -', as we're trying to
> get rid of the except clause.

Is it possible to get rid of it entirely? I was thinking we could
provide more natural defaults, but the user might still need to
declare when things are different.

> So you can still get your old behaviour
> for function pointers by not using the except clause and having it
> write unraisable exceptions in the function, but in Cython space you'd
> simply get better semantics (that is, propagating exceptions).

I agree with Dag that this is very similar to the GIL issue, and
should probably be tackled similarly.

- Robert