[Cython] [cython-users] C++: how to handle failures of 'new'?

[Dag Sverre Seljebotn]

On 07/03/2012 09:15 PM, Robert Bradshaw wrote:
> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>
>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>> wrote:
>>>>
>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>
>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>
>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>
>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> I don't know what happens if a C++ exception is not being caught, but
>>>>>>>> I
>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>> visible than
>>>>>>>
>>>>>>>
>>>>>>> Yep.
>>>>>>>
>>>>>>>> just printing a warning when a Python exception is being ignored due
>>>>>>>> to a
>>>>>>>> missing declaration. It's really unfortunate that our documentation
>>>>>>>> didn't
>>>>>>>> even mention the need for this, because it's not immediately obvious
>>>>>>>> that
>>>>>>>> Cython won't handle errors in "new", and testing for memory errors
>>>>>>>> isn't
>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>
>>>>>>>> Apart from that, I agree, users have to take care to properly declare
>>>>>>>> the
>>>>>>>> API they are using.
>>>>>>>
>>>>>>>
>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>>>>>>> see a
>>>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>>>
>>>>>>
>>>>>> That would have been my intuition, too.
>>>>>
>>>>>
>>>>> If it's actually embedded, with the main driver in C++, one might want
>>>>> it to propagate up.
>>>>
>>>>
>>>> But what kind of a propagation would that be? On the way out, it could
>>>> induce anything, from side effects to resource leaks to crashes,
>>>> depending
>>>> on what the state of the surrounding code is. It would leave the whole
>>>> system in an unpredictable state. I cannot imagine anyone really wanting
>>>> this.
>>>>
>>>>
>>>>>>> So shouldn't we just make --cplus turn *all* external functions and
>>>>>>> methods
>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>>>>>>> manual
>>>>>>> translation, but always have a catch(...)".
>>>>>>>
>>>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>> unwinds and
>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>
>>>>>>> "except -1" should then actually mean both; "except + except -1". So
>>>>>>> it's
>>>>>>> more a question of just adding catch(...) *everywhere*, than making
>>>>>>> "except
>>>>>>> +" the default.
>>>>>>
>>>>>>
>>>>>> I have no idea if there is a performance impact, but if there isn't,
>>>>>> always
>>>>>> catching all exceptions sounds like a reasonable thing to do. After
>>>>>> all, we
>>>>>> have no support for catching C++ exceptions on user side.
>>>>>
>>>>>
>>>>> This is a bit like following every C call with "except *" (though the
>>>>> performance ratios are unclear). It just seems a lot to wrap every
>>>>> single line of a non-trivial C++ using function with try..catch
>>>>> blocks.
>>
>>
>> It seems "a lot" of just what exactly? Generated code? Binary size? Time
>> spent in GCC parser?
>
> All of the above. And we should take a look at the runtime overhead
> (which is hopefully nil, but who knows.)
>
>> Though I guess one might want to try to pull out the try-catch to at least
>> only one per code line rather than one per SimpleCallNode.
>
> Or even higher, if possible. It's still a lot.
>
>> "except *" only has a point when calling functions using the CPython API,
>> but most external C functions are pure C, not CPython-API-using-functions.
>> OTOH, all external C++ functions are C++ :-)
>
> Fair point.
>
>> (Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
>> defaults would be a tad different.)
>
> For sure.
>
>>>> But if users are correct about their declarations, we'd end up with the
>>>> same thing. I think it's worth a try.
>>>
>>>
>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>> because exception handling is so broken in C++ anyways.
>>
>>
>> Except for the fact that any code touching "new" could be raising
>> exceptions? That propagates.
>
> I would guess most of the time people don't bother catching these and
> let the program die, as there's often no sane recovery (the same as
> MemoryErrors in Python, though I guess C++ is less often used from an
> event loop).

The point is to provide the user with the stack trace of exactly where 
you ran out of memory. Even if you can't follow it deep into C++-land, 
the Python-side stack trace can be a worth a lot.

>
>> There is a lot of C++ code out there using exceptions. I'd guess that both
>> mathematical code and Google-written code is unlike most C++ code out there
>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
>> and that doesn't have much point unless you throw an exception now and then
>> (OK, there's the occasional return statement where it matters well).
>
> True, I've seen a small subset of the C++ code that's out there. Maybe
> numerical computations use it a lot?

One library I'm using for distributed dense linear algebra (Elemental) 
does. Numerical computations libraries are very different from one another.

Dag