[issue30297] Recursive starmap causes Segmentation fault

[Raymond Hettinger]

Raymond Hettinger added the comment:

I'm not sure this can be fixed reasonably.  It would entail taking every possible iterator in all of Python and adding some sort of recursion depth check.  That approach risks introducing bugs, risks breaking existing code that relies on heavy iteration depth (we know YouTube uses such code), and it would slow-down iteration everywhere (possibly by quite a bit since iteration is a thin and fast protocol).

Also, this is a general case of recursive crashers (there are a number of open bugs on this) that are hard to detect and prevent.

IMO, what is really needed is something that monitors the C stack and gracefully detects the situation just before a crash and gracefully unwinds the calls.  This isn't easy to do but it would be a broad, complete, performant, and minimally invasive solution to a problem that is irritating but only arises very rarely (and typically only in code that isn't really sane to begin with). 

The usual recursion depth checks on function calls are reasonable because 1) their overhead is minimal compared to overall function call overhead and 2) they detect common programming errors.   

To me, the starmap() case is different because the problem is rare and because iteration is a tight fast protocol.  

One other thought:  While a RuntimeError would be much preferred to a crash, the programmer has to respond in the same way, by changing their code.  In the pbkdf2_bin() example, the code can't be made to run as-is.  The approach has to be changed to a list based approach (even using hand-rolled iterators instead of starmap() won't help).

A last thought:  I presume that the whole reason for using starmap() instead of "yield from" or somesuch is the person writing code needed good performance, so taking away performance defeats the original intent.

----------
assignee:  -> rhettinger
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30297>
_______________________________________