[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

Hi Antoine, On 05/03/2021 4:07 pm, Antoine Pitrou wrote:

On Fri, 5 Mar 2021 15:03:59 +0000 Mark Shannon wrote:

...

There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague. That's my fault; I should have done more implementation first :( FWIW, I have an implementation that should be portable. https://github.com/python/cpython/compare/master...markshannon:pep-overflow-...

I looked through this diff and I'm not sure how this works robustly.

This seems to assume: - each thread's stack size is known and is a compile-time constant We just need to know that the stack size is at least the compile-time constant, we don't need to know exactly what it is.

- Python owns the thread and can compute the stack limit reliably from the current stack frame There are two points of entry into Python code. From the high level API and thread_run(). Since thread_run is called when a thread is started, it is guaranteed to have the full stack available. The high level API (PyRun_SimpleStringFlags and friends) is a bit more

- the OS allocates stack pages in units of BLOCK_SIZE or more (currently 8kiB) The BLOCK_SIZE is just a number. A larger numbers means fewer slow checks, but wastes more stack. It doesn't matter what the underlying

If the stack size is less than the compile-time constant, then a crash is a possibility. However, since PEP 651 would reduce stack consumption, a crash would have been less likely than it currently is. For all supported platforms, there is a default stack size and it is considerably larger than the chosen value of 512k. problematic. If there isn't sufficient stack space, then a crash is a possibility. But, this is no worse than the status quo, and probably better due to reduced C stack use. platform's granularity is.

- the OS doesn't use a segmentation scheme that limits stack accesses with a finer granularity than page (or "block") size I don't think that would matter.

- it's ok to write memory beyond the current stack top (I can imagine verification tools such as Valgrind complaining about that, though this can be worked around using suppressions) It's not much of a stack if you can't grow it :) I wouldn't be surprised if valgrind complains, though.

I would be curious if you could elaborate on these points.

The above scheme does require some care from users of the high-level C-API and should be tested thoroughly for new platforms. But it is a *lot* more robust than what we have now. Cheers, Mark.