On 1/19/2021 10:01 AM, Mark Shannon wrote:

...

...

The following program will run safely to completion:

I interpreted this to mean 'works now', on whatever system you tested this on. You question suggests that you meant "fails now but will work with a successful patch for the PEP".

...

...

     sys.setrecursionlimit(1_000_000)

On Windows, this recursion limit increase lets the function run about 2160 loops (when run with IDLE) instead of 1000, but makes the behavior WORSE by replacing the exception with a silent failure. This could be considered worse than a crash.

...

...

     def f(n):          if n:              f(n-1)

     f(500_000)

I'm not sure whether you are saying that this doesn't work now, that it can't work, or that it shouldn't work.

If that it doesn't work now, then I agree.

On my Win 10, setting the recursion limit to 2160 (2135 in IDLE, which bumps it up to account for additional stack space used by IDLE) results in RecursionError. Much higher and the failure is silent. Always getting a RecursionError would itself be an improvement. So I am saying that current recursion limit handling on Windows is terrible and perhaps worse than on Linux. -- Terry Jan Reedy

On Tue, 19 Jan 2021 15:54:39 +0000 Mark Shannon wrote:

On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

...

On Tue, 19 Jan 2021 13:31:45 +0000 Mark Shannon wrote:

...

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much. It's aim is to make the VM more robust.

On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth() would look like?

It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Incl...

This doesn't tell me how `stack_limit_pointer` is computed or estimated :-)

On Tue, 19 Jan 2021 16:28:46 +0000 Mark Shannon wrote:

On 19/01/2021 4:15 pm, Antoine Pitrou wrote:

...

On Tue, 19 Jan 2021 15:54:39 +0000 Mark Shannon wrote:

...

On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

...

On Tue, 19 Jan 2021 13:31:45 +0000 Mark Shannon wrote:

...

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much. It's aim is to make the VM more robust.

On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth() would look like?

It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Incl...

This doesn't tell me how `stack_limit_pointer` is computed or estimated :-)

It's nothing clever, and the numbers I've chosen are just off the top of my head.

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modu...

What about the main thread? Is `stack_limit_pointer` some kind of thread-local? I suppose nothing stops an OS to use different virtual addresses for the stacks of different threads (especially if randomization of stack addresses is desired). Regards Antoine.

On Tue, 2021-01-19 at 16:22 +0000, Mark Shannon wrote:

On 19/01/2021 3:43 pm, Sebastian Berg wrote:

...

On Tue, 2021-01-19 at 13:31 +0000, Mark Shannon wrote:

...

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much. It's aim is to make the VM more robust.

Abstract ========

This PEP proposes that machine stack overflow is treated differently from runaway recursion. This would allow programs to set the maximum recursion depth to fit their needs and provide additional safety guarantees.

The following program will run safely to completion:

      sys.setrecursionlimit(1_000_000)

      def f(n):           if n:               f(n-1)

      f(500_000)

The following program will raise a StackOverflow, without causing a VM crash:

      sys.setrecursionlimit(1_000_000)

      class X:           def __add__(self, other):               return self + other

      X() + 1

This is appreciated! I recently spend quite a bit of time trying to solve a StackOverflow like this in NumPy (and was unable to fully resolve it).  Of course the code triggering it was bordering on malicious, but it would be nice if it was clear how to not segfault.

Just some questions/notes:

* We currently mostly use `Py_EnterRecursiveCall()` in situations where we need to safe-guard against "almost python" recursions. For example an attribute lookup that returns `self`, or a list containing itself. In those cases the python recursion limit seems a bit nicer (lower and easier to understand). I am not sure it actually matters much, but my question is: Are we sure we want to replace all (or even many) C recursion checks?

Would it help if you had the ability to increase and decrease the recursion depth, as `Py_EnterRecursiveCall()` currently does?

I'm reluctant to expose it, as it might encourage C code authors to use it, rather than `Py_CheckStackDepth()` resulting in crashes.

To be robust, C code must make a call to `Py_CheckStackDepth()`. To check the recursion limit as well would be extra overhead.

...

* Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a new `StackOverflow` exception name is useful. It may create two names for almost identical Python code:  If you unpack a list containing itself compared to a mapping implementing `__getitem__` in Python you would get different exceptions.

True, but they are different. One is a soft limit that can be increased, the other is a hard limit that cannot (at least not easily).

Right. I think my confusion completely resolves around your proposed change of `Py_EnterRecursiveCall()`. A simple example (written in C): def depth(obj, current=0): Py_EnterRecursiveCall() if isinstance(obj, sequence): # has the sequence slots return depth(obj[0], current+1) return current will never hit the "depth" limit for a self containing list or even sequence (as long as `GetItem` can use the C-level slot). But `obj[0]` could nevertheless return a non-trivial object (one with `__del__`, definitely a container with unrelated objects that could use deleting). As the author of the function, I have no knowledge over how much stack space cleaning those up may require? And say someone adds a check for `Py_CheckStackDepth()` inside a dealloc, then this might have to cause a fatal error? Maybe it should even be a fatal error by default in some cases? Also, if the code is slow, the previous recursion may guard against hanging (arguably, if that is the case I probably add an interrupt check, I admit). Long story short, I will trust you guys on it of course, but I am not yet convinced that replacing the check will actually do any good (as opposed to adding and/or providing the additional check) or even be a service to users (since I assume that the vast majority do not crank up the recursion limit to huge values). Cheers, Sebastian

...

* `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because `Py_CheckStackDepth()` would leave plenty of headroom for typical clean-up?

What is "typical" clean up? I would hope that typical cleanup is to return immediately.

...

Can we assume that DECREF's (i.e. list, tuple), will never check the depth, so head-room is usually not necessary?  This is all good, but I am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would be necessary (There are probably many cases where it clearly is, but is it ever for fairly simple code?).

Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need to be very cheap for that to be practical.

If C code is consuming the stack, its responsibility is to not overflow. We can't make you call `Py_CheckStackDepth()`, but we can provide it, so you that will have no excuse for blowing the stack :)

...

What happens if the maximum stack depth is reached while a `StackOverflow` exception is already set?  Will the current "watermark" mechanism remain, or could there be a simple rule that an uncleared `StackOverflow` exception ensures some additional head-room?

When an exception is "set", the C code should be unwinding stack, so those states shouldn't be possible.

We can't give you extra headroom. The C stack is a fixed size. That's why `Py_CheckStackDepthWithHeadRoom()` is provided, if `Py_CheckStackDepth()` fails then it is too late to do much.

Cheers, Mark.

...

Cheers,

Sebastian

...

-----------

The full PEP can be found here: https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers, Mark. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43Y... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N456CVKW... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GSR6DCTX... Code of Conduct: http://python.org/psf/codeofconduct/

On 19/01/2021 5:48 pm, Guido van Rossum wrote:

I'm not clear on how you plan to implement this in CPython.

I can totally see that if a Python function calls another Python function, you can avoid the C stack frame and hence you can have as many Python call levels as you want.

However, there are many scenarios where a Python function calls a C function (e.g. `filter()`, or `dict.__setitem__()`) and that C function at some point calls a Python function (e.g. the `__hash__()` method of the key, or even the `__del__()` method of the value being replaced). Then that Python function can recursively do a similar thing.

Indeed, that is the second case below, where a Python __add__ function recursively performs addition. Most likely, the C stack will get exhausted before the recursion limit is hit, so you'll get a StackOverflow exception.

Are you proposing to also support that kind of thing to go on for a million levels of C stack frames?

No, most likely 10k to 20k calls before a StackOverflow exception.

(Do we even have a cross-platform way of avoiding segfaults due to C stack overflow?)

Arithmetic and comparisons on pointers from within the C stack may not strictly conform to the C standard, but it works just fine. It's pretty standard VM implementation stuff. All the JVMs do this sort of thing. IMO practically beats purity in this case, and crashing less has to be a good thing. A rather hacky proof of concept for the stack overflow handling is here: https://github.com/python/cpython/compare/master...markshannon:pep-overflow-... Cheers, Mark.

On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon mailto:mark@hotpy.org> wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much. It's aim is to make the VM more robust.

Abstract ========

This PEP proposes that machine stack overflow is treated differently from runaway recursion. This would allow programs to set the maximum recursion depth to fit their needs and provide additional safety guarantees.

The following program will run safely to completion:

     sys.setrecursionlimit(1_000_000)

     def f(n):          if n:              f(n-1)

     f(500_000)

The following program will raise a StackOverflow, without causing a VM crash:

     sys.setrecursionlimit(1_000_000)

     class X:          def __add__(self, other):              return self + other

     X() + 1

-----------

The full PEP can be found here: https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers, Mark. _______________________________________________ Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43Y... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido http://python.org/~guido) /Pronouns: he/him //(why is my pronoun here?)/ http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)

For in-process tools, then the API will continue to work. For out-of-process debuggers, then the author's are on their own. But that's always been the case. The source code is public, and it won't be any more opaque than it is at the moment :)

I agree with that, but we need to have in mind that this will break these tools and therefore we need to be aware on how difficult will be to overcome the new situation. Is not the same that a small or medium change is required than the case where is virtually impossible to overcome. If we were designing the interpreter from scratch that would not matter, but with an existing ecosystem of tools, it does. Breaking tools in an established ecosystem has always been a major complaint of users when we do major releases. Notice as well that the gdb helpers *are* an out-of-process tool. Of course, I don't imply that this should be a show stopper or similar, but it should be taken into consideration along all the other pros and cons. Regards from cloudy London, Pablo Galindo Salgado On Wed, 20 Jan 2021 at 15:12, Mark Shannon wrote:

Hi Pablo,

...

Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote: preliminary questions: how is this going to affect

...

the "py-bt" command of the gdb helpers ( https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L... ) and other similar tools that produce a unified Python-C backtrace? There are several debuggers that rely on the fact that they can merge the C stack and the Python stack by substituting every call to "_PyEval_EvalFrameDefault" and friends for the respective Python frame obtained separately or by inspecting the frame object in the stack (like gdb does).

I don't see any problem fixing up the gdb helpers. The code will need to be aware that C frames will be one-to-many to Python frames, but there's nothing tricky there.

...

Is this change going to affect these tools or they will continue working

...

is there any workaround to produce the unified C/Python call stack given

as before? In case this change will affect this tools, the Python stack and the C stack?

It depends on what you mean by "similar tools". For in-process tools, then the API will continue to work. For out-of-process debuggers, then the author's are on their own. But that's always been the case. The source code is public, and it won't be any more opaque than it is at the moment :)

Cheers, Mark.

...

Kind regards, Pablo Galindo Salgado _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7...

...

Code of Conduct: http://python.org/psf/codeofconduct/

Do you have any specific examples?

From my previous message: gdb helpers, lldb helpers, TotalView debugger, py-spy and I know of at least 3 closed source tools.

I feel the responsibility for this belongs with the tool authors. If they aren't using the C-API, or requesting an alternative interface should the API be insufficient, then they can't really complain.

The responsibility to fix it is on the authors, yes, but is not that back and white. In the worst case, what users will see is that tools that worked now don't and in the worst-worst case users will see that those tools will not work again because is virtually impossible with the new system. And that, as I said is not a show stopper, but it matters. In any case, if we are able to fix the gdb helpers in the stdlib, then I am not afraid because it means that everyone could fix their own thing. Cheers, Pablo Galindo Salgado On Wed, 20 Jan 2021 at 16:06, Mark Shannon wrote:

Hi Pablo,

On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:

...

...

It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)

...

For in-process tools, then the API will continue to work. For out-of-process debuggers, then the author's are on their own. But that's always been the case. The source code is public, and it won't be any more opaque than it is at the moment :)

I agree with that, but we need to have in mind that this will break these tools and therefore we need to be aware on how difficult will be to overcome the new situation. Is not the same that a small or medium change is required than the case where is virtually impossible to overcome. If we were designing the interpreter from scratch that would not matter, but with an existing ecosystem of tools, it does. Breaking tools in an established ecosystem has always been a major complaint of users when we do major releases.

Do you have any specific examples?

I feel the responsibility for this belongs with the tool authors. If they aren't using the C-API, or requesting an alternative interface should the API be insufficient, then they can't really complain.

...

Notice as well that the gdb helpers *are* an out-of-process tool.

True, but they are in the standard library, so they are our responsibility to fix.

Cheers, Mark.

...

Of course, I don't imply that this should be a show stopper or similar, but it should be taken into consideration along all the other pros and

cons.

...

Regards from cloudy London, Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon mailto:mark@hotpy.org> wrote:

Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote: > Hi Mark, > > Thanks for gathering this proposal! Looks very interesting. I have some preliminary questions: how is this going to affect > the "py-bt" command of the gdb helpers (

https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L... )

...

> and other similar tools that produce a unified Python-C backtrace? There are several debuggers that > rely on the fact that they can merge the C stack and the Python stack by substituting every call to "_PyEval_EvalFrameDefault" > and friends for the respective Python frame obtained separately or by inspecting the frame object in the stack (like gdb does).

I don't see any problem fixing up the gdb helpers. The code will need to be aware that C frames will be one-to-many to Python frames, but there's nothing tricky there.

> > Is this change going to affect these tools or they will continue working as before? In case this change will affect this tools, > is there any workaround to produce the unified C/Python call stack given the Python stack and the C stack?

It depends on what you mean by "similar tools". For in-process tools, then the API will continue to work. For out-of-process debuggers, then the author's are on their own. But that's always been the case. The source code is public, and it won't be any more opaque than it is at the moment :)

Cheers, Mark.

> > Kind regards, > Pablo Galindo Salgado > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org > To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7...

...

> Code of Conduct: http://python.org/psf/codeofconduct/ >