Victor Stinner :

Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR?

The application will often want the EINTR return (exception) instead of having the function resume on its own.

Can you please elaborate? What do you mean by "get wrong"?

Proper handling of signals is difficult and at times even impossible. For example it is impossible to wake up reliably from the select(2) system call when a signal is generated (which is why linux now has pselect). Marko

Ethan Furman :

On 08/31/2014 02:19 PM, Marko Rauhamaa wrote:

...

The application will often want the EINTR return (exception) instead of having the function resume on its own.

Examples?

As an ignorant person in this area, I do not know why I would ever want to have EINTR raised instead just getting the results of, say, my read() call.

Say you are writing data into a file and it takes a long time (because there is a lot of data or the medium is very slow or there is a hardware problem). You might have designed in a signaling scheme to address just this possibility. Then, the system call had better come out right away without trying to complete the full extent of the call. If a signal is received when read() or write() has completed its task partially (> 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). Marko

Le 1 sept. 2014 00:17, "Marko Rauhamaa" a écrit :

If a signal is received when read() or write() has completed its task partially (> 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write().

This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. (When Python gets a signal, the C signal handler is immediatly called. The handler sets a flag which is cheched before executing an instruction. The Python signal handler can be called between two Python instructions. In some cases, it may be called earlier in functions checking manually the flag. IMO the exact behaviour is undefined. Python tries to call the Python signal handler as soon as possible, with a low performance overhead.) Victor

On Mon, 01 Sep 2014 19:15:33 +1200 Greg Ewing wrote:

Victor Stinner wrote:

...

Le 1 sept. 2014 00:17, "Marko Rauhamaa" mailto:marko@pacujo.net> a écrit :

...

If a signal is received when read() or write() has completed its task partially (> 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write().

This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR.

I think there's a problem here, though. As thing stand, a signal handler that doesn't raise an exception can set a flag, and code after the read() can test it.

Under the proposed scheme, the signal handler has to be made to raise an exception so that the read will be broken out of in the EINTR case.

The PEP already addresses this remark: ""Applications relying on the fact that system calls are interrupted with ``InterruptedError`` will hang. The authors of this PEP don't think that such application exist. If such applications exist, they are not portable and are subject to race conditions (deadlock if the signal comes before the system call).""" Regards Antoine.

Victor Stinner :

But I don't get you point. How does this PEP make the situation worse?

Did I say it would? I just wanted to make sure the system call resumption doesn't become mandatory. Haven't thought through what the exception raising technique would entail. It might be perfectly ok apart from being a change to the signal handler API.

I don't know issues of signals with select() (and without a file descriptor used to wake up it).

A signal handler often sets a flag, which is inspected when select() returns. The problem is when a signal arrives between testing the flag and calling select(). The pselect() system call allows you to block signals and have the system call unblock them correctly to avoid the race. Marko

Victor Stinner wrote:

As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception.

I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. -- Greg

On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg wrote:

On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing wrote:

...

Victor Stinner wrote:

...

As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception.

I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't?

I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities.

I'm inclined to agree about keeping the os module thin. If we were to recreate Python today, from scratch, it might make sense to hide this by default, but now there's almost certainly code out there that depends on the current behavior.

But I also agree that it's hard to pin down which higher level Python library calls are going to be using system calls. My http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem with window resizing for a while (SIGWINCH), and although I use it pretty much daily now without problems, I'm still not sure I got 100% of the possibilities covered.

Fortunately, wrapping a system call can be as simple as:

def retry_on_eintr(function, *args, **kw): ''' Retry a system call until one of the following happens: 1) It succeeds 2) It errors with something other than EINTR '''

while True: try: return function(*args, **kw) except OSError: nothing, extra, nothing2 = sys.exc_info() dummy = nothing dummy = nothing2 if extra.errno == errno.EINTR: continue else: raise

Note that os.read() and os.write() need different handling.

Personally, I really want Python to handle EINTR for me. And indeed, that has been what we have been doing for a while now, piece by piece, bug by bug. Victor just wants to systematize and document that, and I think that's a good idea. We've been consistently treating lack of handling of EINTR as a bug. If there are *real* cases where that causes a backward compatibility problem, then we need to know. But so far, we have gotten zero complaints about the cases that we have fixed. --David PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket.

On Mon, 01 Sep 2014 08:30:27 +0300, Marko Rauhamaa wrote:

"R. David Murray" :

...

PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket.

Under what circumstances would a nonblocking socket generate an EINTR?

Windows. Enough said? The exact error message was: BlockingIOError on my non-blocking socket: 'a non-blocking socket operation could not be completed immediately" Needless to say, I was not expecting this, and was about to tear my remaining hair out about having to completely restructure the code in order to be able to handle an EINTR on a read on an FD that I got back from select as ready, until I realized that the way the code had evolved the only thing I still needed the select for was the timeout, and that the EINTR bug in sockets with a timeout had already been fixed (thank goodness I'm able to use python3.4 for this project). I got lucky, but this is clearly a serious problem for writing selectors based code on Windows. This should tell you just about everything you need to know about why we want to fix this problem so that things work cross platform. --David

Le 1 sept. 2014 02:40, "Greg Ewing" a écrit :

Victor Stinner wrote:

...

As written in the PEP, if you want to be notified of the signal, set a

signal handler which raises an exception.

I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't?

Ok, let's say that a syscall is interrupted by a signal, but rhe signal doesn't raise an exception. So your program can only be interrupted if the signal is received during a syscall, right? I don't think that such program is reliable. Python should not promote such design. It should behave the same if the signal is received during a CPU-bound function. Extract of the PEP: Backward Compatibility: Applications relying on the fact that system calls are interruptedwith ``InterruptedError`` will hang. The authors of this PEP don'tthink that such application exist.If such applications exist, they are not portable and are subject torace conditions (deadlock if the signal comes before the system call).These applications must be fixed to handle signals differently, tohave a reliable behaviour on all platforms and all Python versions.For example, use a signal handler which raises an exception, or use awakeup file descriptor.For applications using event loops, ``signal.set_wakeup_fd()`` is therecommanded option to handle signals. The signal handler writes signalnumbers into the file descriptor and the event loop is awaken to readthem. The event loop can handle these signals without the restrictionof signal handlers. Victor

No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Victor Le 1 sept. 2014 08:12, "Paul Moore" a écrit :

On 31 August 2014 22:38, Victor Stinner wrote:

...

This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt.

Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback...

Paul

There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior. As for the general behavior: all programming languages/platforms handle EINTR transparently. It's high time for Python to have a sensible behavior in this regard. 2014-09-01 8:38 GMT+01:00 Marko Rauhamaa :

Victor Stinner :

...

No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program.

Which raises an interesting question: what happens to the os.read() return value if SIGINT is received?

Marko _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cf.natali%40gmail.com

2014-09-01 12:15 GMT+01:00 Marko Rauhamaa :

Charles-François Natali :

...

...

Which raises an interesting question: what happens to the os.read() return value if SIGINT is received?

There's no return value, a KeywordInterrupt exception is raised. The PEP wouldn't change this behavior.

Slightly disconcerting... but I'm sure overriding SIGINT would cure that. You don't want to lose data if you want to continue running.

...

As for the general behavior: all programming languages/platforms handle EINTR transparently.

C doesn't. EINTR is there for a purpose.

Python is slightly higher level than C, right? I was referring to Java, go, Haskell... Furthermore, that's not true: many operating systems actually restart syscalls by default (including Linux, man 7 signal): """ Interruption of system calls and library functions by signal handlers If a signal handler is invoked while a system call or library function call is blocked, then either: * the call is automatically restarted after the signal handler returns; or * the call fails with the error EINTR. Which of these two behaviors occurs depends on the interface and whether or not the signal handler was established using the SA_RESTART flag (see sigaction(2)). The details vary across UNIX systems; below, the details for Linux. """ The reason the interpreter is subject to so many EINTR is that we *explicitly* clear SA_RESTART because the C-level signal handler must be handled by the interpreter to have a chance to run the Python-level handlers from the main loop. There are many aspects of signal handling in Python that make it different from C: if you want C semantics, stick to C. I do not want to have to put all blocking syscalls within a try/except loop: have a look at the stdlib code, you'll see it's really a pain and ugly. And look at the number of EINTR-related syscalls we've had.

On Mon, 01 Sep 2014 11:47:07 -0400 "R. David Murray" wrote:

...

The two requirements are:

* Allow the application to react to signals immediately in the main flow.

You don't want to be writing your code in Python then. In Python you *never* get to react immediately to signals. The interpreter sets a flag and calls the python signal handler later. Yes, the call is ASAP, but ASAP is *not* "immediately".

Especially if the signal is delivered to another thread (which is OS-dependent), and the main thread is blocked in *another* system call ;-) Regards Antoine.

On 2 September 2014 07:17, Matthew Woodcraft wrote:

(The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.)

Making an exception for sleep() (i.e. still letting it throw EINTR) sounds like a reasonable idea to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

2014-09-02 23:02 GMT+02:00 Matthew Woodcraft :

I think people who use sleep() in their programs could benefit from not having to worry about EINTR as much as anyone else.

The behaviour of time.sleep() is worse than what I expected. On UNIX, if select() fails with EINTR, time.sleep() calls PyErr_CheckSignals(). If the signal handler doesn't raise an exception, time.sleep() returns None and just simply ignores the error. But on Windows, it's the opposite. If time.sleep() is interrupt by CTRL+c, time.sleep() raises an InterruptedError... Good luck to write portable code :-p With the PEP 475, time.sleep(secs) now has a well defined behaviour. It sleeps at least "secs" seconds, retry the syscall on EINTR, and raises an exception if the signal handler raises an exception. Victor

Antoine Pitrou wrote:

Matthew Woodcraft wrote:

...

(The program handles SIGTERM so that it can do a bit of cleanup before exiting, and it uses the signal-handler-sets-a-flag technique. The call that might be interrupted is sleep(), so the program doesn't strictly _rely_ on the existing behaviour; it would just become very slow to exit.)

So, if it's just for process exit, just let the signal handler raise an exception and catch the exception at the top-level.

I wouldn't recommend that anyone take this advice. The signal might come at some time other than the sleep(), and introducing an exception which can mysteriously appear at abitrary points is not a way to make life easier. Nonetheless I'm sure my program would be easy enough to fix to use set_wakeup_fd() or signalfd() if necessary. I'm not saying we shouldn't make this backwards-incompatible change because it will be hard to update existing programs to cope; I'm saying we shouldn't pretend it doesn't count as a backwards-incompatible change. In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported? If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR? (I think it would be helpful if we could tell people "if you want the old EINTR behaviour, just do this simple thing". And I suppose siginterrupt flag=True is a candidate for that.) -M-

In article , Victor Stinner wrote:

2014-09-02 23:03 GMT+02:00 Matthew Woodcraft :

...

In any case I think PEP 475 should be explaining what is going to happen to signal.siginterrupt(). Will setting flag=True be supported?

I first proposed to deprecate the function, but Charles-François thinks that it's unrelated to the PEP (it can be addressed later).

The function will still be available and work.

...

If so, will doing so change the behaviour of those parts of the stdlib which have already been modified to retry after EINTR?

I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code.

That seems good to me. I think it's worth writing in the PEP. But if Python is going to provide its own higher-level model of signal handling, then I think signal.siginterrupt() will need to be documented in terms of that model; it should be saying something along the lines of "if a signal arrives while Python is blocking in a system call then InterruptedError will be raised", and I suppose it will need to say what happens if the signal handler also raised an exception.

...

(I think it would be helpful if we could tell people "if you want the old EINTR behaviour, just do this simple thing". And I suppose siginterrupt flag=True is a candidate for that.)

Why do you want the old behaviour?

Purely to avoid breaking existing programs, particularly in ways which will require effort to fix. I think Python's backward-compatibility rules are basically "the behavior of an API must not change without going through the deprecation process". If we're going to be making an exception to that here, then it would be much better to say "here's a simple change to make to keep your old program working", rather than "please rewrite your fiddly signal-handling code to use more modern techniques", however much nicer those modern techniques are. Or perhaps it would be better to not make an exception at all, and instead add an 'interrupt_syscalls' boolean keyword argument to signal.signal(), while strongly recommending that people set it False. -M-