[Python-Dev] Status of the PEP 433?

[Victor Stinner]

Hi,

I read again all emails related to PEP 433. There are different
opposition to this change and I tried to make a summary.

My question is: how can I move this PEP forward (accept, defer or
reject it)? Can I accept it myself, or do we need a global agreement,
or should someone else decide for this PEP (who? Antoine maybe)?

--

Most complains concern issues if cloexec is set by default. I don't
understand why so many people feel concerned because my PEP proposes
*to not change anything to the default behaviour*: cloexec will not be
set by default, as it was always the case since Python 1.

You have to explicitly use -e command line option, PYTHONCLOEXEC=1
environment variable, or call sys.setdefaultcloexec(True). If you
*explicitly* change the default value of the cloexec parameter, you
must expect to change the behaviour of your application and maybe
discover "bugs" in your application (application incompatible with
cloexec set by default).

(I don't think that any library would like to play with
sys.setdefaultcloexec(), since most developers agree that it might
break applications.)

--

Ok, let's see all oppositions, presented as a FAQ.
(I hope that I didn't misuse email quotes.)

Q: the PEP does not solve file descriptor inherance after fork()
A: Correct, it only solves issues with exec(), as explained in the PEP.

Q: the PEP changes the POSIX behaviour.
A: Antoine replied that "Python is not POSIX", Nick wrote "While many
of Python's APIs are heavily inspired by POSIX, it still isn't POSIX"
(he was talking about integer division). Perl sets cloexec flag by
default since its first version, something like 25 years ago, except
for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by
default.

Q: The PEP may break applications.
A: Most developers agree that it is (very) unlikely. If file
descriptor inherance matters, subprocess must be used (because it
rocks!) with pass_fds. If subprocess is used without pass_fds,
applications stops working since python 3.2 (since python 3.0 on
Windows). pass_fds will clear the close-on-exec flag on the file
descriptors with the PEP. If an application breaks because it relies
on file descriptor inherance, it's easy to detect it (EBADF error) and
simple to fix it (ex: just call sys.setdefaultcloexec(False) at
startup, or set explicitly cloexec parameter on the few functions
causing trouble).

Q: Performance overhead if cloexec is set by default (extra syscalls).
A: The PEP now contains a benchmark: the overhead *on a
microbenchmark* is between 1 and 3%. I bet that nobody will be able to
notice a difference on a real application.

Q: The default value of cloexec must not be configurable: "libraries
cannot rely on it and have to make file descriptors cloexec on an
individual basis, since the default flag can be disabled".
A: IMO inherance of file descriptor doesn't matter in most cases, so
only a few function must be modified to explicitly set or clear
cloexec flag. Said differently: libraries should not care of the
default value of the cloexec parameter, which should only be
configured by applications (not libraries).

Q: The default value of cloexec must not be configurable: "just by
looking at this code, you have no way to know whether the file
descriptor will be inherited by the child process."
A: Guido wrote "Honestly, what happened to the idea that we're all
adults here? The likelihood that someone is clueless enough to misuse
the flag and yet clueful enough to find out how to change it seems
remote -- and they pretty much deserve what they get."

--

Another alternative (not listed in the PEP yet) is to leave the posix
module (its name depends on the platform: posix, nt or ce) unchanged,
and only modify the os module to add cloexec parameter.

I implemented this alternative in the "posixmod" branch of the PEP 433
repository. It gives a better traceback on error because the cloexec
flag is set in Python, so it's possible to know if the error comes
from the creation of the file descriptor or setting the flag in the
traceback. (Does it really matter? I never seen an error on setting or
clearing the flag.)

The problem is that it only concernes posix and os modules: the PEP
does also modify socket, io and select modules, which don't have "low"
and "high" level modules. So it's possible possible to create a socket
using the POSIX behaviour. (Does it matter, it was said that Python is
not POSIX.)

I'm not convinced that we should do something special for posix/os, I
prefer the actual version of the PEP (modify also posix).

Victor