[Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

[Victor Stinner]

2013/1/25 Charles-François Natali <cf.natali at gmail.com>:
> You can actually count me in the cloexec=False camp, and against the
> idea of a configurable default value.

Oh ok.

> Why cloexec shouldn't be set by default:
> - While it's really tempting to fix one of Unix historical worst
> decisions, I don't think we can set file descriptors cloexec by
> default: this would break some applications (I don't think there would
> be too many of them, but still), but most notably, this would break
> POSIX semantics. (...)

Agreed.

> Why the default value shouldn't be tunable:
> - I think it's useless: if the default cloexec behavior can be altered
> (either by a command-line flag, an environment variable or a sys
> module function), then libraries cannot rely on it and have to make
> file descriptors cloexec on an individual basis, since the default
> flag can be disabled.

In my experience, in most cases, the default value of cloexec just
doesn't matter at all. If your program relies on the state of the
close-on-exec flag: you have to make it explicit, specify cloexec
paramater. Example: os.pipe(cloexec=True). If you don't modify your
application, it will just not work using -e command line option (or
PYTHONCLOEXEC environment variable). But why would you enable cloexec
by default if your application is not compatible?

> So it would basically be useless for the Python
> standard library, and any third-party library. So the only use case is
> for application writers that use raw exec() (since subprocess already
> closes file descriptors > 3, and AFAICT we don't expose a way to
> create processes "manually" on Windows), but there I think they fall
> into two categories: those who are aware of the problem of file
> descriptor inheritance, and who therefore set their FDs cloexec
> manually, and those who are not familiar with this issue, and who'll
> never look up a sys.setdefaultcloexec() tunable (and if they do, they
> might think: "Hey, if that's so nice, why isn't it on by default?
> Wait, it might break applications? I'll just leave the default
> then.").

The problem is that Python can be embeded in application: the
application can start a child process independently of Python.

> - But most importantly, I think such a tunable flag is a really wrong
> idea because it's a global tunable that alters the underlying
> operating system semantics. Consider this code:
> """
> r, w = os.pipe()
> if os.fork() == 0:
>     os.execve(['myprog'])
> """

If this snippet doesn't work with cloexec enabled by default, you have
to write: os.pipe(cloexec=False).

> - We should go over the standard library, and create FDs cloexec if
> they're not handed over to the caller, either because they're
> opened/closed before returning, or because the underlying file
> descriptor is kept private (not fileno() method, although it's
> relatively rare). That's the approach chosen by glibc, and it makes
> sense: if another thread forks() while a thread is in the middle of
> getpwnam(), you don't want to leak an open file descriptor to
> /etc/passwd (or /etc/shadow).

I started to make cloexec explicit *everywhere* in the Python stdlib.
I reverted my commit because I think that only a few application start
child processes, so doing extra work (additionnal syscalls to set
cloexec) would slowdown Python for no gain. See my revert commit to
see how many functions need to be modified:
http://hg.python.org/features/pep-433/rev/963e450fc24f

That's why I really like the idea of being able to configure the
default value of the cloexec parameter. By default: no overhead nor
backward compatibility issue. Whereas you can set close-on-exec flag
*everywhere* if you are concern by all issues of inheriting file
descriptors (cases listed in the PEP).

In the stdlib, I only specified cloexec parameter where it was
required to ensure that it works with any default value. Only a few
modules were modified (subprocess, multiprocessing, ...).

Victor