On Tue, Jul 23, 2013 at 4:21 PM, Richard Oudkerk wrote:

On 23/07/2013 11:45pm, Victor Stinner wrote:

...

Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on *handles*, as indicated in its name. On Windows, file*descriptors* are never inherited (are always closed) in child processes. I don't think that it is possible to inherit file descriptors on Windows.

Actually, you can inherit fds if you use os.spawnv() instead of subprocess.Popen().

Wow. Indeed you can -- I just tested this myself. How is this accomplished? I guess the CRT has a backchannel to talk to itself when it creates a process using spawn*? This is the only reason I can think of for the odd default in the CRT of opening file descriptors inheritable by default, which Victor discovered. (But it doesn't explain why os.pipe() creates uninheritable fds.) If it weren't for this I would definitely vote to change the default on Windows throughout the stdlib to create file descriptors whose handles aren't inheritable. (Perhaps with a different policy for stdin/stdout/stderr, which seem to be treated specially at the handle level.) I'm about ready to give up hope that we'll ever have a decent way to deal with this. But I'm also ready to propose that all this is such a mess that we *should* change the default fd/handle inheritance to False, *across platforms*, and damn the torpedoes -- i.e. accept breaking all existing 3rd party UNIX code for subprocess creation that bypasses the subprocess module, as well as breaking uses of os.spawn*() on both platforms that depend on FD inheritance beyond stdin/stdout/stderr). With the new, sane default, all we need instead of PEP 446 is a way to make an FD inheritable after it's been created, which can be a single os.make_inheritable(fd) call that you must apply to the fileno() of the stream or socket object you want inherited (or directly to a FD you created otherwise, e.g. with os.pipe()). On Windows, this should probably only work with os.spawn*(), since otherwise you need *handle* inheritance, not *FD* inheritance, and that's a non-portable concept anyway. We can fix multiprocessing any anything else in the stdlib that this breaks, I presume. To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day. -- --Guido van Rossum (python.org/~guido)

On Wed, Jul 24, 2013 at 11:13 AM, Richard Oudkerk wrote:

...

Wow. Indeed you can -- I just tested this myself. How is this accomplished? I guess the CRT has a backchannel to talk to itself when it creates a process using spawn*?

CreateProcess() takes a STARTUPINFO argument with undocumented fields cbReserved2, lpReserved2. They are used to pass an array of fds.

Does it also inherit sockets (which take up a different namespace than regular FDs in CRT, unlike UNIX)?

...

But I'm also ready to propose that all this is such a mess that we

*should* change the default fd/handle inheritance to False, *across platforms*, and damn the torpedoes -- i.e. accept breaking all existing 3rd party UNIX code for subprocess creation that bypasses the subprocess module, as well as breaking uses of os.spawn*() on both platforms that depend on FD inheritance beyond stdin/stdout/stderr).

+1

Thanks! This was a difficult conclusion to come to. "Damn the torpedoes" is occasionally a nice meme. :-(

...

We can fix multiprocessing any anything else in the stdlib that this breaks, I presume.

In the experimental branch of multiprocessing, child processes no longer inherit unnecessary handles.

And if we default individual handles to uninheritable, we can presumably fix the ones that multiprocessing creates with the express purpose of being inherited easily. (If it even uses that -- I haven't read the source code, maybe it uses named pipes? :-) -- --Guido van Rossum (python.org/~guido)

2013/7/24 Richard Oudkerk :

...

Wow. Indeed you can -- I just tested this myself. How is this accomplished? I guess the CRT has a backchannel to talk to itself when it creates a process using spawn*?

CreateProcess() takes a STARTUPINFO argument with undocumented fields cbReserved2, lpReserved2. They are used to pass an array of fds.

So would it be possible to implement the pass_fds parameter of subprocess using spawnl() or the undocumented fields? And is it possible to close all handles except one (implement "pass_handles")? The idea would be to use something subprocess.Popen(cmd, pass_fds=[pipe_rfd], close_fds=True) or subprocess.Popen(cmd, pass_handles=[pipe_rhandle], close_handles=True) instead of subprocess.Popen(cmd, close_fds=False).

In the experimental branch of multiprocessing, child processes no longer inherit unnecessary handles.

Where is this branch? How did you create the channel between the manager and the worker? Victor

On Wed, 24 Jul 2013 10:56:05 -0700 Guido van Rossum wrote:

But I'm also ready to propose that all this is such a mess that we *should* change the default fd/handle inheritance to False, *across platforms*, and damn the torpedoes -- i.e. accept breaking all existing 3rd party UNIX code for subprocess creation that bypasses the subprocess module, as well as breaking uses of os.spawn*() on both platforms that depend on FD inheritance beyond stdin/stdout/stderr).

So I suppose you mean "change it to False except for stdin/stdout/stderr"?

With the new, sane default, all we need instead of PEP 446 is a way to make an FD inheritable after it's been created, which can be a single os.make_inheritable(fd) call that you must apply to the fileno() of the stream or socket object you want inherited (or directly to a FD you created otherwise, e.g. with os.pipe()). On Windows, this should probably only work with os.spawn*(), since otherwise you need *handle* inheritance, not *FD* inheritance, and that's a non-portable concept anyway.

I'm not sure how *fd* inheritance could work without *handle* inheritance. (since a fd seems to just be a proxy for a handle)

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

What do you call "daemon"? An actual Unix-like daemon? Regards Antoine.

On Wed, 24 Jul 2013 15:25:50 -0700 Guido van Rossum wrote:

...

...

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

What do you call "daemon"? An actual Unix-like daemon?

Yeah, a background process with parent PID 1 and not associated with any terminal group.

But is that relevant to the PEP? A daemon only uses fork(), not exec(). Or have I misunderstood your concern? Regards Antoine.

On Wed, Jul 24, 2013 at 3:35 PM, Antoine Pitrou wrote:

On Wed, 24 Jul 2013 15:25:50 -0700 Guido van Rossum wrote:

...

...

...

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

What do you call "daemon"? An actual Unix-like daemon?

Yeah, a background process with parent PID 1 and not associated with any terminal group.

But is that relevant to the PEP? A daemon only uses fork(), not exec(). Or have I misunderstood your concern?

Actually it's common for "daemonization" libraries to run an arbitrary executable as a daemon, and then it would be relevant. E.g. zdaemon (https://pypi.python.org/pypi/zdaemon) does this. (Disclosure: I wrote the first version of zdaemon when I worked for Zope Corp over a decade ago. :-) -- --Guido van Rossum (python.org/~guido)

I checked the python-daemon module: it closes all open file descriptors except 0, 1, 2. It has a files_preserve attribute to keep some FD opens. It redirects stdin, stdout and stderr to /dev/null and keep these file descriptors open. If python-daemon is used to execute a new program, the files_preserve list can be used to mark these file descriptors as inherited. The zdaemon.zdrun module closes all open file descriptors except 0, 1, 2. It uses also dup2() to redirect stdout and stderr to the write end of a pipe. Victor 2013/7/25 Eric V. Smith :

On 7/24/2013 6:25 PM, Guido van Rossum wrote:

...

...

...

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

What do you call "daemon"? An actual Unix-like daemon?

Yeah, a background process with parent PID 1 and not associated with any terminal group.

There's PEP 3143 and https://pypi.python.org/pypi/python-daemon. I've used it often, with great success.

-- Eric. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

Guido van Rossum writes:

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

Work continues on the PEP 3143-compatible ‘python-daemon’, porting it to Python 3 and aiming for inclusion in the standard library. Interested parties are invited to join us on the discussion forums URL:http://lists.alioth.debian.org/cgi-bin/mailman/admin/python-daemon-devel. -- \ “Politics is not the art of the possible. It consists in | `\ choosing between the disastrous and the unpalatable.” —John | _o__) Kenneth Galbraith, 1962-03-02 | Ben Finney

On 25 Jul, 2013, at 4:18, Ben Finney wrote:

Ben Finney writes:

...

Work continues on the PEP 3143-compatible ‘python-daemon’, porting it to Python 3 and aiming for inclusion in the standard library.

At first glance the library appears to close all open files, with an option to exclude some specific file descriptors (that is, you need to pass a list of files that shouldn't be closed). That makes it a lot harder to do some initialization before daemonizing. I prefer to perform at least some initialization early in program startup to be able to give sensible error messages. I've had too many initscripts that claimed to have started a daemon sucessfully, only to have that daemon stop right away because it noticed a problem right after it detached itself. Ronald

At PyPI URL:http://pypi.python.org/pypi/python-daemon/, and development co-ordinated at Alioth URL:https://alioth.debian.org/projects/python-daemon/.

...

Interested parties are invited to join us on the discussion forums

The correct link for the ‘python-daemon-devel’ forum is URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-de.... For announcements only, we have URL:http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-daemon-an....

-- \ “This sentence contradicts itself — no actually it doesn't.” | `\ —Douglas Hofstadter | _o__) | Ben Finney

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

Le Thu, 25 Jul 2013 12:08:18 +1000, Ben Finney a écrit :

Guido van Rossum writes:

...

To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day.

Work continues on the PEP 3143-compatible ‘python-daemon’, porting it to Python 3 and aiming for inclusion in the standard library.

The PEP hasn't been formally accepted yet, however. Skimming back through the archives, one sticking point was the default value of the "umask" parameter. Setting the umask to 0 if the user didn't ask for something else is a disaster, security-wise. Another problem I've had when using it is that the `pidfile` combines the notion of pidfile and process lock in one unique attribute. This is quite inflexible when you're using something else than Skip Montanaro's "lockfile" library. I'm using a separate lock based on locket.py: https://github.com/mwilliamson/locket.py because it is based on POSIX advisory locks, and therefore doesn't suffer from the "stale pid file" issues you get when a process (or the whole system) crashes. Therefore I'd be -1 on the PEP until those issues are alleviated. Regards Antoine.

Antoine Pitrou writes:

Therefore I'd be -1 on [PEP 3143] until those issues are alleviated.

Cameron Simpson writes:

I have always found the convention that daemons have a umask of 0 to be utterly bogus, because almost all library code relies on the umask to set default security policy for initial file permissions.

Prone to rant on this at length if required...

Thanks folks. We'd love to have this discussion over at the ‘python-daemon-devel’ discussion forum if you want to have it in more detail. -- \ “We are all agreed that your theory is crazy. The question that | `\ divides us is whether it is crazy enough to have a chance of | _o__) being correct.” —Niels Bohr (to Wolfgang Pauli), 1958 | Ben Finney

Le Fri, 26 Jul 2013 14:08:35 +0200, Victor Stinner a écrit :

After having written 2 PEP on the topic, I slowly agree that make all file descriptors non-inheritable is the best *compromise*. It solves most, or all, issues.

Even stdin/stdout/stderr? I think inheriting them is the sane default.

The main drawback is the additionnal syscalls: on some platforms, 2 additional syscalls are need to make a file descriptor non-inheritable for each creation of file descriptor. According to my benchmark on the implementation of the PEP 433: the overhead of making a file descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0 µs) on Linux 3.6.

1% and 3% of what? You're telling us there's a 0.1µs overhead. It's positively tiny. Regards Antoine.

On Fri, 26 Jul 2013 22:17:47 +0200 Victor Stinner wrote:

2013/7/26 Antoine Pitrou :

...

...

The main drawback is the additionnal syscalls: on some platforms, 2 additional syscalls are need to make a file descriptor non-inheritable for each creation of file descriptor. According to my benchmark on the implementation of the PEP 433: the overhead of making a file descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0 µs) on Linux 3.6.

1% and 3% of what? You're telling us there's a 0.1µs overhead. It's positively tiny.

Copy-paste of the link:

""" On Linux, setting the close-on-flag has a low overhead on performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us - O_CLOEXEC: 1% slower (7.9 us) - ioctl(): 3% slower (8.0 us) - fcntl(): 3% slower (8.0 us) """

You aren't answering my question: slower than what? Benchmarking is useless if you aren't telling us what exactly you are benchmarking.

The overhead is between 0.1 and 0.2 µs (100 and 200 ns) according to my micro-benchmark.

That's what I figured out (see above). It's tiny. Antoine.

On Sat, 27 Jul 2013 00:18:40 +0200 Victor Stinner wrote:

2013/7/26 Antoine Pitrou :

...

On Fri, 26 Jul 2013 22:17:47 +0200

...

""" On Linux, setting the close-on-flag has a low overhead on performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us - O_CLOEXEC: 1% slower (7.9 us) - ioctl(): 3% slower (8.0 us) - fcntl(): 3% slower (8.0 us) """

You aren't answering my question: slower than what?

Ah, you didn't understand the labels. bench_cloexec.py runs a benchmark on os.open(path, os.O_RDONLY, cloexec=False) and os.open(path, os.O_RDONLY, cloexec=True) with different implementation of making the file descriptor non-inheritable.

close-on-flag not set: 7.8 us => C code: open(path, O_RDONLY)

O_CLOEXEC: 1% slower (7.9 us) => C code: open(path, O_RDONLY|CLOEXEC) => 1% slower than open(path, O_RDONLY)

ioctl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) => 3% slower than open(path, O_RDONLY)

fcntl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags | FD_CLOEXEC) => 3% slower than open(path, O_RDONLY)

Ok, so I think this it is a totally reasonable compromise. People who bother about a 3% slowdown when calling os.open() can optimize the hell out of their code using Cython for all I care :-) Regards Antoine.

On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith wrote:

On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou wrote:

...

On Sat, 27 Jul 2013 00:18:40 +0200 Victor Stinner wrote:

...

2013/7/26 Antoine Pitrou :

...

On Fri, 26 Jul 2013 22:17:47 +0200

...

""" On Linux, setting the close-on-flag has a low overhead on performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us - O_CLOEXEC: 1% slower (7.9 us) - ioctl(): 3% slower (8.0 us) - fcntl(): 3% slower (8.0 us) """

You aren't answering my question: slower than what?

Ah, you didn't understand the labels. bench_cloexec.py runs a benchmark on os.open(path, os.O_RDONLY, cloexec=False) and os.open(path, os.O_RDONLY, cloexec=True) with different implementation of making the file descriptor non-inheritable.

close-on-flag not set: 7.8 us => C code: open(path, O_RDONLY)

O_CLOEXEC: 1% slower (7.9 us) => C code: open(path, O_RDONLY|CLOEXEC) => 1% slower than open(path, O_RDONLY)

ioctl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) => 3% slower than open(path, O_RDONLY)

fcntl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags | FD_CLOEXEC) => 3% slower than open(path, O_RDONLY)

Ok, so I think this it is a totally reasonable compromise.

People who bother about a 3% slowdown when calling os.open() can optimize the hell out of their code using Cython for all I care :-)

+1 ;)

and +1 for making the sane default of noinherit / cloexec / whatever-others-call-it by default for all fds/handles ever opened by Python. It stops ignoring the issue (ie: the status quo of matching the default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

Do we even need a new PEP, or should we just do it? Or can we adapt Victor's PEP 446? -- --Guido van Rossum (python.org/~guido)

2013/7/27 Nick Coghlan :

...

Do we even need a new PEP, or should we just do it? Or can we adapt Victor's PEP 446?

Adapting the PEP sounds good - while I agree with switching to a sane default, I think the daemonisation thread suggests there may need to be a supporting API to help force FDs created by nominated logging handlers to be inherited.

Why would a Python logging handler be used in a child process? If the child process is a fresh Python process, it starts with the default logging handlers (no handler). Files opened by the logging module must be closed on exec(). Victor

2013/7/27 Guido van Rossum :

Do we even need a new PEP, or should we just do it? Or can we adapt Victor's PEP 446?

I can rewrite the PEP 446 to: * make all file descriptors and handles non-inheritable * remove the cloexec parameter * remove everything about non-blocking sockets (O_NONBLOCK), it should be discussed in a new PEP (it's no more related to O_CLOEXEC / HANDLE_INHERIT_FLAG) Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd, inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)? Or do you prefer a simple os.make_inheritable(fd) with no inheritable parameter? I prefer an explicit parameter, so it's also possible to force again non-inheritable, which also makes sense if the file descriptor was not created by Python. Victor

P.S. perhaps more important than a PEP rewrite is a working patch to see how realistic this is. Could you make the alpha 1 release? On Saturday, July 27, 2013, Guido van Rossum wrote:

On Saturday, July 27, 2013, Victor Stinner wrote:

...

2013/7/27 Guido van Rossum :

...

Do we even need a new PEP, or should we just do it? Or can we adapt Victor's PEP 446?

I can rewrite the PEP 446 to:

* make all file descriptors and handles non-inheritable * remove the cloexec parameter * remove everything about non-blocking sockets (O_NONBLOCK), it should be discussed in a new PEP (it's no more related to O_CLOEXEC / HANDLE_INHERIT_FLAG)

Sounds good.

...

Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd, inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?

Yes.

...

Or do you prefer a simple os.make_inheritable(fd) with no inheritable parameter? I prefer an explicit parameter, so it's also possible to force again non-inheritable, which also makes sense if the file descriptor was not created by Python.

Agreed.

...

Victor

-- --Guido van Rossum (on iPad)

-- --Guido van Rossum (on iPad)

On Fri, Jul 26, 2013 at 5:08 AM, Victor Stinner wrote:

After having written 2 PEP on the topic, I slowly agree that make all file descriptors non-inheritable is the best *compromise*. It solves most, or all, issues.

Right.

The main drawback is the additionnal syscalls: on some platforms, 2 additional syscalls are need to make a file descriptor non-inheritable for each creation of file descriptor. According to my benchmark on the implementation of the PEP 433: the overhead of making a file descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0 µs) on Linux 3.6. http://www.python.org/dev/peps/pep-0433/#performances

Remember that this is going to be Python 3.4 and newer. AFAICT setting O_CLOEXEC on open works on OSX (at least the man page on OSX 10.8 has it), on newer Linuxes, and the equivalent on Windows. So even if it does cost an extra syscall on older systems, those systems will be obsolete before Python 3.4 becomes mainstream there. And it does look like the syscalls are pretty cheap. I also don't think I'm not particularly worried about the cost of syscalls for making a socket (non)blocking -- although we should probably avoid the second fcntl() call if the first call shows the flag is already set the way we want it. -- --Guido van Rossum (python.org/~guido)