[Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

[Victor Stinner]

2013/7/5 Cameron Simpson <cs at zip.com.au>:
> | Both set O_NONBLOCK flag (UNIX)
>
> Oh, how embarassing.

You said that the PEP is not cristal clear. Do you have a suggestion
to make it more clear?

Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and
HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set
means inheritable, whereas O_CLOEXEC set means *not* inheritable)

> | > This is deducable from your PEP, but I was at first confused, and
> | > initially expected get_blocking/set_blocking functions in New
> | > Functions.
> |
> | socket objects already have get_blocking() and set_blocking() methods.
>
> Ah, ok then. As far as it goes. Shouldn't there be a general purpose
> os.get_blocking() and os.set_blocking() functions that operate on
> any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()?

I didn't propose to add these two functions, because I'm not sure that
they are really useful.

We can add os.get_blocking() and os.set_blocking() on UNIX, but these
functions would not be available on Windows. On Windows,
os.set_blocking() only makes sense for sockets, and sockets already
have a set_blocking() method. On UNIX, it makes sense because
set_blocking(fd, True) can be implemented as a single syscall on some
platforms (using ioctl), whereas most developers implement it using
two syscalls (fcntl to get current flags, and fcntl to set the
O_NONBLOCK flag) maybe because it is more portable.

But we cannot add a new "blocking" parameter to all functions creating
file descriptors, like open(), because of Windows. Or do you like the
"raise NotImplementedError on Windows" option?

> But WHY? I think socket file decriptors should be treated little
> differently from other file descriptors.

Because of Windows. On Windows, sockets and files are two different
things. sockets have no "file descriptor", they have a HANDLE. For
example, sockobj.fileno() on Windows returns a huge number, not
something like 3 or 10.

The C type HANDLE is larger than a file descriptor on Windows 64-bit
(sizeof(void*) > sizeof(int)) and so functions must be modified to use
a wider type (to parse their argument), and must support the HANDLE
type (_open_osfhandle() can be used to have a single version of the
code).

> Otherwise, I think it should be separated out into a separate
> proposal if you're proposing it just for sockets; make this proposal
> just close-on-exec and forget the blocking stuff for this specific
> PEP.

The reason for addressing close-on-exec and blocking parameters in the
same PEP is that new versions of operating systems support setting the
two flags at the creation a new file descriptor and a new socket. Read
the article linked at the end of the PEP for the background:
http://udrepper.livejournal.com/20407.html

The Python call socket.socket(..., cloexec=True, blocking=False) only
calls one syscall on Linux >= 2.6.27.

> | > I would expect Popen and friends to need to both clear the flag to
> | > get the descriptors across the fork() call, and _possibly_ to set
> | > the flag again after the fork. Naively, I would expect the the flag
> | > to be as it was before the Popen call, after the call.
> |
> | As explained in the "Inheritance of file descriptors" section, the
> | close-on-exec flag has no effect on fork:
> |
> | "The flag has no effect on fork(), all file descriptors are inherited
> | by the child process."
>
> Agreed, see my second post where I explained I'd misthought it.
> However, as stated there, I think the side effects should be fairly
> overtly stated in the docuemntation.

Sorry, I don't understand. Which "side effect"?

The close-on-exec flag only affects inheritance of file descriptors at
the execv() syscall, not at fork(). And execv() can be called without
fork().

Popen must clear close-on-exec flag on files from its pass_fds
parameter for convinience. It would be surprising to not inherit a
file descriptor passed to Popen in pass_fds, don't you think so?

os.execv() has no pass_fds parameter, and it is a thin wrapper on the
syscall. Popen is a high-level API, that's why it prepares more things
before calling the new program.

> | The close-on-exec flag is cleared after the fork and before the
> | exec(). Pseudo-code for Popen:
> |
> | pid = os.fork()
> | if pid:
> |   return pid
> | # child process
> | for fd in pass_fds: os.set_cloexec(fd, False)
> | os.execv(...)
>
> Fine. No state restore is fine with me. I think it should be as
> clear as possible in the doco.

I don't understand. I already wrote "The flag has no effect on fork(),
all file descriptors are inherited by the child process" in the PEP.
It is not enough? Do you have a suggestion to explain it better?

Victor