<div dir="ltr">FWIW, looking into Python's Lib/socket.py module, it would seem that the dup() method of a _sockobject() just transfers the underlying fd, not performing a new socket create. So a caller could know about that behavior and work to call setblocking(0) only when it has a socket not seen before.<div>
<br></div><div style>In eventlet's case, it would appear the code could be taught this fact, and adjusted accordingly. I'll propose a patch for eventlet that will eliminate most of these calls in their case.</div>
<div style><br></div><div style>Thanks for your time,</div><div style><br></div><div style>-peter</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Jan 20, 2013 at 1:31 AM, Peter Portante <span dir="ltr"><<a href="mailto:peter.a.portante@gmail.com" target="_blank">peter.a.portante@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I don't have a concrete case where a socket object's setblocking() method is called with a value in one module, handed off to another module (which does not know what the first did with it) which in turn also calls setblocking() with the same value. It certainly seems that that not is a common pattern, but perhaps one could argue a valid pattern, since the state of blocking/nonblocking is maintained in the kernel behind the fcntl() system calls.<div>
<div><br></div><div><div>Here is what I am seeing concretely.</div><div><br></div><div>This is the syscall pattern from eventlet/wsgi.py + eventlet/greenio.py (after removing redundants call to set_nonblocking (see <a href="https://bitbucket.org/portante/eventlet/commits/cc27508f4bbaaea566aecb51cf6c8b4629b083bd)" target="_blank">https://bitbucket.org/portante/eventlet/commits/cc27508f4bbaaea566aecb51cf6c8b4629b083bd)</a>). First, these are the call stacks for the three calls to the set_nonblocking() method, made in one HTTP request; the greenio.py:set_nonblocking() method wraps the socketmodule.c:setblocking() method:</div>
<div><br></div><div><font face="courier new, monospace">pid 1385</font></div><div><font face="courier new, monospace"> File "/usr/bin/swift-object-server", line 22, in <module></font></div><div><font face="courier new, monospace"> run_wsgi(conf_file, 'object-server', default_port=6000, **options)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 194, in run_wsgi</font></div><div><font face="courier new, monospace"> run_server(max_clients)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 158, in run_server</font></div><div><font face="courier new, monospace"> wsgi.server(sock, app, NullLogger(), custom_pool=pool)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 598, in server</font></div><div><font face="courier new, monospace"> <b>client_socket = sock.accept()</b></font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 163, in accept</font></div><div><font face="courier new, monospace"> return type(self)(client), addr</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 133, in __init__</font></div><div><font face="courier new, monospace"> <b>set_nonblocking(fd)</b></font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">pid 1385</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl</font></div>
<div><font face="courier new, monospace"> func(*args, **kwargs)</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 516, in process_request</font></div>
<div><font face="courier new, monospace"> proto = self.protocol(socket, address, self)</font></div><div><font face="courier new, monospace"> File "/usr/lib64/python2.6/SocketServer.py", line 616, in __init__</font></div>
<div><font face="courier new, monospace"> self.setup()</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 174, in setup</font></div><div><font face="courier new, monospace"> <b>self.rfile = conn.makefile('rb', self.rbufsize)</b></font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 219, in makefile</font></div><div><font face="courier new, monospace"> return _fileobject(self.dup(), *args, **kw)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 214, in dup</font></div><div><font face="courier new, monospace"> newsock = type(self)(sock)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 133, in __init__</font></div><div><font face="courier new, monospace"> <b>set_nonblocking(fd)</b></font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">pid 1385</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl</font></div>
<div><font face="courier new, monospace"> func(*args, **kwargs)</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 516, in process_request</font></div>
<div><font face="courier new, monospace"> proto = self.protocol(socket, address, self)</font></div><div><font face="courier new, monospace"> File "/usr/lib64/python2.6/SocketServer.py", line 616, in __init__</font></div>
<div><font face="courier new, monospace"> self.setup()</font></div><div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 175, in setup</font></div><div><font face="courier new, monospace"> <b>self.wfile = conn.makefile('wb', self.wbufsize)</b></font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 219, in makefile</font></div><div><font face="courier new, monospace"> return _fileobject(self.dup(), *args, **kw)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 214, in dup</font></div><div><font face="courier new, monospace"> newsock = type(self)(sock)</font></div>
<div><font face="courier new, monospace"> File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 133, in __init__</font></div><div><font face="courier new, monospace"> <b>set_nonblocking(fd)</b></font></div>
<div><br></div><div>The first one above is expected, the next two unexpectedly result in fcntl() calls on the same fd. The strace looks like:</div><div><br></div><div><div><font face="courier new, monospace">accept(8, {sa_family=AF_INET, sin_port=htons(54375), sin_addr=inet_addr("127.0.0.1")}, [16]) = 14</font></div>
<div><font face="courier new, monospace">fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)</font></div><div><font face="courier new, monospace">fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0</font></div><div><font face="courier new, monospace"># </font><b style="font-family:'courier new',monospace">self.rfile = conn.makefile('rb', self.rbufsize)</b></div>
<div><font face="courier new, monospace">fcntl(14, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)</font></div><div><font face="courier new, monospace">fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0</font></div>
<div><font face="courier new, monospace"># </font><b style="font-family:'courier new',monospace">self.wfile = conn.makefile('wb', self.wbufsize)</b></div><div><font face="courier new, monospace">fcntl(14, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)</font></div>
<div><font face="courier new, monospace">fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0</font></div><div><font face="courier new, monospace">recvfrom(14, "GET /sdb1/234456/AUTH_del0/gprfc"..., 8192, 0, NULL, NULL) = 353</font></div>
<div><font face="courier new, monospace">getsockname(14, {sa_family=AF_INET, sin_port=htons(6010), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0</font></div><div>...</div><div><br></div><div>It appears that conn.makefile() is attempting to dup() the fd, but rfile and wfile end up with objects that share the same fd contained in conn.</div>
<div><br></div><div>For eventlet/wsgi.py based webservers, OpenStack Swift is the one I am working with right now, handles millions of requests a day on our customer systems. Seems like these suggested code changes are trivial compared to the number of system calls that can be saved.<br>
<div><br></div><div>Thanks for indulging on this topic,</div><div><br></div><div>-peter</div></div></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Jan 20, 2013 at 12:38 AM, Guido van Rossum <span dir="ltr"><<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On Sat, Jan 19, 2013 at 8:49 PM, Peter Portante<br>
<<a href="mailto:peter.a.portante@gmail.com" target="_blank">peter.a.portante@gmail.com</a>> wrote:<br>
> I noticed while stracing a process that sock.setblocking() calls always<br>
> result in pairs of fcntl() calls on Linux. Checking 2.6.8, 2.7.3, and 3.3.0<br>
> Modules/socketmodule.c, the code seems to use the following (unless I have<br>
> missed something):<br>
><br>
> delay_flag = fcntl(s->sock_fd, F_GETFL, 0);<br>
> if (block)<br>
> delay_flag &= (~O_NONBLOCK);<br>
> else<br>
> delay_flag |= O_NONBLOCK;<br>
> fcntl(s->sock_fd, F_SETFL, delay_flag);<br>
><br>
> Perhaps a check to see the flags changed might be worth making?<br>
><br>
> int orig_delay_flag = fcntl(s->sock_fd, F_GETFL, 0);<br>
> if (block)<br>
> delay_flag = orig_delay_flag & (~O_NONBLOCK);<br>
> else<br>
> delay_flag = orig_delay_flag | O_NONBLOCK;<br>
> if (delay_flag != orig_delay_flag)<br>
> fcntl(s->sock_fd, F_SETFL, delay_flag);<br>
><br>
> OpenStack Swift using the Eventlet module, which sets the accepted socket<br>
> non-blocking, resulting in twice the number of fcntl() calls. Not a killer<br>
> on performance, but it seems simple enough to save a system call here.<br>
<br>
</div></div>This would seem to be a simple enough fix, but it seems you are only<br>
fixing it if a *redundant* call to setblocking() is made (i.e. one<br>
that attempts to set the flag to the value it already has). Why would<br>
this be a common pattern? Even if it was, is the cost of one extra<br>
fcntl() call really worth making the code more complex?<br>
<span><font color="#888888"><br>
--<br>
--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)<br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>