[IPython-dev] ZMQ Segfault - IPython cluster
Dave Hirschfeld
dave.hirschfeld at gmail.com
Thu Aug 21 07:21:36 EDT 2014
I'm running an IPython cluster on Windows HPC x64, Python 2.7.
When I start over ~80-90 engines I notice that the the controller segfaults
with the only error message printed in the terminal being:
Assertion failed: fds.size () <= FD_SETSIZE
(bundled\zeromq\src\select.cpp:68)
Looking on the net there's a fair few posts about this. My take on this is
that it's a bug in libzmq - it should never just kill the host process which
is exactly what the assert does (at least on windows).
The below script should segfault python when N > 1023.
```
import zmq, sys
N = 1024
print 'ZMQ v',zmq.zmq_version(),' N =',N
ctx = zmq.Context()
try:
for _ in range(N):
sock = ctx.socket(zmq.PAIR)
sock.close()
finally:
del ctx
```
The problem arises with the number of connections required by the IPython
cluster which it seems can easily exceed this number.
I hadn't noticed this problem previously but that could well be because I
wasn't using so many engines. Has anyone else come across this problem
before?
With 90 engines running the Task Manager reports the controller process as
using 1353 handles and netstat says it has 1132 open sockets:
In [44]: controller_pid = '3744'
In [45]: socks = !netstat -a -o -n
In [46]: len(socks)
Out[46]: 2861
In [47]: len(socks.grep(controller_pid, field=-1))
Out[47]: 1132
Is this expected? If so, the default (compile time) value for FD_SETSIZE
should really be increased IMHO - especially in light of the severity
(segfault) of the problem.
I've recompiled pyzmq/libzmq manually with FD_SETSIZE=8192 and haven't had
any further problems.
Regards,
Dave
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'cp1252',
'ipython_path':
'C:\\Python\\envs\\quantdev\\1.4.0.post673.g1d57b50\\lib\\site-
packages\\IPython',
'ipython_version': '2.1.0',
'os_name': 'nt',
'platform': 'Windows-2008ServerR2-6.1.7601-SP1',
'sys_executable':
'C:\\Python\\envs\\quantdev\\1.4.0.post673.g1d57b50\\python.exe',
'sys_platform': 'win32',
'sys_version': '2.7.8 |Continuum Analytics, Inc.| (default, Jul 2 2014,
15:12:11) [MSC v.1500 64 bit (AMD64)]'}
More information about the IPython-dev
mailing list