[pypy-dev] [External] pypy3 Multiprocessing Socket Issue

[Moulton, Tom]

I'm using pypy3.7-v7.3.4-linux64 (Python v3.7.10) but have also
tried pypy3.7-v7.3.5rc1-linux64 & pypy-c-jit-102427-77787b8f4c49-linux64.
I'm seeing an issue that does NOT occur when using regular Python 3.7
(CPython v3.7.10).

My application uses Python's multiprocessing library to take advantage of
multiple cores on the CPU and as such I have shared resources that get
utilized between these processes (sockets, queues, etc.), arbitrated
through appropriate locking mechanisms to address resource contention.  The
issue seems to occur with respect to the sharing of sockets between these
processes.  That is, after 30 or more seconds I start to
continually receive the following output, with NO stack trace to where it
is occurring in my application...

Traceback (most recent call last):
  File "/opt/pypy/lib-python/3/multiprocessing/resource_sharer.py", line
142, in _serve
  File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 453, in
accept
  File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 599, in
accept
  File "/opt/pypy/lib-python/3/socket.py", line 212, in accept
OSError: [Errno 24] Too many open files

Clearly, under pypy3, my application is exceeding the number of open files
supported by the operating system (OS) and I can see this occur during the
~30 second lead up, where if I run "lsof -i" (in another shell), I can see
the number of TCP/IP connections increasing steadily (associated with my
application) up to the OS limit, which I can only assume is associated
with, when one my processes accesses this shared socket resource.  Again,
this does NOT occur when running my application under regular Python
(CPython) and I verify this using the same "lsof -i" method.

Thoughts on how I might overcome this issue so I can see if using pypy3
over CPython 3 garners better performance for my specific application?

Regards,

Tom Moulton

-- 
This communication, including any attachments, may contain Proprietary, 
Export Controlled, Legally Privileged or other Confidential Information. 
Any unauthorized review, printing, copying, transmission, dissemination, or 
use of the information is strictly prohibited. If you are not the intended 
recipient, please indicate to the sender that you have received this email 
in error, and delete the copy you received.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/pypy-dev/attachments/20210520/1099f7b5/attachment.html>