[External] pypy3 Multiprocessing Socket Issue
I'm using pypy3.7-v7.3.4-linux64 (Python v3.7.10) but have also tried pypy3.7-v7.3.5rc1-linux64 & pypy-c-jit-102427-77787b8f4c49-linux64. I'm seeing an issue that does NOT occur when using regular Python 3.7 (CPython v3.7.10). My application uses Python's multiprocessing library to take advantage of multiple cores on the CPU and as such I have shared resources that get utilized between these processes (sockets, queues, etc.), arbitrated through appropriate locking mechanisms to address resource contention. The issue seems to occur with respect to the sharing of sockets between these processes. That is, after 30 or more seconds I start to continually receive the following output, with NO stack trace to where it is occurring in my application... Traceback (most recent call last): File "/opt/pypy/lib-python/3/multiprocessing/resource_sharer.py", line 142, in _serve File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 453, in accept File "/opt/pypy/lib-python/3/multiprocessing/connection.py", line 599, in accept File "/opt/pypy/lib-python/3/socket.py", line 212, in accept OSError: [Errno 24] Too many open files Clearly, under pypy3, my application is exceeding the number of open files supported by the operating system (OS) and I can see this occur during the ~30 second lead up, where if I run "lsof -i" (in another shell), I can see the number of TCP/IP connections increasing steadily (associated with my application) up to the OS limit, which I can only assume is associated with, when one my processes accesses this shared socket resource. Again, this does NOT occur when running my application under regular Python (CPython) and I verify this using the same "lsof -i" method. Thoughts on how I might overcome this issue so I can see if using pypy3 over CPython 3 garners better performance for my specific application? Regards, Tom Moulton -- This communication, including any attachments, may contain Proprietary, Export Controlled, Legally Privileged or other Confidential Information. Any unauthorized review, printing, copying, transmission, dissemination, or use of the information is strictly prohibited. If you are not the intended recipient, please indicate to the sender that you have received this email in error, and delete the copy you received.
On 20/5/21 7:45 pm, Moulton, Tom via pypy-dev wrote:
Without seeing your code it is difficult to be specific. Typically this is caused by not closing allocated resources. CPython uses reference counting, and so it implicitly releases resources when they go out of scope. PyPy does not. Are you calling socket.close() or fid.close() when you are done with them? Also see this section of our documentation https://doc.pypy.org/en/latest/cpython_differences.html#differences-related-... Matti
On 20/5/21 7:45 pm, Moulton, Tom via pypy-dev wrote:
Without seeing your code it is difficult to be specific. Typically this is caused by not closing allocated resources. CPython uses reference counting, and so it implicitly releases resources when they go out of scope. PyPy does not. Are you calling socket.close() or fid.close() when you are done with them? Also see this section of our documentation https://doc.pypy.org/en/latest/cpython_differences.html#differences-related-... Matti
participants (2)
-
Matti Picus -
Moulton, Tom