[pypy-dev] Performance issues of socket.recv_into()

[hubo]

Unfortunately, "to receive in the middle of a buffer where there is already some data in 
the beginning of the buffer and you want all data concatenated", that is exactly what I really want... This test script is just a simplified version. So I am really sad to find out that recv_into() is not very efficient.

What I am really insterested in is that it seems it works exactly as two lines of Python code which first do a recv() and then copy the data to the buffer, which means maybe the copy is done with GIL locked. I think that is the source of the huge performance difference. In CPython, recv_into() executes without GIL locked, so multiple threads can use multiple CPUs which improves the performance a lot.

2016-08-09 

hubo 



发件人：Armin Rigo <arigo at tunes.org>
发送时间：2016-08-09 17:30
主题：Re: [pypy-dev] Performance issues of socket.recv_into()
收件人："hubo"<hubo at jiedaibao.com>
抄送："PyPy Developer Mailing List"<pypy-dev at python.org>

Hi, 

On 8 August 2016 at 16:09, hubo <hubo at jiedaibao.com> wrote: 
> P.S. it is interesting that though I thought recv_into() should be more 
> efficient thant recv() since it reduces extra object creation / destruction, 
> the test result shows that recv() outperforms recv_into(), even with 
> CPython. With CPython, it seems server with recv() costs less CPU time than 
> recv_into(), but having the same I/O performance. 

As you found out, recv_into() is not better than recv() in your use 
case.  (There are other use cases where it can be useful, e.g. to 
receive in the middle of a buffer where there is already some data in 
the beginning of the buffer and you want all data concatenated.) 

Indeed, the current PyPy implementation of recv_into() is definitely 
bad.  It just does a regular recv(), and then it manually copies the 
data into the buffer!  So it always done one more copy of the data 
than recv(). 

I just fixed it in e53ea5c9c384.  Now in PyPy (like in CPython), both 
recv() and recv_into() don't copy the received data at all.  The 
difference between the two in this case is completely lost in the 
noise---it's the difference between doing a malloc or not, which costs 
nothing compared to transferring 2MB of data from the kernel to 
userspace. 


A bientôt, 

Armin. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160809/b84a4d86/attachment.html>