[Python-ideas] reducing multiprocessing.Queue contention

[Richard Oudkerk]

On 23/01/2013 11:16am, Charles-François Natali wrote:
> Connection.send() and Connection.recv() have to serialize the data
> using pickle before writing them to the underlying file descriptor.
> While the locking is necessary to guarantee atomic read/write (well,
> it's not necessary if you're writing to a pipe less than PIPE_BUF, and
> writes seem atomic on Windows), the locks don't have to be held while
> the data is serialized.

But you can only rely on the atomicity of writing less than PIPE_BUF 
bytes if you know that no other process is currently trying to send a 
message longer than PIPE_BUF.  Otherwise the short message could be 
embedded in the long message (even if the process sending the long 
message is holding the lock).

-- 
Richard