[ python-Bugs-756104 ] Calling socket.recv() with a large number breaks

[SourceForge.net]

Bugs item #756104, was opened at 2003-06-17 14:25
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756104&group_id=5470

Category: Python Library
Group: Python 2.2.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Alex R.M. Turner (plexq)
Assigned to: Nobody/Anonymous (nobody)
Summary: Calling socket.recv() with a large number breaks

Initial Comment:
I have a backup script that calls socket.recv() passing
the amount of data that is left to get for a given file
as the argument.  For very large files (I have one here
that is 1.9Gig) it seems to break horribly after
receiving about 130Meg.  I have tried to track it down
precisely with basic debugging (putting in print
statements in the python code) but it just seams to
freak out around the 130meg mark, and it just doesn't
make sense.  If I change the argument passed to recv to
32767 it works just fine.  I have attatched the loop
that reads data from the socket (In a working state,
the bad code is commented out).

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2004-07-26 20:41

Message:
Logged In: YES 
user_id=31435

potterru, I don't believe plexq was using _fileobject.read() -- 
he said he was using socket.recv(), and all the comments 
later were consistent with that.

The code you found does appear to *intend* max():  code 
following the snippet you quoted clearly expects that it may 
have read more than "left" bytes, and it would not be 
worrying about that had min() been intended.  I agree the 
code is pretty inscrutable regardless, but we'd have to ask 
Guido why he wrote it that way.

In any case, since this bug report is about socket.recv(), if 
you want to make a case that _fileobject.read() is buggy you 
should open a new report for that.

----------------------------------------------------------------------

Comment By: Igor E. Poteryaev (potterru)
Date: 2004-07-26 07:23

Message:
Logged In: YES 
user_id=48596

It looks like bug in module socket.py in _fileobject.read method. 
... 
while True: 
    left = size - buf_len 
    recv_size = max(self._rbufsize, left) 
    data = self._sock.recv(recv_size) 
 
This code should read not more than *left* or *buffer size* 
 
So, it should be min instead of max ! 
 
Should I file a bug/patch for this ? 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-06-18 14:51

Message:
Logged In: YES 
user_id=31435

I assume you're running on Linux.  Python's implementation of 
recv asks the platform malloc for a buffer of the size 
requested, and raises an exception if malloc returns NULL.  
Unfortunately, malloc on Linux has a nasty habit of returning 
non-NULL even if there's no chance you can actually use the 
amount of memory requested.  There's really nothing Python 
can do about that.

So back to Jeremy's comment:  try the same thing in C.  If 
you get ridiculous behavior there too, it's a platform C/OS 
bug, and Python won't be able to hide it from you.

----------------------------------------------------------------------

Comment By: Alex R.M. Turner (plexq)
Date: 2003-06-18 14:22

Message:
Logged In: YES 
user_id=60034

That as maybe - it might be worth putting a suggested
maximum in the docs.  However I would say that given that an
IPv6 packet could be as large as 2Gig, it's not unreasonable
to ask for a packet as large as 1 Gig.  Wether the problem
is in glibc or python I don't know, although it seems that
asking for a buffer of 1.3 Gig in size, and passing that to
recv() would be odd behaviour on a current system in C given
that most systems couldn't allocate that much memory to a
buffer ;).  I have written fairly extensive socket code in
C/C++ before, and I never used anything larger than 65536
for the obvious reason that you can't receive anything
bigger than that in IPv4 (and most NICs can't handle
anything that big either).  I figured it would be
interesting to see what happened :).  I have a penchant for
being the only person in history to do quite a few things
apparently!

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-06-18 12:29

Message:
Logged In: YES 
user_id=31435

BTW, if you're new to socket programming, do read Python's 
socket programming HOWTO:

http://www.amk.ca/python/howto/sockets/

I expect you're the only person in history to try passing such 
a large value to recv() -- even if it worked, you'd almost 
certainly run out of memory trying to allocate buffer space for 
1.9GB.  sockets are a low-level facility, and it's common to 
pass a relatively small power of 2 (for best match with 
hardware and network realities).

----------------------------------------------------------------------

Comment By: Jeremy Hylton (jhylton)
Date: 2003-06-18 12:02

Message:
Logged In: YES 
user_id=31392

What happens when you write a C program that does the same
thing?  I expect you'll see similar problems.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756104&group_id=5470