sending very large packets over the network

[Azazello]

On Aug 1, 8:50 pm, Gary Herron <gher... at islandtraining.com> wrote:
> Walker Lindley wrote:
> > OK, I'm back with another networking question. I'm trying to seend
> > large amounts of information over TCP (the length of data being given
> > to send() is on the order of 16000 characters in length).
> > Unfortunately on the receiving end, the packets appear to be
> > truncated. So I wrote some code that continuously tries to send bigger
> > and bigger packets until it fails and noticed that it never fails at
> > the same length. I'm not even sure these two things are related, but
> > is there some undocumented (or documented and I missed it) maximum
> > size for data you can pass to send()?
>
> For ethernet connections the size is often about 1500.  But the size
> depends on the underlying protocol, and even if you know the underlying
> protocol along the full route, you can't rely on packets that are
> received being the same as those that are sent.
>
> TCP/IP is a *stream* connection.  What you are guaranteed is this:  All
> the bytes that are sent from one end will be received eventually on the
> other end, in the proper order.  (Or failing that, you will receive an
> error notification.)  No guarantee is made about the packet sizes on
> either end, and you can't even rely on the packets being the same in
> number or length on the two ends.
>
> Your send code can try sending a packet of any size, but it must be
> prepared to examine the number of bytes actually sent, and retry with
> the remainder in a loop until all bytes are sent.    Similarly your
> receiving code must loop around the receive accepting whatever sized
> packets makes it through the connection.
>
> In many TCP/IP connections, it seems that the packets received are
> one-for-one with the packets sent, but relying on this *IS AN ERROR*
> that will bite you.  It fails to across the internet (sometimes) and
> when (at least some) wireless cards are involved.
>
> You may be better off using a package that knows all this and handles it
> properly.  Modules asyncore and asynchat are one possibility.
>
> Gary Herron
>
>
>
>
>
> > the sample code is as follows
> > #server
> > import socket
>
> > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> > s.bind(("", 2000))
> > s.listen(0)
> > sock, addrinfo = s.accept()
> > for i in range(2 ** 16):
> >  length = int(sock.recv(16))
> >  print "excpecting data of length:", length
> >  data = sock.recv(length)
> >  print "received data of length:", len(data)
> >  print
> > s.close()
> > sock.close()
>
> > #client
> > import socket
> > import time
>
> > s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> > s.bind(("", 2001))
> > s.connect(("localhost", 2000))
> > for i in range(2 ** 16):
> >  packet = "h" * i
> >  s.send("%16d" % len(packet))
> >  print "attempting to send data of length:", len(packet)
> >  tmp = s.send(packet)
> >  print "actually sent data of length:", tmp
> >  print
> >  time.sleep(.001)
> > s.close()
>
> > i just put them in different files and ran them from the command line.
> > Any help or suggestions would be greatly appreciated. Thanks.
>
> > -Walker
>
> > --
> > This e-mail is licensed under the Creative Commons
> > Attribution-NoDerivs 2.5 License. To view a copy of this license,
> > visithttp://creativecommons.org/licenses/by-nd/2.5/or send a letter
> > to Creative Commons, 543 Howard Street, 5th Floor, San Francisco,
> > California, 94105, USA.- Hide quoted text -
>
> - Show quoted text -

I'm not an expert on networking so take my advice with a grain of
salt!

My guess is that you're encountering race-like conditions between the
send and recv. commmands because you're running your communication
command 2^16 times, hoping that send and recv. are synchronous on
every step.  This is especially true when you're running other
commands in your data sending loops! Prints take quite a lot of time.
Also, the send command in C (i'm not sure if this is true in python)
does not guarantee that all of your data gets out.

I'm assuming that you're somewhat new to networking so I would
recommend shying away from running an Asynchronous polling server.
Although it is probably a better way to do this.

My suggestions:

Write a while loop in the server with some end of stream checking,
timeouts, et cetera. This will give you a bit of flexibility and
feedback when things go to pot.  Use the sendall() command  in your
client because it throws an exception if data isn't properly sent. and
will eliminate a unsightly for loop.

Client

socket.sendall(data)

Something akin to this old piece of test code

datastream = ''

## A timeout exception is thrown if the receiver is waiting on the
line for .25 seconds
socket.settimeout(.25)

    while 1:
	try:
            buf = socket.recv(1024) ## If data on line put in buffer
        except socket.timeout:      ## Data buffer timeout,  breaks
while loop
            break
	if buf == '': ## If an empty buffer, jump to top of loop (beginning
of file check)
	    continue

        datastream = datastream + str(buf)

        if not len(buf) == 1024:  ## If the buffer is not full break
signaling end of file.
            break

Hope this helps!  And like i said i'm pretty new to this so be
careful!  Foundations of Network Programming by John Goerzen is a
great book if you want some good information on the topic.

Tyler