[Twisted-Python] UDP File Copy
Hi I'm busy writing a small P2P client with Twisted. The primary use will be on a reliable local LAN segment. I would like to use UDP to copy files and the client must be able to accept multiple connections. I have not been able to find any example code via google, any suggestions ? Regs Marinus Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by visiting our website and viewing the webpage at the following address: http://www.deloitte.com/za/disclaimer. The Disclaimer is deemed to form part of the content of this email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre@Deloitte.co.za.
On Mon, 4 Apr 2005 13:17:05 +0200, "van Aswegen, Marinus \(ZA - Johannesburg\)" <mvanaswegen@deloitte.co.za> wrote:
Hi
I'm busy writing a small P2P client with Twisted. The primary use will be on a reliable local LAN segment. I would like to use UDP to copy files and the client must be able to accept multiple connections. I have not been able to find any example code via google, any suggestions ?
Regs
Use TCP instead. UDP might be an optimization to make later (but probably not: I can push 11MB/sec over my LAN with TCP, UDP is not going to do much better). Jp
On Monday 04 April 2005 22:53, Jp Calderone wrote:
Use TCP instead. UDP might be an optimization to make later (but probably not: I can push 11MB/sec over my LAN with TCP, UDP is not going to do much better).
In the context of the old 'fsp' transfer program, I think it was Robert Elz who made the comment: "Those who do not understand TCP are doomed to repeat it". There are times when a UDP-based protocol makes sense. File transfer almost certainly isn't one of them. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Apr 5, 2005 3:35 AM, Anthony Baxter <anthony@interlink.com.au> wrote:
On Monday 04 April 2005 22:53, Jp Calderone wrote:
Use TCP instead. UDP might be an optimization to make later (but probably not: I can push 11MB/sec over my LAN with TCP, UDP is not going to do much better).
In the context of the old 'fsp' transfer program, I think it was Robert Elz who made the comment: "Those who do not understand TCP are doomed to repeat it".
There are times when a UDP-based protocol makes sense. File transfer almost certainly isn't one of them.
the only time that UDP is really appliciable for file transfer is when you are doing Multicast AND Reliable Multicast with Foward Error Correction and other techniques. This is a rare case on the INTERNET since very few routers still support Multicast. Intra-net where you are broadcasting updates or patches to a cluster of machines it makes since, but for peer-to-peer it just doesn't reap any benefits on the internet proper. -- If you don't know what you want, you probably need a nap.
Intra-net where you are broadcasting updates or patches to a cluster of machines it makes since, but for peer-to-peer it just doesn't reap any benefits on the internet proper.
You are wrong, UDP is more and more adopted by end-user P2P software because of its ability to transparently go through - in and out - many types of NAT devices (which are themselves more and more popular among home users). Incoming TCP traffic, on the other hand, often needs to be specifically enabled on the NAT box. For juicy details, you can read the RFC for the STUN protocol (Simple Traversal of User Datagram Protocol - UDP - Through Network Address Translators - NATs): http://www.faqs.org/rfcs/rfc3489.html Regards Antoine.
I was not arguing its popularity, I was arguing its added complexity, which is negligable. None of my equipment that is doing NAT requires any special configuration to enable TCP. Maybe old equipement but I don't have anything less than a year old. I don't see the benefit outside of extremely low-latency game requirements of data that is constantly changing anyway. For REILABLE file transfer it gets complicated quickly which is like the othe person implied that if you are re-implementing all the TCP features in software over UDP why do it. I don't buy the NAT arguement as enough of a reason to justify the complexity. -- If you don't know what you want, you probably need a nap.
UDP has it's place, but typically vanilla file transfer isn't it. UDP traffic typically is harder to NAT rather than easier. Since it is a stateless protocol, what is meant by a 'reply'? How do you keep a forward and reverse translation table for the communications? The answers to these questions are to wrap these 'stateful' UDP sessions inside a psuedo-connection. In otherwords, simulate TCP behavior. The reality of today's internet is that NAT is prevalent, especially in home use. There simply isn't enough IP space in IPv4 to accomodate everybody, so we are left with dealing with the mess that RFC 1918 addressing, NAT and all the other work arounds have given us. STUN came out of trying to get VoIP working in a NATed environment. Since the latency demands of any VoIP application is fairly extreme and a single lost packet isn't detrimental to the data, UDP makes sense for that application. To get it to function in a NAT world however requires some intellegence in software to determine that NAT exists, how it functions and what to do about it. Enter STUN. While you might be able to squeeze some extra speed out of UDP for P2P apps, it isn't without complexity costs somewhere. You basically have the options of ignoring packet loss (VoIP/Skype) or implementing your own packet reliability mechanism (TFTP) within the application. Something like BitTorrent could use UDP, since it is sending bits of data from many sources to many sources. (It doesn't use UDP, it uses TCP.) UDP makes a lot of sense for one-to-many or many-to-many relationships for networking traffic. That's what multicast is all about, which uses UDP. Unfortunately, it's not internet routable so isn't a good general solution to P2P traffic. Tom On Apr 5, 2005 11:26 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Intra-net where you are broadcasting updates or patches to a cluster of machines it makes since, but for peer-to-peer it just doesn't reap any benefits on the internet proper.
You are wrong, UDP is more and more adopted by end-user P2P software because of its ability to transparently go through - in and out - many types of NAT devices (which are themselves more and more popular among home users). Incoming TCP traffic, on the other hand, often needs to be specifically enabled on the NAT box.
For juicy details, you can read the RFC for the STUN protocol (Simple Traversal of User Datagram Protocol - UDP - Through Network Address Translators - NATs): http://www.faqs.org/rfcs/rfc3489.html
Regards
Antoine.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Le mardi 05 avril 2005 à 15:31 -0400, Tom Carmichael a écrit :
UDP has it's place, but typically vanilla file transfer isn't it.
UDP traffic typically is harder to NAT rather than easier. Since it is a stateless protocol, what is meant by a 'reply'? How do you keep a forward and reverse translation table for the communications?
It is only harder for people who design NAT devices. ;) But the real difference here between UDP and TCP, is that when you open an outbound TCP connection, incoming TCP traffic will only be allowed *for that connection*. Whereas if you send an outbound UDP message, *all incoming UDP traffic* towards the originating address/port will be enabled (at least on some NAT boxes). This makes it easy for P2P systems to exchange messages between two NATted clients without any proxying/tunnelling mechanism: you just have to "punch a hole" by sending sporadic outgoing UDP traffic from your UDP socket and then you can receive all UDP traffic coming to that socket. You can't do that with TCP. Regards Antoine.
On Wed, 2005-04-06 at 11:18 +0200, Antoine Pitrou wrote:
But the real difference here between UDP and TCP, is that when you open an outbound TCP connection, incoming TCP traffic will only be allowed *for that connection*. Whereas if you send an outbound UDP message, *all incoming UDP traffic* towards the originating address/port will be enabled (at least on some NAT boxes).
This makes it easy for P2P systems to exchange messages between two NATted clients without any proxying/tunnelling mechanism: you just have to "punch a hole" by sending sporadic outgoing UDP traffic from your UDP socket and then you can receive all UDP traffic coming to that socket. You can't do that with TCP.
You can't do that with NATted UDP, either. Quite many firewalls want to match all of the (srcIP, srcPort, dstIP, dstPort) to match.
Hi again,
You can't do that with NATted UDP, either. Quite many firewalls want to match all of the (srcIP, srcPort, dstIP, dstPort) to match.
Well, instead of claiming I'm an expert (!), I will hereby copy an e-mail that's been posted a few months ago on the p2p-hackers mailing-list by one of the Limewire developers. It talks about the proportion of NAT devices which are compatible with the aforementioned hack ;) Regards Antoine. From:p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildsonent: Wednesday, December 01, 2004 10:51 AM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? David, Our operating environment has not yet stabilized - early beta testers didn't get the proper port stability tests and our entire user base has not been upgraded. However, early indications are that 50 to 60 percent of firewalled users will benefit. Again though, the harshest firewalled users probably have not had a good experience with P2P software so we are working with a self selected crew. This method does work with the Windows XP firewall so we expect it to be widely successful as XP becomes further adopted. One improvement that we can make is to our existing scheme is to handle firewalls/NATs that use a sequential port assignment algorithm for each attempt. Rather than just trying a fixed port, we could try to negotiate a connection with that port +1, +2 and +3. I'm not sure what percentage of users would really benefit from this though. As has been mentioned here recently, we certainly have noticed that incoming TCP connections are often possible after the same type of UDP pinging. We have no numbers on that though. Thanks -greg
Tommi Virtanen wrote:
On Wed, 2005-04-06 at 11:18 +0200, Antoine Pitrou wrote:
But the real difference here between UDP and TCP, is that when you open an outbound TCP connection, incoming TCP traffic will only be allowed *for that connection*. Whereas if you send an outbound UDP message, *all incoming UDP traffic* towards the originating address/port will be enabled (at least on some NAT boxes).
This makes it easy for P2P systems to exchange messages between two NATted clients without any proxying/tunnelling mechanism: you just have to "punch a hole" by sending sporadic outgoing UDP traffic from your UDP socket and then you can receive all UDP traffic coming to that socket. You can't do that with TCP.
You can't do that with NATted UDP, either. Quite many firewalls want to match all of the (srcIP, srcPort, dstIP, dstPort) to match.
Exactly. It depends entirely on what you do with NAT. Some NAT boxes are stricter than others - but in principle I block anything weird or undesirable - most notably protocol hacks of this nature designed to circumvent firewalling. Eugene -- -- =============================================== Reedflute Software Solutions Telephone -> +27 18 293 3236 General information -> info@reedflute.com Project information -> projects@reedflute.com Web -> www.reedflute.com ===============================================
On Wednesday 06 April 2005 01:26, Antoine Pitrou wrote:
You are wrong, UDP is more and more adopted by end-user P2P software because of its ability to transparently go through - in and out - many types of NAT devices (which are themselves more and more popular among home users). Incoming TCP traffic, on the other hand, often needs to be specifically enabled on the NAT box.
Bwahahahahahahaha. No, really - thanks for this. I haven't laughed so hard all day.
For juicy details, you can read the RFC for the STUN protocol (Simple Traversal of User Datagram Protocol - UDP - Through Network Address Translators - NATs): http://www.faqs.org/rfcs/rfc3489.html
STUN helps, but doesn't alter the fact that UDP + NAT is an absolute A-grade disaster. If you think that UDP will somehow make your software more reliable in the face of NATs, you're in for a rude rude surprise. In case you're wondering, I know what I'm talking about here - I've implemented STUN as part of shtoom, and it really only helps a small amount. NAT + UDP == a world of hurt. The only robust way to traverse a NAT with UDP is to use UPnP. And yes, I still regard UPnP as one of the worst protocols ever invented. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
See also http://airhook.org/ and http://sourceforge.net/projects/khashmir -- малюк [ m@maluke.com // ICQ: 39027534 ]
It also bears mentioning that Glyph's nascent Q2Q library does precisely what has been described in this thread: implement TCP-like transmission controls on top of UDP to allow you to punch a hole through NAT. Except Q2Q also does SSL, and is tied to JUICE (by far the easiest protocol API I've ever seen in my life), so it's much cooler than just spewing UDP packets :) -- Alex Levy WWW: http://mesozoic.geecs.org/ "Never let your sense of morals prevent you from doing what is right." -- Salvor Hardin, Isaac Asimov's _Foundation_
Alex Levy wrote:
It also bears mentioning that Glyph's nascent Q2Q library does precisely what has been described in this thread: implement TCP-like transmission controls on top of UDP to allow you to punch a hole through NAT. Except Q2Q also does SSL, and is tied to JUICE (by far the easiest protocol API I've ever seen in my life), so it's much cooler than just spewing UDP packets :)
Where can we find this Q2Q library and JUICE? google does not has an aswer.
For now, it's at svn://divmod.org/svn/Quotient/branches/glyph/pycon-q2q/ -- Alex Levy WWW: http://mesozoic.geecs.org/ "Never let your sense of morals prevent you from doing what is right." -- Salvor Hardin, Isaac Asimov's _Foundation_
participants (11)
-
Alex Levy -
Anthony Baxter -
Antoine Pitrou -
Eugene Coetzee -
jarrod roberson -
Jp Calderone -
maluke -
Patrick Lauber -
Tom Carmichael -
Tommi Virtanen -
van Aswegen, Marinus (ZA - Johannesburg)