On 09:23 am, p.mayers@imperial.ac.uk wrote:

...

I'm thinking basically all p2p applications will need this sort of "reliability" over UDP for e.g. file transfer, so I'm hoping there is something already written.

http://divmod.org/trac/wiki/DivmodVertex

Haven't used it, so can't comment further onit.

As it happens, I have, a little, so I can :). To make a long story short, if you need the features that TCP provides, but you want the simplicity (and widespread de-jure "standard" implementation in routers) of hole-punching via UDP rather than complex dual-connection TCP startup, the easiest thing you can do is just go ahead and implement TCP on top of UDP. We did this for vertex, in the "ptcp" module (pseudo-TCP) which is a very literal-minded implementation of the TCP specification on top of UDP. It's a fully-functional proof of concept at this point, but it needs a lot of maintenance. The performance is terrible because we didn't implement window sizing yet (the window has a static size), there are a few edge-cases on connection tear-down, and the tests aren't quite granular enough (for example, rather than testing specific latency cases, they just probabilistically test packet loss). It also used a predecessor to the now-standard-in-twisted "AMP" protocol for handshaking, and should be upgraded to use the new one. Plus, my own personal bugaboo: it doesn't integrate with cred properly. However, if you need connection-oriented peer-to-peer traffic, contributing to Vertex is going to get you where you need to be faster than any ad-hoc solution. We *have* already implemented the full TCP state machine and connection-management logic in Python. I've personally tested that it can transfer an entire 700MB ISO image across the real, public internet correctly, even in its current state. You will find willing assistance from me and probably some other Divmod folks & fans, because this is a project that many of us would like to see through to completion; we just don't have a lot of spare cycles (or business motivation) at the moment. Depending on what you need, you might want to focus your maintenance efforts on improving PTCP's tests and documentation so as to make it an independently usable unit from Vertex's connection setup. Obviously we'd like it if you could help us to get the whole thing to work, but I have to assume that by the time you're asking this question, you've already done some work on an application protocol to set up the connection.

On 12:27 pm, swp5jhu02@sneakemail.com wrote:

Thank you for all your comments!

I think I'll be looking at Vertex because I also need the actual UDP hole- punching - the whole Vertex solution.

Yaaay!

My dream or idea is to have p2p file sharing with webdav and real mounted file systems on *NIX and Windows.

Oh man. This is *exactly* what I wanted to do with Vertex. Of course, you need a good Twisted-integrated version of FUSE first, but that shouldn't be too hard ;).

Possibly later have /home/peter/projects be automatically synchronized between all my machines.

That seems like more of a job for a version control system. Synchronization is a tricky, tricky problem. It might be more appropriate to continue this discussion on the divmod- dev mailing list. I'll be looking forward to it!

On 05:33 am, iny+news@iki.fi wrote:

glyph@divmod.com writes:

...

Oh man. This is *exactly* what I wanted to do with Vertex. Of course, you need a good Twisted-integrated version of FUSE first, but that shouldn't be too hard ;).

Is there a better way to integrate FUSE and Twisted than use one thread for FUSE and one for Twisted reactor without signal handlers? (Using the FUSE Python bindings of course.)

FUSE just gives a file descriptor to userspace. A better way to integrate with Twisted would simply be to put that file descriptor into the reactor and parse it asynchronously, just like any other protocol. The Python FUSE bindings obscure the issue because, unlike the C libfuse, they assume that your filesystem I/O is blocking, which severely limits the performance of python-based filesystems. (You cannot receive more requests for I/O in your filesystem until the previous one has been completed with the pyfuse bindings, but you can in C.)

I have one filesystem for personal use done this way and it does seem to work. (A simple filesystem for browsing zip contents,)

It certainly works well enough for simple cases. But it would be more robust and higher performance to do it the "normal" Twisted way :).