PythonD: 4Suite or Twisted?

[David Bolen]

Moshe Zadka <m at moshez.org> writes:

> As far as I know, Perspective Broker is not slated to ever do UDP.
> Unreliable transports are really bad for remote-object protocols which
> are based on calling methods. I believe it is fairly easy, however,
> to transport Jelly over UDP, if all you need is "unreliable high-level
> messaging".

I guess to me a method invocation for a remote-object protocol is
perfect for a simple datagram request/response scenario (providing the
payload is small enough for a datagram) and is often what we use for
some of our internal stuff.  A simple timeout error that can be
handled by the upper level application can work fine.

There's nothing wrong with using an unreliable transport for
remote-object protocols if it fits the problem domain.  In many cases,
the reliability of the stream protocol can actually get in the way by
imposing unnecessary timeouts and underlying retry mechanisms that the
higher level request structure doesn't need.  Not to mention that
managing state on a common server for a lot of stream connections
(even if the streams themselves are largely idle) can become a scaling
problem.

But I'm also fine if there's no real intention to let the PB work in
such an environment, as that's a perfectly good design choice.

> You're right, but it's not the Twisted tide you're swimming against.
> Most of the time, UDP is the wrong solution, period. When it is,
> for example for media-streaming, it needs to be treated differently.
> There is no sense of writing reliable transports on top of UDP when
> TCP exists.

"Most of the time" is a broad statement - one that I'm probably in
agreement with.  But there are large classes of applications where TCP
just isn't necessarily the best protocol, and those applications can
still benefit from higher level authentication, remote invocation,
etc...

For example, when you have a large mesh of communicating nodes sending
short bursts of traffic to each other (as is typical in network
management or distributed processing applications).  Constantly
setting up and tearing down TCP sessions (with their overhead) is
costly, and maintaining the mesh of connections is expensive in system
and network resources.  Having been a core backbone person when HTTP
first came on the scene, the choice of new TCP session per request was
devastating to traffic patterns and utilization, to which HTTP 1.1
(shared session) was a crucially needed improvement.

Anyway, using UDP datagrams (even with a rudimentary retry mechanism
when necessary - and for announcement only services it may not even be
necessary if a miss is not critical and will be covered by a later
transmission) can have a lot going for it in terms of scaleability and
performance in some situations.  Should it be misused by games trying
to flood backbones by building a retry mechanism without some sort of
exponential fallback?  Definitely not.  But in the right circumstance
a stateless UDP server can be a really good paradigm.  There's a
spectrum between completely unreliable (raw UDP) and the overhead
imposed by TCP (particularly in setup/teardown) for its reliability.
Sometimes TCP is just too "heavy" and a UDP base is more appropriate.

Hmm, this probably slipped into more of a TCP/UDP protocol issue than
Twisted specific one.  I appreciate the feedback on Twisted though.
I'll look into the Jelly over UDP question.

-- David