Some notes on a high-performance Python application.

[Heiko Wundram]

Am Mittwoch, 26. März 2008 18:54:29 schrieb Michael Ströder:
> Heiko Wundram wrote:
> > Am Mittwoch, 26. März 2008 17:33:43 schrieb John Nagle:
> >> ...
> >>
> >>     Using MySQL as a queueing engine across multiple servers is unusual,
> >> but it works well.  It has the nice feature that the queue ordering
> >> can be anything you can write in a SELECT statement. So we put "fair
> >> queueing" in the rating scheduler; multiple requests from the same IP
> >> address compete with each other, not with those from other IP addresses.
> >> So no one site can use up all the rating capacity.
> >> ...
> >>     Does anyone else architect their systems like this?
> >
> > A Xen(tm) management system I've written at least shares this aspect in
> > that the RPC subsystem for communication between the frontend and the
> > backends is basically a (MySQL) database table which is regularily
> > queried by all backends that work on VHosts to change the state (in the
> > form of a command) according to what the user specifies in the (Web-)UI.
>
> I vaguely remember that this database approach was teached at my former
> university as a basic mechanism for distributed systems at least since
> 1992, but I'd guess much longer...

I didn't say it was unusual or frowned upon (and I was also taught this at uni 
IIRC as a means to "easily" distribute systems which don't have specific 
requirements for response time to RPC requests), but anyway, as you noted for 
Biztalk, it's much easier to hit bottlenecks with a polling-style RPC than 
with a "true" RPC system, as I've come to experience when the number of nodes 
(i.e., backends) grew over the last year and a half.

That's what's basically causing a re-consideration to move from DB-style RPC 
to socket-based RPC, which is going to happen at some point in time for the 
system noted above (but I've sinced changed jobs and am now only a consulting 
developer for that anyway, so it won't be my job to do the dirty migration 
and the redesign ;-)).

-- 
Heiko Wundram