Advice on a TCP passthru daemon for HTTP proxy rotation

[Bryan Olson]

Robin Haswell wrote:
> Hey there
> 
> Soon we will have many squid proxies on many seperate connections for use
> by our services. I want to make them available to users via a single HTTP
> proxy - however, I want fine-grained control over how the squid proxies
> are selected for each connection. This is so I can collect statistics,
> control usage on each proxy, monitor what's going on - etc. However I
> don't want to implement the HTTP proxy protocol in Python, and would
> much rather let my daemon run as a man-in-the-middle for TCP, similar to
> this netcat command:
> 
> rob at aranea:~$ mknod backpipe p
> rob at aranea:~$ nc -l -p 8080 < backpipe | nc ganesh 8080 > backpipe
> 
> Basically when my daemon received a connection (call it "c1"), it makes a
> connection to one of my squid proxies ("c2"), then all data which gets
> read from c1 is written to c2 - all data read from c2 is written to c1.
> I'm pretty sure there's an elegant way to do this but I was wondering if
> anyone had any input? I've tried GIYF'ing this but it's difficult to
> search for :P

Try searching for "port forwarding". Basic port forwarding is
a straightforward sockets programming exercise, but there are
trickier considerations in building it to face the Internet
and front a large number of of servers. Sounds like your TCP
proxy will be a potential single point of failure, and is the
first machine to which users connect. Not the place to try
out new stuff.

You might look into existing proxies and "load balancers"
before coding your own. There's no rule against fronting
Squid proxies with Squid proxies. You may want different
statistics from what they log, but proxying at the TCP
level, you're limited as to what information you can get.

May I ask why you need many Squid proxies? Unless you're
using them to terminating SSL/TLS connections, just one of
them can probably out-run a Python port-forwarder. Squid
usually serves a large percentage of the volume from its
memory cache, while a port forwarder has to read it all
from TCP connections, which is more work.


-- 
--Bryan