taking python enterprise level?...
nanyaks at googlemail.com
Mon Mar 1 13:11:07 CET 2010
On Feb 25, 5:18 pm, "D'Arcy J.M. Cain" <da... at druid.net> wrote:
> On Thu, 25 Feb 2010 15:29:34 +0000
> "Martin P. Hellwig" <martin.hell... at dcuktec.org> wrote:
> > On 02/25/10 13:58, D'Arcy J.M. Cain wrote:
> > > On Thu, 25 Feb 2010 02:26:18 -0800 (PST)
> > <cut>
> > > Our biggest problem was in
> > > a network heavy element of the app and that was low level TCP/IP stuff
> > > that rather than being Python's problem was something we used Python to
> > > fix.
> > <cut>
> > Out off interest, could you elaborate on that?
> Somewhat - there is an NDA so I can't give exact details. It was
> crucial to our app that we sync up databases in Canada and the US (later
> Britain, Europe and Japan) in real time with those transactions. Our
> problem was that even though our two server systems were on the
> backbone, indeed with the same major carrier, we could not keep them in
> sync. We were taking way to long to transact an update across the
> The problem had to do with the way TCP/IP works, especially closer to
> the core. Our provider was collecting data and sending it only after
> filling a buffer or after a timeout. The timeout was short so it
> wouldn't normally be noticed and in most cases (web pages e.g.) the
> connection is opened, data is pushed and the connection is closed so
> the buffer is flushed immediately. Our patterns were different so we
> were hitting the timeout on every single transaction and there was no
> way we would have been able to keep up.
> Our first crack at fixing this was to simply add garbage to the packet
> we were sending. Making the packets an order of magnitude bigger sped
> up the proccessing dramatically. That wasn't a very clean solution
> though so we looked for a better way.
> That better way turned out to asynchronous update transactions. All we
> did was keep feeding updates to the remote site and forget about ACKS.
> We then had a second process which handled ACKS and tracked which
> packets had been properly transferred. The system had IDs on each
> update and retries happened if ACKS didn't happen soon enough.
> Naturally we ignored ACKS that we had already processed.
> All of the above (and much more complexity not even discussed here) was
> handled by Python code and database manipulation. There were a few
> bumps along the way but overall it worked fine. If we were using C or
> even assembler we would not have sped up anything and the solution we
> came up with would have been horrendous to code. As it was I and my
> chief programmer locked ourselves in the boardroom and had a working
> solution before the day was out.
sure it wouldnt have sped it up a bit, even a bit?; probably the
development and maintenance time would be a nightmare but it should
speed the app up a bit...
> Python wins again.
> D'Arcy J.M. Cain <da... at druid.net> | Democracy is three wolveshttp://www.druid.net/darcy/ | and a sheep voting on
> +1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
seriously added to the reputation of python, from my own
More information about the Python-list