Speeding up ftp, async/threadpool?
andrew-pythonlist at puzzling.org
Thu Feb 13 23:15:28 CET 2003
On Thu, Feb 13, 2003 at 06:29:42AM -0800, Ville Vainio wrote:
> Andrew Bennetts <andrew-pythonlist at puzzling.org> wrote in message news:<mailman.1045130385.10906.python-list at python.org>...
> > > optimizing this with a threadpool (which is easy to implement), but
> > > OTOH it seems that an async ftp client could juggle hundreds of ftp
> > > downloads/uploads at the time, effectively saturating the network. How
> > > come such thing hasn't been implemented yet? I'm thinking of an API
> > > like:
> > If you can't wait for me to tidy up Twisted's FTP (and frankly I wouldn't
> > hold my breath ;), I can give you some pointers on how to do it with current
> > Twisted. It is straightforward, just unnecessarily tedious.
> But please do give the pointers, you might still convince me. I have
> actually been trying to find a good excuse to get more familiar with
> Twisted :-).
The best starting point is probably this example:
And the FTPClient API docs are also relevant:
Large chunks of that might not make sense without understanding some basic
Twisted, though. To change that script to run multiple FTPClients in
parallel, just duplicate the calls to ClientCreator/creator.connectTCP.
You'd probably want to use the same callback function for all instances that
took files to retrieve off a list somewhere. Oh, and to do the "waitforall"
part of your API, use a DeferredList...
Let me know if you want more info than just these brief pointers :)
More information about the Python-list