Speeding up ftp, async/threadpool?

[Andrew Bennetts]

On Thu, Feb 13, 2003 at 06:29:42AM -0800, Ville Vainio wrote:
> Andrew Bennetts <andrew-pythonlist at puzzling.org> wrote in message news:<mailman.1045130385.10906.python-list at python.org>...
> 
> > > optimizing this with a threadpool (which is easy to implement), but
> > > OTOH it seems that an async ftp client could juggle hundreds of ftp
> > > downloads/uploads at the time, effectively saturating the network. How
> > > come such thing hasn't been implemented yet? I'm thinking of an API
> > > like:
> 
> > If you can't wait for me to tidy up Twisted's FTP (and frankly I wouldn't
> > hold my breath ;), I can give you some pointers on how to do it with current
> > Twisted.  It is straightforward, just unnecessarily tedious.
> 
[...]
> But please do give the pointers, you might still convince me. I have
> actually been trying to find a good excuse to get more familiar with
> Twisted :-).

The best starting point is probably this example:
    http://twistedmatrix.com/documents/examples/ftpclient.py
And the FTPClient API docs are also relevant:
    http://twistedmatrix.com/documents/TwistedDocs/current/api/public/twisted.protocols.ftp.FTPClient.html

Large chunks of that might not make sense without understanding some basic
Twisted, though.  To change that script to run multiple FTPClients in
parallel, just duplicate the calls to ClientCreator/creator.connectTCP.
You'd probably want to use the same callback function for all instances that
took files to retrieve off a list somewhere.  Oh, and to do the "waitforall"
part of your API, use a DeferredList...

Let me know if you want more info than just these brief pointers :)

-Andrew.