Speeding up ftp, async/threadpool?

[Andrew Bennetts]

On Wed, Feb 12, 2003 at 11:43:32PM -0800, Ville Vainio wrote:
> Trying to do a mget-tish thing with ftplib for a large number of small
> files, one can see how slow a protocol ftp is. I've thought of
> optimizing this with a threadpool (which is easy to implement), but
> OTOH it seems that an async ftp client could juggle hundreds of ftp
> downloads/uploads at the time, effectively saturating the network. How
> come such thing hasn't been implemented yet? I'm thinking of an API
> like:
> 
> a = asyncftp(connections=200)
> a.login("host1.com","myuid","mypasswd")
> a.login("host2.com","myuid2","mypasswd2")
> 
> a.getfile("host1.com","/home/myuid/foo.txt",target="/tmp/foo.txt")
> a.getfile("host2.com","/home/myuid/bar.txt",target="/tmp/bar.txt")
> 
> a.go()
> a.waitforall()

You could do this right now with Twisted, but with the current FTP code it
would be much uglier than this.  Fixing it so that the interface is more
convenient for this sort of thing is on my todo list...

If you can't wait for me to tidy up Twisted's FTP (and frankly I wouldn't
hold my breath ;), I can give you some pointers on how to do it with current
Twisted.  It is straightforward, just unnecessarily tedious.

-Andrew.