[Tutor] why is os.path.walk so slow?

[Wayne Werner]

On Wed, Nov 4, 2009 at 6:16 AM, Garry Willgoose <
garry.willgoose at newcastle.edu.au> wrote:

> <snip>
>
> This is very fast for a directory on my local machine but significantly
> slower on the remote machine. Not surprising but I would have expected that
> the run time for the remote directory would be limited by my broadband speed
> but when I look at upload/download in real time it's less than 10% of
> maximum. Is this just par for the course or is there something I can do that
> better utilizes my broadband bandwidth?


I'm not sure if there's a correlation, but there probably is. What OS are
you (and the remote system) using? What service are you using to connect?

By way of disclosure, I don't have a lot of experience in this category, but
I would venture that whatever service you're using has to send/receive
requests for each file/dir that os.walk checks.

I don't know precisely how os.walk works, but I'm guessing the differences
are as follows:

Local Machine:
python->os.walk->local system calls

Remote Machine:
python->os.walk->call to client->data through the tubes->remote host->system
calls

Even if that's not completely correct, you're still going to have extra
steps as you walk through the remote system, and the bottleneck will
probably not be on the internet connection (and your tests seem to verify
this).

It would work better, I think, if you were able to run a script on the
remote system and return the results.

HTH,
Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20091104/033228ea/attachment.htm>