[Python-Dev] Issue #11051: system calls per import

[Guido van Rossum]

On Sun, Jan 30, 2011 at 11:33 PM, "Martin v. Löwis" <martin at v.loewis.de> wrote:
>> Maybe also
>>
>>    * Read and cache the directory contents and search it ourselves
>>      instead of making a system call for every possible name.
>
> I wouldn't do that - I would expect that this is actually slower than
> making the system calls, because the system might get away with not
> reading the entire directory (whereas it will have to when we explicitly
> ask for that).

Hm. Long (very long) ago I had to implement just that, and it was much
faster. But this was over NFS. Still, I think the directory would have
to be truly enormous before reading its contents (which doesn't access
all the inodes) is slower than statting a few dozen of its entries. At
least on most *nix filesystems.

Another thing to consider: on App Engine (which despite of all its
architectural weirdness uses a -- mostly -- standard Linux filesystem
for the Python code of the app) someone measured that importing from a
zipfile is much faster than importing from the filesystem. I would
imagine this extends to other contexts too, and it makes sense because
the zipfile directory gets cached in memory so no stat() calls are
necessary.

(Basically I am biased to believe that stat() is a pretty slow system
call -- this may just be old NFS lore though.)

-- 
--Guido van Rossum (python.org/~guido)