__file__ access extremely slow

[John Machin]

Steven D'Aprano <steven <at> REMOVE.THIS.cybersource.com.au> writes:

> 
> On Fri, 05 Jun 2009 02:21:07 +0000, Steven D'Aprano wrote:
> 
> > You corrected this to:
> > 
> > for module in sys.modules.itervalues():
> >        try:
> >                path = module.__file__
> >        except (AttributeError, ImportError):
> >                return
> > 
> > (1) You're not importing anything inside the try block. Why do you think
> > ImportError could be raised?
> > 
> > (2) This will stop processing on the first object in sys.modules that
> > doesn't have a __file__ attribute. Since these objects aren't
> > *guaranteed* to be modules,

Definitely not guaranteed to be modules. Python itself drops non-modules in
there! Python 2.3 introduced four keys mapped to None -- one of these was
dropped in 2.4, but the other three are still there in 2.5 and 2.6:

C:\junk>\python23\python -c "import sys; print [k for (k, v) in
sys.modules.items() if v is None]"
['encodings.encodings', 'encodings.codecs', 'encodings.exceptions',
'encodings.types']

C:\junk>\python24\python -c "import sys; print [k for (k, v) in
sys.modules.items() if v is None]"
['encodings.codecs', 'encodings.exceptions', 'encodings.types']

> this is a subtle bug waiting to bite you.
> 
> In fact, not all modules have a __file__ attribute.
> 
> >>> import errno
> >>> errno.__file__
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'module' object has no attribute '__file__'

Yep, none of the built-in modules has a __file__ attribute.

So, as already pointed out, the loop is likely to stop rather early, making the
huge 0.5 seconds look even more suspicious.

Looking forward to seeing the OP's timing code plus a count of the actual number
of loop gyrations ...

Cheers,
John