Mysterious delay in Python module loading (fwd)

[Bob Kline]

On Tue, 8 Jan 2002, Jason Orendorff wrote:

> > I tried breaking down the load times for the separate Python modules, to
> > see if one of them were doing something differently from the others, but
> > the extra load time seems to be spread out fairly evenly between them,
> > making it look more like a problem with the module loading or finding
> > process, rather than an anomaly with one of the individual modules.
> >
> > PYTHONPATH is identical on both machines.  All of our own Python scripts
> > are byte for byte identical.  All of the .pyc files for our own scripts
> > have been recompiled on the machine using them.
> [...]
> > I have confirmed that none of the locations in sys.path
> > are out on a network (they're all on the local hard disks).
>
> Rats.  I was thinking maybe something in sys.path was a CD-ROM
> drive, or some kind of removable media that takes a moment for the
> OS to check.  Oh well.
>
> Anyway, what exactly is the value of sys.path?  If you set it to
> just [ "C:\\Python22\\lib" ], plus the specific other directories
> you need, does that help at all?

I have made some progress in my debugging of the problem.  In fact it
appears that the problem stems from a wide discrepancy in the amount of
time consumed by a call to fopen() on the two machines.  A test repro
written in C with loops to repeatedly attempt to open the files that
Python was trying to open while searching for the module files runs in
under 6 seconds on the development machine, and 382 seconds on the
second machine (some of the fopen calls succeed, most fail, as is the
case when Python is doing its search for the modules).

Unfortunately, this means that in order to go much further I'll probably
have to install the debug kernel, which I'm not very excited about
doing, given the impact such activities tend to have on project
schedules.

I'm reluctant to believe that disk hardware has anything to do with the
discrepancy, since looping through the same path names over and over
will surely have everything the OS needs cached in memory by the second
time through the outer loop.  I've shut down enough services on the
machine to ensure that I've got plenty of memory for caching (Task
Manager shows about half of the 256MB of real memory in use).  On the
other hand, the CPU is almost completely idle during the repro test.

The only remotely likely possibility that comes to mind at this point in
the game is something to do with screwed-up ACLs.

-- 
Bob Kline
mailto:bkline at rksystems.com
http://www.rksystems.com