A smarter(?) package importer.

[Prabhu Ramachandran]

hi,

Thanks to all of you for the tips esp.  Bernhard, Gordon, Rainer and
Michael.

>>>>> "GMcM" == Gordon McMillan <gmcm at hypernet.com> writes:

    >> all of its keys and then fail?  Hmmm, but if you kept caching
    >> failure, you'd increase the number of keys.  If the package
    >> nesting is substantial for every global module this would
    >> insert several new item into sys.modules.

    GMcM> But sys.modules is a dictionary. It's hashed, not scanned.
    GMcM> If you *don't* cache failure, you get 3 (4 on Windows)
    GMcM> attempted opens per directory that you could have avoided.

Well, I'm not sure it does 3/4 opens but yes it does do 3/4
imp.find_modules which will be more expensive than a
sys.modules[mod_name].

Anyway, I did the following:

  (1) added caching to knee.py and fixed a bug in it.

  (2) I also added caching to my_import.py (what a lousy name!
  <hint>).

  (3) cleaned up the pkg stuff so it now tests for caching, tests if
  importing is ok etc.

Its again available here:

    http://av.stanford.edu/~prabhu/download/

A new test_import.tar.gz package which is far nicer and the modifed
knee.py can be found.

I'd really appreciate if you folks can check it out.


Here are some crude timing estimates while using my_import:

# this has been duly done a few times so dont worry this is not first
# time import time

# normal import
>>> import time
>>> s = time.time (); import scipy; print time.time()-s
1.38626110554
>>> 

$ python
>>> import my_import   
>>> import time
>>> s = time.time (); import scipy; print time.time()-s
1.48667407036

Which if you ask me is pretty good considering what it can do.  Also,
this kind of proves that this code works for large projects as well.

Here are some crude timings of code with and without cache (its
actually just a single line that adds caching).

no cache: 1.5991319418, cache: 1.49252903461

which isnt much but is measurable. :)

So, I'll ask again, how about making this behaviour standard? :)

prabhu