Extending the import mechanism - what is recommended?

[Steve Holden]

dbr517 at gmail.com wrote:
> On Jan 29, 2:36 pm, Steve Holden <st... at holdenweb.com> wrote:
>> dbr... at gmail.com wrote:
>>> I need to extend the import mechanism to support another file type.
>>> I've already written the necessary C library to read the file and
>>> return a python code object.
>>> I found one example which just sub-classed imputil.ImportManager like
>>> this:
>>> from myLib import pye_code as pye_code
>>> class MyImporter(imputil.ImportManager):
>>>     def __init__(self):
>>>         imputil.ImportManager.__init__(self)
>>>         self.add_suffix('.pye', self.import_pye)
>>>         self.install()
>>>     def import_pye(self, filepath, fileinfo, filename):
>>>         data = pye_code(filepath)
>>>         return 0, data, {}
>>> This actually works fine if the module is just a few lines of code,
>>> but it won't chain to the "built-in" importers; if the module that I'm
>>> importing does something as simple as 'import re', it fails.
>>> It may be that my confusion here is because (even after reading the
>>> code), I'm not clear on the purposes of imputil.ImportManager vs.
>>> imputil.Importer :-(
>>> What is the "preferred" way to do this type of extension?  One other
>>> note; at this time, I just need to import individual module files with
>>> this extension; I don't need to import packages.
>> Here's an importer I wrote some time ago to bring modules in from a
>> relational database. I won't trouble you with the code to compile the
>> modules/packages and add them into the database, but perhaps the
>> attached code will be enough to show you what you are doing that's not
>> working.
>>
[...]
>> [dbimp.py]#
>> # Import modules from a database
>> #
>> # NOTE: could use sys version info to select appropriate module version
>> #       - could ask whether to install if absent ... heh, heh :-)
>> #
>> import sys, db, marshal
>> VER = sys.hexversion
> ----------------------------------->
>> def install():
>>     sys.path_hooks.append(dbimporter)
>>     sys.path_importer_cache.clear() # probably not necessary
>>     sys.path.insert(0, "*db*") # probably not needed with a metea-path hook?
> 
> Steve -
> 
> Thanks!  Got this to work with one interesting problem . . . if I use
> sys.path.insert(0....) and insert my hook at the head of the path,
> then I can't import anything EXCEPT my special modules . . . If I use
> sys.path.append("*pye*") then I'm OK.
> 
> One thing I changed was that my load_module function raises
> ImportError if it fails; my understanding from reading PEP302 is that
> that's what's SUPPOSED to happen . . .
> 
> At any rate, I can now import my custom modules . . . thanks!
> 
Excellent news, thanks for letting me know. I can't say that code was 
particularly well-tested, so it's possible that my version makes some 
error that stops the import from running against further path elements 
if the search by the custom importer fails.

May have a few minutes to look at this later, but not for a while. 
Anyway, glad the code helped. Perhaps someone else with more PEP302-fu 
would be able to point out my error.

Any other changes you'd like to feed back besides the ImportError?

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/