<enable-ramble-mode> :-) On Sat, 20 Nov 1999, Greg Stein wrote:
... The point about dynamic modules actually discovered a basic problem that I need to resolve now. The current imputil assumes that if a particular Importer loaded the top-level module in a package, then that Importer is responsible for loading all other modules within that package. In my particular test, I tried to import "xml.parsers.pyexpat". The two package modules were handled by SysPathImporter. The pyexpat module is a dynamic load module, so it is *not* handled by the Importer -- bam. Failure.
Basically, each part of "xml.parsers.pyexpat" may need to use a different Importer...
I've thought about this and decided the issue is with my particular Importer, rather than the imputil design. The PathImporter traverses a set of paths and establishes a package hierarchy based on a filesystem layout. It should be able to load dynamic modules from within that filesystem area. A couple alternatives, and why I don't believe they work as well: * A separate importer to just load dynamic libraries: this would need to replicate PathImporter's mapping of Python module/package hierarchy onto the filesystem. There would also be a sequencing issue because one Importer's paths would be searched before the other's paths. Current Python import rules establishes that a module earlier in sys.path (whether a dyn-lib or not) is loaded before one later in the path. This behavior could be broken if two Importers were used. * A design whereby other types of modules can be placed into the filesystem and multiple Importers are used to load parts of the path (e.g. PathImporter for xml.parsers and DynLibImporter for pyexpat). This design doesn't work well because the mapping of Python module/package to the filesystem is established by PathImporter -- try to mix a "private" mapping design among Importers creates too much coupling. There is also an argument that the design is fundamentally incorrect :-). I would argue against that, however. I'm not sure what form an argument *against* imputil would be, so I'm not sure how to preempty it :-). But we can get an idea of various arguments by hypothesizing different scenarios and requireing that the imputil design satisifies them. In the above two alternatives, they were examing the use of a secondary Importer to load things out of the filesystem (and it explained why two Importers in whatever configuration is not a good thing). Let's state for argument's sake that files of some type T must be placable within the filesystem (i.e. according to the layout defined by PathImporter). We'll also say that PathImporter doesn't understand T, since the latter was designed later or is private to some app. The way to solve this is to allow PathImporter to recognize it through some configuration of the instance (e.g. self.recognized_types). A set of hooks in the PathImporter would then understand how to map files of type T to a code or module object. (alternatively, a generalized set of hooks at the Importer class level) Note that you could easily have a utility function that scans sys.importers for a PathImporter instance and adds the data to recognize a new type -- this would allow for simple installation of new types. Note that PathImporter inherently defines a 1:1 mapping from a module to a file. Archives (zip or jar files) cannot be recognized and handled by PathImporter. An archive defines an entirely different style of mapping between a module/package and a file in the filesystem. Of course, an Importer that uses archives can certainly look for them in sys.path. The imputil design is derived directly from the "import" statement. "Here is a module/package name, give me a module." (this is embodied in the get_code() method in Importer) The find/load design established by ihooks is very filesystem-based. In many situations, a find/load is very intertwined. If you want to take the URL case, then just examine the actual network activity -- preferably, you want a single transaction (e.g. one HTTP GET). Find/load implies two transactions. With nifty context handling between the two steps, you can get away with a single transaction. But the point is that the design requires you to get work around its inherent two-step mechanism and establish a single step. This is weird, of course, because importing is never *just* a find or a load, but always both. Well... since I've satisfied to myself that PathImporter needs to load dynamic lib modules, I'm off to code it... Cheers, -g -- Greg Stein, http://www.lyra.org/