[Ironpython-users] Resource-based ZIP importer / meta_path importer

[Keith Rome]

Following up on this topic from a few weeks ago....

I did not file an issue on it, because it isn't directly a part of the python spec, so it's more "something I would like to see" and less "something isn't working right".

Anyhow, I have been doing some tinkering.  I feel like I am right at the goal line, and just need a tiny bit of help to cross over it.

The zipimport module is an implementation of "PEP 273: Import Modules from Zip Archives". My first tinkerings were with modifying the zipimport class and ZimImportModule class to facilitate using a URI path as in my suggestion in the previous email. I ran into a brick wall with this however, because PythonContext.InitializeModule() still makes calls to both System.IO.Path directly as well as the GetFullPath() method of the current PAL. Both of those things prevent me from being able to use anything but a path value that is valid on the current file system (i.e., I can't use a "<protocol>://<address>/<path><resource>" scheme).

In reading the details of PEP 273, I also learned about "PEP 302: New Import Hooks" (http://www.python.org/dev/peps/pep-0302/). This PEP looks to be exactly what I want, because it is totally transparent to sys.path, and therefore has no need for any of the path canonicalization workarounds that I was running into with a PEP 273 solution. PEP 302 appears to already be implemented in IronPython... at least sys.meta_path is there, and the IronPython.Runtime.Importer class attempts to make use of it.

So I fully implemented a zip resource based meta_path importer by following the example of zipimport but adapting it for how meta_path is intended to work. It appears to work fine for simple imports. But there is a major problem I have encountered. The meta_path functionality does not appear to be re-entrant, and I have not yet been able to figure out why.

Implementing a meta_path importer is fairly simple (easier than a path hook importer like zipimport). I just have to implement find_module() and load_module() and append my importer to sys.meta_path. These work as expected. But the final step of the importer protocol dictates that "If the module is a Python module (as opposed to a built-in module or a dynamically loaded extension), it should execute the module's code in the module's global name space (module.__dict__)." This is easy enough, and works exactly like in zipimport. PythonContext.CompileModule() produces an executable ScriptCode object, and I simply Run() it using the module's Scope.

This then proceeds to execute the module initialization code, which invariably does some imports of its own. As a test case, I use the following script: "from multiprocessing import process". The multiprocessing module attempts to "import os" among other things. I would expect this to invoke find_module('os') and subsequently load_module('os') on my meta_path importer. But it does not. For whatever reason, when initializing a module, any internal attempts to resolve further imports seem to be ignoring meta_path importers. These imports fail because they should have been serviced by my meta_path importer, which was not invoked.


1.       Has anyone implemented a meta_path importer in IronPython that could maybe shed some light on this?

2.       Is this working as intended? (doesn't seem to be)

3.       Is the PEP 302 support perhaps incomplete?

4.       Am I maybe missing a step somewhere?

I have followed the advice in PEP 302. In addition to everything zipimport does, I am also setting __file__ to "<resource>", and setting __path__ to an empty List. I also ensure that the module is added to sys.modules before executing the initialization code, and I check sys.modules at the very start of my load_module() method to prevent double-loading. Although it would seem that none of those things would have any bearing on this particular issue.


Keith Rome
Senior Consultant and Architect
MCPD-EAD, MCSD, MCDBA, MCTS-WPF, MCTS-TFS, MCTS-WSS
Wintellect | 770.617.4016 | krome at wintellect.com<mailto:rome at wintellect.com>
www.wintellect.com<http://www.wintellect.com/>

From: Slide [mailto:slide.o.mix at gmail.com]
Sent: Monday, March 12, 2012 10:03 AM
To: Keith Rome
Cc: ironpython-users at python.org
Subject: Re: [Ironpython-users] IronPython 2.7.2 Released

This is a great idea, this functionality does not currently exist, could you please file a new issue at http://ironpython.codeplex.com?

Thanks,

slide
On Sun, Mar 11, 2012 at 11:23 PM, Keith Rome <rome at wintellect.com<mailto:rome at wintellect.com>> wrote:
Thanks for the great work!

Regarding the new zipimport functionality - is it possible to specify an embedded resource name for the path to the zip? For example, if I wanted to distribute a library by just packaging it up within an assembly? Or is it necessary to write the resource out to a file on disk first?

Example:
Instead of: sys.path.insert(0, '/path/lib.zip')
Something like: sys.path.insert(0, 'resource:path.lib.zip')

The primary reason behind wanting to deliver via embedded resource is to prevent tinkering/tampering by end users. But there are additional reasons such as simpler deployment under Silverlight and not having to deal with versioning of extra "satellite" external resources when using our scripting runtime environment in multiple projects. All of those issues go away when we can package everything directly within the assembly(s).

I am currently using a custom PlatformAdaptationLayer to supply the file contents at runtime and msbuild wildcard inclusion of the libraries and all subfolders to embed them as resources (something that msbuild tolerates, but Visual Studio does not). It would be much cleaner to just use zipimport, as well as much easier to maintain (and the distributed assembly would also be much smaller due to zip compression).


Keith Rome
Senior Consultant and Architect
MCPD-EAD, MCSD, MCDBA, MCTS-WPF, MCTS-TFS, MCTS-WSS
Wintellect | 770.617.4016<tel:770.617.4016> | krome at wintellect.com<mailto:krome at wintellect.com>
www.wintellect.com<http://www.wintellect.com>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20120402/b76cf3cc/attachment.html>