[Python-Dev] Does Zip Importer have to be Special?

[Phil Thompson]

On 24/07/2014 6:48 pm, Brett Cannon wrote:
> On Thu Jul 24 2014 at 1:07:12 PM, Phil Thompson 
> <phil at riverbankcomputing.com>
> wrote:
> 
>> I have an importer for use in applications that embed an interpreter
>> that does a similar job to the Zip importer (except that the storage 
>> is
>> a C data structure rather than a .zip file). Just like the Zip 
>> importer
>> I need to import my importer and add it to sys.path_hooks. However the
>> earliest opportunity I have to do this is after the Py_Initialize() 
>> call
>> returns - but this is too late because some parts of the standard
>> library have already needed to be imported.
>> 
>> My current workaround is to include a modified version of 
>> _bootstrap.py
>> as a frozen module that has the necessary steps added to the end of 
>> its
>> _install() function.
>> 
>> The Zip importer doesn't have this problem because it gets special
>> treatment - the call to its equivalent code is hard-coded and happens
>> exactly when needed.
>> 
>> What would help is a table of functions that were called where
>> _PyImportZip_Init() is currently called. By default the only entry in
>> the table would be _PyImportZip_Init. There would be a way of 
>> modifying
>> the table, either like how PyImport_FrozenModules is handled or how
>> Inittab is handled.
>> 
>> ...or if there is a better solution that I have missed that doesn't
>> require a modified _bootstrap.py.
>> 
> 
> Basically you want a way to specify arguments into
> importlib._bootstrap._install() so that sys.path_hooks and 
> sys.meta_path
> were configurable instead of hard-coded (it could also be done just 
> past
> importlib being installed, but that's a minor detail). Either way there 
> is
> technically no reason not to allow for it, just lack of motivation 
> since
> this would only come up for people who embed the interpreter AND have a
> custom importer which affects loading the stdlib as well (any reason 
> you
> can't freeze the stdblib as a solution?).

Not really. I'd lose the compression my importer implements.

(Are there any problems with freezing packages rather than simple 
modules?)

> We could go the route of some static array that people could modify.
> Another option would be to allow for the specification of a single 
> function
> which is called just prior to importing the rest of the stdlib,
> 
> The problem with all of this is you are essentially asking for a hook 
> to
> let you have code have access to the interpreter state before it is 
> fully
> initialized. Zipimport and the various bits of code that get loaded 
> during
> startup are special since they are coded to avoid touching anything 
> that
> isn't ready to be used. So if we expose something that allows access 
> prior
> to full initialization it would have to be documented as having no
> guarantees of interpreter state, etc. so we are not held to some API 
> that
> makes future improvements difficult.
> 
> IOW allowing for easy patching of Python is probably the best option I 
> can
> think of. Would tweaking importlib._bootstrap._install() to accept
> specified values for sys.meta_path and sys.path_hooks be enough so that 
> you
> can change the call site for those functions?

My importer runs under PathFinder so it needs sys.path as well (and 
doesn't need sys.meta_path).

Phil