[Python-ideas] Refactoring the import system to be object-oriented at the top level

[Nick Coghlan]

(Disclaimer: this is complicated Py-in-the-sky stuff, and I'm handwaving
away a lot of major problems with the concept, not least of which is the
sheer amount of work involved. I just wanted to get the idea published
somewhere while I was thinking about it)

I'm in the process of implementing a runpy.run_path function for 2.7/3.2
to allow Python code to use the zipfile and directory execution feature
provided by the CPython command line in 2.6/3.1. It turns out the global
state used for the import system is causing some major pain in the
implementation. It's solvable, but it will probably involve a couple of
rather ugly hacks and the result sure as hell isn't going to be thread-safe.

Anyway, the gist of the idea in the subject line is to take all of the
PEP 302 data stores and make them attributes of an ImportEngine class.
This would affect at least:

sys.modules
sys.path
sys.path_hooks
sys.path_importer_cache
sys.meta_path
sys.dont_write_bytecode

The underlying import machinery would migrate to instance methods of the
new class.

The standard import engine instance would be stored in a new sys module
attribute (e.g. 'sys.import_engine'). For backwards compatibility, the
existing sys attributes would remain as references to the relevant
instance attributes of the standard engine.

Modules would get a new special attribute (e.g. '__import_engine__')
identifying the import engine that was used to import that module.
__import__ would be modified to take the new special attribute into account.

The main immediate benefit from my point of view would be to allow runpy
to create *copies* of the standard import engine so that
runpy.run_module and runpy.run_path could go do their thing without
impacting the rest of the interpreter. At the moment that really isn't
feasible, hence the lack of thread safety in that module.

I suspect such a change would greatly simplify experimentation with
Python security models as well: restricted code could be given a
restricted import engine rather than the restrictions having to be baked
in to the standard import engine.

>From an OO design point of view, it's a classic migration of global
state and multiple functions to manipulate that state into a single
global instance of a new class.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------