<div dir="ltr">I've updated PEP 451 to address comments and clear a few things up. Most notably, I added a list of terms at the beginning.<div><br></div><div>The PEP is pretty close to done and feedback has simmered down. Does anyone object to my posting the next update to python-dev?</div>
<div><br></div><div>There are two main open questions:</div><div><br></div><div>1. How does ModuleSpec interact with existing import-sensitive modules in the standard library?</div><div>2. PJE's concerns about reload semantics and lazy loading.</div>
<div><br></div><div>Regarding the first, I'm not too concerned with the ability to adapt those modules to ModuleSpec without much effort. However, I will be doing a thorough check before I'll ask for pronouncement.</div>
<div><br></div><div>About lazy loading, from what I understand, importlib.reload() broke backward compatibility with regards to PJE's use case when it switched to depending on __loader__. Perhaps it was even before that. I'll have to check. Regardless, PEP 451 does not change the semantic of reload() from what they currently are.</div>
<div><br></div><div>The PEP could restore the previous semantics without a lot of work (if module.__spec__ is not set, then call find_spec(), set it, and reload using that). However, if reload() backward compatibility got broken somewhere along the lines, that sounds like a bug that should be addressed separately.</div>
<div><br></div><div>-eric</div><div><br></div><div><br></div><div>=====================================================================</div><div><br></div><div><div>PEP: 451</div><div>Title: A ModuleSpec Type for the Import System</div>
<div>Version: $Revision$</div><div>Last-Modified: $Date$</div><div>Author: Eric Snow <<a href="mailto:ericsnowcurrently@gmail.com">ericsnowcurrently@gmail.com</a>></div><div>Discussions-To: <a href="mailto:import-sig@python.org">import-sig@python.org</a></div>
<div>Status: Draft</div><div>Type: Standards Track</div><div>Content-Type: text/x-rst</div><div>Created: 8-Aug-2013</div><div>Python-Version: 3.4</div><div>Post-History: 8-Aug-2013, 28-Aug-2013, 18-Sep-2013</div><div>Resolution:</div>
<div><br></div><div><br></div><div>Abstract</div><div>========</div><div><br></div><div>This PEP proposes to add a new class to importlib.machinery called</div><div>"ModuleSpec". It will provide all the import-related information used</div>
<div>to load a module and will be available without needing to load the</div><div>module first. Finders will directly provide a module's spec instead of</div><div>a loader (which they will continue to provide indirectly). The import</div>
<div>machinery will be adjusted to take advantage of module specs, including</div><div>using them to load modules.</div><div><br></div><div><br></div><div>Terms and Concepts</div><div>==================</div><div><br></div>
<div>The changes in this proposal are an opportunity to make several</div><div>existing terms and concepts more clear, whereas currently they are</div><div>(unfortunately) ambiguous. New concepts are also introduced in this</div>
<div>proposal. Finally, it's worth explaining a few other existing terms</div><div>with which people may not be so familiar. For the sake of context, here</div><div>is a brief summary of all three groups of terms and concepts. A more</div>
<div>detailed explanation of the import system is found at</div><div>[import_system_docs]_.</div><div><br></div><div>finder</div><div>------</div><div><br></div><div>A "finder" is an object that identifies the loader that the import</div>
<div>system should use to load a module. Currently this is accomplished by</div><div>calling the finder's find_module() method, which returns the loader.</div><div><br></div><div>Finders are strictly responsible for providing the loader, which they do</div>
<div>through their find_module() method. The import system then uses that</div><div>loader to load the module.</div><div><br></div><div>loader</div><div>------</div><div><br></div><div>A "loader" is an object that is used to load a module during import.</div>
<div>Currently this is done by calling the loader's load_module() method. A</div><div>loader may also provide APIs for getting information about the modules</div><div>it can load, as well as about data from sources associated with such a</div>
<div>module.</div><div><br></div><div>Right now loaders (via load_module()) are responsible for certain</div><div>boilerplate import-related operations. These are:</div><div><br></div><div>1. perform some (module-related) validation;</div>
<div>2. create the module object;</div><div>3. set import-related attributes on the module;</div><div>4. "register" the module to sys.modules;</div><div>5. exec the module;</div><div>6. clean up in the event of failure while loading the module.</div>
<div><br></div><div>This all takes place during the import system's call to</div><div>Loader.load_module().</div><div><br></div><div>origin</div><div>------</div><div><br></div><div>This is a new term and concept. The idea of it exists subtly in the</div>
<div>import system already, but this proposal makes the concept explicit.</div><div><br></div><div>"origin" is the import context means the system (or resource within a</div><div>system) from which a module originates. For the purposes of this</div>
<div>proposal, "origin" is also a string which identifies such a resource or</div><div>system. "origin" is applicable to all modules.</div><div><br></div><div>For example, the origin for built-in and frozen modules is the</div>
<div>interpreter itself. The import system already identifies this origin as</div><div>"built-in" and "frozen", respectively. This is demonstrated in the</div><div>following module repr: "<module 'sys' (built-in)>".</div>
<div><br></div><div>In fact, the module repr is already a relatively reliable, though</div><div>implicit, indicator of a module's origin. Other modules also indicate</div><div>their origin through other means, as described in the entry for</div>
<div>"location".</div><div><br></div><div>It is up to the loader to decide on how to interpret and use a module's</div><div>origin, if at all.</div><div><br></div><div>location</div><div>--------</div><div><br>
</div><div>This is a new term. However the concept already exists clearly in the</div><div>import system, as associated with the ``__file__`` and ``__path__``</div><div>attributes of modules, as well as the name/term "path" elsewhere.</div>
<div><br></div><div>A "location" is a resource or "place", rather than a system at large,</div><div>from which a module is loaded. It qualifies as an "origin". Examples</div><div>of locations include filesystem paths and URLs. A location is</div>
<div>identified by the name of the resource, but may not necessarily identify</div><div>the system to which the resource pertains. In such cases the loader</div><div>would have to identify the system itself.</div><div><br>
</div><div>In contrast to other kinds of module origin, a location cannot be</div><div>inferred by the loader just by the module name. Instead, the loader</div><div>must be provided with a string to identify the location, usually by the</div>
<div>finder that generates the loader. The loader then uses this information</div><div>to locate the resource from which it will load the module. In theory</div><div>you could load the module at a given location under various names.</div>
<div><br></div><div>The most common example of locations in the import system are the</div><div>files from which source and extension modules are loaded. For these</div><div>modules the location is identified by the string in the ``__file__``</div>
<div>attribute. Although ``__file__`` isn't particularly accurate for some</div><div>modules (e.g. zipped), it is currently the only way that the import</div><div>system indicates that a module has a location.</div><div>
<br></div><div>A module that has a location may be called "locatable".</div><div><br></div><div>cache</div><div>-----</div><div><br></div><div>The import system stores compiled modules in the __pycache__ directory</div>
<div>as an optimization. This module cache that we use today was provided by</div><div>PEP 3147. For this proposal, the relevant API for module caching is the</div><div>``__cache__`` attribute of modules and the cache_from_source() function</div>
<div>in importlib.util. Loaders are responsible for putting modules into the</div><div>cache (and loading out of the cache). Currently the cache is only used</div><div>for compiled source modules. However, this proposal explicitly allows</div>
<div><br></div><div>package</div><div>-------</div><div><br></div><div>The concept does not change, nor does the term. However, the</div><div>distinction between modules and packages is mostly superficial.</div><div>Packages *are* modules. They simply have a ``__path__`` attribute and</div>
<div>import may add attributes bound to submodules. The typical perceived</div><div>difference is a source of confusion. This proposal explicitly</div><div>de-emphasizes the distinction between packages and modules where it</div>
<div>makes sense to do so.</div><div><br></div><div><br></div><div>Motivation</div><div>==========</div><div><br></div><div>The import system has evolved over the lifetime of Python. In late 2002</div><div>PEP 302 introduced standardized import hooks via finders and</div>
<div>loaders and sys.meta_path. The importlib module, introduced</div><div>with Python 3.1, now exposes a pure Python implementation of the APIs</div><div>described by PEP 302, as well as of the full import system. It is now</div>
<div>much easier to understand and extend the import system. While a benefit</div><div>to the Python community, this greater accessibilty also presents a</div><div>challenge.</div><div><br></div><div>As more developers come to understand and customize the import system,</div>
<div>any weaknesses in the finder and loader APIs will be more impactful. So</div><div>the sooner we can address any such weaknesses the import system, the</div><div>better...and there are a couple we can take care of with this proposal.</div>
<div><br></div><div>Firstly, any time the import system needs to save information about a</div><div>module we end up with more attributes on module objects that are</div><div>generally only meaningful to the import system. It would be nice to</div>
<div>have a per-module namespace in which to put future import-related</div><div>information and to pass around within the import system. Secondly,</div><div>there's an API void between finders and loaders that causes undue</div>
<div>complexity when encountered. The PEP 420 (namespace packages)</div><div>implementation had to work around this. The complexity surfaced again</div><div>during recent efforts on a separate proposal. [ref_files_pep]_</div>
<div><br></div><div>The `finder`_ and `loader`_ sections above detail current responsibility</div><div>of both. Notably, loaders are not required to provide any of the</div><div>functionality of their load_module() through other methods. Thus,</div>
<div>though the import-related information about a module is likely available</div><div>without loading the module, it is not otherwise exposed.</div><div><br></div><div>Furthermore, the requirements assocated with load_module() are</div>
<div>common to all loaders and mostly are implemented in exactly the same</div><div>way. This means every loader has to duplicate the same boilerplate</div><div>code. importlib.util provides some tools that help with this, but</div>
<div>it would be more helpful if the import system simply took charge of</div><div>these responsibilities. The trouble is that this would limit the degree</div><div>of customization that load_module() could easily continue to facilitate.</div>
<div><br></div><div>More importantly, While a finder *could* provide the information that</div><div>the loader's load_module() would need, it currently has no consistent</div><div>way to get it to the loader. This is a gap between finders and loaders</div>
<div>which this proposal aims to fill.</div><div><br></div><div>Finally, when the import system calls a finder's find_module(), the</div><div>finder makes use of a variety of information about the module that is</div>
<div>useful outside the context of the method. Currently the options are</div><div>limited for persisting that per-module information past the method call,</div><div>since it only returns the loader. Popular options for this limitation</div>
<div>are to store the information in a module-to-info mapping somewhere on</div><div>the finder itself, or store it on the loader.</div><div><br></div><div>Unfortunately, loaders are not required to be module-specific. On top</div>
<div>of that, some of the useful information finders could provide is</div><div>common to all finders, so ideally the import system could take care of</div><div>those details. This is the same gap as before between finders and</div>
<div>loaders.</div><div><br></div><div>As an example of complexity attributable to this flaw, the</div><div>implementation of namespace packages in Python 3.3 (see PEP 420) added</div><div>FileFinder.find_loader() because there was no good way for</div>
<div>find_module() to provide the namespace search locations.</div><div><br></div><div>The answer to this gap is a ModuleSpec object that contains the</div><div>per-module information and takes care of the boilerplate functionality</div>
<div>involved with loading the module.</div><div><br></div><div><br></div><div>Specification</div><div>=============</div><div><br></div><div>The goal is to address the gap between finders and loaders while</div><div>changing as little of their semantics as possible. Though some</div>
<div>functionality and information is moved to the new ModuleSpec type,</div><div>their behavior should remain the same. However, for the sake of clarity</div><div>the finder and loader semantics will be explicitly identified.</div>
<div><br></div><div>Here is a high-level summary of the changes described by this PEP. More</div><div>detail is available in later sections.</div><div><br></div><div>importlib.machinery.ModuleSpec (new)</div><div>------------------------------------</div>
<div><br></div><div>A specification for a module's import-system-related state. See the</div><div>`ModuleSpec`_ section below for a more detailed description.</div><div><br></div><div>* ModuleSpec(name, loader, \*, origin=None, loader_state=None, is_package=None)</div>
<div><br></div><div>Attributes:</div><div><br></div><div>* name - a string for the name of the module.</div><div>* loader - the loader to use for loading.</div><div>* origin - the name of the place from which the module is loaded,</div>
<div> e.g. "builtin" for built-in modules and the filename for modules</div><div> loaded from source.</div><div>* submodule_search_locations - list of strings for where to find</div><div> submodules, if a package (None otherwise).</div>
<div>* loader_state - a container of extra module-specific data for use</div><div> during loading.</div><div>* cached (property) - a string for where the compiled module should be</div><div> stored.</div><div>* parent (RO-property) - the name of the package to which the module</div>
<div> belongs as a submodule (or None).</div><div>* has_location (RO-property) - a flag indicating whether or not the</div><div> module's "origin" attribute refers to a location.</div><div><br></div><div>Instance Methods:</div>
<div><br></div><div>* module_repr() - provide a repr string for the spec'ed module;</div><div> non-locatable modules will use their origin (e.g. "built-in").</div><div>* init_module_attrs(module) - set any of a module's import-related</div>
<div> attributes that aren't already set.</div><div><br></div><div>importlib.util Additions</div><div>------------------------</div><div><br></div><div>These are ModuleSpec factory functions, meant as a convenience for</div>
<div>finders. See the `Factory Functions`_ section below for more detail.</div><div><br></div><div>* spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None)</div><div> - build a spec from file-oriented information and loader APIs.</div>
<div>* from_loader(name, loader, \*, origin=None, is_package=None) - build</div><div> a spec with missing information filled in by using loader APIs.</div><div><br></div><div>This factory function is useful for some backward-compatibility</div>
<div>situations:</div><div><br></div><div>* spec_from_module(module, loader=None) - build a spec based on the</div><div> import-related attributes of an existing module.</div><div><br></div><div>Other API Additions</div>
<div>-------------------</div><div><br></div><div>* importlib.find_spec(name, path=None) will work exactly the same as</div><div> importlib.find_loader() (which it replaces), but return a spec instead</div><div> of a loader.</div>
<div><br></div><div>For loaders:</div><div><br></div><div>* importlib.abc.Loader.exec_module(module) will execute a module in its</div><div> own namespace. It replaces importlib.abc.Loader.load_module(), taking</div><div>
over its module execution functionality.</div><div>* importlib.abc.Loader.create_module(spec) (optional) will return the</div><div> module to use for loading.</div><div><br></div><div>For modules:</div><div><br></div><div>
* Module objects will have a new attribute: ``__spec__``.</div><div><br></div><div>API Changes</div><div>-----------</div><div><br></div><div>* InspectLoader.is_package() will become optional.</div><div><br></div><div>Deprecations</div>
<div>------------</div><div><br></div><div>* importlib.abc.MetaPathFinder.find_module()</div><div>* importlib.abc.PathEntryFinder.find_module()</div><div>* importlib.abc.PathEntryFinder.find_loader()</div><div>* importlib.abc.Loader.load_module()</div>
<div>* importlib.abc.Loader.module_repr()</div><div>* The parameters and attributes of the various loaders in</div><div> importlib.machinery</div><div>* importlib.util.set_package()</div><div>* importlib.util.set_loader()</div>
<div>* importlib.find_loader()</div><div><br></div><div>Removals</div><div>--------</div><div><br></div><div>These were introduced prior to Python 3.4's release, so they can simply</div><div>be removed.</div><div><br>
</div><div>* importlib.abc.Loader.init_module_attrs()</div><div>* importlib.util.module_to_load()</div><div><br></div><div>Other Changes</div><div>-------------</div><div><br></div><div>* The import system implementation in importlib will be changed to make</div>
<div> use of ModuleSpec.</div><div>* importlib.reload() will make use of ModuleSpec.</div><div>* Import-related module attributes (other than ``__spec__``) will no</div><div> longer be used directly by the import system.</div>
<div>* Import-related attributes should no longer be added to modules</div><div> directly.</div><div>* The module type's ``__repr__()`` will be a thin wrapper around a pure</div><div> Python implementation which will leverage ModuleSpec.</div>
<div>* The spec for the ``__main__`` module will reflect the appropriate</div><div> name and origin.</div><div><br></div><div>Backward-Compatibility</div><div>----------------------</div><div><br></div><div>* If a finder does not define find_spec(), a spec is derived from</div>
<div> the loader returned by find_module().</div><div>* PathEntryFinder.find_loader() still takes priority over</div><div> find_module().</div><div>* Loader.load_module() is used if exec_module() is not defined.</div><div>
<br></div><div>What Will not Change?</div><div>---------------------</div><div><br></div><div>* The syntax and semantics of the import statement.</div><div>* Existing finders and loaders will continue to work normally.</div>
<div>* The import-related module attributes will still be initialized with</div><div> the same information.</div><div>* Finders will still create loaders (now storing them in specs).</div><div>* Loader.load_module(), if a module defines it, will have all the</div>
<div> same requirements and may still be called directly.</div><div>* Loaders will still be responsible for module data APIs.</div><div>* importlib.reload() will still overwrite the import-related attributes.</div><div><br>
</div><div>Responsibilities</div><div>----------------</div><div><br></div><div>Here's a quick breakdown of where responsibilities lie after this PEP.</div><div><br></div><div>finders:</div><div><br></div><div>* create loader</div>
<div>* create spec</div><div><br></div><div>loaders:</div><div><br></div><div>* create module (optional)</div><div>* execute module</div><div><br></div><div>ModuleSpec:</div><div><br></div><div>* orchestrate module loading</div>
<div>* boilerplate for module loading, including managing sys.modules and</div><div> setting import-related attributes</div><div>* create module if loader doesn't</div><div>* call loader.exec_module(), passing in the module in which to exec</div>
<div>* contain all the information the loader needs to exec the module</div><div>* provide the repr for modules</div><div><br></div><div><br></div><div>What Will Existing Finders and Loaders Have to Do Differently?</div><div>
==============================================================</div><div><br></div><div>Immediately? Nothing. The status quo will be deprecated, but will</div><div>continue working. However, here are the things that the authors of</div>
<div>finders and loaders should change relative to this PEP:</div><div><br></div><div>* Implement find_spec() on finders.</div><div>* Implement exec_module() on loaders, if possible.</div><div><br></div><div>The ModuleSpec factory functions in importlib.util are intended to be</div>
<div>helpful for converting existing finders. from_loader() and</div><div>from_file_location() are both straight-forward utilities in this</div><div>regard. In the case where loaders already expose methods for creating</div>
<div>and preparing modules, ModuleSpec.from_module() may be useful to</div><div>the corresponding finder.</div><div><br></div><div>For existing loaders, exec_module() should be a relatively direct</div><div>conversion from the non-boilerplate portion of load_module(). In some</div>
<div>uncommon cases the loader should also implement create_module().</div><div><br></div><div><br></div><div>ModuleSpec Users</div><div>================</div><div><br></div><div>ModuleSpec objects have 3 distinct target audiences: Python itself,</div>
<div>import hooks, and normal Python users.</div><div><br></div><div>Python will use specs in the import machinery, in interpreter startup,</div><div>and in various standard library modules. Some modules are</div><div>import-oriented, like pkgutil, and others are not, like pickle and</div>
<div>pydoc. In all cases, the full ModuleSpec API will get used.</div><div><br></div><div>Import hooks (finders and loaders) will make use of the spec in specific</div><div>ways. First of all, finders may use the spec factory functions in</div>
<div>importlib.util to create spec objects. They may also directly adjust</div><div>the spec attributes after the spec is created. Secondly, the finder may</div><div>bind additional information to the spec (in finder_extras) for the</div>
<div>loader to consume during module creation/execution. Finally, loaders</div><div>will make use of the attributes on a spec when creating and/or executing</div><div>a module.</div><div><br></div><div>Python users will be able to inspect a module's ``__spec__`` to get</div>
<div>import-related information about the object. Generally, Python</div><div>applications and interactive users will not be using the ``ModuleSpec``</div><div>factory functions nor any the instance methods.</div><div><br>
</div><div><br></div><div>How Loading Will Work</div><div>=====================</div><div><br></div><div>This is an outline of what happens in ModuleSpec's loading</div><div>functionality::</div><div><br></div><div> def load(spec):</div>
<div> if not hasattr(spec.loader, 'exec_module'):</div><div> module = spec.loader.load_module(<a href="http://spec.name">spec.name</a>)</div><div> spec.init_module_attrs(module)</div><div>
return sys.modules[<a href="http://spec.name">spec.name</a>]</div><div><br></div><div> module = None</div><div> if hasattr(spec.loader, 'create_module'):</div><div> module = spec.loader.create_module(spec)</div>
<div> if module is None:</div><div> module = ModuleType(<a href="http://spec.name">spec.name</a>)</div><div> spec.init_module_attrs(module)</div><div><br></div><div> sys.modues[<a href="http://spec.name">spec.name</a>] = module</div>
<div> try:</div><div> spec.loader.exec_module(module)</div><div> except Exception:</div><div> del sys.modules[<a href="http://spec.name">spec.name</a>]</div><div> raise</div><div>
return sys.modules[<a href="http://spec.name">spec.name</a>]</div><div><br></div><div>These steps are exactly what Loader.load_module() is already</div><div>expected to do. Loaders will thus be simplified since they will only</div>
<div>need to implement exec_module().</div><div><br></div><div>Note that we must return the module from sys.modules. During loading</div><div>the module may have replaced itself in sys.modules. Since we don't have</div>
<div>a post-import hook API to accommodate the use case, we have to deal with</div><div>it. However, in the replacement case we do not worry about setting the</div><div>import-related module attributes on the object. The module writer is on</div>
<div>their own if they are doing this.</div><div><br></div><div><br></div><div>ModuleSpec</div><div>==========</div><div><br></div><div>Attributes</div><div>----------</div><div><br></div><div>Each of the following names is an attribute on ModuleSpec objects. A</div>
<div>value of None indicates "not set". This contrasts with module</div><div>objects where the attribute simply doesn't exist. Most of the</div><div>attributes correspond to the import-related attributes of modules. Here</div>
<div>is the mapping. The reverse of this mapping is used by</div><div>ModuleSpec.init_module_attrs().</div><div><br></div><div>========================== ==============</div><div>On ModuleSpec On Modules</div>
<div>========================== ==============</div><div>name __name__</div><div>loader __loader__</div><div>package __package__</div><div>origin __file__*</div>
<div>cached __cached__*,**</div><div>submodule_search_locations __path__**</div><div>loader_state \-</div><div>has_location \-</div><div>========================== ==============</div>
<div><br></div><div>| \* Set on the module only if spec.has_location is true.</div><div>| \*\* Set on the module only if the spec attribute is not None.</div><div><br></div><div>While package and has_location are read-only properties, the remaining</div>
<div>attributes can be replaced after the module spec is created and even</div><div>after import is complete. This allows for unusual cases where directly</div><div>modifying the spec is the best option. However, typical use should not</div>
<div>involve changing the state of a module's spec.</div><div><br></div><div>**origin**</div><div><br></div><div>"origin" is a string for the name of the place from which the module</div><div>originates. See `origin`_ above. Aside from the informational value,</div>
<div>it is also used in module_repr(). In the case of a spec where</div><div>"has_location" is true, ``__file__`` is set to the value of "origin".</div><div>For built-in modules "origin" would be set to "built-in".</div>
<div><br></div><div>**has_location**</div><div><br></div><div>As explained in the `location`_ section above, many modules are</div><div>"locatable", meaning there is a corresponding resource from which the</div>
<div>module will be loaded and that resource can be described by a string.</div><div>In contrast, non-locatable modules can't be loaded in this fashion, e.g.</div><div>builtin modules and modules dynamically created in code. For these, the</div>
<div>name is the only way to access them, so they have an "origin" but not a</div><div>"location".</div><div><br></div><div>"has_location" is true if the module is locatable. In that case the</div>
<div>spec's origin is used as the location and ``__file__`` is set to</div><div>spec.origin. If additional location information is required (e.g.</div><div>zipimport), that information may be stored in spec.loader_state.</div>
<div><br></div><div>"has_location" may be implied from the existence of a load_data() method</div><div>on the loader.</div><div><br></div><div>Incidently, not all locatable modules will be cachable, but most will.</div>
<div><br></div><div>**submodule_search_locations**</div><div><br></div><div>The list of location strings, typically directory paths, in which to</div><div>search for submodules. If the module is a package this will be set to</div>
<div>a list (even an empty one). Otherwise it is None.</div><div><br></div><div>The name of the corresponding module attribute, ``__path__``, is</div><div>relatively ambiguous. Instead of mirroring it, we use a more explicit</div>
<div>name that makes the purpose clear.</div><div><br></div><div>**loader_state**</div><div><br></div><div>A finder may set loader_state to any value to provide additional</div><div>data for the loader to use during loading. A value of None is the</div>
<div>default and indicates that there is no additional data. Otherwise it</div><div>can be set to any object, such as a dict, list, or</div><div>types.SimpleNamespace, containing the relevant extra information.</div><div>
<br></div><div>For example, zipimporter could use it to pass the zip archive name</div><div>to the loader directly, rather than needing to derive it from origin</div><div>or create a custom loader for each find operation.</div>
<div><br></div><div>loader_state is meant for use by the finder and corresponding loader.</div><div>It is not guaranteed to be a stable resource for any other use.</div><div><br></div><div>Factory Functions</div><div>-----------------</div>
<div><br></div><div>**spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None)**</div><div><br></div><div>Build a spec from file-oriented information and loader APIs.</div><div><br></div><div>
* "origin" will be set to the location.</div><div>* "has_location" will be set to True.</div><div>* "cached" will be set to the result of calling cache_from_source().</div><div><br></div><div>
* "origin" can be deduced from loader.get_filename() (if "location" is</div><div> not passed in.</div><div>* "loader" can be deduced from suffix if the location is a filename.</div><div>* "submodule_search_locations" can be deduced from loader.is_package()</div>
<div> and from os.path.dirname(location) if locatin is a filename.</div><div><br></div><div>**from_loader(name, loader, \*, origin=None, is_package=None)**</div><div><br></div><div>Build a spec with missing information filled in by using loader APIs.</div>
<div><br></div><div>* "has_location" can be deduced from loader.get_data.</div><div>* "origin" can be deduced from loader.get_filename().</div><div>* "submodule_search_locations" can be deduced from loader.is_package()</div>
<div> and from os.path.dirname(location) if locatin is a filename.</div><div><br></div><div>**spec_from_module(module, loader=None)**</div><div><br></div><div>Build a spec based on the import-related attributes of an existing</div>
<div>module. The spec attributes are set to the corresponding import-</div><div>related module attributes. See the table in `Attributes`_.</div><div><br></div><div>Omitted Attributes and Methods</div><div>------------------------------</div>
<div><br></div><div>The following ModuleSpec methods are not part of the public API since</div><div>it is easy to use them incorrectly and only the import system really</div><div>needs them (i.e. they would be an attractive nuisance).</div>
<div><br></div><div>* _create() - provide a new module to use for loading.</div><div>* _exec(module) - execute the spec into a module namespace.</div><div>* _load() - prepare a module and execute it in a protected way.</div>
<div>* _reload(module) - re-execute a module in a protected way.</div><div><br></div><div>Here are other omissions:</div><div><br></div><div>There is no "PathModuleSpec" subclass of ModuleSpec that separates out</div>
<div>has_location, cached, and submodule_search_locations. While that might</div><div>make the separation cleaner, module objects don't have that distinction.</div><div>ModuleSpec will support both cases equally well.</div>
<div><br></div><div>While "is_package" would be a simple additional attribute (aliasing</div><div>self.submodule_search_locations is not None), it perpetuates the</div><div>artificial (and mostly erroneous) distinction between modules and</div>
<div>packages.</div><div><br></div><div>Conceivably, a ModuleSpec.load() method could optionally take a list of</div><div>modules with which to interact instead of sys.modules. That</div><div>capability is left out of this PEP, but may be pursued separately at</div>
<div>some other time, including relative to PEP 406 (import engine).</div><div><br></div><div>Likewise load() could be leveraged to implement multi-version</div><div>imports. While interesting, doing so is outside the scope of this</div>
<div>proposal.</div><div><br></div><div>Others:</div><div><br></div><div>* Add ModuleSpec.submodules (RO-property) - returns possible submodules</div><div> relative to the spec.</div><div>* Add ModuleSpec.loaded (RO-property) - the module in sys.module, if</div>
<div> any.</div><div>* Add ModuleSpec.data - a descriptor that wraps the data API of the</div><div> spec's loader.</div><div>* Also see [cleaner_reload_support]_.</div><div><br></div><div><br></div><div>Backward Compatibility</div>
<div>----------------------</div><div><br></div><div>ModuleSpec doesn't have any. This would be a different story if</div><div>Finder.find_module() were to return a module spec instead of loader.</div><div>In that case, specs would have to act like the loader that would have</div>
<div>been returned instead. Doing so would be relatively simple, but is an</div><div>unnecessary complication. It was part of earlier versions of this PEP.</div><div><br></div><div>Subclassing</div><div>-----------</div>
<div><br></div><div>Subclasses of ModuleSpec are allowed, but should not be necessary.</div><div>Simply setting loader_state or adding functionality to a custom</div><div>finder or loader will likely be a better fit and should be tried first.</div>
<div>However, as long as a subclass still fulfills the requirements of the</div><div>import system, objects of that type are completely fine as the return</div><div>value of Finder.find_spec().</div><div><br></div><div><br>
</div><div>Existing Types</div><div>==============</div><div><br></div><div>Module Objects</div><div>--------------</div><div><br></div><div>Other than adding ``__spec__``, none of the import-related module</div><div>attributes will be changed or deprecated, though some of them could be;</div>
<div>any such deprecation can wait until Python 4.</div><div><br></div><div>A module's spec will not be kept in sync with the corresponding import-</div><div>related attributes. Though they may differ, in practice they will</div>
<div>typically be the same.</div><div><br></div><div>One notable exception is that case where a module is run as a script by</div><div>using the ``-m`` flag. In that case ``module.__spec__.name`` will</div><div>reflect the actual module name while ``module.__name__`` will be</div>
<div>``__main__``.</div><div><br></div><div>A module's spec is not guaranteed to be identical between two modules</div><div>with the same name. Likewise there is no guarantee that successive</div><div>calls to importlib.find_spec() will return the same object or even an</div>
<div>equivalent object, though at least the latter is likely.</div><div><br></div><div>Finders</div><div>-------</div><div><br></div><div>Finders are still responsible for creating the loader. That loader will</div><div>
now be stored in the module spec returned by find_spec() rather</div><div>than returned directly. As is currently the case without the PEP, if a</div><div>loader would be costly to create, that loader can be designed to defer</div>
<div>the cost until later.</div><div><br></div><div>**MetaPathFinder.find_spec(name, path=None)**</div><div><br></div><div>**PathEntryFinder.find_spec(name)**</div><div><br></div><div>Finders will return ModuleSpec objects when find_spec() is</div>
<div>called. This new method replaces find_module() and</div><div>find_loader() (in the PathEntryFinder case). If a loader does</div><div>not have find_spec(), find_module() and find_loader() are</div><div>used instead, for backward-compatibility.</div>
<div><br></div><div>Adding yet another similar method to loaders is a case of practicality.</div><div>find_module() could be changed to return specs instead of loaders.</div><div>This is tempting because the import APIs have suffered enough,</div>
<div>especially considering PathEntryFinder.find_loader() was just</div><div>added in Python 3.3. However, the extra complexity and a less-than-</div><div>explicit method name aren't worth it.</div><div><br></div><div>
Loaders</div><div>-------</div><div><br></div><div>**Loader.exec_module(module)**</div><div><br></div><div>Loaders will have a new method, exec_module(). Its only job</div><div>is to "exec" the module and consequently populate the module's</div>
<div>namespace. It is not responsible for creating or preparing the module</div><div>object, nor for any cleanup afterward. It has no return value.</div><div>exec_module() will be used during both loading and reloading.</div>
<div><br></div><div>exec_module() should properly handle the case where it is called more</div><div>than once. For some kinds of modules this may mean raising ImportError</div><div>every time after the first time the method is called. This is</div>
<div>particularly relevant for reloading, where some kinds of modules do not</div><div>support in-place reloading.</div><div><br></div><div>**Loader.create_module(spec)**</div><div><br></div><div>Loaders may also implement create_module() that will return a</div>
<div>new module to exec. It may return None to indicate that the default</div><div>module creation code should be used. One use case, though atypical, for</div><div>create_module() is to provide a module that is a subclass of the builtin</div>
<div>module type. Most loaders will not need to implement create_module(),</div><div><br></div><div>create_module() should properly handle the case where it is called more</div><div>than once for the same spec/module. This may include returning None or</div>
<div>raising ImportError.</div><div><br></div><div>.. note::</div><div><br></div><div> exec_module() and create_module() should not set any import-related</div><div> module attributes. The fact that load_module() does is a design flaw</div>
<div> that this proposal aims to correct.</div><div><br></div><div>Other changes:</div><div><br></div><div>PEP 420 introduced the optional module_repr() loader method to limit</div><div>the amount of special-casing in the module type's ``__repr__()``. Since</div>
<div>this method is part of ModuleSpec, it will be deprecated on loaders.</div><div>However, if it exists on a loader it will be used exclusively.</div><div><br></div><div>Loader.init_module_attr() method, added prior to Python 3.4's</div>
<div>release , will be removed in favor of the same method on ModuleSpec.</div><div><br></div><div>However, InspectLoader.is_package() will not be deprecated even</div><div>though the same information is found on ModuleSpec. ModuleSpec</div>
<div>can use it to populate its own is_package if that information is</div><div>not otherwise available. Still, it will be made optional.</div><div><br></div><div>One consequence of ModuleSpec is that loader ``__init__`` methods will</div>
<div>no longer need to accommodate per-module state. The path-based loaders</div><div>in importlib take arguments in their ``__init__()`` and have</div><div>corresponding attributes. However, the need for those values is</div>
<div>eliminated by module specs.</div><div><br></div><div>In addition to executing a module during loading, loaders will still be</div><div>directly responsible for providing APIs concerning module-related data.</div><div>
<br></div><div><br></div><div>Other Changes</div><div>=============</div><div><br></div><div>* The various finders and loaders provided by importlib will be</div><div> updated to comply with this proposal.</div><div>* The spec for the ``__main__`` module will reflect how the interpreter</div>
<div> was started. For instance, with ``-m`` the spec's name will be that</div><div> of the run module, while ``__main__.__name__`` will still be</div><div> "__main__".</div><div>* We add importlib.find_spec() to mirror</div>
<div> importlib.find_loader() (which becomes deprecated).</div><div>* importlib.reload() is changed to use ModuleSpec.load().</div><div>* importlib.reload() will now make use of the per-module import</div><div> lock.</div>
<div><br></div><div><br></div><div>Reference Implementation</div><div>========================</div><div><br></div><div>A reference implementation will be available at</div><div><a href="http://bugs.python.org/issue18864">http://bugs.python.org/issue18864</a>.</div>
<div><br></div><div><br></div><div>Open Issues</div><div>==============</div><div><br></div><div>\* The impact of this change on pkgutil (and setuptools) needs looking</div><div>into. It has some generic function-based extensions to PEP 302. These</div>
<div>may break if importlib starts wrapping loaders without the tools'</div><div>knowledge.</div><div><br></div><div>\* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc,</div><div>inspect.</div><div><br>
</div><div>For instance, pickle should be updated in the ``__main__`` case to look</div><div>at ``module.__spec__.name``.</div><div><br></div><div>\* Impact on some kinds of lazy loading modules. [lazy_import_concerns]_</div>
<div><br></div><div><br></div><div>References</div><div>==========</div><div><br></div><div>.. [ref_files_pep] <a href="http://mail.python.org/pipermail/import-sig/2013-August/000658.html">http://mail.python.org/pipermail/import-sig/2013-August/000658.html</a></div>
<div><br></div><div>.. [import_system_docs] <a href="http://docs.python.org/3/reference/import.html">http://docs.python.org/3/reference/import.html</a></div><div><br></div><div>.. [cleaner_reload_support] <a href="https://mail.python.org/pipermail/import-sig/2013-September/000735.html">https://mail.python.org/pipermail/import-sig/2013-September/000735.html</a></div>
<div><br></div><div>.. [lazy_import_concerns] <a href="https://mail.python.org/pipermail/python-dev/2013-August/128129.html">https://mail.python.org/pipermail/python-dev/2013-August/128129.html</a></div><div><br></div><div>
<br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div><div><br></div><div> </div><div>..</div><div> Local Variables:</div><div> mode: indented-text</div>
<div> indent-tabs-mode: nil</div><div> sentence-end-double-space: t</div><div> fill-column: 70</div><div> coding: utf-8</div><div> End:</div><div><br></div></div></div>