<div dir="ltr">If this can lead to the deprecation of .pth files then I support the idea, but I think there are technical issues in terms of implementation that have not been throught through yet. This is going to require an implementation (even if it isn't in importlib._bootstrap but as a subclass of importlib.machinery.FileFinder or something) to see how you plan to make all of this work before this PEP can move beyond this SIG.<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 18, 2013 at 6:10 PM, Eric Snow <span dir="ltr"><<a href="mailto:ericsnowcurrently@gmail.com" target="_blank">ericsnowcurrently@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<br><br>Nick talked me into writing this PEP, so blame him for the idea. <wink> I haven't had a chance to polish it up, but the content communicates the proposal well enough to post here. Let me know what you think. Once some concensus is reached I'll commit the PEP and post to python-dev. I have a rough implementation that'll I'll put online when I get a chance.<br>
<br>If Guido is not interested maybe Brett would like to be BDFL-Delegate. :)<br><br>-eric<br><br><br><div>PEP: 4XX</div><div>Title: Per-Module Import Path</div><div>Version: $Revision$</div><div>Last-Modified: $Date$</div>
<div>Author: Eric Snow <<a href="mailto:ericsnowcurrently@gmail.com" target="_blank">ericsnowcurrently@gmail.com</a>></div><div> Nick Coghlan <<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></div>
<div>BDFL-Delegate: ???</div>
<div>Discussions-To: <a href="mailto:import-sig@python.org" target="_blank">import-sig@python.org</a></div><div>Status: Draft</div><div>Type: Standards Track</div><div>Content-Type: text/x-rst</div><div>Created: 17-Jul-2013</div>
<div>Python-Version: 3.4</div>
<div>Post-History: 18-Jul-2013</div><div>Resolution:</div><div><br></div><div><br></div><div>Abstract</div><div>=======</div><div><br></div><div>Path-based import of a module or package involves traversing ``sys.path``</div>
<div>or a package path to locate the appropriate file or directory(s).</div><div>Redirecting from there to other locations is useful for packaging and</div><div>for virtual environments. However, in practice such redirection is</div>
<div>currently either `limited or fragile <Existing Alternatives>`_.</div><div><br></div><div>This proposal provides a simple filesystem-based method to redirect from</div><div>the normal module search path to other locations recognized by the</div>
<div>import system. This involves one change to path-based imports, adds one</div><div>import-related file type, and introduces a new module attribute. One</div><div>consequence of this PEP is the deprecation of ``.pth`` files.</div>
<div><br></div><div><br></div><div>Motivation</div><div>=========</div><div><br></div><div>One of the problems with virtual environments is that you are likely to</div><div>end up with duplicate installations of lots of common packages, and</div>
<div>keeping them up to date can be a pain.</div><div><br></div><div>One of the problems with archive-based distribution is that it can be</div></div></blockquote><div><br></div><div>You say "One of the problems" at the start of 3/4 of the paragraphs in this section. Variety is the spice of life. =) Try "Another problem is that for archive-based", etc.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>tricky to register the archive as a Python path entry when needed</div><div>
without polluting the path of applications that don't need it.</div></div></blockquote><div><br></div><div>How is this unique to archive-based distributions compared to any other scenario where all distributions are blindly added to sys.path?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>One of the problems with working directly from a source checkout is</div>
<div>getting the relevant source directories onto the Python path, especially</div>
<div>when you have multiple namespace package fragments spread across several</div><div>subdirectories of a large repository.</div><div><br></div></div></blockquote><div><br></div><div>E.g., a source checkout for the coverage.py project might be stored in the directory ``coveragepy``, but the actual source code is stored in ``coveragepy/coverage``, requiring ``coveragepy`` to be on sys.path in order to access the package.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>The `current solutions <Existing Alternatives>`_ all have their flaws.</div>
<div>Reference files are intended to address those deficiencies.</div><div><br></div><div><br></div><div>Specification</div><div>===========</div><div><br></div><div>Change to the Import System</div><div>-----------------------------</div>
<div><br></div><div>Currently, during `path-based import` of a module, the following happens for each `path entry` of `sys.path` or of the `__path__` of the module's parent:</div><div><br></div><div>1. look for `<path entry>/<name>/__init__.py` (and other supported suffixes),</div>
<div> * return `loader`;</div><div>2. look for `<path entry>/<name>.py` (and other supported suffixes),</div><div> * return loader;</div><div>3. look for `<path entry>/<name>/`,</div><div> * extend namespace portions path.</div>
<div><br></div></div></blockquote><div><br></div><div>Please capitalize the first letter of each bullet point (here and the rest of the PEP). Reads better since they are each separate sentences.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div></div><div>Once the path is exhausted, if no `loader` was found and the `namespace portions` path is non-empty, then a `NamespaceLoader` is returned with that path.</div><div><br></div><div>This proposal inserts a step before step 1 for each `path entry`:</div>
<div><br></div><div>0. look for `<path entry>/<name>.ref`</div></div></blockquote><div><br></div><div>Why .ref? Why not .path?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div> a. get loader for `<fullname>` (absolute module name) using path found in `.ref` file (see below) using `the normal mechanism`[link to language reference],</div>
<div> * stop processing the path entry if `.ref` file is empty;</div></div></blockquote><div><br></div><div>You should clarify how you plan to "get loader". You will have to find the proper finder as well in case the .ref file references a zip file or something which requires a different finder than the one which came across the .ref file.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div> b. check for `NamespaceLoader`,</div><div> * extend namespace portions path;</div>
<div> c. otherwise, return loader.</div><div><br></div><div>
Note the following consequences:</div><div><br></div><div>* if a ref file is found, it takes precedence over module files and package directories under the same path entry (see `Empty Ref Files as Markers`_);</div><div>* that holds for empty ref files also;</div>
<div>* the loader for a ref file, if any, comes from the full import system (i.e. `sys.meta_path`) rather than just the path-based import system;</div><div>* `.ref` files can indirectly provide fragments for namespace packages.</div>
</div></blockquote><div><br></div><div>This ramification for namespace packages make the changed semantic proposal a bit trickier than you are suggesting since you are essentially doing recursive path entry search. And is that possible? If I have a .ref file that refers to a path which itself has a .ref will that then lead to another search? I mean it seems like you going to be doing ``return importlib.find_loader(fullname, paths_found_in_ref) if paths_found_in_ref else None, []`` from within a finder which finds a .ref file, which itself would support a recursive search.</div>
<div><br></div><div>Everything below should come before the import changes. It's hard to follow what is really be proposed for semantics without knowing e.g. .ref files can have 0 or more paths and just a single path, etc.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>Reference Files</div><div>---------------</div><div><br></div><div>A new kind of file will live alongside package directories and module source files: reference files. These files have the following characteristics:</div>
<div><br></div><div>* named `<module name>.ref` in contrast to `<module name>.py` (etc.) or `<module name>/`;</div><div>* placed under `sys.path` entries or package path (just like modules and packages).</div>
<div><br></div><div>Reference File Format</div><div>----------------------</div><div><br></div><div>The contents of a reference file will conform to the following format:</div><div><br></div><div>* contain zero or more path entries, just like sys.path;</div>
<div>* one path entry per line;</div><div>* path entry order is preserved;</div><div>* may contain comment lines starting with "#", which are ignored;</div><div>* may contain blank lines, which are ignored;</div>
<div>* must be UTF-8 encoded.</div><div><br></div><div>Directory Path Entries</div><div>----------------------</div><div><br></div><div>Directory names are by far the most common type of path entry. Here is how they are constrained in reference files:</div>
<div><br></div><div>* may be absolute or relative;</div><div>* must be forward slash separated regardless of platform;</div><div>* each must be the parent directory where the module will be looked for.</div><div><br></div>
<div>To be clear, reference files (just like `sys.path`) deliberately reference the *parent* directory to be searched (rather than the module or package directory). So they work transparently with `__pycache__` and allow searching for `.dist-info <PEP 376>`_ directories through them.</div>
<div><br></div><div>Relative directory names will be resolved based on the directory containing the ref file, rather than the current working directory. Allowing relative directory names allows you to include sensible ref files in a source repo.</div>
<div><br></div><div>Empty Ref Files as Markers</div><div>-----------------------------</div><div><br></div><div>Handling `.ref` files first allows for the use of empty ref files as markers to indicate "this is not the module you are looking for". Here are two situations where that helps.</div>
</div></blockquote><div><br></div><div>"Here" -> "There"</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<div><br></div><div>First, an empty ref file helps resolve conflicts between script names and package names. When the interpreter is started with a filename, the directory of that script is added to the front of `sys.path`. This may be a problem for later imports where the intended module or package is on a regular path entry.</div>
<div><br></div><div>If an import references the script's name, the file will get run again by the import system as a module (only `__main__` was added to `sys.modules` earlier) [PEP 395]_. This is a further problem if you meant to import a module or package in another path entry.</div>
<div><br></div><div>The presence of an empty ref file in the script's directory would essentially render it invisible to the import system. This problem and solution apply for all of the files or directories in the script's directory.</div>
<div><br></div><div>Second, the namespace package mechanism has a side-effect: a directory without a __init__.py may be incorrectly treated as a namespace package fragment. The presence of an empty ref file indicates such a directory should be ignored.</div>
<div><br></div><div>A Module Attribute to Expose Contributing Ref Files</div><div>---------------------------------------------</div><div><br></div><div>Knowing the origin of a module is important when tracking down problems, particularly import-related ones. Currently, that entails looking at `<module>.__file__` and `<module.__package__>.__path__` (or `sys.path`).</div>
<div><br></div><div>With this PEP there can be a chain of ref files in between the currently available path and a module's __file__. Having access to that list of ref files is important in order to determine why one file was selected over another as the origin for the module. When an unexpected file gets used for one of your imports, you'll care about this!</div>
<div><br></div><div>In order to facilitate that, modules will have a new attribute: `__indirect__`. It will be a tuple comprised of the chain of ref files, in order, used to locate the module's __file__. An empty tuple or with one item will be the most common case. An empty tuple indicates that no ref files were used to locate the module.</div>
</div></blockquote><div><br></div><div>This complicates things even further. How are you going to pass this info along a call chain through find_loader()? Are we going to have to add find_loader3() to support this (nasty side-effect of using tuples instead of types.SimpleNamespace for the return value)? Some magic second value or type from find_loader() which flags the values in the iterable are from a .ref file and not any other possible place? This requires an API change and there isn't any mention of how that would look or work.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<div><br></div><div>Examples</div><div>--------</div><div><br></div><div>XXX are these useful?</div></div></blockquote><div><br></div><div>Yes if you change this to pip or setuptools and and also make it so it shows how you could point to version-specific distributions.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Top-level module (`import spam`)::</div><div><br></div><div> ~/venvs/ham/python/site-packages/</div>
<div> spam.ref</div><div><br></div><div> spam.ref:</div><div> # use the system installed module</div><div> /python/site-packages</div><div><br></div><div> /python/site-packages:</div><div> spam.py</div>
<div><br></div><div> spam.__file__:</div><div> "/python/site-packages/spam.py"</div><div><br></div><div> spam.__indirect__:</div><div> ("~/venvs/ham/python/site-packages/spam.ref",)</div><div>
<br></div><div>Submodule (`python -m myproject.tests`)::</div><div><br></div><div> ~/myproject/</div><div> setup.py</div><div> tests/</div><div> __init__.py</div><div> __main__.py</div><div> myproject/</div>
<div> __init__.py</div><div> tests.ref</div><div><br></div><div> tests.ref:</div><div> ../</div><div><br></div><div> myproject.__indirect__:</div><div> ()</div><div><br></div><div> myproject.tests.__file__:</div>
<div> "~/myproject/tests/__init__.py"</div><div><br></div><div> myproject.tests.__indirect__:</div><div> ("~/myproject/myproject/tests.ref",)</div><div><br></div><div>Multiple Path Entries::</div>
<div><br></div><div> myproj/</div><div> __init__.py</div><div> mod.ref</div><div><br></div><div> mod.ref:</div><div> # fall back to the old one</div><div> /python/site-packages/mod-new/</div><div> /python/site-packages/mod-old/</div>
<div><br></div><div> /python/site-packages/</div><div> mod-old/</div><div> mod.py</div><div><br></div><div> myproj.mod.__file__:</div><div> "/python/site-packages/mod-old/mod.py"</div><div>
<br>
</div><div> myproj.mod.__indirect__:</div><div> ("myproj/mod.ref",)</div><div><br></div><div>Chained Ref Files::</div><div><br></div><div> venvs/ham/python/site-packages/</div><div> spam.ref</div><div>
<br></div><div> venvs/ham/python/site-packages/spam.ref:</div><div> # use the system installed module</div><div> /python/site-packages</div><div><br></div><div> /python/site-packages/</div><div> spam.ref</div>
<div><br></div><div> /python/site-packages/spam.ref:</div><div> # use the clone</div><div> ~/clones/myproj/</div><div><br></div><div> ~/clones/myproj/</div><div> spam.py</div><div><br></div><div> spam.__file__:</div>
<div> "~/clones/myproj/spam.py"</div><div><br></div><div> spam.__indirect__:</div><div> ("venvs/ham/python/site-packages/spam.ref", "/python/site-packages/spam.ref")</div><div><br>
</div><div>Reference Implementation</div><div>------------------------</div><div><br></div><div>A reference implementation is available at <TBD>.</div><div><br></div><div>XXX double-check zipimport support</div><div>
<br></div><div><br></div><div>Deprecation of .pth Files</div><div>=============================</div><div><br></div><div>The `site` module facilitates the composition of `sys.path`. As part of that, `.pth` files are processed and entries added to `sys.path`. Ref files are intended as a replacement.</div>
<div><br></div><div>XXX also deprecate .pkg files (see pkgutil.extend_path())?</div><div><br></div><div>Consequently, `.pth` files will be deprecated.</div></div></blockquote><div><br></div><div>Link to "Existing Alternatives" discussion as to why this deprecation is desired.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Deprecation Schedule</div><div>-------------------------</div>
<div><br></div><div>1. documented: 3.4,</div><div>2. warnings: 3.5 and 3.6,</div><div>3. removal: 3.7</div><div><br></div><div>XXX Deprecate sooner?</div><div><br></div><div><br></div><div>Existing Alternatives</div><div>
=================</div><div><br></div><div>.pth Files</div><div>----------</div><div><br></div><div>`*.pth` files have the problem that they're global: if you add them to `site-packages`, they will be processed at startup by *every* Python application run using that Python installation. </div>
</div></blockquote><div><br></div><div>"... thanks to them being processed by the site module instead of by the import system and individual finders."</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>This is an undesirable side effect of the way `*.pth` processing is defined, but can't be changed due to backwards compatibility issues.</div>
<div><br></div><div>Furthermore, `*.pth` files are processed at interpreter startup...</div></div></blockquote><div><br></div><div>That's a moot point; .ref files can be as well if they are triggered as part of an import.</div>
<div><br></div><div>A bigger concern is that they execute arbitrary Python code which could be viewed as an unexpected security risk. Some might complain about the difficulty then of loading non-standard importers, but that really should be the duty of the code performing the import and not the distribution itself; IOW I would argue that it is up to the user to get things in line to use a distribution in the format they choose to use it instead of the distribution dictating how it should be bundled.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>.egg-link files</div><div>--------------</div><div><br></div><div>`*.egg-link` files are much closer to the proposed `*.ref` files. The difference is that `*.egg-link` files are designed to work with `pkg_resources` and `distribution names`, while `*.ref files` are designed to work with package and module names as an automatic part of the import system.</div>
<div><br></div><div>Symlinks</div><div>---------</div><div><br></div><div>Actual symlinks have the problem that they aren't really practical on Windows, and also that they don't support non-versioned references to versioned `dist-info` directories.</div>
<div><br></div><div>Design Alternatives</div><div>===================</div><div><br></div><div>Ignore Empty Ref Files</div><div>----------------------</div><div><br></div><div>An empty ref file would be ignored rather than effectively stopping the processing of the path entry. This loses the benefits outlined above of empty ref files as markers.</div>
<div><br></div><div>ImportError for Empty Ref Files</div><div>-------------------------------</div><div><br></div><div>An empty ref file would result in an ImportError. The only benefit to this would be to disallow empty ref files and make it clear when they are encountered.</div>
<div><br></div><div>Handle Ref Files After Namespace Packages</div><div>-----------------------------------------</div><div><br></div><div>Rather than handling ref files first, they could be handled last. Thus they would have lower priority than namespace package fragments. This would be insignificantly more backward compatible. However, as with ignoring empty ref files, handling them last would prevent their use as markers for ignoring a path entry.</div>
<div><br></div><div>Send Ref File Path Through Path Import System Only</div><div>--------------------------------------------------</div><div><br></div><div>As indicated above, the path entries in a ref file are passed back through the metapath finders to get the loader. Instead we could use just the path-based import system. This would prevent metapath finders from having a chance to handle the module under a different path.</div>
<div><br></div><div>Restrict Ref File Path Entries to Directories</div><div>---------------------------------------------</div><div><br></div><div>Rather than allowing anything for the path entries in a ref file, they could be restricted to just directories. This is by far the common case. However, it would add complexity without any justification for not allowing metapath importers a chance at the module under a new path.</div>
<div><br></div><div>Restrict Directories in Ref File Path Entries to Absolute</div><div>---------------------------------------------------------</div><div><br></div><div>Directory path entries in ref files can be relative or absolute. Limiting to just absolute directory names would be an artificial change to existing constraints on path entries without any justification. Furthermore, it would prevent simple use of ref files in code bases relative to project roots.</div>
<div><br></div><div><br></div><div>Future Extensions</div><div>===============</div><div><br></div><div>Longer term, we should also allow *versioned* `*.ref` files that can be used to reference modules and packages that aren't available for ordinary import (since they don't follow the "name.ref" format), but are available to tools like `pkg_resources` to handle parallel installs of different versions.</div>
<div><br></div><div><br></div><div>References</div><div>==========</div><div><br></div><div>.. [0] ...</div><div> ()</div><div><br></div><div><br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div>
<div><br></div><div>^L</div><div>..</div><div> Local Variables:</div><div> mode: indented-text</div><div> indent-tabs-mode: nil</div><div> sentence-end-double-space: t</div><div> fill-column: 70</div><div> coding: utf-8</div>
<div> End:</div></div>
<br>_______________________________________________<br>
Import-SIG mailing list<br>
<a href="mailto:Import-SIG@python.org">Import-SIG@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/import-sig" target="_blank">http://mail.python.org/mailman/listinfo/import-sig</a><br>
<br></blockquote></div><br></div></div>