[Import-SIG] Eliminating implicit __main__ relative imports

[Nick Coghlan]

On 28 March 2017 at 20:25, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The gist of the idea is to ask what if, instead of doing:
>
>     sys.path.insert(0, <the script directory>)
>
> we instead did this little dance to add an anonymous top-level
> main-relative namespace package:
>
>     mod = mod.ModuleType("")
>     mod.__path__ = [<the script directory>]
>     sys.modules[""] = mod
>     __main__.__package__ = ""
>
> That's already enough to allow explicit relative imports via "mod =
> importlib.import_module('name', package='')", but a couple of sanity
> checks elsewhere in importlib guarding against empty module names
> would need to be relaxed to allow the "from . import name" syntax.

While this approach is cute & clever, I'm now leaning back towards the
approach I wrote up in http://bugs.python.org/issue29929#msg290689,
which would be to make *__main__ itself* a runtime pseudo-package by
setting `__main__.__path__` appropriately.

The big advantage of that approach is that it already "just works",
even at the syntactic level:

    $ echo "print(__name__)" > relative.py
    $ python3
    Python 3.5.3 (default, Mar 21 2017, 17:21:33)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> __path__ = [os.getcwd()]
    >>> from . import relative
    __main__.relative
    >>>

It's also going to be more compatible with other tools that work with
sys.modules and other interfaces, and expect packages to always have a
non-empty name. The backwards-and-forwards compatible check is then to
look for "__path__" in the module globals:

    if "__path__" in globals():
        from . import relative_module_name
    else:
        import relative_module_name

This is also robust against dunder-variables being set in the builtins
module, as I discovered today that that has "__package__" explicitly
set to the empty string.

That then leaves the question of what to do about the double-import
trap (http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-double-import-trap)
as any change along these lines will make all main relative imports
accessible under two names:

* their implicitly relative top-level import name
* their explicit relative name under the "__main__." namespace

As a first step, I think actually following through on this idea would
require also accepting PEP 499 so that modules executed via the -m
switch were added to sys.modules under both "__name__" and
"__spec__.name": https://www.python.org/dev/peps/pep-0499/

It would also mean that in cases where `__main__` is a pseudo-package,
we would bind it in sys.modules as both `__main__` and
`__main__.<filename>` (so "import __main__" and "from . import
<filename>" gave you the same answer).

Beyond that, rather than actively preventing double-import errors,
we'd just emit RuntimeWarning messages to make them easier to debug
when they happened:

* for `sys.main_path_entry`, we'd warn for any import satisfied from
that directory that *didn't* start with the `__main__.` prefix (e.g.
"Implicit relative import {name} from main path entry {dirpath}").
* we'd add a check to the import system that when populating __path__
for a package based __spec__.submodule_search_locations, we'd warn for
any entries that also appeared in sys.path ((e.g. "{name} package
directory {dirpath} also appears in sys.path")).

The latter check wouldn't be completely free, but it should be fast
relative to actual module imports (and we could build a dynamic cache
as a set if the linear search through sys.path turned out to be overly
slow).

The transition plan would then be to emit those warnings as
DeprecationWarning in 3.7, upgrade to a visible-by-default
RuntimeWarning in 3.8, and then *maybe* consider omitting
sys.main_path_entry from sys.path by default in 3.9.

I'm not going to pursue this idea further until after PEP 538 is
squared away, but it's currently seeming feasible to me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia