[Python-Dev] BDFL-Delegate appointments for several PEPs

[Cameron Simpson]

On 25Mar2019 03:52, Terry Reedy <tjreedy at udel.edu> wrote:
>On 3/25/2019 12:27 AM, Cameron Simpson wrote:
>>>>I was thinking about IDLE and its tangled web of circular inports, 
>>>>but I am now convinced that this change will not affect it.  Indeed, 
>>>>idlelib/pyshell.py already implements idea of the proposal, ending 
>>>>with
>>>>
>>>>if __name__ == "__main__":
>>>>    sys.modules['pyshell'] = sys.modules['__main__']
>>>>    main()
>>>
>>>After more investigation, I realized that to stop having duplicate 
>>>modulue:
>>>1. The alias should be 'idlelib.pyshell', not 'pyshell', at least 
>>>when imports are all absolute.
>>
>>The PEP499 patch effectively uses __main__.__spec__.name for the name 
>>of the alias. Does that simplify your issue?
[...]
>>What about (untested):
>>
>>    if __name__ == '__main__':
>>        if __spec__.name not in sys.modules:
>>            sys.modules[__spec__.name] = sys.modules['__main__']
>
>When I start pyshell in my master repository directory on windows with
>  python -m idlelib.pyshell
>__spec__.name is 'idlelib.pyshell, which I currently hard-coded.
>When I start with what should be equivalent
>  python f:/dev/3x/lib/idlelib/pyshell.py
>__spec__ is None and __spec__.name an attribute error.

Um, yes. I presume that since no "import" has been done, there's no 
import spec (.__spec__).

Clearly the above needs to accomodate this, possibly with a fallback 
guess. Is sniffing the end components of __file__ at all sane? ending in 
idlelib/pyshell.py or pyshell.py? Or is that just getting baroque?

I don't think these are strictly the same from some kind of purist 
viewpoint: the path might be anything - _is_ it reasonable to suppose 
that it has a module name (== importable/finding through the import 
path)?

>>>If I run python f:/dev/3x/lib/idlelib/pyshell.py, the PEP patch would 
>>>have to notice that pyshell is a module within idlelib and alias 
>>>'__main__' to 'idlelib.pyshell', not 'pyshell'.  Would the same be 
>>>true if within-package import were all relative?
>>
>>I think so because we're using .__spec__.name, which I though was 
>>post import name resolution.
>
>You must be doing something different when __spec__ is None ;-).  I 
>tested the patch and it does not raise AttributeError with the command 
>above.

Indeed. I may have fudged a bit when I said "The PEP499 patch 
effectively uses __main__.__spec__.name". It modifies runpy.py's 
_run_module_as_main function, and that is called for the "python -m 
module_name" invocation, so it can get the module spec because it has a 
module name.

So the patch doesn't have to cope with __spec__ being None.

As you say, __spec__ is None for "python path/to/file.py" so __spec__ 
isn't any use there. Apologies.

[...]
>>Test 3, with the pyshell.py sys.modules assignment commented out:
>>
>>    [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib ./python.exe -i -m idlelib.pyshell
[...]
>>    >>> sys.modules['__main__']
>>    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>>    >>> sys.modules['pyshell']
>>    Traceback (most recent call last):
>>      File "<stdin>", line 1, in <module>
>>    KeyError: 'pyshell'
>>    >>> sys.modules['idlelib.pyshell']
>>    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>>    >>> id(sys.modules['__main__'])
>>    4552379336
>>    >>> id(sys.modules['idlelib.pyshell'])
>>    4552379336
>>
>>Here we've got __main__ and idlelib.pyshell the same module and no 
>>'pyshell' in sys.modules.
>
>>I don't think I understand your "relative import" scenario.
>
>If files other that pyshell used relative 'import ./pyshell' instead 
>of absolute 'import idlelib.pyshell', would the sys.modules key still 
>be 'idlelib.pyshell' or 'pyshell'?   Which is to ask, would the alias 
>needed to avoid a second pyshell module still be 'idlelib.pyshell' or 
>'pyshell'?

Ok.

As I understand it Python 3 imports are absolute: without a leading dot 
a name is absolute, so "import pyshell" should install 
sys.module['pyshell'] _provided_ that 'pyshell' can be found in the 
module search path.

Conversely, an "import .pyshell" is an import relative to the current 
module's package name, equivalent to an import of the absolute path 
"package.name.pyshell", for whatever the package name is. So (a) you can 
only import '.pyshell' from within a package containing a 'pyshell.py' 
file and (b) you can't import import '.pyshell' if you're not in a 
package.

I stuffed a "test2.py" into the local idlelib like this:

    import sys
    print("running", __file__, __name__)
    print(repr(sorted(sys.modules)))
    print(repr(sys.paht))
    from pyshell import idle_showwarning
    print(repr(sorted(sys.modules)))

and fiddled with the "from pyshell import idle_showwarning" line.

(I'm presuming this is what you have in mind, since "import ./pyshell" 
elicits a syntax error.)

Using "./python.exe -m idlelib.test2":

  Plain "pyshell" gets an ImportError - no such module.

  Using ".pyshell" imports the pyshell module as "idlelib.pyshell" in 
  sys.modules.

Which was encouraging until I went "./python.exe Lib/idlelib/test2.py".  
This puts Lib/idlelib (as an absolute path) at the start of sys.path.

A plain "pyshell" import works and installs sys.modules['pyshell'].

Conversely, trying the ".pyshell" import gets:

  ModuleNotFoundError: No module named '__main__.pyshell'; '__main__' is not a package

So we can get 'pyshell' or 'idlelib.pyshell' into sys.modules depending 
how we invoke python.

HOWEVER, if you're importing the 'pyshell' from idlelib _as found in the 
module search path_, whether absolutely as 'idlelib.pyshell' or 
relatives as '.pyshell' from within the idlelib package, you should 
always get 'idlelib.pyshell' in the sys.modules map.

And I don't think you should need to worry about a circular import 
importing some top level name 'pyshell' because that's not using the 
idlelib package, so I'd argue it isn't your problem.

Thoughts?

Cheers,
Cameron Simpson <cs at cskk.id.au>