[Python-ideas] Running scripts with relative imports directly, was: Re: proposal: "python -m foo"

[Paul Sokolovsky]

Hello,

On Thu, 6 Aug 2015 13:57:19 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

[]

> >     "I'm -1 on this and on any other proposed twiddlings of the
> > __main__ machinery. The only use case seems to be running scripts
> > that happen to be living inside a module's directory, which I've
> > always seen as an antipattern. To make me change my mind you'd have
> > to convince me that it isn't."
> >
> >     <URL:https://mail.python.org/pipermail/python-3000/2007-April/006793.html>
> >
> > He doesn't describe (that I can find) what makes him think it's an
> > antipattern, so I'm not clear on how he expects to be convinced
> > it's a valid pattern.

While Nick's PEP-0395 lists enough things in Python import system which
may confuse casual users (and which thus should raise alarm for all
Python advocates, who think it's nice, easy-to-use language), I'd like
to elaborate on my particular usecase.

So, when you start a new "python library", you probably start it as a
single-file module. And it's easy to play with it for both you and your
users - just make another file in the same dir as your module, add
"import my_module" to it, voila. At some point, you may decide that
library is too big for a single file, and needs splitting. Which of
course means converting a module to a package. And of course, when you
import "utils", you want to be sure it imports your utils, not
something else, which means using relative imports.

But then suddenly, you no longer can drop your test scripts in the
same directory where code is (like you did it before), but need to drop
it level up, which may be not always convenient. And it would be one
thing if it required extra step to run scripts located inside package
dir (casual-user-in-me hunch feeling would be that PYTHONPATH needs to
be set to ..), but we talk about not being able to do it at all. 


> It's an anti-pattern because doing it fundamentally confuses the
> import system's internal state:
> https://www.python.org/dev/peps/pep-0395/#why-are-my-imports-broken

Excellent PEP, Nick, it took some time to read thru it and references,
but it answered all my questions. After reading it, it's hard to
disagree that namespace packages support, a simplification and
clarification in itself, conflicted and blocked an automagic way to
resolve imports confusion for the user. But then indeed a logical
solution is to give user's power to explicitly resolve this issue, if
implicit no longer can work - by letting -m accept relative module
paths, as you show below.

It's also a discovery for me that -m's functionality appears to be
handled largely on stdlib side using runpy module, and that's the main
purpose of that module. I'll look into prototyping relative import
support when I have time.

And Nick, if you count votes for reviving PEP-395, +1 for that, IMHO,
it's much worthy work than e.g. yet another (3rd!) string templating
variant (still adhoc and limited, e.g. not supporting
control statements).

> 
> Relative imports from the main module just happen to be a situation
> where the failure is an obvious one rather than subtle state
> corruption.
> 
> > Nonetheless, that appears to be the hurdle you'd need to
> > confront.
> 
> This came up more recently during the PEP 420 discussions, when the
> requirement to include __init__.py to explicitly mark package
> directories was eliminated. This means there's no longer any way for
> the interpreter to reliably infer from the filesystem layout precisely
> where in the module hierarchy you intended a module to live. See
> https://www.python.org/dev/peps/pep-0420/#discussion for references.
> 
> However, one of the subproposals from PEP 395 still offers a potential
> fix: https://www.python.org/dev/peps/pep-0395/#id24
> 
> That proposes to allow explicit relative imports at the command line,
> such that Paul's example could be correctly invoked as:
> 
>     python3 -m ..pkg.foo
> 
> It would also be possible to provide a syntactic shorthand for
> submodules of top level packages:
> 
>     python3 -m .foo
> 
> The key here is that the interpreter is being explicitly told that the
> current directory is inside a package, as well as how far down in the
> package hierarchy it lives, and can adjust the way it sets sys.path[0]
> accordingly before proceeding on to import "pkg.foo" as __main__.
> 
> That should be a relatively uncomplicated addition to
> runpy._run_module_as_main that could be rolled into Cameron's PEP
> plans. Steps required:
> 
> * count leading dots in the supplied mod_name
> * remove "leading_dots-1" trailing directory names from sys.path[0]
> * strip the leading dots from mod_name before continuing with the rest
> of the function
> * in the special case of only 1 leading dot, remove the final
> directory segment from sys.path[0] and prepend it to mod_name with a
> dot separator
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com