[Python-Dev] Supporting packages on the command line

[Nick Coghlan]

Just van Rossum wrote:
> I understand the difficulty of implementing it, but as a user I find it
> a really really stupid restriction. I routively run (doc)tests of
> individual modules, which usually are submodules of a package. Using -m
> to do this would help me tremendously. As it stands, -m doesn't help me
> at all. I'd even go so far and -1 the entire feature if it doesn't
> support submodules. I guess it's too late for that :)

This may be more a case of my misjudging py-dev's likely collective 
reaction than anything else.

Support for '-m' was lukewarm enough when it last came up, that I didn't 
expect to get a good reaction if I suggested adding a stdlib module in 
order to enhance it to support packages.

While I wrote a patch to enable it (#1043356 - it uses the simple 
C-level strategy of  'try to locate at the top level, if that doesn't 
work, hand it over to the Python version'), we seemed to be too close to 
the beta to push for inclusion this time around. Add in the fact that I 
was about to be moving back to Brisbane after being Oregon for three 
months. . . (I'm back in Brisbane now, though)

At the moment, '-m's behaviour is pretty easy to explain:
  "Look for a top-level module with the specified name. If it is found, 
run it as if its name had been fully specified on the command line. If 
it is not found, report an error"

The behaviour currently implemented in the enhancement patch is:
  "Look for a top-level module with the specified name. If it is found, 
run it as if its name had been fully specified on the command line. If 
it is not found, attempt to import any containing package, then look for 
the module within that package. Run the located module as for a 
top-level module. If it is still not found, report an error.

Note: For modules within packages, this differs slightly from running 
them directly from the command line by filename. Using this switch, the 
script's containing package is fully imported prior to execution of the 
script. This does not happen when the script's filename is used directly."

As an implementation detail, the top-level scan is done in C, the scan 
that understands packages is done in Python. The main reasons for that 
are that the top-level scan gets used to *find* the Python version if 
it's needed, and even a simple scan looking for dots is a pain in C 
(although that *would* likely be slightly quicker than the current 
'failed lookup' approach for scripts inside modules, it would also be 
slightly slower for top-level modules, as well as adding more code to 
main.c).

Selling the additional complexity was the main reason I didn't expect to 
get a good reaction to this idea with the 2.4 beta almost out the door.

I'm happy to make whatever changes to that patch are needed for 
inclusion (e.g. changing the module name, adding it to the docs 
underwhatever name is chosen) - I guess it's mainly Anthony's call 
whether he's prepared to accept such a change after the 2.4b1 release.

Cheers,
Nick.

P.S. I'd also like some feedback on a quirk of the current version of 
that patch - as noted on SF, at the moment it potentially tramples on 
argv[0] at the C-level, which seems questionable given the existence of 
Py_GetArgcArgv(). The simplest way around that is to *not* set 
sys.argv[0] correctly when running pyrun/execmodule implicitly (i.e. 
sys.argv[0] may contain the name of the interpreter, or a command line 
switch, rather than the name of pyrun/execmodule - document this 
possibility with a comment in the __main__ block of pyrun/execmodule).