Idea/PEP draft: Add support of __main__.py to argparse.ArgumentParser.prog
Hello, I have a proposal to improve support of __main__.py (python -m foobar) invocation to argparse.ArgumentParser You can find attached a PEP draft. Unfortunately, I'm not very confident on how to add that PEP and ... honnestly not very used to github. In short, can you help me and advise on this? Thanks a lot and best regards, Michaël Hooreman
There’s no need for this to be a PEP at this point. You only write those after discussion on -ideas, once you have the ideas nailed down, answers to all of the objections, all of the bikeshedding issues out in the open, and consensus that the idea is ready for a final decision. I’m not even sure this needs a mailing list discussion. I may be wrong, but getting the wrong prog name sounds like a clear bug, so you could just post a bug report on b.p.o and link a pull request on GitHub. If there really are issues to discuss, someone on b.p.o will tell you to take it to -issues or -dev as appropriate. Meanwhile, skimming your proposal: It says it doesn’t work on a Windows. It claims to work on POSIX systems but I’m pretty sure it will not work on macOS, or really anything but Linux, if it’s using the /proc file system for anything nontrivial. Sent from my iPhone
One more thing: from reading your proposal, it sounds like your actual issue may be that you’ve got something you want to distribute to end-users who aren’t Python experts to run as `python -m spam`, and the usage messages they get back are confusing them? If so, why not just use a setuptools entry point script, so your users can just run `spam` instead of `python -m spam`? That isn’t appropriate for scripts like pip or venv that are all about the Python installation itself (someone with multiple Pythons installed will almost certainly need to run python3.7 -m pip, python3.8 -m pip, pypy3 -m pip, ~/bin/python -m pip, etc. and coming up with names for all of those—especially automatically—would be silly), but that’s not the case for most scripts. Sent from my iPhone
Thanks a lot for your comments Andrew, I'm replying to your two emails using this, just to avoid confusions. On the need of a PEP or a bug fix, well, I'm learning ;-) As I'm a good student, I've written a draft PEP... My idea was more a proposal to be discussed because wanting python -m foobar instead of __main__.py is only an opinion. So, I'm not sure if this can be considered as a bug. Also, I'm not sure to be "good enough" to make proposal for code changes... The portability issue can be removed by using f"{sys.executable} -m foobar", but it is less accurate versus what the user has really typed. What's better? Having a different behavior if /proc is available or not, or always be "less accurate"? Is it indended for end users? Well, IMHO, users of command lines are technical users in 99% of the cases. End users only want web and GUI... My use case is about commands which are automatically launched, or manually by "well trained" people. Having the usage string which reflecs the really used command is better to avoid confusions. For that last point, again according to my opinion, the fact that it is intended for end users or not is not the point. The real question is "do argparse work as we expect or not?". Best regards. Le 23/08/19 à 17:50, Andrew Barnert a écrit :
Hello Michaël, in your PEP your wrote: --- The standard library's ``argparse.ArgumentParser`` uses by default ``sys.argv[0]`` for his ``prog`` property. It means that when using the ``python -m foobar`` invocation, which calls under the hood the ``foobar.__main__`` module, the ``prog`` will be ``__main__.py``, giving as usage: ``usage: __main__.py [-h] ...``. --- I did a quick check with the file `prog_name.py` which contained: if __name__ == '__main__': from argparse import ArgumentParser ap = ArgumentParser(description='Test default prog name') args = ap.parse_args() then put it on PYTHONPATH and invoked it with: python -m prog_name --help and got the correct reply: usage: prog_name.py [-h] Test default prog name optional arguments: -h, --help show this help message and exit I checked on Python 3.7.3 (Windows) and Python 2.7.13 (raspbian), on both systems it behaves the same way. In what environment exactly do you see the problem? Richard
Hi Richard, The issue is with package foobar containing __main__.py file (not with a module called as a script, which is your case), and called via python -m foobar See the last paragraph of https://docs.python.org/3/library/__main__.html : « For a package, the same effect can be achieved by including a __main__.py module, the contents of which will be executed when the module is run with -m. » Best regards, Michaël Le ven. 23 août 2019 22:32, Richard Musil <risa2000x@gmail.com> a écrit :
On Fri, Aug 23, 2019 at 11:54 PM Michael Hooreman <michael@hooreman.be> wrote:
I have misread your original explanation. I can see it now. Now, another (possibly again naïve) idea :) Could you get the package name simply by parsing the path of the script? I guess it will only work if the package is normally stored in the filesystem. For example, since you already know that you are in `__main__.py` and this file is a part of some package: prog = os.path.basename(os.path.dirname(os.path.abspath(sys.argv[0]))) ap = ArgumentParser(prog=prog, description='Test default prog name') The thing which I see arguable is that ArgumentParser is technically correct when it shows `__main__.py` as an executed program/script and the fact that this particular file has a special role in the package hierarchy has no relevance for ArgumentParser operation. In other words, it is also possible to execute a script called `__main__.py` without any package. Richard
Thanks Richard, In fact, I should add more than the python executable in prog: it must contain -m and the package « classpath », but I can try arpund this. I'll come back with the results asap. Running __main.py__ as a script works *only* if we use absolute imports. And I consider that using absolute imports in a package to itself is a very bad practice. So, __main.py__ is not an executable. Best regards. Le sam. 24 août 2019 03:24, Richard Musil <risa2000x@gmail.com> a écrit :
On Sat, Aug 24, 2019 at 9:25 AM Michael Hooreman <michael@hooreman.be> wrote:
I probably should clarify my original statement. I did not mean that `__main__.py` in a package should be directly executable. What I meant was that someone can write a standalone script and being completely oblivious of its special meaning of the name in the package hierarchy choose to name it `__main__.py`. In this particular case it would be correct to expect that ArgumentParser shows the name of the scripts as `__main__.py`. It is not really a use case and I only gave it as an example that from the ArgumentParser's perspective no special treatment can be derived based on the name alone. Adding more complex logic to ArgumentParser to figure out if it runs "inside the package" seems like an overkill, if it could be resolved in the client code which already has most of the knowledge (that it runs in `__main__.py` and in particular in `__main__.py` which has a special role in the package). Richard
Thanks Richard, You are right. So, my doubt about the fact that it must be changed or not is meaningful, and argparse should not be changed. I'll keep using the « overriding » class, and plane to make a pip package for that. Thanks to everybody for those discussions. Le sam. 24 août 2019 09:53, Richard Musil <risa2000x@gmail.com> a écrit :
On Aug 24, 2019, at 00:25, Michael Hooreman <michael@hooreman.be> wrote:
In fact, I should add more than the python executable in prog: it must contain -m and the package « classpath »
But that’s not a valid prog. If you try to execute the program “python -m spam” with arguments “eggs” and “cheese” (whether with exec, or something higher level like the shell, or Python’s subprocess module), you’re just going to get an error because there’s nothing named “python -m spam” on the PATH. If you’re really trying to reproduce what the user typed (modulo any quoting and escaping, which you can only guess at because the shell doesn’t preserve the distinction between “spam 'my arg'” and “spam my\ arg”, etc.), the prog is “python”, and the args are [“python”, “-m”, “spam”, “eggs”, “cheese”]. I’d ask again why you’re not using a setuptools entrypoint script here.
What being part of the program and what are arguments is not meaningful at this point: I only want an accurate usage string. I don't make a setuptools entry point here for the good reason that this is used as part of a system, which is not using standard deliverables. You have to consider that they are other uses cases of python than some packages to deploy with setuptools. Best regards. Le 24/08/19 à 17:01, Andrew Barnert a écrit :
Let me clarify a bit. Setuptools and standalone scripts is indeed the best classical approach, but that does not adapts to my use case. I'm working with a contunuously, fast evolving system, and there is no "identified release" (continuous delivery). I'm under the impression that you tell me that I must use setuptools entrypoint script so, in other words, a "standard executable" script. But, it this is THE only way correct way to do, why do python allows __main__.py? I don't use standalone script because there are many entry points to my library, which has a very depth (sub-)packages structure, and on one hand having dozens of scripts in a ./bin directory lacks of efficiency, on the other hand I don't want to use absolute imports within a library just to be able to have standalone scripts, knowing that it is also the job of __main__.py. Best regards. Le 24/08/19 à 17:37, Michael Hooreman a écrit :
On Aug 24, 2019, at 08:48, Michael Hooreman <michael@hooreman.be> wrote:
So what? You don’t need an identified release to use pip. Let’s say people have to git pull your code every time they want a new feature or fix, and different people are running code dozens of commits behind or ahead of each other or even on different branches. That’s about the worst-case scenario for managing distribution, right? But pip still handles that just fine. Just pip install from the git repo, and you get the latest code (without even having to have a local clone of the repo, or know anything about git). Or maybe you have insane security requirements, and you deliver updates by having armed guards physically hand out USB sticks with today’s code. Again, pip handles that fine; just pip install /dev/stick/path/to/pkg/setup.py instead of cp -a /dev/stick/path/to/pkg ~/local/pkg. And so on.
I'm under the impression that you tell me that I must use setuptools entrypoint script so, in other words, a "standard executable" script.
No, I’m not telling you that you _must_ use it. But I am telling you that you _can_ probably use it, and if you can, it will automatically solve your problem.
But, it this is THE only way correct way to do, why do python allows __main__.py?
This is a silly question. Imagine you were asking how to insert a value into the middle of a tuple, and, after telling you how you can sort of do the immutable equivalent of what you’re asking, I also pointed out that usually when you want to insert into the middle of stuff, you want a list, which makes it trivial to do exactly what you want. Would you demand to know why Python allows tuples when they don’t solve your problem? Of course not. But that’s what you’re asking here. But if you want an answer anyway, I can think of multiple reasons to add and then maintain the -m functionality even though scripts are useful more often. The first one, I already gave you. There are packages where you’re likely to want to run the same thing with multiple Python installations, and which Python installation is important. For example, you don’t want ipython to start a REPL for any random install, you want a REPL for a specific one. You don’t want pip to install into any installation, you want it to install into a specific one. And so on. These packages have to be designed around -m, so you can run python3.8 -m pip or pypy3 -m ipython or whatever. (But notice that most such packages actually still do provide a shortcut script for the convenience of people who only have one installation.) There are also a lot of packages that _can_ be used as scripts but most people don’t. For example, in the stdlib, many people use the json, http.server, etc. modules in their own code; most of them never run those modules as scripts. But a few people do. So, it’s worth making then runnable, but not worth advertising that fact by adding a script in the PATH. There are also packages that are mainly used straight out of the dev tree. For an extreme example, a package that’s needed to run your setup.py build stage had better be runnable before you’ve done setup.py build, and -m is a way to do that. Are those enough reasons why it might make sense for Python to include -m even though it may not be the best answer to your problem?
OK, I don’t think you know how entrypoint scripts work. (It’s not an easily discoverable feature, so _most_ people don’t know, sadly.) You don’t have to write a complicated script for every entrypoint, you just add an entry to a list in your setup.py file and the scripts get generated at install time. And, even more importantly, you don’t need to change your code to use absolute imports. The generated script imports the entrypoint from its package using the normal mechanism, so it’s still part of a package. And if that package was a subpackage or another package, the entrypoint is still part of a subpackage. And so on. If you really do have dozens of functions that everyone needs to run, you might consider writing a driver that steals the first arg and uses it to decide which entrypoint to call—sort of like the way tools like git work. You can still leave the rest of the argparsing up to each entrypoint if you want (which is a good idea if the separate commands have unrelated APIs—or if you need to be able to run some of the commands out of the dev tree with -m even though most of the people you’re distributing to won’t often do so). But either way, this isn’t done for your automatically—there are third party libs that make it easy, but it’s still not as easy as just adding an entry to your setup.py script—so it may not be worth the effort for your use.
Yes, but many of those use cases actually _can_ still benefit from setuptools, so it’s worth knowing what it can do rather than assuming it’s useless to anyone who isn’t deploying public monolithic releases to PyPI, much less getting angry if someone even mentions using it as a suggestion.
Thanks for those advises, Andrew To tell you a bit more about my context, I'm working on an end-to-end data science projets, which is: - using 90% of python, but also different things - those thing are sometimes impacting system configuration - there's a development accounts, I'm working there without any need of setup.py, compilation, whatever: I run what I develop, period. This includes also cython code, which is compiled on the fly. - commits are on svn - corporate policy - deployment on production servers is a dedicated script, copying on prod users and servers via ssh, and automatic restart of what's needed - ... - for all those reasons, I have no use case for setuptools until now So, yes, I must admit that I only know the basis of setuptools and I'm a newbie with git. But, well, everybody has something to learn. I'll consider what you say, and try to understand what entrypoint is. I have understood entrypoint as a wording for an entry point (main) script. Sorry. I just hope that I won't have to spend days on adapting my workflow, but well, that's life. [Private joke] I must say that I find disappointing to receive a "tell me what you need, I'll explain how to not need that". We are all IT guys, and I also do that. Now, I see what it is from the other side of the court :-) [End of private joke] Best regards Le sam. 24 août 2019 21:40, Andrew Barnert <abarnert@yahoo.com> a écrit :
# Python packaging, setup.py, console_scripts entrypoints, and __main__.py ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py thecode.py ```python !cat setup.py ``` from setuptools import setup setup(name='example', version='0.1', description='Example', url='http://github.com/.../...', author='Example', author_email='example@example.com', license='...', packages=['example'], zip_safe=False, entry_points={ 'console_scripts': [ 'example = example.thecode:main' ]} ) ```python !cat example/thecode.py ``` import sys def main(argv=None): if argv is None: argv = sys.argv print("argparse here", argv) ```python !cat example/__main__.py ``` import sys from .thecode import main def othermain(): print("othermain") return main(sys.argv) if __name__ == "__main__": othermain() ```python !cat ./requirements.txt ``` -e . ```python !pip install -r ./requirements.txt # (this is equivalent to `pip install -e .`) ``` Obtaining file:///home/user/-wrk/-ve37/s_example/src/s_example (from -r ./requirements.txt (line 1)) Installing collected packages: example Found existing installation: example 0.1 Uninstalling example-0.1: Successfully uninstalled example-0.1 Running setup.py develop for example Successfully installed example ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py thecode.py example.egg-info: dependency_links.txt not-zip-safe SOURCES.txt entry_points.txt PKG-INFO top_level.txt ```python # This is generated when you `pip install -e` !cat ./example.egg-info/entry_points.txt ``` [console_scripts] example = example.thecode:main ```python # this runs the 'example' console_scripts entry_point !example ``` argparse here ['/home/user/-wrk/-ve37/s_example/bin/example'] ```python # this runs example/__main__ !python -m example ``` othermain argparse here ['/home/user/-wrk/-ve37/s_example/src/s_example/example/__main__.py'] ... When you want the project to be reproducible: - you have tests (eg in tests/) - you have a requirements.txt that lists exact versions of all dependencies - you build the entire environment (including any requisite OS packages) from zero (with a virtualenv and/or docker), then run the tests There are a number of cookiecutters (project templates) which have the whole project setup: - https://cookiecutter.readthedocs.io/en/latest/readme.html#available-cookiecu... - cookiecutter-pypackage includes pytest setup - https://cookiecutter.readthedocs.io/en/latest/readme.html#data-science - https://cookiecutter.readthedocs.io/en/latest/readme.html#reproducible-scien... I prepared this as a Jupyter notebook, then did 'Save as' > 'Markdown'. Because it has a requirements.txt, if I put this code in a git repo and launch it in mybinder.org (repo2docker), it will build a docker container (also containing jupyter) and launch a free cloud instance; so that others can review the code and notebooks in a read/write environment (where changes are not persisted because the instance is later just deleted) - REES: Reproducible Environment Specification https://repo2docker.readthedocs.io/en/latest/specification.html - environment.yml - Install a Python environment - Pipfile and/or Pipfile.lock - Install a Python environment - requirements.txt - Install a Python environment - setup.py - Install Python packages - Project.toml - Install a Julia environment - REQUIRE - Install a Julia environment (legacy) - install.R - Install an R/RStudio environment - apt.txt - Install packages with apt-get - DESCRIPTION - Install an R package - manifest.xml - Install Stencila - postBuild - Run code after installing the environment - start - Run code before the user sessions starts - runtime.txt - Specifying runtimes - default.nix - the nix package manager - Dockerfile - Advanced environments
https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#... On Sat, Aug 24, 2019 at 5:11 PM Wes Turner <wes.turner@gmail.com> wrote:
```python !cat ../../bin/example ``` #!/home/user/-wrk/-ve37/s_example/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'example','console_scripts','example' __requires__ = 'example' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('example', 'console_scripts', 'example')() ) On Sat, Aug 24, 2019 at 5:16 PM Wes Turner <wes.turner@gmail.com> wrote:
Source: https://github.com/westurner/setuptoolsexample Read-only GitHub notebook view: https://github.com/westurner/setuptoolsexample/blob/master/examplenb.ipynb Read/Write mybinder.org (BinderHub (repo2docker)) notebook: https://mybinder.org/v2/gh/westurner/setuptoolsexample/master?filepath=examp... On Sat, Aug 24, 2019 at 5:22 PM Wes Turner <wes.turner@gmail.com> wrote:
On Aug 24, 2019, at 13:16, Michael Hooreman <michael@hooreman.be> wrote:
I'll consider what you say, and try to understand what entrypoint is. I have understood entrypoint as a wording for an entry point (main) script. Sorry.
Well, it _is_ that, but the point is that setuptools can auto-generate those main scripts for you, without making you rearrange any of your code. Which is very cool, and it’s a shame more people don’t know how to use it.
I just hope that I won't have to spend days on adapting my workflow, but well, that's life.
In your case, hours adapting your scripts to munge the usage output might well be a better use of your time than days learning a new tool and adapting your workflow to it. That doesn’t mean we should necessarily change the stdlib of Python to encourage more people to build workflows like yours in the future instead of more easily-manageable ones. But it does mean that if a lot of people are already in your situation, and there’s something that would help all of them, a change could well be worth doing. That’s why I responded specifically to your proposal, and only added the stuff about setuptools entrypoints as an afterthought. Your proposal might well be useful—but not if it breaks all POSIX systems but Linux, gives a prog that isn’t usable as a prog, etc. I raised those issues to see if you have a solution (or if anyone else does).
Personally, I generally try to explain “Here’s how to do that” before “Here’s how _not_ to do that.” Especially since “Here’s how to do that” often illustrates what a huge pain it is to do that, which convinces people better than just saying “Don’t do that.” :) But in this case, you already had a solution to offer, so it’s not really the same thing.
Indeed. I usually do a workflow which makes my needs compatible with the standards, but I've started this when I was a python beginner. I'm aware since months that I should use setuptools, but it was not on my priority, because that bad decision had no impact. Now, time to change had come... Maybe one suggestion: I think it would be great to document it as a limitation in argparse. Don't you think? Regarding a "patching lib", I don't plan to add anything to pip until I found a portable solution. Sorry if my message was incorrect. Thanks again to everyone. Le sam. 24 août 2019 23:20, Andrew Barnert <abarnert@yahoo.com> a écrit :
So, to rephrase your request, you want argparse.ArgumentParser.prog to contain "{sys.executable} -m {modulename}" so that the help text correctly indicates that it was invoked that way? PEP notes: - POSIX only is maybe not a good strategy - Can cmdline contain other unescaped \0's? - This would be a sight performance regression for the most common cases. (__main__.py is typically only used by stdlib modules; which is what it was added for) Solutions / workarounds: - console_script entry_points - How does pip make this work? When I run `python -u -m pip -h` it prints `{sys.executable} -m pip` as the progname under usage (with *optparse*) - Is this platform portable? Does __package__ always contain the correct module name? prog = None if os.path.basename(sys.argv[0]) == "__main__.py": prog = f"{sys.executable} -m {__package__}" ... - thecode.py (prog=...) - output from `!examplecli -h` and `!python -m example -h` # Python packaging, setup.py, console_scripts entrypoints, and \_\_main__.py - Source: https://github.com/westurner/setuptoolsexample - Read-only GitHub notebook view: https://github.com/westurner/setuptoolsexample/blob/master/examplenb.ipynb - Read-only Nbviewer notebook view: https://nbviewer.jupyter.org/github/westurner/setuptoolsexample/blob/master/... - Read/Write mybinder.org (BinderHub (repo2docker)) notebook: https://mybinder.org/v2/gh/westurner/setuptoolsexample/master?filepath=examp... References: - https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#... ```python !rm -rf ./example.egg-info/ !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py __pycache__ thecode.py ```python !cat setup.py ``` from setuptools import setup setup(name='example', version='0.1', description='Example', url='http://github.com/.../...', author='Example', author_email='example@example.com', license='...', packages=['example'], zip_safe=False, entry_points={ 'console_scripts': [ 'examplecli = example.thecode:main' ]} ) ```python !cat example/thecode.py ``` import argparse import os import sys def main(argv=None): if argv is None: argv = sys.argv print("argv:", argv) prog = None if os.path.basename(sys.argv[0]) == "__main__.py": prog = f"{sys.executable} -m {__package__}" prs = argparse.ArgumentParser(prog=prog) prs.parse_args() ```python !cat example/__main__.py ``` import sys from .thecode import main def othermain(): print("othermain") return main(sys.argv) if __name__ == "__main__": othermain() ```python !cat ./requirements.txt ``` -e . ```python !pip install -r ./requirements.txt # (this is equivalent to `pip install -e .`) ``` Obtaining file:///home/user/-wrk/-ve37/sexample/src/sexample (from -r ./requirements.txt (line 1)) Installing collected packages: example Found existing installation: example 0.1 Uninstalling example-0.1: Successfully uninstalled example-0.1 Running setup.py develop for example Successfully installed example ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py __pycache__ thecode.py example.egg-info: dependency_links.txt not-zip-safe SOURCES.txt entry_points.txt PKG-INFO top_level.txt ```python # This is generated when you `pip install -e` !cat ./example.egg-info/entry_points.txt ``` [console_scripts] examplecli = example.thecode:main ```python !ls ../../bin/ ``` activate jsonschema pip3 activate.csh jupyter pip3.7 activate.fish jupyter-bundlerextension postactivate activate.ps1 jupyter-kernel postdeactivate activate_this.py jupyter-kernelspec preactivate activate.xsh jupyter-migrate predeactivate easy_install jupyter-nbconvert pygmentize easy_install-3.7 jupyter-nbextension python examplecli jupyter-notebook python3 get_env_details jupyter-run python3.7 iptest jupyter-serverextension python-config iptest3 jupyter-troubleshoot wheel ipython jupyter-trust ipython3 pip ```python !cat ../../bin/examplecli ``` #!/home/user/-wrk/-ve37/sexample/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'example','console_scripts','examplecli' __requires__ = 'example' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('example', 'console_scripts', 'examplecli')() ) ```python # this runs the 'example' console_scripts entry_point !examplecli -h ``` argv: ['/home/user/-wrk/-ve37/sexample/bin/examplecli', '-h'] usage: examplecli [-h] optional arguments: -h, --help show this help message and exit ```python # this runs example/__main__ !python -m example -h ``` othermain argv: ['/home/user/-wrk/-ve37/sexample/src/sexample/example/__main__.py', '-h'] usage: /home/user/-wrk/-ve37/sexample/bin/python -m example [-h] optional arguments: -h, --help show this help message and exit On Saturday, August 24, 2019, Michael Hooreman <michael@hooreman.be> wrote:
There’s no need for this to be a PEP at this point. You only write those after discussion on -ideas, once you have the ideas nailed down, answers to all of the objections, all of the bikeshedding issues out in the open, and consensus that the idea is ready for a final decision. I’m not even sure this needs a mailing list discussion. I may be wrong, but getting the wrong prog name sounds like a clear bug, so you could just post a bug report on b.p.o and link a pull request on GitHub. If there really are issues to discuss, someone on b.p.o will tell you to take it to -issues or -dev as appropriate. Meanwhile, skimming your proposal: It says it doesn’t work on a Windows. It claims to work on POSIX systems but I’m pretty sure it will not work on macOS, or really anything but Linux, if it’s using the /proc file system for anything nontrivial. Sent from my iPhone
One more thing: from reading your proposal, it sounds like your actual issue may be that you’ve got something you want to distribute to end-users who aren’t Python experts to run as `python -m spam`, and the usage messages they get back are confusing them? If so, why not just use a setuptools entry point script, so your users can just run `spam` instead of `python -m spam`? That isn’t appropriate for scripts like pip or venv that are all about the Python installation itself (someone with multiple Pythons installed will almost certainly need to run python3.7 -m pip, python3.8 -m pip, pypy3 -m pip, ~/bin/python -m pip, etc. and coming up with names for all of those—especially automatically—would be silly), but that’s not the case for most scripts. Sent from my iPhone
Thanks a lot for your comments Andrew, I'm replying to your two emails using this, just to avoid confusions. On the need of a PEP or a bug fix, well, I'm learning ;-) As I'm a good student, I've written a draft PEP... My idea was more a proposal to be discussed because wanting python -m foobar instead of __main__.py is only an opinion. So, I'm not sure if this can be considered as a bug. Also, I'm not sure to be "good enough" to make proposal for code changes... The portability issue can be removed by using f"{sys.executable} -m foobar", but it is less accurate versus what the user has really typed. What's better? Having a different behavior if /proc is available or not, or always be "less accurate"? Is it indended for end users? Well, IMHO, users of command lines are technical users in 99% of the cases. End users only want web and GUI... My use case is about commands which are automatically launched, or manually by "well trained" people. Having the usage string which reflecs the really used command is better to avoid confusions. For that last point, again according to my opinion, the fact that it is intended for end users or not is not the point. The real question is "do argparse work as we expect or not?". Best regards. Le 23/08/19 à 17:50, Andrew Barnert a écrit :
Hello Michaël, in your PEP your wrote: --- The standard library's ``argparse.ArgumentParser`` uses by default ``sys.argv[0]`` for his ``prog`` property. It means that when using the ``python -m foobar`` invocation, which calls under the hood the ``foobar.__main__`` module, the ``prog`` will be ``__main__.py``, giving as usage: ``usage: __main__.py [-h] ...``. --- I did a quick check with the file `prog_name.py` which contained: if __name__ == '__main__': from argparse import ArgumentParser ap = ArgumentParser(description='Test default prog name') args = ap.parse_args() then put it on PYTHONPATH and invoked it with: python -m prog_name --help and got the correct reply: usage: prog_name.py [-h] Test default prog name optional arguments: -h, --help show this help message and exit I checked on Python 3.7.3 (Windows) and Python 2.7.13 (raspbian), on both systems it behaves the same way. In what environment exactly do you see the problem? Richard
Hi Richard, The issue is with package foobar containing __main__.py file (not with a module called as a script, which is your case), and called via python -m foobar See the last paragraph of https://docs.python.org/3/library/__main__.html : « For a package, the same effect can be achieved by including a __main__.py module, the contents of which will be executed when the module is run with -m. » Best regards, Michaël Le ven. 23 août 2019 22:32, Richard Musil <risa2000x@gmail.com> a écrit :
On Fri, Aug 23, 2019 at 11:54 PM Michael Hooreman <michael@hooreman.be> wrote:
I have misread your original explanation. I can see it now. Now, another (possibly again naïve) idea :) Could you get the package name simply by parsing the path of the script? I guess it will only work if the package is normally stored in the filesystem. For example, since you already know that you are in `__main__.py` and this file is a part of some package: prog = os.path.basename(os.path.dirname(os.path.abspath(sys.argv[0]))) ap = ArgumentParser(prog=prog, description='Test default prog name') The thing which I see arguable is that ArgumentParser is technically correct when it shows `__main__.py` as an executed program/script and the fact that this particular file has a special role in the package hierarchy has no relevance for ArgumentParser operation. In other words, it is also possible to execute a script called `__main__.py` without any package. Richard
Thanks Richard, In fact, I should add more than the python executable in prog: it must contain -m and the package « classpath », but I can try arpund this. I'll come back with the results asap. Running __main.py__ as a script works *only* if we use absolute imports. And I consider that using absolute imports in a package to itself is a very bad practice. So, __main.py__ is not an executable. Best regards. Le sam. 24 août 2019 03:24, Richard Musil <risa2000x@gmail.com> a écrit :
On Sat, Aug 24, 2019 at 9:25 AM Michael Hooreman <michael@hooreman.be> wrote:
I probably should clarify my original statement. I did not mean that `__main__.py` in a package should be directly executable. What I meant was that someone can write a standalone script and being completely oblivious of its special meaning of the name in the package hierarchy choose to name it `__main__.py`. In this particular case it would be correct to expect that ArgumentParser shows the name of the scripts as `__main__.py`. It is not really a use case and I only gave it as an example that from the ArgumentParser's perspective no special treatment can be derived based on the name alone. Adding more complex logic to ArgumentParser to figure out if it runs "inside the package" seems like an overkill, if it could be resolved in the client code which already has most of the knowledge (that it runs in `__main__.py` and in particular in `__main__.py` which has a special role in the package). Richard
Thanks Richard, You are right. So, my doubt about the fact that it must be changed or not is meaningful, and argparse should not be changed. I'll keep using the « overriding » class, and plane to make a pip package for that. Thanks to everybody for those discussions. Le sam. 24 août 2019 09:53, Richard Musil <risa2000x@gmail.com> a écrit :
On Aug 24, 2019, at 00:25, Michael Hooreman <michael@hooreman.be> wrote:
In fact, I should add more than the python executable in prog: it must contain -m and the package « classpath »
But that’s not a valid prog. If you try to execute the program “python -m spam” with arguments “eggs” and “cheese” (whether with exec, or something higher level like the shell, or Python’s subprocess module), you’re just going to get an error because there’s nothing named “python -m spam” on the PATH. If you’re really trying to reproduce what the user typed (modulo any quoting and escaping, which you can only guess at because the shell doesn’t preserve the distinction between “spam 'my arg'” and “spam my\ arg”, etc.), the prog is “python”, and the args are [“python”, “-m”, “spam”, “eggs”, “cheese”]. I’d ask again why you’re not using a setuptools entrypoint script here.
What being part of the program and what are arguments is not meaningful at this point: I only want an accurate usage string. I don't make a setuptools entry point here for the good reason that this is used as part of a system, which is not using standard deliverables. You have to consider that they are other uses cases of python than some packages to deploy with setuptools. Best regards. Le 24/08/19 à 17:01, Andrew Barnert a écrit :
Let me clarify a bit. Setuptools and standalone scripts is indeed the best classical approach, but that does not adapts to my use case. I'm working with a contunuously, fast evolving system, and there is no "identified release" (continuous delivery). I'm under the impression that you tell me that I must use setuptools entrypoint script so, in other words, a "standard executable" script. But, it this is THE only way correct way to do, why do python allows __main__.py? I don't use standalone script because there are many entry points to my library, which has a very depth (sub-)packages structure, and on one hand having dozens of scripts in a ./bin directory lacks of efficiency, on the other hand I don't want to use absolute imports within a library just to be able to have standalone scripts, knowing that it is also the job of __main__.py. Best regards. Le 24/08/19 à 17:37, Michael Hooreman a écrit :
On Aug 24, 2019, at 08:48, Michael Hooreman <michael@hooreman.be> wrote:
So what? You don’t need an identified release to use pip. Let’s say people have to git pull your code every time they want a new feature or fix, and different people are running code dozens of commits behind or ahead of each other or even on different branches. That’s about the worst-case scenario for managing distribution, right? But pip still handles that just fine. Just pip install from the git repo, and you get the latest code (without even having to have a local clone of the repo, or know anything about git). Or maybe you have insane security requirements, and you deliver updates by having armed guards physically hand out USB sticks with today’s code. Again, pip handles that fine; just pip install /dev/stick/path/to/pkg/setup.py instead of cp -a /dev/stick/path/to/pkg ~/local/pkg. And so on.
I'm under the impression that you tell me that I must use setuptools entrypoint script so, in other words, a "standard executable" script.
No, I’m not telling you that you _must_ use it. But I am telling you that you _can_ probably use it, and if you can, it will automatically solve your problem.
But, it this is THE only way correct way to do, why do python allows __main__.py?
This is a silly question. Imagine you were asking how to insert a value into the middle of a tuple, and, after telling you how you can sort of do the immutable equivalent of what you’re asking, I also pointed out that usually when you want to insert into the middle of stuff, you want a list, which makes it trivial to do exactly what you want. Would you demand to know why Python allows tuples when they don’t solve your problem? Of course not. But that’s what you’re asking here. But if you want an answer anyway, I can think of multiple reasons to add and then maintain the -m functionality even though scripts are useful more often. The first one, I already gave you. There are packages where you’re likely to want to run the same thing with multiple Python installations, and which Python installation is important. For example, you don’t want ipython to start a REPL for any random install, you want a REPL for a specific one. You don’t want pip to install into any installation, you want it to install into a specific one. And so on. These packages have to be designed around -m, so you can run python3.8 -m pip or pypy3 -m ipython or whatever. (But notice that most such packages actually still do provide a shortcut script for the convenience of people who only have one installation.) There are also a lot of packages that _can_ be used as scripts but most people don’t. For example, in the stdlib, many people use the json, http.server, etc. modules in their own code; most of them never run those modules as scripts. But a few people do. So, it’s worth making then runnable, but not worth advertising that fact by adding a script in the PATH. There are also packages that are mainly used straight out of the dev tree. For an extreme example, a package that’s needed to run your setup.py build stage had better be runnable before you’ve done setup.py build, and -m is a way to do that. Are those enough reasons why it might make sense for Python to include -m even though it may not be the best answer to your problem?
OK, I don’t think you know how entrypoint scripts work. (It’s not an easily discoverable feature, so _most_ people don’t know, sadly.) You don’t have to write a complicated script for every entrypoint, you just add an entry to a list in your setup.py file and the scripts get generated at install time. And, even more importantly, you don’t need to change your code to use absolute imports. The generated script imports the entrypoint from its package using the normal mechanism, so it’s still part of a package. And if that package was a subpackage or another package, the entrypoint is still part of a subpackage. And so on. If you really do have dozens of functions that everyone needs to run, you might consider writing a driver that steals the first arg and uses it to decide which entrypoint to call—sort of like the way tools like git work. You can still leave the rest of the argparsing up to each entrypoint if you want (which is a good idea if the separate commands have unrelated APIs—or if you need to be able to run some of the commands out of the dev tree with -m even though most of the people you’re distributing to won’t often do so). But either way, this isn’t done for your automatically—there are third party libs that make it easy, but it’s still not as easy as just adding an entry to your setup.py script—so it may not be worth the effort for your use.
Yes, but many of those use cases actually _can_ still benefit from setuptools, so it’s worth knowing what it can do rather than assuming it’s useless to anyone who isn’t deploying public monolithic releases to PyPI, much less getting angry if someone even mentions using it as a suggestion.
Thanks for those advises, Andrew To tell you a bit more about my context, I'm working on an end-to-end data science projets, which is: - using 90% of python, but also different things - those thing are sometimes impacting system configuration - there's a development accounts, I'm working there without any need of setup.py, compilation, whatever: I run what I develop, period. This includes also cython code, which is compiled on the fly. - commits are on svn - corporate policy - deployment on production servers is a dedicated script, copying on prod users and servers via ssh, and automatic restart of what's needed - ... - for all those reasons, I have no use case for setuptools until now So, yes, I must admit that I only know the basis of setuptools and I'm a newbie with git. But, well, everybody has something to learn. I'll consider what you say, and try to understand what entrypoint is. I have understood entrypoint as a wording for an entry point (main) script. Sorry. I just hope that I won't have to spend days on adapting my workflow, but well, that's life. [Private joke] I must say that I find disappointing to receive a "tell me what you need, I'll explain how to not need that". We are all IT guys, and I also do that. Now, I see what it is from the other side of the court :-) [End of private joke] Best regards Le sam. 24 août 2019 21:40, Andrew Barnert <abarnert@yahoo.com> a écrit :
# Python packaging, setup.py, console_scripts entrypoints, and __main__.py ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py thecode.py ```python !cat setup.py ``` from setuptools import setup setup(name='example', version='0.1', description='Example', url='http://github.com/.../...', author='Example', author_email='example@example.com', license='...', packages=['example'], zip_safe=False, entry_points={ 'console_scripts': [ 'example = example.thecode:main' ]} ) ```python !cat example/thecode.py ``` import sys def main(argv=None): if argv is None: argv = sys.argv print("argparse here", argv) ```python !cat example/__main__.py ``` import sys from .thecode import main def othermain(): print("othermain") return main(sys.argv) if __name__ == "__main__": othermain() ```python !cat ./requirements.txt ``` -e . ```python !pip install -r ./requirements.txt # (this is equivalent to `pip install -e .`) ``` Obtaining file:///home/user/-wrk/-ve37/s_example/src/s_example (from -r ./requirements.txt (line 1)) Installing collected packages: example Found existing installation: example 0.1 Uninstalling example-0.1: Successfully uninstalled example-0.1 Running setup.py develop for example Successfully installed example ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py thecode.py example.egg-info: dependency_links.txt not-zip-safe SOURCES.txt entry_points.txt PKG-INFO top_level.txt ```python # This is generated when you `pip install -e` !cat ./example.egg-info/entry_points.txt ``` [console_scripts] example = example.thecode:main ```python # this runs the 'example' console_scripts entry_point !example ``` argparse here ['/home/user/-wrk/-ve37/s_example/bin/example'] ```python # this runs example/__main__ !python -m example ``` othermain argparse here ['/home/user/-wrk/-ve37/s_example/src/s_example/example/__main__.py'] ... When you want the project to be reproducible: - you have tests (eg in tests/) - you have a requirements.txt that lists exact versions of all dependencies - you build the entire environment (including any requisite OS packages) from zero (with a virtualenv and/or docker), then run the tests There are a number of cookiecutters (project templates) which have the whole project setup: - https://cookiecutter.readthedocs.io/en/latest/readme.html#available-cookiecu... - cookiecutter-pypackage includes pytest setup - https://cookiecutter.readthedocs.io/en/latest/readme.html#data-science - https://cookiecutter.readthedocs.io/en/latest/readme.html#reproducible-scien... I prepared this as a Jupyter notebook, then did 'Save as' > 'Markdown'. Because it has a requirements.txt, if I put this code in a git repo and launch it in mybinder.org (repo2docker), it will build a docker container (also containing jupyter) and launch a free cloud instance; so that others can review the code and notebooks in a read/write environment (where changes are not persisted because the instance is later just deleted) - REES: Reproducible Environment Specification https://repo2docker.readthedocs.io/en/latest/specification.html - environment.yml - Install a Python environment - Pipfile and/or Pipfile.lock - Install a Python environment - requirements.txt - Install a Python environment - setup.py - Install Python packages - Project.toml - Install a Julia environment - REQUIRE - Install a Julia environment (legacy) - install.R - Install an R/RStudio environment - apt.txt - Install packages with apt-get - DESCRIPTION - Install an R package - manifest.xml - Install Stencila - postBuild - Run code after installing the environment - start - Run code before the user sessions starts - runtime.txt - Specifying runtimes - default.nix - the nix package manager - Dockerfile - Advanced environments
https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#... On Sat, Aug 24, 2019 at 5:11 PM Wes Turner <wes.turner@gmail.com> wrote:
```python !cat ../../bin/example ``` #!/home/user/-wrk/-ve37/s_example/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'example','console_scripts','example' __requires__ = 'example' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('example', 'console_scripts', 'example')() ) On Sat, Aug 24, 2019 at 5:16 PM Wes Turner <wes.turner@gmail.com> wrote:
Source: https://github.com/westurner/setuptoolsexample Read-only GitHub notebook view: https://github.com/westurner/setuptoolsexample/blob/master/examplenb.ipynb Read/Write mybinder.org (BinderHub (repo2docker)) notebook: https://mybinder.org/v2/gh/westurner/setuptoolsexample/master?filepath=examp... On Sat, Aug 24, 2019 at 5:22 PM Wes Turner <wes.turner@gmail.com> wrote:
On Aug 24, 2019, at 13:16, Michael Hooreman <michael@hooreman.be> wrote:
I'll consider what you say, and try to understand what entrypoint is. I have understood entrypoint as a wording for an entry point (main) script. Sorry.
Well, it _is_ that, but the point is that setuptools can auto-generate those main scripts for you, without making you rearrange any of your code. Which is very cool, and it’s a shame more people don’t know how to use it.
I just hope that I won't have to spend days on adapting my workflow, but well, that's life.
In your case, hours adapting your scripts to munge the usage output might well be a better use of your time than days learning a new tool and adapting your workflow to it. That doesn’t mean we should necessarily change the stdlib of Python to encourage more people to build workflows like yours in the future instead of more easily-manageable ones. But it does mean that if a lot of people are already in your situation, and there’s something that would help all of them, a change could well be worth doing. That’s why I responded specifically to your proposal, and only added the stuff about setuptools entrypoints as an afterthought. Your proposal might well be useful—but not if it breaks all POSIX systems but Linux, gives a prog that isn’t usable as a prog, etc. I raised those issues to see if you have a solution (or if anyone else does).
Personally, I generally try to explain “Here’s how to do that” before “Here’s how _not_ to do that.” Especially since “Here’s how to do that” often illustrates what a huge pain it is to do that, which convinces people better than just saying “Don’t do that.” :) But in this case, you already had a solution to offer, so it’s not really the same thing.
Indeed. I usually do a workflow which makes my needs compatible with the standards, but I've started this when I was a python beginner. I'm aware since months that I should use setuptools, but it was not on my priority, because that bad decision had no impact. Now, time to change had come... Maybe one suggestion: I think it would be great to document it as a limitation in argparse. Don't you think? Regarding a "patching lib", I don't plan to add anything to pip until I found a portable solution. Sorry if my message was incorrect. Thanks again to everyone. Le sam. 24 août 2019 23:20, Andrew Barnert <abarnert@yahoo.com> a écrit :
So, to rephrase your request, you want argparse.ArgumentParser.prog to contain "{sys.executable} -m {modulename}" so that the help text correctly indicates that it was invoked that way? PEP notes: - POSIX only is maybe not a good strategy - Can cmdline contain other unescaped \0's? - This would be a sight performance regression for the most common cases. (__main__.py is typically only used by stdlib modules; which is what it was added for) Solutions / workarounds: - console_script entry_points - How does pip make this work? When I run `python -u -m pip -h` it prints `{sys.executable} -m pip` as the progname under usage (with *optparse*) - Is this platform portable? Does __package__ always contain the correct module name? prog = None if os.path.basename(sys.argv[0]) == "__main__.py": prog = f"{sys.executable} -m {__package__}" ... - thecode.py (prog=...) - output from `!examplecli -h` and `!python -m example -h` # Python packaging, setup.py, console_scripts entrypoints, and \_\_main__.py - Source: https://github.com/westurner/setuptoolsexample - Read-only GitHub notebook view: https://github.com/westurner/setuptoolsexample/blob/master/examplenb.ipynb - Read-only Nbviewer notebook view: https://nbviewer.jupyter.org/github/westurner/setuptoolsexample/blob/master/... - Read/Write mybinder.org (BinderHub (repo2docker)) notebook: https://mybinder.org/v2/gh/westurner/setuptoolsexample/master?filepath=examp... References: - https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#... ```python !rm -rf ./example.egg-info/ !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py __pycache__ thecode.py ```python !cat setup.py ``` from setuptools import setup setup(name='example', version='0.1', description='Example', url='http://github.com/.../...', author='Example', author_email='example@example.com', license='...', packages=['example'], zip_safe=False, entry_points={ 'console_scripts': [ 'examplecli = example.thecode:main' ]} ) ```python !cat example/thecode.py ``` import argparse import os import sys def main(argv=None): if argv is None: argv = sys.argv print("argv:", argv) prog = None if os.path.basename(sys.argv[0]) == "__main__.py": prog = f"{sys.executable} -m {__package__}" prs = argparse.ArgumentParser(prog=prog) prs.parse_args() ```python !cat example/__main__.py ``` import sys from .thecode import main def othermain(): print("othermain") return main(sys.argv) if __name__ == "__main__": othermain() ```python !cat ./requirements.txt ``` -e . ```python !pip install -r ./requirements.txt # (this is equivalent to `pip install -e .`) ``` Obtaining file:///home/user/-wrk/-ve37/sexample/src/sexample (from -r ./requirements.txt (line 1)) Installing collected packages: example Found existing installation: example 0.1 Uninstalling example-0.1: Successfully uninstalled example-0.1 Running setup.py develop for example Successfully installed example ```python !ls ** ``` examplenb.ipynb requirements.txt setup.py example: __init__.py __main__.py __pycache__ thecode.py example.egg-info: dependency_links.txt not-zip-safe SOURCES.txt entry_points.txt PKG-INFO top_level.txt ```python # This is generated when you `pip install -e` !cat ./example.egg-info/entry_points.txt ``` [console_scripts] examplecli = example.thecode:main ```python !ls ../../bin/ ``` activate jsonschema pip3 activate.csh jupyter pip3.7 activate.fish jupyter-bundlerextension postactivate activate.ps1 jupyter-kernel postdeactivate activate_this.py jupyter-kernelspec preactivate activate.xsh jupyter-migrate predeactivate easy_install jupyter-nbconvert pygmentize easy_install-3.7 jupyter-nbextension python examplecli jupyter-notebook python3 get_env_details jupyter-run python3.7 iptest jupyter-serverextension python-config iptest3 jupyter-troubleshoot wheel ipython jupyter-trust ipython3 pip ```python !cat ../../bin/examplecli ``` #!/home/user/-wrk/-ve37/sexample/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'example','console_scripts','examplecli' __requires__ = 'example' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('example', 'console_scripts', 'examplecli')() ) ```python # this runs the 'example' console_scripts entry_point !examplecli -h ``` argv: ['/home/user/-wrk/-ve37/sexample/bin/examplecli', '-h'] usage: examplecli [-h] optional arguments: -h, --help show this help message and exit ```python # this runs example/__main__ !python -m example -h ``` othermain argv: ['/home/user/-wrk/-ve37/sexample/src/sexample/example/__main__.py', '-h'] usage: /home/user/-wrk/-ve37/sexample/bin/python -m example [-h] optional arguments: -h, --help show this help message and exit On Saturday, August 24, 2019, Michael Hooreman <michael@hooreman.be> wrote:
participants (4)
-
Andrew Barnert -
Michael Hooreman -
Richard Musil -
Wes Turner