PEP 389: argparse - new command line parsing module

On Sun, Sep 27, 2009 at 8:41 PM, Benjamin Peterson <benjamin@python.org> wrote:
I feel like I'm repeating the PEP, but here it is again anyway. There will be messages in the docs and pending deprecation warnings (which don't show up by default but can be requested) starting in Python 2.7 and 3.2. Regular deprecation warnings wouldn't show up until Python 3.4, 3 years away. This compromise was intended exactly to address the issue you brought up about people getting over the Python 3 transition. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

-1 for deprecating getopt. getopt is super-simple and especially useful for c programmers learning python. +1 for argparse.+1 for eventual deprecation of optparse - optparse and argparse have a very similar syntax and having both is just confusing. tsboapooowtdi On Mon, Sep 28, 2009 at 6:46 AM, Steven Bethard <steven.bethard@gmail.com>wrote:

On Mon, Sep 28, 2009 at 6:03 AM, Jon Ribbens <jon+python-dev@unequivocal.co.uk> wrote:
Ok, sounds like there are a number of supporters for keeping getopt around and just deprecating optparse. For those who'd like to keep getopt around, I have a few questions: * Would you be opposed to a note in the getopt documentation suggesting argparse as an alternative? * Would you like argparse to grow an add_getopt_arguments method (as in my other post)? * If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

* Would you be opposed to a note in the getopt documentation suggesting argparse as an alternative?
from the top of http://docs.python.org/library/getopt.html - "A more convenient, flexible, and powerful alternative is the optparse module."I think this statement should be emphasized better but it's already there so no worries. * Would you like argparse to grow an add_getopt_arguments method (as in my
other post)?
* If argparse grew an add_getopt_arguments, would you still want to keep
getopt around? And if so, why?
Argparse/optparse use a parser instance and getopt doesn't. I think that's a saved line you have to keep saving so this would be the better syntax imho: optlist, args = argparse.getopt(args, 'abc:d:') I was thinking on the general issue of what's better for the std-lib: 1. Mini modules - 3+ small modules in each subject that can be useful in different contexts. For example the xml libs. or 2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this. I think in general python std-lib went the path of mini modules. Its advantages 1. Each mini module can be coherent and while python isn't really OnlyOneWayToDoIt, each module in itself has OnlyOneWayToDoIt. 2. Documentation is less cluttered by a large amount of functions that all do the same thing but differently. This is very important for python's learning curve. disadvantages: 1. You have to craft your documentation carefully as to be very explicit on why you have all these modules for the same thing. Probably each module should explain it's purpose in relation to the other modules and this explanation should appear at the top of the documentation of each of the mini modules in the subject. Today it would be very hard for me to figure out how the xml modules interrelate, urllib has improved in python3 but still isn't perfect. 2. I could be using some silly way of doing things (getopt) while I really wanted to do it the good way (argparse). I think that while adding getopt functionality to argparse might be nice but would eventually cause clutter.

Yuvgoog Greenle <ubershmekel <at> gmail.com> writes:
+1 for adding argparse and eventually deprecating optparse, -0 for deprecating getopt.
2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this.
I'm not sure that it is, if you mean code size. In Python 2.5, logging is 1300 SLOC, less than say tarfile, pickletools, pydoc and decimal. Regards, Vinay Sajip

On Tue, 29 Sep 2009 12:28:39 am Steven Bethard wrote:
+1
* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?
0
* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?
Simplicity of the learning curve. Using it is as simple as: getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) Does argparse allow anything as simple as that? -- Steven D'Aprano

On Mon, Sep 28, 2009 at 8:27 AM, Steven D'Aprano <steve@pearwood.info> wrote:
You forgot the for-loop, nested if/else statements and type conversions. ;-)
Does argparse allow anything as simple as that?
If you mean, does argparse allow configuration to be specified using the getopt style ("a:b", ["alpha=", "beta"]), no, it currently doesn't. But if this is useful functionality, and would reasonably replace the getopt use cases, then I'd be happy to add it. In the simplest version, we might add something like:: options, args = argparse.getopt("a:b", ["alpha=", "beta"]) where you could then use "options" without any looping:: alpha = options.a or options.alpha beta = options.b or options.beta But if people still like the traditional getopt loop with nested if/elses better, then we might as well just keep getopt around and not add anything to argparse. I'm fine with that too, I just want to make sure that there isn't an obvious deficiency in argparse that we could easily fix. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Mon, Sep 28, 2009 at 08:49, Steven Bethard <steven.bethard@gmail.com> wrote:
=) I do wonder if people who are advocating for getopt sticking around realize how much extra code must be written to make sure what it gives back to you is in some sane manner. Let's take ``getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])`` as an example and simply assume that 'alpha' takes a string as an argument and that it's required and that 'beta' is a boolean flag. To pull everything out you could do:: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) options_dict = dict(options) alpha = options_dict.get('-a', options_dict.get('--alpha', '')) beta = '-b' in options_dict or '--beta' in options_dict main(alpha, beta, args) Obviously if one of the getopt supporters has a better way of doing this then please speak up. Now, Steven, can you show how best to do this in argparse? I am willing to bet that the total number of lines to do this is not that much more and does not require you to know to use 'or' or the dict constructor along with dict.get() in order to keep it compact. I can only imagine what some newbie might try to do in order to be correct (if they even try). -Brett

On Mon, Sep 28, 2009 at 12:22 PM, Brett Cannon <brett@python.org> wrote:
Here's the same option parsing in argparse: parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') args = parser.parse_args() main(args.alpha, args.beta) Or if those final positional arguments were actually meaningful, then you would add one more argument like this:: parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') parser.add_argument('gammas', nargs='*') args = parser.parse_args() main(args.alpha, args.beta, args.gammas) Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

for a live demo of how getopt is useful and flexible, I like how Audacity uses it: http://www.google.com/codesearch/p?hl=en&sa=N&cd=6&ct=rc#_hWFOhGz9lE/mezzo/scons/sconsign.py&q=getopt%20%22import%20getopt%22%20file:%5C.py$&l=264 To answer your question, it goes like this: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) for o, a in options: if o in ('-a', '--alpha'): alpha = a elif o in ('-b', '--beta'): beta = a main(alpha, beta, args) Notice a few things: 1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter). 2. There is no chance the parser will print things I don't want it to print. 3. You can do whatever you want using this flow - call a function, a binary operator (ie | is used in audacity scons example above) 4. in argparse this flow can be emulated and would be nicer, in ('-a', '--alpha') becomes == 'alpha' In a perfect world, getopt would be the low level parser that argparse and optparse rely on. This is not too far fetched btw, all that needs to be done is add another optional argument to getopt that would allow '-' and '--' to be replaced with arbitrary signs, OR moving the parsing code from argparse into getopt. On Mon, Sep 28, 2009 at 10:57 PM, Steven Bethard <steven.bethard@gmail.com>wrote:

2009/9/28 Yuvgoog Greenle <ubershmekel@gmail.com>:
That one does worry me. I'd rather argparse (or any library function) didn't call sys.exit on my behalf - it should raise an exception. Is it actually true that argparse exits? (I can imagine that it might if --help was specified, for example. An exception may not be right here, but I still don't like the idea of a straight exit - I've used too many C libraries that think they know when I want to exit).
2. There is no chance the parser will print things I don't want it to print.
That may also be bad - for example, Windows GUI-mode programs raise an error if they write to stdout/stderr. I could imagine using argparse for such a program, and wanting to do something with --help other than write to stdout and exit (a message box, for example). And yet, I'd want access to the text argparse would otherwise write to stdout. Paul.

Paul Moore wrote:
You can override ArgumentParser.error() to raise an exception.
It looks like this might not be so easy to do. I'd suggest adding a file-like object to the constructor, defaulting to sys.stderr; or maybe an ArgumentParser.print() method that can be overridden. Eric.

On Tue, Sep 29, 2009 at 1:31 PM, Paul Moore <p.f.moore@gmail.com> wrote:
This is behavior that argparse inherits from optparse, but I believe it's still what 99.9% of users want. If you're writing a command line interface, you don't want a stack trace when there's an error message (which is what you'd get if argparse just raised exceptions) you want an exit with an error code. That's what command line applications are supposed to do. If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do.
There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On approximately 9/29/2009 1:57 PM, came the following characters from the keyboard of Steven Bethard:
Documenting both of these options would forestall people from thinking it is only useful for console applications. An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good. In making the method public you might want to rename it to something other than print_message. If all the messages are errors, including "error" in the name would be good. If not, classifying the errors vs non-errors as an extra parameter might be good (in other words... inform the user and continue, or inform the user and exit). Separating the message from the classification is not good, because that leads to dialog boxes having only an "OK" button, and depending on the message, it can be really inappropriate to display an "OK" button... buttons named "Sorry", "Alas!", and "Exit" or "Quit" are often more appropriate, even when there is no user choice possible. Clearly if someone is writing a GUI, they are willing to write lots of lines of code to do things... a couple more well-documented lines to integrate argparse into their chosen GUI doesn't seem onerous. Especially if they can cut, paste, and hack from the above-suggested example code, like they do for the rest of their GUI code. Well, at least, cut, paste, and hack is how I write GUIs when I bother. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

On Tue, Sep 29, 2009 at 3:04 PM, Glenn Linderman <v+python@g.nevcal.com> wrote:
I'm totally fine with people thinking it is only useful for console applications. That's what it's intended for. That said, if there are people out there who want to use it for other applications, I'm happy to make things easier for them if I know concretely what they want.
An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good.
I don't really use GUI libraries, so I wouldn't be able to come up with such an example. I'd also rather not make API changes based on speculative use cases, so before I spend time documenting these things, I'd really like to hear from someone who has already, say, used getopt or optparse in conjunction with a GUI library, and what feedback they have about that. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On approximately 9/29/2009 4:38 PM, came the following characters from the keyboard of Steven Bethard:
Hmm. Maybe that is partly why (sadly) so many GUI programs don't offer much in the way of command line options. Of course, many of their users in Windowsland wouldn't know how to tweak the shortcut to invoke them with options anyway, which might be another part. Fortunately, there are some good Windows programs that do offer rich GUIs and also rich command line options (e.g. IrfanView and ImgBurn)
I'm not a Tk user, just coming to Python from Perl, where I used Win32::GUI, but when I was looking for a functional and portable GUI development package, Perl didn't have one (still doesn't have one that supports Unicode and printing), and Python does... so here I come to Python and Qt. From my experience in Perl GUI Windowsland, the primary issue with command line parsing is displaying the message in a dialog instead of STDOUT. But the message and the user choices have to be known at the same time to design the dialog box. And, there is nothing so aggravating as to be shown an error message, and the only option is a button that says "OK", when it clearly isn't OK. So the tone/type of the messages also needs to be known, even when there are no user choices. The --help option could display the help message, and offer OK. Many errors (particularly unrecoverable ones) should display the error message, and offer an Exit button, or just the close box. A few (although probably only highly customized user options) might want to give the user multiple recovery options. So, I guess I'm in the unfortunate position of seeing the need, but not having used these specific technologies enough to offer an example either. So far, I've only used optparse (just now hearing about argparse in this thread) for command line programs in Python, and I am still just experimenting with Qt, and don't have enough familiarity with it yet to think that what I'm doing is best practices. I think it would be sad to a new replacement for optparse that didn't GUI usage, though, at least in a limited form. The concepts I describe seem sufficient from experience in other environments, and I would think they would be sufficient in Python too, but I'm too new here to state that definitely. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

On 09/29/2009 04:38 PM, Steven Bethard wrote:
Note: on Unix systems, --help should still print to the terminal, not pop up a GUI text box with the help information. So being able to override the behaviour might be good but it is more than a simple, GUI vs console distinction. Are we talking about anything else than --help output (for the printing question)? About exit(), I agree with others about wanting to catch the exception myself and then choosing to exit from the code. I'm not sure that it's actually useful in practice, though...it might just feel cleaner but not actually be that helpful. -Toshio

I haven't checked if it's possible, but I suggest Argparse have it's own exception class that inherits from SystemExit and that exception would be thrown. ParseError, or something similar. I suggest this just because it would be more readable I guess and would exactly explain why this code exits. On Sat, Oct 3, 2009 at 8:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle <ubershmekel@gmail.com> wrote:
I've never seen such an idiom before (subclassing SystemExit) but it would certainly be possible create an ArgumentParserExit exception like that. Then you would have your choice of overriding .exit() or catching the exception. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

Steven Bethard wrote:
Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sat, Oct 3, 2009 at 8:17 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I'm certainly fine with that -- I'm just trying to make sure I've addressed whatever needs addressed to get argparse in.
I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands.
Maybe I misunderstood, but I think this is already the default argparse behavior, no? import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo') subparsers = parser.add_subparsers() parser1 = subparsers.add_parser('1') parser1.add_argument('--bar') parser2 = subparsers.add_parser('2') parser2.add_argument('baz') parser.parse_args(['--help'])
positional arguments: {1,2} optional arguments: -h, --help show this help message and exit --foo FOO
optional arguments: -h, --help show this help message and exit --bar BAR
positional arguments: baz optional arguments: -h, --help show this help message and exit Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

Steven Bethard wrote:
Cool. I didn't realise that help for subcommands was already implemented. :-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sat, Oct 3, 2009 at 7:21 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Check it out: def ParseAndRun(): crazy_external_function_that_might_exit() # Argparse blah blah parser.parse_args() if __name__ == "__main__": try: ParseAndRun() except SystemExit: # was it crazy_external_function_that_might_exit or an argparse error? I know this might come through as bike shedding but it's just customary python that every module have it's own exception types as to not mix them up with others. --yuv

Yuvgoog Greenle wrote:
Then subclass and override .exit() as discussed - or put proper exception handling around the call to parse_args() (optionally rethrowing with whatever custom exception type you wish). Michael
--yuv
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sun, 4 Oct 2009 03:38:31 am Yuvgoog Greenle wrote:
Does it matter? What difference does it make?
You are mistaken.
Plain, ordinary AttributeError, not OSModuleSubclassOfAttributeError. I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types. There's no reason to treat SystemExit as special in this regard. optparse, for example, calls sys.exit(), which operates by raising SystemExit. Ordinary SystemExit, not a special module-specific subclass. -- Steven D'Aprano

On Sat, Oct 3, 2009 at 8:29 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types.
I might be wrong on this. Your point is extra true for modules in the standard library (which is what we're talking about for argparse). I just think that if a parser error is causing the SystemExit, I would rather catch a parser error than catching a SystemExit for the sake of readability. It saves me the comments: # Catching SystemExit because parse_args() throws SystemExit on parser errors. # Subclassing ArgumentParser and overriding exit because I don't want to exit() upon parser errors. So I'm sorry if what I said was irrelevant. I've never written or taken part of writing a std-lib module. --yuv

On Sun, 4 Oct 2009 04:46:19 am Yuvgoog Greenle wrote:
But why are you catching the error? As a general rule, you *want* your command line app to exit if it can't understand the arguments you pass to it. What else can it do? Guess what you wanted? Assuming you have a reason for catching the exception, I don't see that there's any difference in readability between these: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except argparse.ParserError: process_error() else: main(ns) and: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except SystemExit: process_error() else: main(ns) You don't need a comment warning that you are catching SystemExit because parse_args raises SystemExit, any more than you need a comment saying that you are catching ValueError because some function raises ValueError. The fact that you are catching an exception implies that the function might raise that exception. A comment like: "Catching SystemExit because parse_args() throws SystemExit on parser errors." is up them with comments like this: x += 1 # Add 1 to x. -- Steven D'Aprano

* Steven D'Aprano wrote:
It's semantically different. You usually don't catch SystemExit directly, because you want your programs to be stopped. Additionally, a library exiting your program is badly designed, as it's unexpected. Thatswhy such a comment is useful. Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse. That way all parties here should be satisifed. (I do the same all the time in my signal handlers - that's another reason I'd rather not catch SystemExit directly as well :-) nd -- "Umfassendes Werk (auch fuer Umsteiger vom Apache 1.3)" -- aus einer Rezension <http://pub.perlig.de/books.html#apache2>

On Sun, 4 Oct 2009 05:35:04 am André Malo wrote:
Exactly -- so why catch it at all? But even if there is a good reason to catch it, there's still no reason to subclass SystemExit unless argparse wants to distinguish different types of fatal error, and allow code to catch some but not all. But frankly, if I'm having a hard time thinking of a reason to catch SystemExit, I'm having an even harder time thinking why you'd want to (say) catch SystemExitTooManyArguments but not SystemExitMissingArgument.
Additionally, a library exiting your program is badly designed, as it's unexpected.
It's not unexpected for an argument parser. Do you know any applications that run when given invalid arguments? As a general rule, what can the application do? Guess what you wanted?
Thatswhy such a comment is useful.
The comment doesn't tell you anything that wasn't obvious from the code. It is pointless.
Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse.
In the following code snippet: try: ns = argparse.parse_args() except SystemExit: ... is there any confusion between SystemExit raised by parse_args and SystemExit raised by other components? *What* other components? If SystemExit was raised in that try block, where could it have come from other than parse_args? Do you write comments like these? try: value = mydict[key] except KeyError: # Catch KeyError raised by dict lookup ... try: n = mylist.index(x) except ValueError: # Catch ValueError raised by mylist.index ... Useless comments are worse than no comments, because useless comments waste the readers' time and they risk becoming out of sync with the code and turning into misleading comments.
That way all parties here should be satisifed.
No. It wastes the time of the argparse developer, it wastes the time of people learning argparse, it wastes the time of people who read the code and wonder what's the difference between SystemExit and ArgparseSystemExit. (Answer: there is no difference.) -- Steven D'Aprano

Steven D'Aprano wrote:
There are uses of argparse outside of command line apps. For example, I use it to parse --options for IPython %magic commands. Of course, I just subclass ArgumentParser and replace the .error() method. I require a particular IPython exception type in order for the error to be recognized correctly in the %magic subsystem. The other use case, as mentioned earlier, was for debugging your parser on the interactive prompt. A custom subclass may also be able to hold more machine-readable information about the error than the formatted error message, but I don't have any particular use cases about what may be the most useful. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sun, 4 Oct 2009 08:21:46 am Robert Kern wrote:
Exactly. There are uses for catching SystemExit, which is why it's an exception and not an unconditional exit. But they're rare, and not difficult to deal with: in your case, having argparse raise a subclass of SystemExit won't help you, you would still need to subclass, and in other cases, you can just catch SystemExit.
Nobody has requested that the exception expose more information. They've requested that argparse paint the SystemExit a slightly different shade of yellow to the colour it already is -- this is pure bike-shedding. Subclassing SystemExit just in case someday in the indefinite future there comes a need to add extra information to the exception falls foul of You Ain't Gonna Need It. Keep it simple -- if, someday, such a need becomes apparent, then subclass. -- Steven D'Aprano

Yuvgoog Greenle wrote:
Not in my Python world it isn't. While that is sometimes the right answer, more often the right answer is to properly scope your try/except statement in order to avoid catching exceptions from code you aren't interested in. If you end up stuck with a library call that lacks both granularity and specificity of exceptions then you live with it (and maybe file a bug report/feature request with the library author). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On 2009-09-29 18:38 PM, Steven Bethard wrote:
I use argparse (and previously optparse) frequently to handle the command line arguments of GUI apps. I tend to use them in the same way as CLI programs, though, since I usually only use command line arguments when starting the GUIs from the terminal. I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs. I'm not sure what would make a program "GUI-mode" or not. Certainly, I have written Python programs that use wxPython and PyQt on Windows that print to stdout/stderr, and they appear to work fine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2009/9/30 Robert Kern <robert.kern@gmail.com>:
I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs.
:-)
It's the difference between using python.exe and pythonw.exe. There's a flag in the executable header saying whether the executable is "console mode" or "gui mode". GUI mode programs are run (by the OS) without a console (or if run from a console prompt, they automatically detach from that console, much like Unix programs which fork themselves to leave the terminal group (did I get the terminology right?) but done by the OS). As a result, the program has no valid stdin/out/err handles. Any attempt to write to them causes the program to crash. Traceback (most recent call last): File "hello.py", line 13, in <module> main() File "hello.py", line 7, in main sys.stdout.flush() IOError: [Errno 9] Bad file descriptor (Question - is it *ever* possible for a Unix program to have invalid file descriptors 0,1 and 2? At startup - I'm assuming anyone who does os.close(1) knows what they are doing!) Paul.

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:
Normally you don't close fd 0, 1 & 2 but rather redirect them to /dev/null (a black hole). That saves you from nastiness when a library or something misbahaves and tries to use one of those. It would have been nice if windows GUI-mode programs as you describe them would do something similar. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:
Yes, at startup you just have the file descriptors your parent process gave you. This may or may not include 0, 1, and 2, and may or may not include other descriptors too. I seem to recall there have been security holes caused in the past by people assuming 0-2 will exist.

Paul Moore wrote:
Of course; simply use the >&- pseudo-redirection, which has been a standard sh feature (later inherited by ksh and bash, but not csh) for ~30 years. The error message is amusing, too: $ python -c 'print "foo"' >&- close failed in file object destructor: Error in sys.excepthook: Original exception was: Adding an explicit flush results in a more understandable error message: Traceback (most recent call last): File "<string>", line 1, in <module> IOError: [Errno 9] Bad file descriptor

Hrvoje Niksic <hrvoje.niksic <at> avl.com> writes:
Python 3 complains at startup and is a bit more explicit: $ ./python -c '1' >&- Fatal Python error: Py_Initialize: can't initialize sys standard streams OSError: [Errno 9] Bad file descriptor Abandon Of course, if it is stderr that you explicitly close, you lose all reporting: $ ./python -c '1' 2>&- Abandon Regards Antoine.

2009/10/8 Paul Moore <p.f.moore@gmail.com>:
When running pythonw, fileno(stdout) is negative, so sys.stdout is set to None, and print() silently returns. But the situation is not perfect, see http://bugs.python.org/issue6501 where Apache provides a stdout, but python cannot determine the encoding because there is no console. -- Amaury Forgeot d'Arc

2009/9/29 Steven Bethard <steven.bethard@gmail.com>:
Thanks, that's fine for me (as things stand, no need to publicly expose and document _print_message). BTW, the helpful and responsive way you reply to queries is much appreciated. I'm +1 on the PEP (although I see why people want getopt to stay, so I'm happy to leave that and only deprecate optparse). Paul.

Steven Bethard wrote:
Instead of forcing the user to override the ArgumentParser class to change how errors are handled, I suggest adding a separate method ArgumentParser.parse_args_with_exceptions() that raises exceptions instead of writing to stdout/stderr and never calls sys.exit(). Then implement ArgumentParser.parse_args() as a wrapper around parse_args_with_exceptions(): class ArgparseError(Exception): """argparse-specific exception type.""" pass class ArgumentError(ArgparseError): # ... class ArgumentParser(...): # ... def parse_args_with_exceptions(...): # like the old parse_args(), except raises exceptions instead of # writing to stdout/stderr or calling sys.exit()... def parse_args(self, *args, **kwargs): try: self.parse_args_with_exceptions(*args, **kwargs) except ArgparseError as e: self.print_usage(_sys.stderr) self.exit(status=2, message=(_('%s: error: %s\n') % (self.prog, e,))) # perhaps catch other exceptions that need special handling... def error(self, message): raise ArgparseError(message) The exception classes should hold enough information to be useful to non-command-line users, and obviously contain error messages that are output to stderr by default. This would allow non-command-line users to call parse_args_with_exceptions() and handle the exceptions however they like. Michael

Greg Ewing wrote:
I was going to mention that. Between catching SystemExit, the '-i' switch to the interpeter and the PYTHONINSPECT environment variable, investigating applications that call sys.exit isn't *that* difficult. It's a far cry from the instant exits of a C level abort call. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

As Yuvgoog Greenle says, the canonical getopt way is to write alpha = None beta = False options, args = getopt.getopt(sys.argv[1:],"a:b",['alpha=','beta']): for opt, val in options: if arg in ('-a','--alpha'): alpha = val elif arg in ('-b','--beta'): beta = True main(alpha, beta, args) Even though this is many more lines, I prefer it over optparse/argparse: this code has only a single function call, whereas the argparse version has three function calls to remember. The actual processing uses standard Python data structures which I don't need to look up in the documentation.
Now, Steven, can you show how best to do this in argparse?
This demonstrates my point: you were able to use getopt right away (even though not in the traditional way), whereas you need to ask for help on using argparse properly.
See above - getopt users don't care about compactness in the processing.
Depends on the background of the newbie. If they come from C, they immediately recognize the way of doing things. Regards, Martin

On Mon, Sep 28, 2009 at 8:44 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I don't think this is fair at all. I am totally unable to write getopt code without checking the documentation -- I don't remember the format string syntax, nor what the function returns. But that's just how library modules work -- if you don't know the library, you have to read the documentation. The only reason getopt is easier for you is that you're already familiar with the API from C. That said, I can certainly understand that folks who use getopt in C would have an easy transition to getopt in Python and a harder transition to argparse. You didn't directly answer my question of whether adding an "add_getopt_arguments" function would meet your needs, but I infer from your answer above that it wouldn't because you'd still need to make multiple function calls which is your primary complaint. So at this point, I think it's clear that there's nothing I can reasonably add to argparse to make getopt people more comfortable that isn't just duplicating what's already in getopt. So let's just deprecate optparse (as in the PEP), and leave getopt alone (other than adding a note to the documentation suggesting argparse as an alternative). -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Mon, Sep 28, 2009 at 20:44, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Actually, I had to read the docs for getopt. And I chose to not even try argparse when the creator of the module is cc'ed on the email and can obviously do a better example using his own code then I could. -Brett

Martin> alpha = None Martin> beta = False Martin> options, args = getopt.getopt(sys.argv[1:],"a:b",['alpha=','beta']): Martin> for opt, val in options: ... Martin> Even though this is many more lines, I prefer it over Martin> optparse/argparse: this code has only a single function call, ... Agreed. I have never completely wrapped my brain around optparse. Getopt I just remember. Skip

skip@pobox.com wrote:
I have never completely wrapped my brain around optparse. Getopt I just remember.
Seems to me that optparse and argparse are fairly similar in their APIs, and that argparse isn't going to be significantly easier to fit in one's brain than optparse. There's an art to coming up with an API that makes simple things easy and other things possible. I'm not convinced that argparse represents a subsantial enough advancement in that art to justify replacing optparse with it in the stdlib, and thereby forcing everyone to learn a similar-but-different API. -- Greg

On Sep 29, 2009, at 6:51 PM, Greg Ewing wrote:
There's no question it is if you're doing more complicated stuff, like extending it or using subcommands. After I converted my code from optparse to argparse, I ended up with less stuff that was more regular and easier to understand. It convinced me that argparse is a win. -Barry

Greg Ewing wrote:
As someone that has written multiple optparse based utility scripts, I would say that yes, argparse is a *huge* improvement. Several things that I implemented for my own use in a rather clumsy fashion (subcommands, aggregated parsers, non-interspersed arguments) have far more elegant support built directly into argparse. For the getopt-vs-opt/argparse discussion, I believe the major distinction is in the relative balance between procedural and declarative style of argument handling. getopt is very procedural - you define a minimal amount regarding the options you accept, but then do the bulk of the command line processing yourself optparse is declarative to some degree, but forces you to drop back to a procedural style to handle arguments and subcommands. argparse takes things even further in a declarative direction by adding explicit support for positional arguments and subcommands. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Barry Warsaw wrote:
People assess the mental cost differently though - for getopt, they tend to assign the cost to the script they're currently writing, while for optparse/argparse they assign the cost to those two libraries being difficult to learn :) Keeping getopt around *and* including a "add_getopt_arguments" method in argparse is probably the best of both worlds, in that it allows for relatively straightforward evolution of an application: 1. Start with getopt 2. As the getopt argument parsing gets twisty and arcane and maintaining the help string becomes a nightmare, move to argparse with the "add_getopt_arguments" method. 3. Over time, convert more arguments to proper argparse arguments with full documentation. (Note that getting a good help string "for free" is one of the biggest gains I currently find in using optparse over getopt. This is especially so once you start making use of option groups) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Tue, Sep 29, 2009 at 6:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
One thing that wouldn't be good in this transition is that "add_getopt_arguments" can only generate the part of the help string that says "-a" and "--author" exist as options -- it can't add the additional bit of text that says what they do because that's not provided to the getopt API. I suspect this makes the transition less convenient because with getopt you were probably already manually maintaining a usage message that had at least some of this information, and switching to "add_getopt_arguments" would mean throwing this information away. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Sep 29, 2009, at 9:18 PM, Nick Coghlan wrote:
Maybe. I haven't been following this entire thread, but I don't see much point in making it easy to go from getopt to argparse. The two are so different that I think you're either going to jump all in or not. Maybe that's just me though as I really don't see much use for getopt any more (even for throwaway scripts). -Barry

On Tue, Sep 29, 2009 at 9:30 PM, Barry Warsaw <barry@python.org> wrote:
Maybe. I haven't been following this entire thread, but I don't see much point in making it easy to go from getopt to argparse.
I'm with you on this; specific getopt uses are more likely to be switched to opt/argparse than to shift gradually via hybrid APIs.
Heh. I never liked getopt anyway. :-) -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller

Brett Cannon <brett <at> python.org> writes:
Obviously if one of the getopt supporters has a better way of doing this then please speak up.
I'm not a getopt supporter per se - I'm +1 for argparse - but Opster provides some nice syntax sugar on top of getopt, and can be seen here: http://blogg.ingspree.net/blog/2009/09/14/opster/ I've contacted Steven about a similar approach for argparse, and we're waiting for Yuvgoog's work on argparse(func) to be done to see if there's a feasible similar approach.

On a tangent -- a use case that I see happening frequently but which is pretty poorly supported by optparse is a command-line that has a bunch of general flags, then a 'subcommand name', and then more flags, specific to the subcommand. Most here are probably familiar with this pattern from SVN, Hg, and other revision control systems (P4 anyone?) with a rich command-line interface. There are some variants, e.g. whether global and subcommand-specific flags may overlap, and whether flags may follow positional args (Hg and SVN seem to differ here a bit). I've helped write at least two tools at Google that have this structure; both used different approaches, and neither was particularly easy to get right. Getting all the different --help output to make sense was mostly a manual process. (E.g. "foo --help" should print the general flags and the list of known subcommands, whereas "foo --help subcommand" should print flags and other usage info about the specific subcommand.) Also switching out to different calls based on the subcommand should/might be part of this. I would be willing to live with a third option parser in the stdlib if it solved this problem well. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 2009-09-30 11:38 AM, Guido van Rossum wrote:
I don't think argparse supports the "foo --help subcommand" OOB. I think it would be simple to modify argparse to make it do so. It does support general options followed by a subcommand with options, though. http://argparse.googlecode.com/svn/tags/r101/doc/other-methods.html#sub-comm... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sep 30, 2009, at 12:58 PM, Robert Kern wrote:
Right. I've made it kind of work in Mailman 3, but it would be nice for argparse to support this out of the box. Note that I think you want two forms: foo help subcommand foo subcommand --help to basically print the same help text. This is the way bzr does it for example and it works great. Other than that, I think argparse supports Guido's use case very nicely. -Barry

On Wed, Sep 30, 2009 at 02:22:53PM -0400, Barry Warsaw wrote:
From my cursory reading of the documentation, it looks like argparse can only add subparsers for subcommands. Is there any way to add subparsers
In some commands, options as well as subcommands can change subsequent parsing. The iptables command is a good example of a command-line program that follows this practice. From the man page: after [a module name is specified], various extra command line options become available, depending on the specific module. You can specify multiple extended match modules in one line, and you can use the -h or --help options after the module has been specified to receive help specific to that module. In the case of iptables, module names are specified as options, not as subcommands. based on options instead (as iptables does)? Also, is it possible to add these subparsers dynamically? For example, you would want to be able to load a module immediately after parsing the name instead of having to keep a predetermined list of all module names. I'm pretty sure that bzr dynamically loads modules this way. Can argparse help with this? Sorry for all of the questions. I ask them because I have some experience with adding the above features to optparse, and it was a lot of work to get it right. It also seems like there are a lot of programs that need to load modules dynamically. I would be really excited if argparse made this easier than optparse did. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

On 2009-09-30 15:17 PM, Andrew McNabb wrote:
I have not done so, but I suspect so. The implementation of .add_subparsers() adds to the positional argument list, but one could be written to append to the option list.
Not out-of-box, but it looks fairly straightforward to plug in. The subparser logic is mostly encapsulated in the _SubparsersAction class. You can register a new class for it: parser.register('action', 'parsers', MySubParsersAction) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Wed, Sep 30, 2009 at 1:17 PM, Andrew McNabb <amcnabb@mcnabbs.org> wrote:
Currently this is not supported, but it would be a nice feature.
You can probably already do this. I'm not 100% sure what you want to do, but it's certainly possible to define an argparse.Action that loads a module when it's invoked. It might look something like:: class MyAction(argparse.Action): def __call__(self, parser, namespace, value, option_string=None): mod = __import__(value) # or whatever Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Wed, Sep 30, 2009 at 02:40:20PM -0700, Steven Bethard wrote:
This looks much easier than what I was able to do in optparse. Cool. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

On Wed, Sep 30, 2009 at 09:21, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I am reluctant to jump on these argument parsing decorators until they have existed outside the standard library for a while and have settled down as I suspect some people want to use annotations to do type conversion/checking, others don't, etc. -Brett

On Mon, Sep 28, 2009 at 07:28:39AM -0700, Steven Bethard wrote:
No, I'd encourage such a note.
* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?
That looks nice indeed.
* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?
If clearly documented as "Hey, getopt users, check this out!" then I'd be indifferent to the proposed deprecation of getopt. Regards Floris

-1 for deprecating getopt. getopt is super-simple and especially useful for c programmers learning python. +1 for argparse.+1 for eventual deprecation of optparse - optparse and argparse have a very similar syntax and having both is just confusing. tsboapooowtdi On Mon, Sep 28, 2009 at 6:46 AM, Steven Bethard <steven.bethard@gmail.com>wrote:

On Mon, Sep 28, 2009 at 6:03 AM, Jon Ribbens <jon+python-dev@unequivocal.co.uk> wrote:
Ok, sounds like there are a number of supporters for keeping getopt around and just deprecating optparse. For those who'd like to keep getopt around, I have a few questions: * Would you be opposed to a note in the getopt documentation suggesting argparse as an alternative? * Would you like argparse to grow an add_getopt_arguments method (as in my other post)? * If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

* Would you be opposed to a note in the getopt documentation suggesting argparse as an alternative?
from the top of http://docs.python.org/library/getopt.html - "A more convenient, flexible, and powerful alternative is the optparse module."I think this statement should be emphasized better but it's already there so no worries. * Would you like argparse to grow an add_getopt_arguments method (as in my
other post)?
* If argparse grew an add_getopt_arguments, would you still want to keep
getopt around? And if so, why?
Argparse/optparse use a parser instance and getopt doesn't. I think that's a saved line you have to keep saving so this would be the better syntax imho: optlist, args = argparse.getopt(args, 'abc:d:') I was thinking on the general issue of what's better for the std-lib: 1. Mini modules - 3+ small modules in each subject that can be useful in different contexts. For example the xml libs. or 2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this. I think in general python std-lib went the path of mini modules. Its advantages 1. Each mini module can be coherent and while python isn't really OnlyOneWayToDoIt, each module in itself has OnlyOneWayToDoIt. 2. Documentation is less cluttered by a large amount of functions that all do the same thing but differently. This is very important for python's learning curve. disadvantages: 1. You have to craft your documentation carefully as to be very explicit on why you have all these modules for the same thing. Probably each module should explain it's purpose in relation to the other modules and this explanation should appear at the top of the documentation of each of the mini modules in the subject. Today it would be very hard for me to figure out how the xml modules interrelate, urllib has improved in python3 but still isn't perfect. 2. I could be using some silly way of doing things (getopt) while I really wanted to do it the good way (argparse). I think that while adding getopt functionality to argparse might be nice but would eventually cause clutter.

Yuvgoog Greenle <ubershmekel <at> gmail.com> writes:
+1 for adding argparse and eventually deprecating optparse, -0 for deprecating getopt.
2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this.
I'm not sure that it is, if you mean code size. In Python 2.5, logging is 1300 SLOC, less than say tarfile, pickletools, pydoc and decimal. Regards, Vinay Sajip

On Tue, 29 Sep 2009 12:28:39 am Steven Bethard wrote:
+1
* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?
0
* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?
Simplicity of the learning curve. Using it is as simple as: getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) Does argparse allow anything as simple as that? -- Steven D'Aprano

On Mon, Sep 28, 2009 at 8:27 AM, Steven D'Aprano <steve@pearwood.info> wrote:
You forgot the for-loop, nested if/else statements and type conversions. ;-)
Does argparse allow anything as simple as that?
If you mean, does argparse allow configuration to be specified using the getopt style ("a:b", ["alpha=", "beta"]), no, it currently doesn't. But if this is useful functionality, and would reasonably replace the getopt use cases, then I'd be happy to add it. In the simplest version, we might add something like:: options, args = argparse.getopt("a:b", ["alpha=", "beta"]) where you could then use "options" without any looping:: alpha = options.a or options.alpha beta = options.b or options.beta But if people still like the traditional getopt loop with nested if/elses better, then we might as well just keep getopt around and not add anything to argparse. I'm fine with that too, I just want to make sure that there isn't an obvious deficiency in argparse that we could easily fix. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Mon, Sep 28, 2009 at 08:49, Steven Bethard <steven.bethard@gmail.com> wrote:
=) I do wonder if people who are advocating for getopt sticking around realize how much extra code must be written to make sure what it gives back to you is in some sane manner. Let's take ``getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])`` as an example and simply assume that 'alpha' takes a string as an argument and that it's required and that 'beta' is a boolean flag. To pull everything out you could do:: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) options_dict = dict(options) alpha = options_dict.get('-a', options_dict.get('--alpha', '')) beta = '-b' in options_dict or '--beta' in options_dict main(alpha, beta, args) Obviously if one of the getopt supporters has a better way of doing this then please speak up. Now, Steven, can you show how best to do this in argparse? I am willing to bet that the total number of lines to do this is not that much more and does not require you to know to use 'or' or the dict constructor along with dict.get() in order to keep it compact. I can only imagine what some newbie might try to do in order to be correct (if they even try). -Brett

On Mon, Sep 28, 2009 at 12:22 PM, Brett Cannon <brett@python.org> wrote:
Here's the same option parsing in argparse: parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') args = parser.parse_args() main(args.alpha, args.beta) Or if those final positional arguments were actually meaningful, then you would add one more argument like this:: parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') parser.add_argument('gammas', nargs='*') args = parser.parse_args() main(args.alpha, args.beta, args.gammas) Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

for a live demo of how getopt is useful and flexible, I like how Audacity uses it: http://www.google.com/codesearch/p?hl=en&sa=N&cd=6&ct=rc#_hWFOhGz9lE/mezzo/scons/sconsign.py&q=getopt%20%22import%20getopt%22%20file:%5C.py$&l=264 To answer your question, it goes like this: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) for o, a in options: if o in ('-a', '--alpha'): alpha = a elif o in ('-b', '--beta'): beta = a main(alpha, beta, args) Notice a few things: 1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter). 2. There is no chance the parser will print things I don't want it to print. 3. You can do whatever you want using this flow - call a function, a binary operator (ie | is used in audacity scons example above) 4. in argparse this flow can be emulated and would be nicer, in ('-a', '--alpha') becomes == 'alpha' In a perfect world, getopt would be the low level parser that argparse and optparse rely on. This is not too far fetched btw, all that needs to be done is add another optional argument to getopt that would allow '-' and '--' to be replaced with arbitrary signs, OR moving the parsing code from argparse into getopt. On Mon, Sep 28, 2009 at 10:57 PM, Steven Bethard <steven.bethard@gmail.com>wrote:

2009/9/28 Yuvgoog Greenle <ubershmekel@gmail.com>:
That one does worry me. I'd rather argparse (or any library function) didn't call sys.exit on my behalf - it should raise an exception. Is it actually true that argparse exits? (I can imagine that it might if --help was specified, for example. An exception may not be right here, but I still don't like the idea of a straight exit - I've used too many C libraries that think they know when I want to exit).
2. There is no chance the parser will print things I don't want it to print.
That may also be bad - for example, Windows GUI-mode programs raise an error if they write to stdout/stderr. I could imagine using argparse for such a program, and wanting to do something with --help other than write to stdout and exit (a message box, for example). And yet, I'd want access to the text argparse would otherwise write to stdout. Paul.

Paul Moore wrote:
You can override ArgumentParser.error() to raise an exception.
It looks like this might not be so easy to do. I'd suggest adding a file-like object to the constructor, defaulting to sys.stderr; or maybe an ArgumentParser.print() method that can be overridden. Eric.

On Tue, Sep 29, 2009 at 1:31 PM, Paul Moore <p.f.moore@gmail.com> wrote:
This is behavior that argparse inherits from optparse, but I believe it's still what 99.9% of users want. If you're writing a command line interface, you don't want a stack trace when there's an error message (which is what you'd get if argparse just raised exceptions) you want an exit with an error code. That's what command line applications are supposed to do. If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do.
There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On approximately 9/29/2009 1:57 PM, came the following characters from the keyboard of Steven Bethard:
Documenting both of these options would forestall people from thinking it is only useful for console applications. An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good. In making the method public you might want to rename it to something other than print_message. If all the messages are errors, including "error" in the name would be good. If not, classifying the errors vs non-errors as an extra parameter might be good (in other words... inform the user and continue, or inform the user and exit). Separating the message from the classification is not good, because that leads to dialog boxes having only an "OK" button, and depending on the message, it can be really inappropriate to display an "OK" button... buttons named "Sorry", "Alas!", and "Exit" or "Quit" are often more appropriate, even when there is no user choice possible. Clearly if someone is writing a GUI, they are willing to write lots of lines of code to do things... a couple more well-documented lines to integrate argparse into their chosen GUI doesn't seem onerous. Especially if they can cut, paste, and hack from the above-suggested example code, like they do for the rest of their GUI code. Well, at least, cut, paste, and hack is how I write GUIs when I bother. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

On Tue, Sep 29, 2009 at 3:04 PM, Glenn Linderman <v+python@g.nevcal.com> wrote:
I'm totally fine with people thinking it is only useful for console applications. That's what it's intended for. That said, if there are people out there who want to use it for other applications, I'm happy to make things easier for them if I know concretely what they want.
An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good.
I don't really use GUI libraries, so I wouldn't be able to come up with such an example. I'd also rather not make API changes based on speculative use cases, so before I spend time documenting these things, I'd really like to hear from someone who has already, say, used getopt or optparse in conjunction with a GUI library, and what feedback they have about that. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On approximately 9/29/2009 4:38 PM, came the following characters from the keyboard of Steven Bethard:
Hmm. Maybe that is partly why (sadly) so many GUI programs don't offer much in the way of command line options. Of course, many of their users in Windowsland wouldn't know how to tweak the shortcut to invoke them with options anyway, which might be another part. Fortunately, there are some good Windows programs that do offer rich GUIs and also rich command line options (e.g. IrfanView and ImgBurn)
I'm not a Tk user, just coming to Python from Perl, where I used Win32::GUI, but when I was looking for a functional and portable GUI development package, Perl didn't have one (still doesn't have one that supports Unicode and printing), and Python does... so here I come to Python and Qt. From my experience in Perl GUI Windowsland, the primary issue with command line parsing is displaying the message in a dialog instead of STDOUT. But the message and the user choices have to be known at the same time to design the dialog box. And, there is nothing so aggravating as to be shown an error message, and the only option is a button that says "OK", when it clearly isn't OK. So the tone/type of the messages also needs to be known, even when there are no user choices. The --help option could display the help message, and offer OK. Many errors (particularly unrecoverable ones) should display the error message, and offer an Exit button, or just the close box. A few (although probably only highly customized user options) might want to give the user multiple recovery options. So, I guess I'm in the unfortunate position of seeing the need, but not having used these specific technologies enough to offer an example either. So far, I've only used optparse (just now hearing about argparse in this thread) for command line programs in Python, and I am still just experimenting with Qt, and don't have enough familiarity with it yet to think that what I'm doing is best practices. I think it would be sad to a new replacement for optparse that didn't GUI usage, though, at least in a limited form. The concepts I describe seem sufficient from experience in other environments, and I would think they would be sufficient in Python too, but I'm too new here to state that definitely. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

On 09/29/2009 04:38 PM, Steven Bethard wrote:
Note: on Unix systems, --help should still print to the terminal, not pop up a GUI text box with the help information. So being able to override the behaviour might be good but it is more than a simple, GUI vs console distinction. Are we talking about anything else than --help output (for the printing question)? About exit(), I agree with others about wanting to catch the exception myself and then choosing to exit from the code. I'm not sure that it's actually useful in practice, though...it might just feel cleaner but not actually be that helpful. -Toshio

I haven't checked if it's possible, but I suggest Argparse have it's own exception class that inherits from SystemExit and that exception would be thrown. ParseError, or something similar. I suggest this just because it would be more readable I guess and would exactly explain why this code exits. On Sat, Oct 3, 2009 at 8:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle <ubershmekel@gmail.com> wrote:
I've never seen such an idiom before (subclassing SystemExit) but it would certainly be possible create an ArgumentParserExit exception like that. Then you would have your choice of overriding .exit() or catching the exception. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

Steven Bethard wrote:
Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sat, Oct 3, 2009 at 8:17 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I'm certainly fine with that -- I'm just trying to make sure I've addressed whatever needs addressed to get argparse in.
I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands.
Maybe I misunderstood, but I think this is already the default argparse behavior, no? import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo') subparsers = parser.add_subparsers() parser1 = subparsers.add_parser('1') parser1.add_argument('--bar') parser2 = subparsers.add_parser('2') parser2.add_argument('baz') parser.parse_args(['--help'])
positional arguments: {1,2} optional arguments: -h, --help show this help message and exit --foo FOO
optional arguments: -h, --help show this help message and exit --bar BAR
positional arguments: baz optional arguments: -h, --help show this help message and exit Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

Steven Bethard wrote:
Cool. I didn't realise that help for subcommands was already implemented. :-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sat, Oct 3, 2009 at 7:21 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Check it out: def ParseAndRun(): crazy_external_function_that_might_exit() # Argparse blah blah parser.parse_args() if __name__ == "__main__": try: ParseAndRun() except SystemExit: # was it crazy_external_function_that_might_exit or an argparse error? I know this might come through as bike shedding but it's just customary python that every module have it's own exception types as to not mix them up with others. --yuv

Yuvgoog Greenle wrote:
Then subclass and override .exit() as discussed - or put proper exception handling around the call to parse_args() (optionally rethrowing with whatever custom exception type you wish). Michael
--yuv
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sun, 4 Oct 2009 03:38:31 am Yuvgoog Greenle wrote:
Does it matter? What difference does it make?
You are mistaken.
Plain, ordinary AttributeError, not OSModuleSubclassOfAttributeError. I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types. There's no reason to treat SystemExit as special in this regard. optparse, for example, calls sys.exit(), which operates by raising SystemExit. Ordinary SystemExit, not a special module-specific subclass. -- Steven D'Aprano

On Sat, Oct 3, 2009 at 8:29 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types.
I might be wrong on this. Your point is extra true for modules in the standard library (which is what we're talking about for argparse). I just think that if a parser error is causing the SystemExit, I would rather catch a parser error than catching a SystemExit for the sake of readability. It saves me the comments: # Catching SystemExit because parse_args() throws SystemExit on parser errors. # Subclassing ArgumentParser and overriding exit because I don't want to exit() upon parser errors. So I'm sorry if what I said was irrelevant. I've never written or taken part of writing a std-lib module. --yuv

On Sun, 4 Oct 2009 04:46:19 am Yuvgoog Greenle wrote:
But why are you catching the error? As a general rule, you *want* your command line app to exit if it can't understand the arguments you pass to it. What else can it do? Guess what you wanted? Assuming you have a reason for catching the exception, I don't see that there's any difference in readability between these: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except argparse.ParserError: process_error() else: main(ns) and: parser = argparse.ArgumentParser() setup_args(parser) try: ns = parser.parse_args() except SystemExit: process_error() else: main(ns) You don't need a comment warning that you are catching SystemExit because parse_args raises SystemExit, any more than you need a comment saying that you are catching ValueError because some function raises ValueError. The fact that you are catching an exception implies that the function might raise that exception. A comment like: "Catching SystemExit because parse_args() throws SystemExit on parser errors." is up them with comments like this: x += 1 # Add 1 to x. -- Steven D'Aprano

* Steven D'Aprano wrote:
It's semantically different. You usually don't catch SystemExit directly, because you want your programs to be stopped. Additionally, a library exiting your program is badly designed, as it's unexpected. Thatswhy such a comment is useful. Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse. That way all parties here should be satisifed. (I do the same all the time in my signal handlers - that's another reason I'd rather not catch SystemExit directly as well :-) nd -- "Umfassendes Werk (auch fuer Umsteiger vom Apache 1.3)" -- aus einer Rezension <http://pub.perlig.de/books.html#apache2>

On Sun, 4 Oct 2009 05:35:04 am André Malo wrote:
Exactly -- so why catch it at all? But even if there is a good reason to catch it, there's still no reason to subclass SystemExit unless argparse wants to distinguish different types of fatal error, and allow code to catch some but not all. But frankly, if I'm having a hard time thinking of a reason to catch SystemExit, I'm having an even harder time thinking why you'd want to (say) catch SystemExitTooManyArguments but not SystemExitMissingArgument.
Additionally, a library exiting your program is badly designed, as it's unexpected.
It's not unexpected for an argument parser. Do you know any applications that run when given invalid arguments? As a general rule, what can the application do? Guess what you wanted?
Thatswhy such a comment is useful.
The comment doesn't tell you anything that wasn't obvious from the code. It is pointless.
Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse.
In the following code snippet: try: ns = argparse.parse_args() except SystemExit: ... is there any confusion between SystemExit raised by parse_args and SystemExit raised by other components? *What* other components? If SystemExit was raised in that try block, where could it have come from other than parse_args? Do you write comments like these? try: value = mydict[key] except KeyError: # Catch KeyError raised by dict lookup ... try: n = mylist.index(x) except ValueError: # Catch ValueError raised by mylist.index ... Useless comments are worse than no comments, because useless comments waste the readers' time and they risk becoming out of sync with the code and turning into misleading comments.
That way all parties here should be satisifed.
No. It wastes the time of the argparse developer, it wastes the time of people learning argparse, it wastes the time of people who read the code and wonder what's the difference between SystemExit and ArgparseSystemExit. (Answer: there is no difference.) -- Steven D'Aprano

Steven D'Aprano wrote:
There are uses of argparse outside of command line apps. For example, I use it to parse --options for IPython %magic commands. Of course, I just subclass ArgumentParser and replace the .error() method. I require a particular IPython exception type in order for the error to be recognized correctly in the %magic subsystem. The other use case, as mentioned earlier, was for debugging your parser on the interactive prompt. A custom subclass may also be able to hold more machine-readable information about the error than the formatted error message, but I don't have any particular use cases about what may be the most useful. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sun, 4 Oct 2009 08:21:46 am Robert Kern wrote:
Exactly. There are uses for catching SystemExit, which is why it's an exception and not an unconditional exit. But they're rare, and not difficult to deal with: in your case, having argparse raise a subclass of SystemExit won't help you, you would still need to subclass, and in other cases, you can just catch SystemExit.
Nobody has requested that the exception expose more information. They've requested that argparse paint the SystemExit a slightly different shade of yellow to the colour it already is -- this is pure bike-shedding. Subclassing SystemExit just in case someday in the indefinite future there comes a need to add extra information to the exception falls foul of You Ain't Gonna Need It. Keep it simple -- if, someday, such a need becomes apparent, then subclass. -- Steven D'Aprano

Yuvgoog Greenle wrote:
Not in my Python world it isn't. While that is sometimes the right answer, more often the right answer is to properly scope your try/except statement in order to avoid catching exceptions from code you aren't interested in. If you end up stuck with a library call that lacks both granularity and specificity of exceptions then you live with it (and maybe file a bug report/feature request with the library author). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On 2009-09-29 18:38 PM, Steven Bethard wrote:
I use argparse (and previously optparse) frequently to handle the command line arguments of GUI apps. I tend to use them in the same way as CLI programs, though, since I usually only use command line arguments when starting the GUIs from the terminal. I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs. I'm not sure what would make a program "GUI-mode" or not. Certainly, I have written Python programs that use wxPython and PyQt on Windows that print to stdout/stderr, and they appear to work fine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2009/9/30 Robert Kern <robert.kern@gmail.com>:
I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs.
:-)
It's the difference between using python.exe and pythonw.exe. There's a flag in the executable header saying whether the executable is "console mode" or "gui mode". GUI mode programs are run (by the OS) without a console (or if run from a console prompt, they automatically detach from that console, much like Unix programs which fork themselves to leave the terminal group (did I get the terminology right?) but done by the OS). As a result, the program has no valid stdin/out/err handles. Any attempt to write to them causes the program to crash. Traceback (most recent call last): File "hello.py", line 13, in <module> main() File "hello.py", line 7, in main sys.stdout.flush() IOError: [Errno 9] Bad file descriptor (Question - is it *ever* possible for a Unix program to have invalid file descriptors 0,1 and 2? At startup - I'm assuming anyone who does os.close(1) knows what they are doing!) Paul.

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:
Normally you don't close fd 0, 1 & 2 but rather redirect them to /dev/null (a black hole). That saves you from nastiness when a library or something misbahaves and tries to use one of those. It would have been nice if windows GUI-mode programs as you describe them would do something similar. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:
Yes, at startup you just have the file descriptors your parent process gave you. This may or may not include 0, 1, and 2, and may or may not include other descriptors too. I seem to recall there have been security holes caused in the past by people assuming 0-2 will exist.

Paul Moore wrote:
Of course; simply use the >&- pseudo-redirection, which has been a standard sh feature (later inherited by ksh and bash, but not csh) for ~30 years. The error message is amusing, too: $ python -c 'print "foo"' >&- close failed in file object destructor: Error in sys.excepthook: Original exception was: Adding an explicit flush results in a more understandable error message: Traceback (most recent call last): File "<string>", line 1, in <module> IOError: [Errno 9] Bad file descriptor

Hrvoje Niksic <hrvoje.niksic <at> avl.com> writes:
Python 3 complains at startup and is a bit more explicit: $ ./python -c '1' >&- Fatal Python error: Py_Initialize: can't initialize sys standard streams OSError: [Errno 9] Bad file descriptor Abandon Of course, if it is stderr that you explicitly close, you lose all reporting: $ ./python -c '1' 2>&- Abandon Regards Antoine.

2009/10/8 Paul Moore <p.f.moore@gmail.com>:
When running pythonw, fileno(stdout) is negative, so sys.stdout is set to None, and print() silently returns. But the situation is not perfect, see http://bugs.python.org/issue6501 where Apache provides a stdout, but python cannot determine the encoding because there is no console. -- Amaury Forgeot d'Arc

2009/9/29 Steven Bethard <steven.bethard@gmail.com>:
Thanks, that's fine for me (as things stand, no need to publicly expose and document _print_message). BTW, the helpful and responsive way you reply to queries is much appreciated. I'm +1 on the PEP (although I see why people want getopt to stay, so I'm happy to leave that and only deprecate optparse). Paul.

Steven Bethard wrote:
Instead of forcing the user to override the ArgumentParser class to change how errors are handled, I suggest adding a separate method ArgumentParser.parse_args_with_exceptions() that raises exceptions instead of writing to stdout/stderr and never calls sys.exit(). Then implement ArgumentParser.parse_args() as a wrapper around parse_args_with_exceptions(): class ArgparseError(Exception): """argparse-specific exception type.""" pass class ArgumentError(ArgparseError): # ... class ArgumentParser(...): # ... def parse_args_with_exceptions(...): # like the old parse_args(), except raises exceptions instead of # writing to stdout/stderr or calling sys.exit()... def parse_args(self, *args, **kwargs): try: self.parse_args_with_exceptions(*args, **kwargs) except ArgparseError as e: self.print_usage(_sys.stderr) self.exit(status=2, message=(_('%s: error: %s\n') % (self.prog, e,))) # perhaps catch other exceptions that need special handling... def error(self, message): raise ArgparseError(message) The exception classes should hold enough information to be useful to non-command-line users, and obviously contain error messages that are output to stderr by default. This would allow non-command-line users to call parse_args_with_exceptions() and handle the exceptions however they like. Michael

Greg Ewing wrote:
I was going to mention that. Between catching SystemExit, the '-i' switch to the interpeter and the PYTHONINSPECT environment variable, investigating applications that call sys.exit isn't *that* difficult. It's a far cry from the instant exits of a C level abort call. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

As Yuvgoog Greenle says, the canonical getopt way is to write alpha = None beta = False options, args = getopt.getopt(sys.argv[1:],"a:b",['alpha=','beta']): for opt, val in options: if arg in ('-a','--alpha'): alpha = val elif arg in ('-b','--beta'): beta = True main(alpha, beta, args) Even though this is many more lines, I prefer it over optparse/argparse: this code has only a single function call, whereas the argparse version has three function calls to remember. The actual processing uses standard Python data structures which I don't need to look up in the documentation.
Now, Steven, can you show how best to do this in argparse?
This demonstrates my point: you were able to use getopt right away (even though not in the traditional way), whereas you need to ask for help on using argparse properly.
See above - getopt users don't care about compactness in the processing.
Depends on the background of the newbie. If they come from C, they immediately recognize the way of doing things. Regards, Martin

On Mon, Sep 28, 2009 at 8:44 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I don't think this is fair at all. I am totally unable to write getopt code without checking the documentation -- I don't remember the format string syntax, nor what the function returns. But that's just how library modules work -- if you don't know the library, you have to read the documentation. The only reason getopt is easier for you is that you're already familiar with the API from C. That said, I can certainly understand that folks who use getopt in C would have an easy transition to getopt in Python and a harder transition to argparse. You didn't directly answer my question of whether adding an "add_getopt_arguments" function would meet your needs, but I infer from your answer above that it wouldn't because you'd still need to make multiple function calls which is your primary complaint. So at this point, I think it's clear that there's nothing I can reasonably add to argparse to make getopt people more comfortable that isn't just duplicating what's already in getopt. So let's just deprecate optparse (as in the PEP), and leave getopt alone (other than adding a note to the documentation suggesting argparse as an alternative). -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Mon, Sep 28, 2009 at 20:44, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Actually, I had to read the docs for getopt. And I chose to not even try argparse when the creator of the module is cc'ed on the email and can obviously do a better example using his own code then I could. -Brett

Martin> alpha = None Martin> beta = False Martin> options, args = getopt.getopt(sys.argv[1:],"a:b",['alpha=','beta']): Martin> for opt, val in options: ... Martin> Even though this is many more lines, I prefer it over Martin> optparse/argparse: this code has only a single function call, ... Agreed. I have never completely wrapped my brain around optparse. Getopt I just remember. Skip

skip@pobox.com wrote:
I have never completely wrapped my brain around optparse. Getopt I just remember.
Seems to me that optparse and argparse are fairly similar in their APIs, and that argparse isn't going to be significantly easier to fit in one's brain than optparse. There's an art to coming up with an API that makes simple things easy and other things possible. I'm not convinced that argparse represents a subsantial enough advancement in that art to justify replacing optparse with it in the stdlib, and thereby forcing everyone to learn a similar-but-different API. -- Greg

On Sep 29, 2009, at 6:51 PM, Greg Ewing wrote:
There's no question it is if you're doing more complicated stuff, like extending it or using subcommands. After I converted my code from optparse to argparse, I ended up with less stuff that was more regular and easier to understand. It convinced me that argparse is a win. -Barry

Greg Ewing wrote:
As someone that has written multiple optparse based utility scripts, I would say that yes, argparse is a *huge* improvement. Several things that I implemented for my own use in a rather clumsy fashion (subcommands, aggregated parsers, non-interspersed arguments) have far more elegant support built directly into argparse. For the getopt-vs-opt/argparse discussion, I believe the major distinction is in the relative balance between procedural and declarative style of argument handling. getopt is very procedural - you define a minimal amount regarding the options you accept, but then do the bulk of the command line processing yourself optparse is declarative to some degree, but forces you to drop back to a procedural style to handle arguments and subcommands. argparse takes things even further in a declarative direction by adding explicit support for positional arguments and subcommands. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Barry Warsaw wrote:
People assess the mental cost differently though - for getopt, they tend to assign the cost to the script they're currently writing, while for optparse/argparse they assign the cost to those two libraries being difficult to learn :) Keeping getopt around *and* including a "add_getopt_arguments" method in argparse is probably the best of both worlds, in that it allows for relatively straightforward evolution of an application: 1. Start with getopt 2. As the getopt argument parsing gets twisty and arcane and maintaining the help string becomes a nightmare, move to argparse with the "add_getopt_arguments" method. 3. Over time, convert more arguments to proper argparse arguments with full documentation. (Note that getting a good help string "for free" is one of the biggest gains I currently find in using optparse over getopt. This is especially so once you start making use of option groups) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Tue, Sep 29, 2009 at 6:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
One thing that wouldn't be good in this transition is that "add_getopt_arguments" can only generate the part of the help string that says "-a" and "--author" exist as options -- it can't add the additional bit of text that says what they do because that's not provided to the getopt API. I suspect this makes the transition less convenient because with getopt you were probably already manually maintaining a usage message that had at least some of this information, and switching to "add_getopt_arguments" would mean throwing this information away. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Sep 29, 2009, at 9:18 PM, Nick Coghlan wrote:
Maybe. I haven't been following this entire thread, but I don't see much point in making it easy to go from getopt to argparse. The two are so different that I think you're either going to jump all in or not. Maybe that's just me though as I really don't see much use for getopt any more (even for throwaway scripts). -Barry

On Tue, Sep 29, 2009 at 9:30 PM, Barry Warsaw <barry@python.org> wrote:
Maybe. I haven't been following this entire thread, but I don't see much point in making it easy to go from getopt to argparse.
I'm with you on this; specific getopt uses are more likely to be switched to opt/argparse than to shift gradually via hybrid APIs.
Heh. I never liked getopt anyway. :-) -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller

Brett Cannon <brett <at> python.org> writes:
Obviously if one of the getopt supporters has a better way of doing this then please speak up.
I'm not a getopt supporter per se - I'm +1 for argparse - but Opster provides some nice syntax sugar on top of getopt, and can be seen here: http://blogg.ingspree.net/blog/2009/09/14/opster/ I've contacted Steven about a similar approach for argparse, and we're waiting for Yuvgoog's work on argparse(func) to be done to see if there's a feasible similar approach.

On a tangent -- a use case that I see happening frequently but which is pretty poorly supported by optparse is a command-line that has a bunch of general flags, then a 'subcommand name', and then more flags, specific to the subcommand. Most here are probably familiar with this pattern from SVN, Hg, and other revision control systems (P4 anyone?) with a rich command-line interface. There are some variants, e.g. whether global and subcommand-specific flags may overlap, and whether flags may follow positional args (Hg and SVN seem to differ here a bit). I've helped write at least two tools at Google that have this structure; both used different approaches, and neither was particularly easy to get right. Getting all the different --help output to make sense was mostly a manual process. (E.g. "foo --help" should print the general flags and the list of known subcommands, whereas "foo --help subcommand" should print flags and other usage info about the specific subcommand.) Also switching out to different calls based on the subcommand should/might be part of this. I would be willing to live with a third option parser in the stdlib if it solved this problem well. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 2009-09-30 11:38 AM, Guido van Rossum wrote:
I don't think argparse supports the "foo --help subcommand" OOB. I think it would be simple to modify argparse to make it do so. It does support general options followed by a subcommand with options, though. http://argparse.googlecode.com/svn/tags/r101/doc/other-methods.html#sub-comm... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sep 30, 2009, at 12:58 PM, Robert Kern wrote:
Right. I've made it kind of work in Mailman 3, but it would be nice for argparse to support this out of the box. Note that I think you want two forms: foo help subcommand foo subcommand --help to basically print the same help text. This is the way bzr does it for example and it works great. Other than that, I think argparse supports Guido's use case very nicely. -Barry

On Wed, Sep 30, 2009 at 02:22:53PM -0400, Barry Warsaw wrote:
From my cursory reading of the documentation, it looks like argparse can only add subparsers for subcommands. Is there any way to add subparsers
In some commands, options as well as subcommands can change subsequent parsing. The iptables command is a good example of a command-line program that follows this practice. From the man page: after [a module name is specified], various extra command line options become available, depending on the specific module. You can specify multiple extended match modules in one line, and you can use the -h or --help options after the module has been specified to receive help specific to that module. In the case of iptables, module names are specified as options, not as subcommands. based on options instead (as iptables does)? Also, is it possible to add these subparsers dynamically? For example, you would want to be able to load a module immediately after parsing the name instead of having to keep a predetermined list of all module names. I'm pretty sure that bzr dynamically loads modules this way. Can argparse help with this? Sorry for all of the questions. I ask them because I have some experience with adding the above features to optparse, and it was a lot of work to get it right. It also seems like there are a lot of programs that need to load modules dynamically. I would be really excited if argparse made this easier than optparse did. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

On 2009-09-30 15:17 PM, Andrew McNabb wrote:
I have not done so, but I suspect so. The implementation of .add_subparsers() adds to the positional argument list, but one could be written to append to the option list.
Not out-of-box, but it looks fairly straightforward to plug in. The subparser logic is mostly encapsulated in the _SubparsersAction class. You can register a new class for it: parser.register('action', 'parsers', MySubParsersAction) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Wed, Sep 30, 2009 at 1:17 PM, Andrew McNabb <amcnabb@mcnabbs.org> wrote:
Currently this is not supported, but it would be a nice feature.
You can probably already do this. I'm not 100% sure what you want to do, but it's certainly possible to define an argparse.Action that loads a module when it's invoked. It might look something like:: class MyAction(argparse.Action): def __call__(self, parser, namespace, value, option_string=None): mod = __import__(value) # or whatever Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Wed, Sep 30, 2009 at 02:40:20PM -0700, Steven Bethard wrote:
This looks much easier than what I was able to do in optparse. Cool. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

On Wed, Sep 30, 2009 at 09:21, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I am reluctant to jump on these argument parsing decorators until they have existed outside the standard library for a while and have settled down as I suspect some people want to use annotations to do type conversion/checking, others don't, etc. -Brett

On Mon, Sep 28, 2009 at 07:28:39AM -0700, Steven Bethard wrote:
No, I'd encourage such a note.
* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?
That looks nice indeed.
* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?
If clearly documented as "Hey, getopt users, check this out!" then I'd be indifferent to the proposed deprecation of getopt. Regards Floris
participants (27)
-
"Martin v. Löwis"
-
Amaury Forgeot d'Arc
-
Andrew McNabb
-
André Malo
-
Antoine Pitrou
-
Barry Warsaw
-
Brett Cannon
-
Eric Smith
-
Floris Bruynooghe
-
Fred Drake
-
Glenn Linderman
-
Greg Ewing
-
Guido van Rossum
-
Hrvoje Niksic
-
Jon Ribbens
-
m h
-
Michael Foord
-
Michael Haggerty
-
Nick Coghlan
-
Paul Moore
-
Robert Kern
-
skip@pobox.com
-
Steven Bethard
-
Steven D'Aprano
-
Toshio Kuratomi
-
Vinay Sajip
-
Yuvgoog Greenle