On Mon, Sep 28, 2009 at 06:59:45AM +0300, Yuvgoog Greenle wrote:

-1 for deprecating getopt. getopt is super-simple and especially useful for c programmers learning python.

+1 for argparse.+1 for eventual deprecation of optparse - optparse and argparse have a very similar syntax and having both is just confusing. tsboapooowtdi

+1 on all of this :-) It would be a shame to see getopt go but optparse -> argparse seems logical. Floris

On Mon, Sep 28, 2009 at 09:38:20AM +0100, Floris Bruynooghe wrote:

On Mon, Sep 28, 2009 at 06:59:45AM +0300, Yuvgoog Greenle wrote:

...

-1 for deprecating getopt. getopt is super-simple and especially useful for c programmers learning python.

+1 for argparse.+1 for eventual deprecation of optparse - optparse and argparse have a very similar syntax and having both is just confusing. tsboapooowtdi

+1 on all of this :-)

It would be a shame to see getopt go but optparse -> argparse seems logical.

+1 from me too - keep getopt, deprecate optparse.

* Would you be opposed to a note in the getopt documentation suggesting argparse as an alternative?

from the top of http://docs.python.org/library/getopt.html - "A more convenient, flexible, and powerful alternative is the optparse module."I think this statement should be emphasized better but it's already there so no worries. * Would you like argparse to grow an add_getopt_arguments method (as in my

other post)?

* If argparse grew an add_getopt_arguments, would you still want to keep

getopt around? And if so, why?

Argparse/optparse use a parser instance and getopt doesn't. I think that's a saved line you have to keep saving so this would be the better syntax imho: optlist, args = argparse.getopt(args, 'abc:d:') I was thinking on the general issue of what's better for the std-lib: 1. Mini modules - 3+ small modules in each subject that can be useful in different contexts. For example the xml libs. or 2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this. I think in general python std-lib went the path of mini modules. Its advantages 1. Each mini module can be coherent and while python isn't really OnlyOneWayToDoIt, each module in itself has OnlyOneWayToDoIt. 2. Documentation is less cluttered by a large amount of functions that all do the same thing but differently. This is very important for python's learning curve. disadvantages: 1. You have to craft your documentation carefully as to be very explicit on why you have all these modules for the same thing. Probably each module should explain it's purpose in relation to the other modules and this explanation should appear at the top of the documentation of each of the mini modules in the subject. Today it would be very hard for me to figure out how the xml modules interrelate, urllib has improved in python3 but still isn't perfect. 2. I could be using some silly way of doing things (getopt) while I really wanted to do it the good way (argparse). I think that while adding getopt functionality to argparse might be nice but would eventually cause clutter.

On Mon, Sep 28, 2009 at 8:27 AM, Steven D'Aprano wrote:

On Tue, 29 Sep 2009 12:28:39 am Steven Bethard wrote:

...

* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?

0

...

* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?

Simplicity of the learning curve. Using it is as simple as:

getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])

You forgot the for-loop, nested if/else statements and type conversions. ;-)

Does argparse allow anything as simple as that?

If you mean, does argparse allow configuration to be specified using the getopt style ("a:b", ["alpha=", "beta"]), no, it currently doesn't. But if this is useful functionality, and would reasonably replace the getopt use cases, then I'd be happy to add it. In the simplest version, we might add something like:: options, args = argparse.getopt("a:b", ["alpha=", "beta"]) where you could then use "options" without any looping:: alpha = options.a or options.alpha beta = options.b or options.beta But if people still like the traditional getopt loop with nested if/elses better, then we might as well just keep getopt around and not add anything to argparse. I'm fine with that too, I just want to make sure that there isn't an obvious deficiency in argparse that we could easily fix. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On Mon, Sep 28, 2009 at 08:49, Steven Bethard wrote:

On Mon, Sep 28, 2009 at 8:27 AM, Steven D'Aprano wrote:

...

On Tue, 29 Sep 2009 12:28:39 am Steven Bethard wrote:

...

* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?

0

...

* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?

Simplicity of the learning curve. Using it is as simple as:

getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])

You forgot the for-loop, nested if/else statements and type conversions. ;-)

=) I do wonder if people who are advocating for getopt sticking around realize how much extra code must be written to make sure what it gives back to you is in some sane manner. Let's take ``getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])`` as an example and simply assume that 'alpha' takes a string as an argument and that it's required and that 'beta' is a boolean flag. To pull everything out you could do:: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) options_dict = dict(options) alpha = options_dict.get('-a', options_dict.get('--alpha', '')) beta = '-b' in options_dict or '--beta' in options_dict main(alpha, beta, args) Obviously if one of the getopt supporters has a better way of doing this then please speak up. Now, Steven, can you show how best to do this in argparse? I am willing to bet that the total number of lines to do this is not that much more and does not require you to know to use 'or' or the dict constructor along with dict.get() in order to keep it compact. I can only imagine what some newbie might try to do in order to be correct (if they even try). -Brett

for a live demo of how getopt is useful and flexible, I like how Audacity uses it: http://www.google.com/codesearch/p?hl=en&sa=N&cd=6&ct=rc#_hWFOhGz9lE/mezzo/scons/sconsign.py&q=getopt%20%22import%20getopt%22%20file:%5C.py$&l=264 To answer your question, it goes like this: options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) for o, a in options: if o in ('-a', '--alpha'): alpha = a elif o in ('-b', '--beta'): beta = a main(alpha, beta, args) Notice a few things: 1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter). 2. There is no chance the parser will print things I don't want it to print. 3. You can do whatever you want using this flow - call a function, a binary operator (ie | is used in audacity scons example above) 4. in argparse this flow can be emulated and would be nicer, in ('-a', '--alpha') becomes == 'alpha' In a perfect world, getopt would be the low level parser that argparse and optparse rely on. This is not too far fetched btw, all that needs to be done is add another optional argument to getopt that would allow '-' and '--' to be replaced with arbitrary signs, OR moving the parsing code from argparse into getopt. On Mon, Sep 28, 2009 at 10:57 PM, Steven Bethard wrote:

On Mon, Sep 28, 2009 at 12:22 PM, Brett Cannon wrote:

...

On Mon, Sep 28, 2009 at 08:49, Steven Bethard wrote:

...

On Mon, Sep 28, 2009 at 8:27 AM, Steven D'Aprano wrote:

...

On Tue, 29 Sep 2009 12:28:39 am Steven Bethard wrote:

...

* Would you like argparse to grow an add_getopt_arguments method (as in my other post)?

0

...

* If argparse grew an add_getopt_arguments, would you still want to keep getopt around? And if so, why?

Simplicity of the learning curve. Using it is as simple as:

getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])

You forgot the for-loop, nested if/else statements and type conversions. ;-)

=) I do wonder if people who are advocating for getopt sticking around realize how much extra code must be written to make sure what it gives back to you is in some sane manner.

Let's take ``getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])`` as an example and simply assume that 'alpha' takes a string as an argument and that it's required and that 'beta' is a boolean flag. To pull everything out you could do::

options, args = getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"]) options_dict = dict(options) alpha = options_dict.get('-a', options_dict.get('--alpha', '')) beta = '-b' in options_dict or '--beta' in options_dict

main(alpha, beta, args)

Obviously if one of the getopt supporters has a better way of doing this then please speak up.

Now, Steven, can you show how best to do this in argparse?

Here's the same option parsing in argparse:

parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') args = parser.parse_args()

main(args.alpha, args.beta)

Or if those final positional arguments were actually meaningful, then you would add one more argument like this::

parser = argparse.ArgumentParser() parser.add_argument('-a', '--alpha') parser.add_argument('-b', '--beta', action='store_true') parser.add_argument('gammas', nargs='*') args = parser.parse_args()

main(args.alpha, args.beta, args.gammas)

Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ubershmekel%40gmail.com

On Mon, Sep 28, 2009 at 8:44 PM, "Martin v. Löwis" wrote:

...

Let's take ``getopt.getopt(sys.argv[1:], "a:b", ["alpha=", "beta"])``

As Yuvgoog Greenle says, the canonical getopt way is to write [snip getopt code] Even though this is many more lines, I prefer it over optparse/argparse: this code has only a single function call, whereas the argparse version has three function calls to remember. The actual processing uses standard Python data structures which I don't need to look up in the documentation.

...

Now, Steven, can you show how best to do this in argparse?

This demonstrates my point: you were able to use getopt right away (even though not in the traditional way), whereas you need to ask for help on using argparse properly.

I don't think this is fair at all. I am totally unable to write getopt code without checking the documentation -- I don't remember the format string syntax, nor what the function returns. But that's just how library modules work -- if you don't know the library, you have to read the documentation. The only reason getopt is easier for you is that you're already familiar with the API from C. That said, I can certainly understand that folks who use getopt in C would have an easy transition to getopt in Python and a harder transition to argparse. You didn't directly answer my question of whether adding an "add_getopt_arguments" function would meet your needs, but I infer from your answer above that it wouldn't because you'd still need to make multiple function calls which is your primary complaint. So at this point, I think it's clear that there's nothing I can reasonably add to argparse to make getopt people more comfortable that isn't just duplicating what's already in getopt. So let's just deprecate optparse (as in the PEP), and leave getopt alone (other than adding a note to the documentation suggesting argparse as an alternative). -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

Martin> alpha = None Martin> beta = False Martin> options, args = getopt.getopt(sys.argv[1:],"a:b",['alpha=','beta']): Martin> for opt, val in options: ... Martin> Even though this is many more lines, I prefer it over Martin> optparse/argparse: this code has only a single function call, ... Agreed. I have never completely wrapped my brain around optparse. Getopt I just remember. Skip

On Tue, Sep 29, 2009 at 2:31 PM, Paul Moore wrote:

2009/9/28 Yuvgoog Greenle :

...

1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter).

That one does worry me. I'd rather argparse (or any library function) didn't call sys.exit on my behalf - it should raise an exception. Is it actually true that argparse exits? (I can imagine that it might if --help was specified, for example. An exception may not be right here, but I still don't like the idea of a straight exit - I've used too many C libraries that think they know when I want to exit).

+1

On Tue, Sep 29, 2009 at 1:31 PM, Paul Moore wrote:

2009/9/28 Yuvgoog Greenle :

...

1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter).

That one does worry me. I'd rather argparse (or any library function) didn't call sys.exit on my behalf - it should raise an exception. Is it actually true that argparse exits? (I can imagine that it might if --help was specified, for example. An exception may not be right here, but I still don't like the idea of a straight exit - I've used too many C libraries that think they know when I want to exit).

This is behavior that argparse inherits from optparse, but I believe it's still what 99.9% of users want. If you're writing a command line interface, you don't want a stack trace when there's an error message (which is what you'd get if argparse just raised exceptions) you want an exit with an error code. That's what command line applications are supposed to do. If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do.

...

2. There is no chance the parser will print things I don't want it to print.

That may also be bad - for example, Windows GUI-mode programs raise an error if they write to stdout/stderr. I could imagine using argparse for such a program, and wanting to do something with --help other than write to stdout and exit (a message box, for example). And yet, I'd want access to the text argparse would otherwise write to stdout.

There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

skip@pobox.com wrote:

I have never completely wrapped my brain around optparse. Getopt I just remember.

Seems to me that optparse and argparse are fairly similar in their APIs, and that argparse isn't going to be significantly easier to fit in one's brain than optparse. There's an art to coming up with an API that makes simple things easy and other things possible. I'm not convinced that argparse represents a subsantial enough advancement in that art to justify replacing optparse with it in the stdlib, and thereby forcing everyone to learn a similar-but-different API. -- Greg

On Tue, Sep 29, 2009 at 3:04 PM, Glenn Linderman wrote:

On approximately 9/29/2009 1:57 PM, came the following characters from the keyboard of Steven Bethard:

...

If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do. [snip] There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case.

Documenting both of these options would forestall people from thinking it is only useful for console applications.

I'm totally fine with people thinking it is only useful for console applications. That's what it's intended for. That said, if there are people out there who want to use it for other applications, I'm happy to make things easier for them if I know concretely what they want.

An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good.

I don't really use GUI libraries, so I wouldn't be able to come up with such an example. I'd also rather not make API changes based on speculative use cases, so before I spend time documenting these things, I'd really like to hear from someone who has already, say, used getopt or optparse in conjunction with a GUI library, and what feedback they have about that. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

On approximately 9/29/2009 4:38 PM, came the following characters from the keyboard of Steven Bethard:

On Tue, Sep 29, 2009 at 3:04 PM, Glenn Linderman wrote:

...

On approximately 9/29/2009 1:57 PM, came the following characters from the keyboard of Steven Bethard:

...

If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do. [snip] There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case. Documenting both of these options would forestall people from thinking it is only useful for console applications.

I'm totally fine with people thinking it is only useful for console applications. That's what it's intended for. That said, if there are people out there who want to use it for other applications, I'm happy to make things easier for them if I know concretely what they want.

Hmm. Maybe that is partly why (sadly) so many GUI programs don't offer much in the way of command line options. Of course, many of their users in Windowsland wouldn't know how to tweak the shortcut to invoke them with options anyway, which might be another part. Fortunately, there are some good Windows programs that do offer rich GUIs and also rich command line options (e.g. IrfanView and ImgBurn)

...

An example of using argparse with Tk (I think that is the only GUI that ships with Python) would also be good.

I don't really use GUI libraries, so I wouldn't be able to come up with such an example. I'd also rather not make API changes based on speculative use cases, so before I spend time documenting these things, I'd really like to hear from someone who has already, say, used getopt or optparse in conjunction with a GUI library, and what feedback they have about that.

I'm not a Tk user, just coming to Python from Perl, where I used Win32::GUI, but when I was looking for a functional and portable GUI development package, Perl didn't have one (still doesn't have one that supports Unicode and printing), and Python does... so here I come to Python and Qt. From my experience in Perl GUI Windowsland, the primary issue with command line parsing is displaying the message in a dialog instead of STDOUT. But the message and the user choices have to be known at the same time to design the dialog box. And, there is nothing so aggravating as to be shown an error message, and the only option is a button that says "OK", when it clearly isn't OK. So the tone/type of the messages also needs to be known, even when there are no user choices. The --help option could display the help message, and offer OK. Many errors (particularly unrecoverable ones) should display the error message, and offer an Exit button, or just the close box. A few (although probably only highly customized user options) might want to give the user multiple recovery options. So, I guess I'm in the unfortunate position of seeing the need, but not having used these specific technologies enough to offer an example either. So far, I've only used optparse (just now hearing about argparse in this thread) for command line programs in Python, and I am still just experimenting with Qt, and don't have enough familiarity with it yet to think that what I'm doing is best practices. I think it would be sad to a new replacement for optparse that didn't GUI usage, though, at least in a limited form. The concepts I describe seem sufficient from experience in other environments, and I would think they would be sufficient in Python too, but I'm too new here to state that definitely. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

Greg Ewing wrote:

skip@pobox.com wrote:

...

I have never completely wrapped my brain around optparse. Getopt I just remember.

Seems to me that optparse and argparse are fairly similar in their APIs, and that argparse isn't going to be significantly easier to fit in one's brain than optparse.

There's an art to coming up with an API that makes simple things easy and other things possible. I'm not convinced that argparse represents a subsantial enough advancement in that art to justify replacing optparse with it in the stdlib, and thereby forcing everyone to learn a similar-but-different API.

As someone that has written multiple optparse based utility scripts, I would say that yes, argparse is a *huge* improvement. Several things that I implemented for my own use in a rather clumsy fashion (subcommands, aggregated parsers, non-interspersed arguments) have far more elegant support built directly into argparse. For the getopt-vs-opt/argparse discussion, I believe the major distinction is in the relative balance between procedural and declarative style of argument handling. getopt is very procedural - you define a minimal amount regarding the options you accept, but then do the bulk of the command line processing yourself optparse is declarative to some degree, but forces you to drop back to a procedural style to handle arguments and subcommands. argparse takes things even further in a declarative direction by adding explicit support for positional arguments and subcommands. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Sep 29, 2009, at 8:25 PM, Nick Coghlan wrote:

getopt is very procedural - you define a minimal amount regarding the options you accept, but then do the bulk of the command line processing yourself

Right, and we shouldn't underestimate the cognitive load this imposes. All those twisty if-conditions are de-facto part of getopt's API. -Barry

Greg Ewing wrote:

Paul Moore wrote:

...

I'd rather argparse (or any library function) didn't call sys.exit on my behalf - it should raise an exception.

Actually, sys.exit() *does* raise an exception (i.e. SystemExit) that you can catch if you want.

I was going to mention that. Between catching SystemExit, the '-i' switch to the interpeter and the PYTHONINSPECT environment variable, investigating applications that call sys.exit isn't *that* difficult. It's a far cry from the instant exits of a C level abort call. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Sep 29, 2009, at 9:18 PM, Nick Coghlan wrote:

Keeping getopt around *and* including a "add_getopt_arguments" method in argparse is probably the best of both worlds, in that it allows for relatively straightforward evolution of an application:

1. Start with getopt 2. As the getopt argument parsing gets twisty and arcane and maintaining the help string becomes a nightmare, move to argparse with the "add_getopt_arguments" method. 3. Over time, convert more arguments to proper argparse arguments with full documentation.

Maybe. I haven't been following this entire thread, but I don't see much point in making it easy to go from getopt to argparse. The two are so different that I think you're either going to jump all in or not. Maybe that's just me though as I really don't see much use for getopt any more (even for throwaway scripts). -Barry

Steven Bethard wrote:

On Tue, Sep 29, 2009 at 1:31 PM, Paul Moore wrote:

...

2009/9/28 Yuvgoog Greenle :

...

1. There is no chance of the script killing itself. In argparse and optparse exit() is called on every parsing error (btw because of this it sucks to debug parse_args in an interpreter). [...]

This is behavior that argparse inherits from optparse, but I believe it's still what 99.9% of users want. If you're writing a command line interface, you don't want a stack trace when there's an error message (which is what you'd get if argparse just raised exceptions) you want an exit with an error code. That's what command line applications are supposed to do.

If you're not using argparse to write command line applications, then I don't feel bad if you have to do a tiny bit of extra work to take care of that use case. In this particular situation, all you have to do is subclass ArgumentParser and override exit() to do whatever you think it should do.

...

...

2. There is no chance the parser will print things I don't want it to print. [...]

There is only a single method in argparse that prints things, _print_message(). So if you want it to do something else, you can simply override it in a subclass. I can make that method public if this is a common use case.

Instead of forcing the user to override the ArgumentParser class to change how errors are handled, I suggest adding a separate method ArgumentParser.parse_args_with_exceptions() that raises exceptions instead of writing to stdout/stderr and never calls sys.exit(). Then implement ArgumentParser.parse_args() as a wrapper around parse_args_with_exceptions(): class ArgparseError(Exception): """argparse-specific exception type.""" pass class ArgumentError(ArgparseError): # ... class ArgumentParser(...): # ... def parse_args_with_exceptions(...): # like the old parse_args(), except raises exceptions instead of # writing to stdout/stderr or calling sys.exit()... def parse_args(self, *args, **kwargs): try: self.parse_args_with_exceptions(*args, **kwargs) except ArgparseError as e: self.print_usage(_sys.stderr) self.exit(status=2, message=(_('%s: error: %s\n') % (self.prog, e,))) # perhaps catch other exceptions that need special handling... def error(self, message): raise ArgparseError(message) The exception classes should hold enough information to be useful to non-command-line users, and obviously contain error messages that are output to stderr by default. This would allow non-command-line users to call parse_args_with_exceptions() and handle the exceptions however they like. Michael

Yuvgoog Greenle writes:

+1 for adding argparse and eventually deprecating optparse, -0 for deprecating getopt.

2. Big modules - 1 uber-module for each subject that does everything. Maybe logging is an example of this.

I'm not sure that it is, if you mean code size. In Python 2.5, logging is 1300 SLOC, less than say tarfile, pickletools, pydoc and decimal. Regards, Vinay Sajip

On a tangent -- a use case that I see happening frequently but which is pretty poorly supported by optparse is a command-line that has a bunch of general flags, then a 'subcommand name', and then more flags, specific to the subcommand. Most here are probably familiar with this pattern from SVN, Hg, and other revision control systems (P4 anyone?) with a rich command-line interface. There are some variants, e.g. whether global and subcommand-specific flags may overlap, and whether flags may follow positional args (Hg and SVN seem to differ here a bit). I've helped write at least two tools at Google that have this structure; both used different approaches, and neither was particularly easy to get right. Getting all the different --help output to make sense was mostly a manual process. (E.g. "foo --help" should print the general flags and the list of known subcommands, whereas "foo --help subcommand" should print flags and other usage info about the specific subcommand.) Also switching out to different calls based on the subcommand should/might be part of this. I would be willing to live with a third option parser in the stdlib if it solved this problem well. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 2009-09-30 11:38 AM, Guido van Rossum wrote:

On a tangent -- a use case that I see happening frequently but which is pretty poorly supported by optparse is a command-line that has a bunch of general flags, then a 'subcommand name', and then more flags, specific to the subcommand. Most here are probably familiar with this pattern from SVN, Hg, and other revision control systems (P4 anyone?) with a rich command-line interface. There are some variants, e.g. whether global and subcommand-specific flags may overlap, and whether flags may follow positional args (Hg and SVN seem to differ here a bit).

I've helped write at least two tools at Google that have this structure; both used different approaches, and neither was particularly easy to get right. Getting all the different --help output to make sense was mostly a manual process. (E.g. "foo --help" should print the general flags and the list of known subcommands, whereas "foo --help subcommand" should print flags and other usage info about the specific subcommand.) Also switching out to different calls based on the subcommand should/might be part of this.

I would be willing to live with a third option parser in the stdlib if it solved this problem well.

I don't think argparse supports the "foo --help subcommand" OOB. I think it would be simple to modify argparse to make it do so. It does support general options followed by a subcommand with options, though. http://argparse.googlecode.com/svn/tags/r101/doc/other-methods.html#sub-comm... -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sep 30, 2009, at 12:58 PM, Robert Kern wrote:

I don't think argparse supports the "foo --help subcommand" OOB. I think it would be simple to modify argparse to make it do so. It does support general options followed by a subcommand with options, though.

Right. I've made it kind of work in Mailman 3, but it would be nice for argparse to support this out of the box. Note that I think you want two forms: foo help subcommand foo subcommand --help to basically print the same help text. This is the way bzr does it for example and it works great. Other than that, I think argparse supports Guido's use case very nicely. -Barry

On 2009-09-30 15:17 PM, Andrew McNabb wrote:

On Wed, Sep 30, 2009 at 02:22:53PM -0400, Barry Warsaw wrote:

...

Right. I've made it kind of work in Mailman 3, but it would be nice for argparse to support this out of the box. Note that I think you want two forms:

foo help subcommand foo subcommand --help

to basically print the same help text. This is the way bzr does it for example and it works great.

In some commands, options as well as subcommands can change subsequent parsing. The iptables command is a good example of a command-line program that follows this practice. From the man page:

after [a module name is specified], various extra command line options become available, depending on the specific module. You can specify multiple extended match modules in one line, and you can use the -h or --help options after the module has been specified to receive help specific to that module.

In the case of iptables, module names are specified as options, not as subcommands.

...

From my cursory reading of the documentation, it looks like argparse can only add subparsers for subcommands. Is there any way to add subparsers based on options instead (as iptables does)?

I have not done so, but I suspect so. The implementation of .add_subparsers() adds to the positional argument list, but one could be written to append to the option list.

Also, is it possible to add these subparsers dynamically? For example, you would want to be able to load a module immediately after parsing the name instead of having to keep a predetermined list of all module names. I'm pretty sure that bzr dynamically loads modules this way. Can argparse help with this?

Not out-of-box, but it looks fairly straightforward to plug in. The subparser logic is mostly encapsulated in the _SubparsersAction class. You can register a new class for it: parser.register('action', 'parsers', MySubParsersAction) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Wed, Sep 30, 2009 at 1:17 PM, Andrew McNabb wrote:

...

From my cursory reading of the documentation, it looks like argparse can only add subparsers for subcommands.  Is there any way to add subparsers based on options instead (as iptables does)?

Currently this is not supported, but it would be a nice feature.

Also, is it possible to add these subparsers dynamically?  For example, you would want to be able to load a module immediately after parsing the name instead of having to keep a predetermined list of all module names. I'm pretty sure that bzr dynamically loads modules this way.  Can argparse help with this?

You can probably already do this. I'm not 100% sure what you want to do, but it's certainly possible to define an argparse.Action that loads a module when it's invoked. It might look something like:: class MyAction(argparse.Action): def __call__(self, parser, namespace, value, option_string=None): mod = __import__(value) # or whatever Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

2009/9/30 Robert Kern :

I am blissfully unaware of the problems Paul mentioned about Windows GUI-mode programs.

:-)

I'm not sure what would make a program "GUI-mode" or not. Certainly, I have written Python programs that use wxPython and PyQt on Windows that print to stdout/stderr, and they appear to work fine.

It's the difference between using python.exe and pythonw.exe. There's a flag in the executable header saying whether the executable is "console mode" or "gui mode". GUI mode programs are run (by the OS) without a console (or if run from a console prompt, they automatically detach from that console, much like Unix programs which fork themselves to leave the terminal group (did I get the terminology right?) but done by the OS). As a result, the program has no valid stdin/out/err handles. Any attempt to write to them causes the program to crash. Traceback (most recent call last): File "hello.py", line 13, in <module> main() File "hello.py", line 7, in main sys.stdout.flush() IOError: [Errno 9] Bad file descriptor (Question - is it *ever* possible for a Unix program to have invalid file descriptors 0,1 and 2? At startup - I'm assuming anyone who does os.close(1) knows what they are doing!) Paul.

On Thu, Oct 01, 2009 at 09:58:59AM +0100, Paul Moore wrote:

(Question - is it *ever* possible for a Unix program to have invalid file descriptors 0,1 and 2? At startup - I'm assuming anyone who does os.close(1) knows what they are doing!)

Yes, at startup you just have the file descriptors your parent process gave you. This may or may not include 0, 1, and 2, and may or may not include other descriptors too. I seem to recall there have been security holes caused in the past by people assuming 0-2 will exist.

I haven't checked if it's possible, but I suggest Argparse have it's own exception class that inherits from SystemExit and that exception would be thrown. ParseError, or something similar. I suggest this just because it would be more readable I guess and would exactly explain why this code exits. On Sat, Oct 3, 2009 at 8:15 AM, Nick Coghlan wrote:

Toshio Kuratomi wrote:

...

About exit(), I agree with others about wanting to catch the exception myself and then choosing to exit from the code.  I'm not sure that it's actually useful in practice, though...it might just feel cleaner but not actually be that helpful.

As others have pointed out, if you want to do that then you can just catch SystemExit.

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia --------------------------------------------------------------- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ubershmekel%40gmail.com

Steven Bethard wrote:

On Sat, Oct 3, 2009 at 3:45 AM, Yuvgoog Greenle wrote:

...

I haven't checked if it's possible, but I suggest Argparse have it's own exception class that inherits from SystemExit and that exception would be thrown.

ParseError, or something similar.

I suggest this just because it would be more readable I guess and would exactly explain why this code exits.

I've never seen such an idiom before (subclassing SystemExit) but it would certainly be possible create an ArgumentParserExit exception like that. Then you would have your choice of overriding .exit() or catching the exception.

Steve

Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

Steven Bethard wrote:

[snip...]

...

I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands.

Maybe I misunderstood, but I think this is already the default argparse behavior, no?

Cool. I didn't realise that help for subcommands was already implemented. :-) Michael

import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo') subparsers = parser.add_subparsers() parser1 = subparsers.add_parser('1') parser1.add_argument('--bar') parser2 = subparsers.add_parser('2') parser2.add_argument('baz') parser.parse_args(['--help'])

...

...

...

import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo') subparsers = parser.add_subparsers() parser1 = subparsers.add_parser('1') parser1.add_argument('--bar') parser2 = subparsers.add_parser('2') parser2.add_argument('baz')

...

...

...

# top level argument help parser.parse_args(['--help'])

usage: [-h] [--foo FOO] {1,2} ...

positional arguments: {1,2}

optional arguments: -h, --help show this help message and exit --foo FOO

...

...

...

# help for subparser 1 parser.parse_args(['1', '--help'])

usage: 1 [-h] [--bar BAR]

optional arguments: -h, --help show this help message and exit --bar BAR

...

...

...

# help for subparser 2 parser.parse_args(['2', '--help'])

usage: 2 [-h] baz

positional arguments: baz

optional arguments: -h, --help show this help message and exit

Steve

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

Yuvgoog Greenle wrote:

On Sat, Oct 3, 2009 at 7:21 PM, Michael Foord wrote:

...

[snip...] Why not just catch SystemExit? If you want a custom exception the overriding .exit() should be sufficient. I'd be much more interested in Guido's suggestion of auto-generated custom help messages for sub-commands.

Check it out:

def ParseAndRun(): crazy_external_function_that_might_exit()

# Argparse blah blah parser.parse_args()

if __name__ == "__main__": try: ParseAndRun() except SystemExit: # was it crazy_external_function_that_might_exit or an argparse error?

I know this might come through as bike shedding but it's just customary python that every module have it's own exception types as to not mix them up with others.

Then subclass and override .exit() as discussed - or put proper exception handling around the call to parse_args() (optionally rethrowing with whatever custom exception type you wish). Michael

--yuv

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

On Sat, Oct 3, 2009 at 8:29 PM, Steven D'Aprano wrote:

I could show a thousand other examples. It simply isn't true that all, or even most, modules have their own exception types.

I might be wrong on this. Your point is extra true for modules in the standard library (which is what we're talking about for argparse). I just think that if a parser error is causing the SystemExit, I would rather catch a parser error than catching a SystemExit for the sake of readability. It saves me the comments: # Catching SystemExit because parse_args() throws SystemExit on parser errors. # Subclassing ArgumentParser and overriding exit because I don't want to exit() upon parser errors. So I'm sorry if what I said was irrelevant. I've never written or taken part of writing a std-lib module. --yuv

* Steven D'Aprano wrote:

You don't need a comment warning that you are catching SystemExit because parse_args raises SystemExit, any more than you need a comment saying that you are catching ValueError because some function raises ValueError. The fact that you are catching an exception implies that the function might raise that exception. A comment like:

"Catching SystemExit because parse_args() throws SystemExit on parser errors."

is up them with comments like this:

x += 1 # Add 1 to x.

It's semantically different. You usually don't catch SystemExit directly, because you want your programs to be stopped. Additionally, a library exiting your program is badly designed, as it's unexpected. Thatswhy such a comment is useful. Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse. That way all parties here should be satisifed. (I do the same all the time in my signal handlers - that's another reason I'd rather not catch SystemExit directly as well :-) nd -- "Umfassendes Werk (auch fuer Umsteiger vom Apache 1.3)" -- aus einer Rezension http://pub.perlig.de/books.html#apache2

On Sun, 4 Oct 2009 05:35:04 am André Malo wrote:

* Steven D'Aprano wrote:

...

You don't need a comment warning that you are catching SystemExit because parse_args raises SystemExit, any more than you need a comment saying that you are catching ValueError because some function raises ValueError. The fact that you are catching an exception implies that the function might raise that exception. A comment like:

"Catching SystemExit because parse_args() throws SystemExit on parser errors."

is up them with comments like this:

x += 1 # Add 1 to x.

It's semantically different. You usually don't catch SystemExit directly, because you want your programs to be stopped.

Exactly -- so why catch it at all? But even if there is a good reason to catch it, there's still no reason to subclass SystemExit unless argparse wants to distinguish different types of fatal error, and allow code to catch some but not all. But frankly, if I'm having a hard time thinking of a reason to catch SystemExit, I'm having an even harder time thinking why you'd want to (say) catch SystemExitTooManyArguments but not SystemExitMissingArgument.

Additionally, a library exiting your program is badly designed, as it's unexpected.

It's not unexpected for an argument parser. Do you know any applications that run when given invalid arguments? As a general rule, what can the application do? Guess what you wanted?

Thatswhy such a comment is useful.

The comment doesn't tell you anything that wasn't obvious from the code. It is pointless.

Here's what I'd do: I'd subclass SystemExit in this case and raise the subclass from argparse.

In the following code snippet: try: ns = argparse.parse_args() except SystemExit: ... is there any confusion between SystemExit raised by parse_args and SystemExit raised by other components? *What* other components? If SystemExit was raised in that try block, where could it have come from other than parse_args? Do you write comments like these? try: value = mydict[key] except KeyError: # Catch KeyError raised by dict lookup ... try: n = mylist.index(x) except ValueError: # Catch ValueError raised by mylist.index ... Useless comments are worse than no comments, because useless comments waste the readers' time and they risk becoming out of sync with the code and turning into misleading comments.

That way all parties here should be satisifed.

No. It wastes the time of the argparse developer, it wastes the time of people learning argparse, it wastes the time of people who read the code and wonder what's the difference between SystemExit and ArgparseSystemExit. (Answer: there is no difference.) -- Steven D'Aprano

Yuvgoog Greenle wrote:

I know this might come through as bike shedding but it's just customary python that every module have it's own exception types as to not mix them up with others.

Not in my Python world it isn't. While that is sometimes the right answer, more often the right answer is to properly scope your try/except statement in order to avoid catching exceptions from code you aren't interested in. If you end up stuck with a library call that lacks both granularity and specificity of exceptions then you live with it (and maybe file a bug report/feature request with the library author). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Hrvoje Niksic writes:

Of course; simply use the >&- pseudo-redirection, which has been a standard sh feature (later inherited by ksh and bash, but not csh) for ~30 years. The error message is amusing, too:

$ python -c 'print "foo"' >&- close failed in file object destructor: Error in sys.excepthook:

Original exception was:

Python 3 complains at startup and is a bit more explicit: $ ./python -c '1' >&- Fatal Python error: Py_Initialize: can't initialize sys standard streams OSError: [Errno 9] Bad file descriptor Abandon Of course, if it is stderr that you explicitly close, you lose all reporting: $ ./python -c '1' 2>&- Abandon Regards Antoine.

2009/10/8 Paul Moore :

2009/10/7 Antoine Pitrou :

...

Python 3 complains at startup and is a bit more explicit:

$ ./python -c '1' >&- Fatal Python error: Py_Initialize: can't initialize sys standard streams OSError: [Errno 9] Bad file descriptor Abandon

Of course, if it is stderr that you explicitly close, you lose all reporting:

$ ./python -c '1' 2>&- Abandon

Hmm, how does Python 3's pythonw work on Windows? (I don't have a Windows installation of Python 3 to hand at the moment)

When running pythonw, fileno(stdout) is negative, so sys.stdout is set to None, and print() silently returns. But the situation is not perfect, see http://bugs.python.org/issue6501 where Apache provides a stdout, but python cannot determine the encoding because there is no console. -- Amaury Forgeot d'Arc