[Python-Dev] Issue 15906; regression in argparse in Python 3.3, 3.2, and 2.7

[R. David Murray]

On Tue, 11 Sep 2012 11:34:30 -0400, Barry Warsaw <barry at python.org> wrote:
> Issue 15906 describes a problem with argparse that is breaking lots of code in
> Ubuntu.  This is a recent regression caused by the fix for issue 12776, and it
> affects Python 2.7, 3.2, and 3.3.
> 
> I posted a diff that should fix the problem, but at the heart of it is a
> semantic ambiguity in argparse that needs clarification.  This needs to be
> cleared up before a proper patch can be applied.  I have submitted a patch for
> what *I* think reasonable semantics should be, but let's see what you think.
> 
> Issue 12776 tried to fix a problem illustrated by this example:
> 
> -----snip snip-----
> import argparse
> 
> parser = argparse.ArgumentParser()
> parser.add_argument('--file', type=open, default='/etc/passwd')
> args = parser.parse_args()
> 
> print(args.file.read())
> 
> args = parser.parse_args(['--file', '/etc/group'])
> print(args.file.read())
> -----snip snip-----
> 
> What this code is (IMO, sensibly) trying to do is say that args.file will
> always be an open file, regardless of whether the path comes from the default
> value or is given on the command line.

Actually, what 12776 was trying to fix was that the conversion function
was *always* called, and thus the file was *always* opened, even when
a file argument was passed in the arguments to be parsed.

> The problem is that this breaks some other, also sensible code:
> 
> -----snip snip-----
> import argparse
> 
> parser = argparse.ArgumentParser()
> parser.add_argument("--test", dest="test", type=str,
>     default=[], action='append')
> 
> args = parser.parse_args(['--test', 'bar'])
> args.test.append('baz')
> 
> args = parser.parse_args()
> args.test.append('baz')
> -----snip snip-----
> 
> This code is saying, I want to accumulate string arguments into a list,
> regardless of whether any arguments are given on the command line.
>
> The fix for issue 12776 broke the last two lines of the second example,
> because in the no-command-line-arguments-given case, arg.test is the *string*
> "[]" and not the actual empty list object.

But apparently in doing so we broke something else.

There is no question that the current state of the tip of each branch is
a regression.   So at a minimum the old behavior needs to be restored,
which is that the above code should do as you say:  args.test should end
up initialized to the empty list.

As interesting aside: if the above is changed to

    args = parser.parse_args(['--test], 'x'])

the append will work.  The fact that that is true is doubtless a clue
as to why the fix for 12776 broke things unexpectedly.

> It seems to me that the semantics could reasonably be implied to mean that the
> type converter should only be applied to the default value when
> action='store', as is the default.  Then in the second example, because
> action='append', the type conversion would not be applied (it makes no sense
> to do so).

There is another possible semantic, which is that when the store type is
append, the converter should be applied to each of the individual items
in the default list.

Which brings us to another issue: as things stand now, if we have, say,
'default=['abc']', then passing in '--test x' on the command line would
result in args.test being equal to

    ['abc', 'x']

which is consistent with optparse but not necessarily the desired
semantics.

> I have attached a diff to issue 15906 that implements these semantics.  If you
> agree, then I will apply this to all of 3.3, 3.2, and 2.7, which are all
> affected by this bug (because the original fix for 12776 was applied to all
> three branches).
>
> Georg, I would like to apply this to the 3.3 branch.

The 12776 fix isn't going to be in 3.3, so I don't think this is a
pressing issue.  We can take our time to make sure we have the correct
fix.  It is, however, a release blocker for 2.7.4, 3.2.4, and 3.3.1.

--David