[Python-ideas] proto-PEP: Fixing Non-constant Default Arguments

[Chris Rebert]

Apologies in advance for the lateness of my reply. I missed a few emails 
in all the shuffle.

Jim Jewett wrote:
> On 1/30/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> Jim Jewett wrote:
>> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> 
>> >> Naive programmers desiring mutable default arguments
>> >> often make the mistake of writing the following:
> 
>> >>      def foo(mutable=some_expr_producing_mutable):
>> >>          #rest of function
> 
>> > Yes, it is an ugly gotcha, but so is the alternative.
>> > If it is changed, then just as many naive programmers (though perhaps
>> > not exactly the same ones) will make the opposite mistake -- and so
>> > will some experienced programmers who are used to current semantics.
> 
>> Anyone (ab)using the current semantics with the above construct is or
>> ought to be aware that such usage is unpythonic.
> 
> Are you worried about naive programmers or not?
[snip]
 >> Additionally, I see
 >> making the opposite mistake as usually only maybe causing a performance
 >> problem, as opposed to causing the program to work incorrectly.
 >
 > That is true only when the argument is a cache.
 > Generally, resetting the storage will make the program work
 > incorrectly, but the program won't fail on the first data tested --
 > which means the bug is less likely to be caught.
 >

I am worried about naive programmers. It seems that the opposite mistake 
(thinking arguments will be evaluated only once, at def-time, rather 
than at each call) won't have non-performance related effects a good bit 
of the time. For many of the remaining cases, the person probably 
is/should be using a constant instead of a default argument. Yes, there 
will be some cases where badness could happen. However, (1) this is 
intended for Py3k, so there will already be a bunch of changes for 
newbies to learn, (2) the chance of weirdness happening is a good bit 
lower compared to the current state of mutable default arguments.

>> Also, due to the
>> hairiness/obscurity of the construct, hardly anyone uses it, so the
>> impact of removing it should be relatively minor.
> 
> I just did found 181 such uses in my own installation.  29 were either
> 3rd-party or test code, but that still leaves over 150 uses in the
> standard library.  It is possible that some of them would also work
> with your semantics, or may even be bugs today -- but you would have
> to go through them one-by-one as part of the PEP process.

I have a script to gather statistics on default argument usage in the 
standard library 90% done. Its results will be included in the next PEP 
draft.

>>      There are currently few, if any, known good uses of the current
>> behavior of mutable default arguments.  The most common one is to
>> preserve function state between calls.  However, as one of the 
>> lists [2] comments, this purpose is much better served by decorators, 
>> classes, or(though less preferred) global variables.
> 
> I disagree.  This is particularly wrong for someone coming from a
> functional background.
> 
>> I assume you disagree with the "purpose is much better served by
>> decorators, classes, or local variables" part as opposed to the "default
>> mutable arguments with current semantics have few good uses" part.
>> Please correct me if I'm in error.
> 
> I disagree with both.  There are many uses for preserving function
> state between calls.  C uses static local variables.   Object Oriented
> languages often use objects.

First, I was referring to using default args for caching being evil, not 
mutable default arguments in general. Just wanted to clarify in case you 
misinterpreted me.
True, there are many uses for preserving function state between calls, 
but how much better/clearer is doing so using default arguments as 
opposed to objects or other approaches? It's an aesthetic decision. 
IMHO, the other ways seem better. YMMV.

>> > A class plus an instantiation seems far too heavyweight for what ought
>> > to be a simple function.  I'm not talking (only) about the runtime;
>> > the number of methods and lines of code is the real barrier for me.
>> > I'll sometimes do it (or use a global) anyhow, but it feels wrong.  If
>> > I had felt that need more often when I was first learning python (or
>> > before I knew about the  __call__ workaround), I might have just
>> > written the language off as no less bloated than java.
> 
>> I'm sorry, but when you have a function sharing state between calls,
>> that just screams to me that you should make it a method of an object
> 
> That's because you drank the OO koolaid.  Python tries not to enforce
> any particular programming style.  Its object model happens to be
> pretty good, but there are still times when OO isn't the answer.

Well, it's a trade-off: support procedural/functional programming a bit 
more, or make default arguments a bit more intuitive and obsolete the 
'foo=None; if foo is None' idiom.

>> If performance is your concern, the decorator
>> version might perform better (I don't really know), or in the extreme
>> case, you could use a global variable, which is definitely faster.
> 
> I think there is something wrong with your benchmark.

Sorry, yeah, you're right. I was being a little fast and loose. But the 
point was that there are (at the price of some elegance) ways to 
optimize such situations even more.

> That said, speed is *not* my concern -- code size is.  If I have to
> add several lines of boilerplate, the code becomes much less readable.
> That is roughly the same justification you have for not liking the
> "if arg is None:  arg=foo()" idiom.  The difference is that you're
> suggesting adding far more than a line or two.

True, however I'm willing to bet that the boilerplate for evaluating 
default arguments only once won't be needed nearly as much, as such 
situations requiring such behavior is not too common. The statistics 
should help show exactly how much mutable default arguments are 
exploited in the wild.

>> Alternatively, we could make the new
>> semantics the default and have syntax to use the old semantics via
>> 'once' or some other keyword. This does nicely emphasize that such
>> semantics would be purely for optimization reasons.
> 
> Using a mutable default is almost never for optimization reasons.

Actually, I was more thinking it might be used to stop immutable default 
arguments from being re-evaluated multiple times, if it turns out that 
re-evaluations really do have major effects on performance. Now that I 
think about it, I guess it could be used to get the old behavior for 
mutable default arguments.

- Chris Rebert