[issue18531] Undocumented different between METH_KEYWORDS and **kws
New submission from Barry A. Warsaw: A colleague discovered an interesting implementation different between C-defined functions and Python-defined functions called with **kws arguments. He tried to do this:
from collections import defaultdict '{foo}{bar}'.format(**defaultdict(str)) ''
He wondered how this could work, especially given:
def what(**kws): ... print(type(kws), repr(kws)) ... what(**defaultdict(str)) <class 'dict'> {}
The Language Reference clearly states that when what() is called, a new dictionary is create, which is populated by keyword arguments not consumed by positional args. http://docs.python.org/3/reference/compound_stmts.html#function-definitions http://docs.python.org/3/reference/expressions.html#calls What isn't described is the behavior when the function is defined in C using METH_KEYWORDS. In that case, the object passed through the ** argument is passed unscathed to the underlying methodobject defined to take METH_KEYWORDS. str.format() is of the latter type so it gets the actual defaultdict. what() of course gets the defined behavior. It would be nice if the implementation details of the function did not leak into this behavior, but this is admittedly a corner case of the CPython implementation. I haven't checked any of the alternative implementations to see what they do. My guess is that CPython won't change for practical and backward compatibility reasons, thus I've opened this issue as a documentation bug. At the very least, an implementation note in the Language Reference should be added. ---------- assignee: docs@python components: Documentation messages: 193559 nosy: barry, docs@python priority: normal severity: normal status: open title: Undocumented different between METH_KEYWORDS and **kws versions: Python 2.7, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Alex Gaynor added the comment: I'll confirm that PyPy raises a KeyError on the format() code. ---------- nosy: +alex _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Changes by Evgeny Kapun <abacabadabacaba@gmail.com>: ---------- nosy: +abacabadabacaba _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Serhiy Storchaka added the comment: Yet one difference is that kwargs can be NULL in C function if no keyword arguments are supplied. Here is a patch that adds tests for exact types of args and kwargs arguments in C functions. I think that we should keep passing NULL as kwargs for performance reasons. But shouldn't we convert dict subclass to exact dict? I think there are good reasons to do this. Some functions can save kwargs for latter use. And this makes the implementation more consistent with PyPy. If defaultdict behavior is needed for formatting, there is format_map(). ---------- keywords: +patch nosy: +serhiy.storchaka type: -> behavior versions: +Python 3.5, Python 3.6 -Python 3.3, Python 3.4 Added file: http://bugs.python.org/file42742/test_args_kwargs_types.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: Removed file: http://bugs.python.org/file42742/test_args_kwargs_types.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: Added file: http://bugs.python.org/file42743/test_args_kwargs_types.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Serhiy Storchaka added the comment: Proposed patch converts var-keyword arguments to exact dict. Since it changes the behavior of str.format() it can be pushed only in 3.6. ---------- Added file: http://bugs.python.org/file42746/ceval_kwargs_exact_dict.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Changes by Barry A. Warsaw <barry@python.org>: ---------- nosy: +eric.smith _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Barry A. Warsaw added the comment: Eric should chime in on the str.format() implications. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Eric V. Smith added the comment: I think it's okay to change this as far as str.format() goes. Interestingly enough, the motivating case to add str.format_map() in issue 6081 was exactly this kind of code. Although I notice that the example in that issue, which it shows as failing, works in both 2.7 and 3.4 (the only versions I have handy). So I'm not sure what changed after 2009 to cause that code to start working, but I'd be okay with it breaking again. But maybe we should track it down, in case it was deliberately changed and is important to someone. In any event, since str.format_map() is well-known (to me, at least!), and is the approved way of getting the behavior of passing a specific map in to the format machinery, I'm okay with changing **dict to create an exact new dict, at least as far as str.format() goes. I can't speak to the overall implications (such as other APIs and performance). I agree it would be a 3.6 only change (and I've change the versions to reflect that). ---------- versions: -Python 2.7, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Barry A. Warsaw added the comment: On May 06, 2016, at 03:41 PM, Eric V. Smith wrote:
I agree it would be a 3.6 only change (and I've change the versions to reflect that).
The reason the other versions were included was because this was originally reported as a documentation bug. It should probably still be documented in those older releases. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Eric V. Smith added the comment: Okay. Adding back 3.5 and 2.7, in case someone wants to document the behavior there. I'm not sure if we fix docs in 3.4 still. ---------- versions: +Python 2.7, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Serhiy Storchaka added the comment: str.format() uses PyDict_GetItem() in 2.7, but PyObject_GetItem() in 3.x. A comment before: /* Use PyObject_GetItem instead of PyDict_GetItem because this code is no longer just used with kwargs. It might be passed a non-dict when called through format_map. */ Thus the behavior of str.format() was changed by accident. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Roundup Robot added the comment: New changeset e4835b1ed7b1 by Serhiy Storchaka in branch 'default': Issue #18531: Single var-keyword argument of dict subtype was passed https://hg.python.org/cpython/rev/e4835b1ed7b1 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Eric V. Smith added the comment: I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely explains things, because the code in issue 6081 also works in 2.7. But I don't think it's terribly important in any event, as long as we understand the 3.x difference. Thanks for researching! ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Serhiy Storchaka added the comment:
I'm not sure PyObject_GetItem instead of PyDict_GetItem in 3.x completely explains things, because the code in issue 6081 also works in 2.7.
It doesn't work to me in current develop 2.7.11+. What version did you test? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Serhiy Storchaka added the comment: Now this is just the documentation issue. I think the change of the behavior in 3.6 should be documented too, but I even don't know where. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Eric V. Smith added the comment: Oops, sorry. I'm an idiot. I was running in a venv where python was python3. Checking with my installed 2.7.10, I see the expected error. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
Roundup Robot added the comment: New changeset e3283b4ae61d by Serhiy Storchaka in branch '3.5': Backported tests for issue #18531. https://hg.python.org/cpython/rev/e3283b4ae61d New changeset 621e0a3249c2 by Serhiy Storchaka in branch '2.7': Backported tests for issue #18531. https://hg.python.org/cpython/rev/621e0a3249c2 ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18531> _______________________________________
participants (6)
-
Alex Gaynor -
Barry A. Warsaw
-
Eric V. Smith
-
Evgeny Kapun
-
Roundup Robot
-
Serhiy Storchaka