The curious case of 255 function arguments
Hello Python Gurus, TL;DR: 3.7 released functions having greater than 255 arguments. Despite explicit checks for this in 2.x, no such limit is actually imposed -- why? In the 3.7 release notes "Other Language Changes" section ( https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes), the first bullet point denotes
More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters. (Contributed by Serhiy Storchaka in bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896 <https://bugs.python.org/issue18896>.)
Now lets get something straight: unless I want to exclusively support Python 3.7 or higher, I must make sure I obey the <255 rule. Use *args // **kwargs, etc. I'm totally ok with that, 2020 is already here in my mind ;) Curiosity is the reason I'm reaching out. Upon further investigation and some discussion with like-minded Python enthusiasts, the code being patched by Serhiy Storchaka is present in e.g., Python 2.7 ( https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016) if (nargs + nkeywords + ngens > 255) { ast_error(n, "more than 255 arguments"); return NULL; } Despite that code, as demonstrated with the supplemental output in the post script, *no 2.x versions fail with >255 arguments*. In contrast, 3.x where x<7 all do fail (as expected) with a SyntaxError. To test this, I tried every minor release of python (excluding v1, arbitrarily choosing the latest patch release of a minor version) with the following snippet via the -c flag /path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i in range(1, 300)]) + "): pass")' Which tries to construct a function def foo(a0, a1, ..., a299): pass I've looked at the C code for a while and it is entirely non-obvious what would lead to python *2* *allowing* >255 arguments. Anybody happen to know how / why the python *2* versions *succeed*? Thank you for reading, this is not a problem, just a burning desire for closure (even if anecdotal) as to how this can be. I deeply love python, and am not complaining! I stumbled across this and found it truly confounding, and thought the gurus here may happen to recall what changed in 3.x that lead the the error condition actually being asserted :) Sincerely, Stephen McDowell P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the installation of all the python versions just to see if it truly was a 2.x / 3.x divide. The results of running `python -V` followed by the `python -c 'exec("def foo...")'` described above, with some extra prints for clarity are as follows (script hackily thrown together in ~30minutes not included, so as not to make your eyes bleed): ******************************************************************************** Python 2.0.1 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.1.3 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.2.3 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.3.7 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.4.6 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.5.6 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.6.9 ==> Greater than 255 Arguments supported ******************************************************************************** Python 2.7.15 ==> Greater than 255 Arguments supported ******************************************************************************** Python 3.0.1 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.1.5 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.2.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.3.7 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.4.9 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.5.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.6.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments ******************************************************************************** Python 3.7.0 ==> Greater than 255 Arguments supported P.P.S. Seriously, I LOVE PYTHON <3 It was so easy to download, configure, build, and install each of these versions, and run the test! Thank you :)
With Python 2.7.15 what fails is a call with explicit arguments (e.g. `foo(0,0,0 ... 0,0)`), not the function definition. Calling with `foo([0]*300)` instead works. On Mon, Aug 6, 2018 at 7:18 AM Stephen McDowell <sjm324@cornell.edu> wrote:
Hello Python Gurus,
TL;DR: 3.7 released functions having greater than 255 arguments. Despite explicit checks for this in 2.x, no such limit is actually imposed -- why?
In the 3.7 release notes "Other Language Changes" section ( https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes), the first bullet point denotes
More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters. (Contributed by Serhiy Storchaka in bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896 <https://bugs.python.org/issue18896>.)
Now lets get something straight: unless I want to exclusively support Python 3.7 or higher, I must make sure I obey the <255 rule. Use *args // **kwargs, etc. I'm totally ok with that, 2020 is already here in my mind ;)
Curiosity is the reason I'm reaching out. Upon further investigation and some discussion with like-minded Python enthusiasts, the code being patched by Serhiy Storchaka is present in e.g., Python 2.7 ( https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)
if (nargs + nkeywords + ngens > 255) { ast_error(n, "more than 255 arguments"); return NULL; }
Despite that code, as demonstrated with the supplemental output in the post script, *no 2.x versions fail with >255 arguments*. In contrast, 3.x where x<7 all do fail (as expected) with a SyntaxError. To test this, I tried every minor release of python (excluding v1, arbitrarily choosing the latest patch release of a minor version) with the following snippet via the -c flag
/path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i in range(1, 300)]) + "): pass")'
Which tries to construct a function
def foo(a0, a1, ..., a299): pass
I've looked at the C code for a while and it is entirely non-obvious what would lead to python *2* *allowing* >255 arguments. Anybody happen to know how / why the python *2* versions *succeed*?
Thank you for reading, this is not a problem, just a burning desire for closure (even if anecdotal) as to how this can be. I deeply love python, and am not complaining! I stumbled across this and found it truly confounding, and thought the gurus here may happen to recall what changed in 3.x that lead the the error condition actually being asserted :)
Sincerely,
Stephen McDowell
P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the installation of all the python versions just to see if it truly was a 2.x / 3.x divide. The results of running `python -V` followed by the `python -c 'exec("def foo...")'` described above, with some extra prints for clarity are as follows (script hackily thrown together in ~30minutes not included, so as not to make your eyes bleed):
******************************************************************************** Python 2.0.1 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.1.3 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.2.3 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.3.7 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.4.6 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.5.6 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.6.9 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.7.15 ==> Greater than 255 Arguments supported
******************************************************************************** Python 3.0.1 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.1.5 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.2.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.3.7 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.4.9 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.5.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.6.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.7.0 ==> Greater than 255 Arguments supported
P.P.S. Seriously, I LOVE PYTHON <3 It was so easy to download, configure, build, and install each of these versions, and run the test! Thank you :) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/agriff%40tin.it
typo... meant of course foo(*([0]*300)) Andrea On Mon, Aug 6, 2018 at 9:57 AM Andrea Griffini <agriff@tin.it> wrote:
With Python 2.7.15 what fails is a call with explicit arguments (e.g. `foo(0,0,0 ... 0,0)`), not the function definition. Calling with `foo([0]*300)` instead works.
On Mon, Aug 6, 2018 at 7:18 AM Stephen McDowell <sjm324@cornell.edu> wrote:
Hello Python Gurus,
TL;DR: 3.7 released functions having greater than 255 arguments. Despite explicit checks for this in 2.x, no such limit is actually imposed -- why?
In the 3.7 release notes "Other Language Changes" section ( https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes), the first bullet point denotes
More than 255 arguments can now be passed to a function, and a function can now have more than 255 parameters. (Contributed by Serhiy Storchaka in bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896 <https://bugs.python.org/issue18896>.)
Now lets get something straight: unless I want to exclusively support Python 3.7 or higher, I must make sure I obey the <255 rule. Use *args // **kwargs, etc. I'm totally ok with that, 2020 is already here in my mind ;)
Curiosity is the reason I'm reaching out. Upon further investigation and some discussion with like-minded Python enthusiasts, the code being patched by Serhiy Storchaka is present in e.g., Python 2.7 ( https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)
if (nargs + nkeywords + ngens > 255) { ast_error(n, "more than 255 arguments"); return NULL; }
Despite that code, as demonstrated with the supplemental output in the post script, *no 2.x versions fail with >255 arguments*. In contrast, 3.x where x<7 all do fail (as expected) with a SyntaxError. To test this, I tried every minor release of python (excluding v1, arbitrarily choosing the latest patch release of a minor version) with the following snippet via the -c flag
/path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i in range(1, 300)]) + "): pass")'
Which tries to construct a function
def foo(a0, a1, ..., a299): pass
I've looked at the C code for a while and it is entirely non-obvious what would lead to python *2* *allowing* >255 arguments. Anybody happen to know how / why the python *2* versions *succeed*?
Thank you for reading, this is not a problem, just a burning desire for closure (even if anecdotal) as to how this can be. I deeply love python, and am not complaining! I stumbled across this and found it truly confounding, and thought the gurus here may happen to recall what changed in 3.x that lead the the error condition actually being asserted :)
Sincerely,
Stephen McDowell
P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the installation of all the python versions just to see if it truly was a 2.x / 3.x divide. The results of running `python -V` followed by the `python -c 'exec("def foo...")'` described above, with some extra prints for clarity are as follows (script hackily thrown together in ~30minutes not included, so as not to make your eyes bleed):
******************************************************************************** Python 2.0.1 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.1.3 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.2.3 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.3.7 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.4.6 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.5.6 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.6.9 ==> Greater than 255 Arguments supported
******************************************************************************** Python 2.7.15 ==> Greater than 255 Arguments supported
******************************************************************************** Python 3.0.1 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.1.5 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.2.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.3.7 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.4.9 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.5.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.6.6 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1 SyntaxError: more than 255 arguments
******************************************************************************** Python 3.7.0 ==> Greater than 255 Arguments supported
P.P.S. Seriously, I LOVE PYTHON <3 It was so easy to download, configure, build, and install each of these versions, and run the test! Thank you :) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/agriff%40tin.it
06.08.18 08:13, Stephen McDowell пише:
I've looked at the C code for a while and it is entirely non-obvious what would lead to python *2* /allowing/ >255 arguments. Anybody happen to know how / why the python *2* versions *succeed*?
The error message is misleading. It should be "more than 255 parameters". This limitation is due to the optimization used in Python 3 for call variables (see https://bugs.python.org/issue12399 for details). In all versions <3.7 there is a limitation on the number of explicit function arguments because of the limitation of the CALL_FUNCTION opcode.
Thank you for reading, this is not a problem, just a burning desire for closure (even if anecdotal) as to how this can be. I deeply love python, and am not complaining! I stumbled across this and found it truly confounding, and thought the gurus here may happen to recall what changed in 3.x that lead the the error condition actually being asserted :)
Read the history of the code. Commit messages usually contain explanations or references to issues.
Hi Andrea and Serhiy, Thank you for your responses and clarifying that it is specifically the CALL_FUNCTION. I tested this in my megascript and it will fail when trying to call the functions directly and receive an error then (Py 2.x: fail at call invocation, Py 3.y w/ y<7: fail at function definition). @Serhiy I looked through the commits and had found https://github.com/python/cpython/commit/5bb8b9134b0bb35a73c76657f41cafa3e43... but the commit that removed the 255 checks also explains that this is specifically about the call function ( https://github.com/python/cpython/commit/214678e44bf7773c0ed9c3684818354001d... ), so indeed I should have been able to answer this myself. The reason why I originally had encountered this was (as discussed in one of the bug reports) from code that was generating a class hierarchy to represent Doxygen's XML schema. The class constructors had >255 arguments, but in executing the code it actually does still work in python 2.x. The reason is because all of the arguments are defaulted to None, and during execution of typical sample XML files, the explicit construction with all
255 arguments virtually never happens.
f.write("def foo_2({0}):\n".format(", ".join(["a{0}=None".format(str(i)) for i in range(300)]))) f.write(" print('foo_2 executed')\n\n") # ... in generated __main__ ... f.write(" foo_2()\n\n") foo_2() will succeed in python 2.x because the CALL_FUNCTION is not explicitly getting more than 255 parameters. Very interesting! Thank you both again for your responses, I am grateful to finally understand the way in which success / failure works here :) -Stephen On Mon, Aug 6, 2018 at 2:17 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
06.08.18 08:13, Stephen McDowell пише:
I've looked at the C code for a while and it is entirely non-obvious what would lead to python *2* /allowing/ >255 arguments. Anybody happen to know how / why the python *2* versions *succeed*?
The error message is misleading. It should be "more than 255 parameters". This limitation is due to the optimization used in Python 3 for call variables (see https://bugs.python.org/issue12399 for details).
In all versions <3.7 there is a limitation on the number of explicit function arguments because of the limitation of the CALL_FUNCTION opcode.
Thank you for reading, this is not a problem, just a burning desire for
closure (even if anecdotal) as to how this can be. I deeply love python, and am not complaining! I stumbled across this and found it truly confounding, and thought the gurus here may happen to recall what changed in 3.x that lead the the error condition actually being asserted :)
Read the history of the code. Commit messages usually contain explanations or references to issues.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/sjm324% 40cornell.edu
participants (3)
-
Andrea Griffini -
Serhiy Storchaka -
Stephen McDowell