[issue18085] Verifying refcounts.dat
New submission from Serhiy Storchaka: I found one error in Doc/data/refcounts.dat (see issue9369, second argument of PyObject_CallMethodObjArgs is `PyObject*`, not `char*`). Perhaps there are other errors. How this file was generated first? Is it possible to write a tool which will automatically verify at least signatures of functions? ---------- assignee: docs@python components: Documentation messages: 190236 nosy: docs@python, serhiy.storchaka, skip.montanaro priority: normal severity: normal status: open title: Verifying refcounts.dat type: behavior versions: Python 2.7, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Roundup Robot added the comment: New changeset 0889ab0d0da1 by Serhiy Storchaka in branch '3.3': Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat. http://hg.python.org/cpython/rev/0889ab0d0da1 New changeset ef9d42b98a3d by Serhiy Storchaka in branch '2.7': Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat. http://hg.python.org/cpython/rev/ef9d42b98a3d New changeset 6d0fd113a2e4 by Serhiy Storchaka in branch 'default': Issue #18085: Fix PyObject_CallMethodObjArgs()'s entry in refcounts.dat. http://hg.python.org/cpython/rev/6d0fd113a2e4 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Fred L. Drake, Jr. added the comment: I'm a little surprised that still exists. The first version was generated manually. ---------- nosy: +fdrake _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Brett Cannon added the comment: Wow, I didn't even know that file existed. It could actually help with some static analysis of the C code to verify that the refcounts are as expected w/o actually having to do inter-procedural analysis to figure out all of the refcount values. I wonder if Dave's gcc work could help either verify the file or benefit from it? ---------- nosy: +brett.cannon, dmalcolm _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Skip Montanaro added the comment: As far as I can tell that file is still manually maintained. I see it mentioned in Doc/conf.py, but nowhere else. It shouldn't be hard to deal with manually, as the C API doesn't change that often. A verifier shouldn't be terribly difficult to write, as Python C source is pretty consistently structured. I'm not volunteering for the task though. <wink> ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Fred L. Drake, Jr. added the comment: Were I adding that today, I'd use a more verbose (but more standard) format, like configparser or JSON. If any further use is going to be made of it, that should be considered. Colon-delimited is a pretty fragile format. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Skip Montanaro added the comment:
Fred L. Drake, Jr. added the comment:
Were I adding that today, I'd use a more verbose (but more standard) format, like configparser or JSON. If any further use is going to be made of it, that should be considered. Colon-delimited is a pretty fragile format.
Given the context, it should be okay until Python is rewritten in C++. :-) I don't know where this is consumed. Presumably by Sphinx somewhere. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Amaury Forgeot d'Arc added the comment: And is it necessary to list all functions there? Many functions share the same behavior: they don't change the ownership of PyObject* passed as argument, and return a new reference. Only document functions that don't conform to this rule, like PyTuple_SET_ITEM and PyImport_AddModule. ---------- nosy: +amaury.forgeotdarc _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Skip Montanaro added the comment:
And is it necessary to list all functions there?
Probably not. However, BITD, I was figuring this stuff out as I was going along by examining the source code and writing down what I found. No regard for common or special cases. S ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Serhiy Storchaka added the comment: Since creating refcounts.dat many functions changed their argument's types from `char*` to `const char*`. Here is a patch which fixes mismatches (perhaps not all). refcounts.dat in 3.x contains PyInt_* functions which don't exist in 3.x. ---------- keywords: +patch Added file: http://bugs.python.org/file30436/refcounts_const_char.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Roundup Robot added the comment: New changeset ffe24e3e7a2a by Serhiy Storchaka in branch '3.3': Issue #18085: Add missed const modifier for some entries in refcounts.dat. http://hg.python.org/cpython/rev/ffe24e3e7a2a New changeset 6587fd3d89ae by Serhiy Storchaka in branch 'default': Issue #18085: Add missed const modifier for some entries in refcounts.dat. http://hg.python.org/cpython/rev/6587fd3d89ae ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: ---------- priority: normal -> low _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Berker Peksag added the comment: Can we now close this as a duplicate of issue 9755? ---------- nosy: +berker.peksag _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Fred L. Drake, Jr. added the comment: I don't think this is a duplicate of issue 9755; this relates to verifying the data, and that revolves around possible process improvements. Whether this issue should be closed is tied to whether the file has been verified, as the issue title suggests. I don't know whether Serhiy verified everything when he made his changes or not; that's not explicit in the issue or commit comments. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Serhiy Storchaka added the comment: No, I hadn't verified all names. I'm sure there are API functions not mentioned in refcounts.dat. See also issue23903. It would be nice to have a tool that parses Python source headers and produce lists of all public names in different formats for manual updating python3.def, refcounts.dat, docs, etc. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- pull_requests: +10476 stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- pull_requests: +10476, 10477 stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- pull_requests: +10476, 10477, 10478 stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- pull_requests: -10477 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- pull_requests: -10478 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: PR 11247 adds more functions and fixes some errors. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: New changeset 83dd4e87a62311cfea5fdd37e8a945b6b07bccee by Serhiy Storchaka in branch 'master': bpo-18085: Update refcounts.dat. (GH-11247) https://github.com/python/cpython/commit/83dd4e87a62311cfea5fdd37e8a945b6b07... ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by miss-islington <mariatta.wijaya+miss-islington@gmail.com>: ---------- pull_requests: +10487 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
miss-islington <mariatta.wijaya+miss-islington@gmail.com> added the comment: New changeset 73fc14d1f8441aef5ee03be627c63e74a6d915d6 by Miss Islington (bot) in branch '3.7': bpo-18085: Update refcounts.dat. (GH-11247) https://github.com/python/cpython/commit/73fc14d1f8441aef5ee03be627c63e74a6d... ---------- nosy: +miss-islington _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.7, Python 3.8 -Python 2.7, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue18085> _______________________________________
participants (8)
-
Amaury Forgeot d'Arc -
Berker Peksag
-
Brett Cannon
-
Fred L. Drake, Jr.
-
miss-islington
-
Roundup Robot
-
Serhiy Storchaka
-
Skip Montanaro