[issue22558] Missing hint to source code - complete
New submission from Friedrich Spee von Langenfeld: As mentioned in Issue22528, in some modules´ documentation, the link to their source code is missing. To check the stdlib, I´ve written the script missing_hint.py. It seems that in the following module entries, there aren´t a link to their source code (the last part of output of missing_hint.py): https://docs.python.org/3/library/aifc.html https://docs.python.org/3/library/base64.html https://docs.python.org/3/library/binhex.html https://docs.python.org/3/library/bz2.html https://docs.python.org/3/library/cgi.html https://docs.python.org/3/library/cgitb.html https://docs.python.org/3/library/chunk.html https://docs.python.org/3/library/code.html https://docs.python.org/3/library/codecs.html https://docs.python.org/3/library/codeop.html https://docs.python.org/3/library/configparser.html https://docs.python.org/3/library/copy.html https://docs.python.org/3/library/copyreg.html https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/datetime.html https://docs.python.org/3/library/decimal.html https://docs.python.org/3/library/difflib.html https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/fnmatch.html https://docs.python.org/3/library/formatter.html https://docs.python.org/3/library/ftplib.html https://docs.python.org/3/library/getpass.html https://docs.python.org/3/library/glob.html https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/imaplib.html https://docs.python.org/3/library/imp.html https://docs.python.org/3/library/io.html https://docs.python.org/3/library/locale.html https://docs.python.org/3/library/lzma.html https://docs.python.org/3/library/macpath.html https://docs.python.org/3/library/mailbox.html https://docs.python.org/3/library/mimetypes.html https://docs.python.org/3/library/nntplib.html https://docs.python.org/3/library/numbers.html https://docs.python.org/3/library/os.html https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/plistlib.html https://docs.python.org/3/library/poplib.html https://docs.python.org/3/library/profile.html https://docs.python.org/3/library/pydoc.html https://docs.python.org/3/library/py_compile.html https://docs.python.org/3/library/quopri.html https://docs.python.org/3/library/re.html https://docs.python.org/3/library/sched.html https://docs.python.org/3/library/selectors.html https://docs.python.org/3/library/shelve.html https://docs.python.org/3/library/shutil.html https://docs.python.org/3/library/smtplib.html https://docs.python.org/3/library/sndhdr.html https://docs.python.org/3/library/socket.html https://docs.python.org/3/library/ssl.html https://docs.python.org/3/library/stat.html https://docs.python.org/3/library/stringprep.html https://docs.python.org/3/library/struct.html https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/telnetlib.html https://docs.python.org/3/library/tempfile.html https://docs.python.org/3/library/timeit.html https://docs.python.org/3/library/traceback.html https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/turtle.html https://docs.python.org/3/library/uuid.html https://docs.python.org/3/library/xdrlib.html Is it possible to add the specific source code link to all of this entries? ---------- assignee: docs@python components: Documentation files: missing_hint.py messages: 228563 nosy: Friedrich.Spee.von.Langenfeld, docs@python priority: normal severity: normal status: open title: Missing hint to source code - complete Added file: http://bugs.python.org/file36812/missing_hint.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
SilentGhost added the comment: Your code produces many false positives, would you care to reduce this list to only those modules that actually do not have a Source code link? ---------- nosy: +SilentGhost _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
R. David Murray added the comment: And that have source code. That said, some modules may have been consciously omitted. ---------- nosy: +r.david.murray, rhettinger _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Friedrich Spee von Langenfeld added the comment: I decremented the shown number of modules by 25. Here is the new list: https://docs.python.org/3/library/base64.html https://docs.python.org/3/library/binhex.html https://docs.python.org/3/library/bz2.html https://docs.python.org/3/library/cgitb.html https://docs.python.org/3/library/chunk.html https://docs.python.org/3/library/code.html https://docs.python.org/3/library/codecs.html https://docs.python.org/3/library/codeop.html https://docs.python.org/3/library/configparser.html https://docs.python.org/3/library/copy.html https://docs.python.org/3/library/copyreg.html https://docs.python.org/3/library/csv.html https://docs.python.org/3/library/datetime.html https://docs.python.org/3/library/decimal.html https://docs.python.org/3/library/difflib.html https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/formatter.html https://docs.python.org/3/library/getpass.html https://docs.python.org/3/library/imp.html https://docs.python.org/3/library/io.html https://docs.python.org/3/library/locale.html https://docs.python.org/3/library/lzma.html https://docs.python.org/3/library/macpath.html https://docs.python.org/3/library/mailbox.html https://docs.python.org/3/library/numbers.html https://docs.python.org/3/library/os.html https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/re.html https://docs.python.org/3/library/selectors.html https://docs.python.org/3/library/socket.html https://docs.python.org/3/library/stat.html # false positive https://docs.python.org/3/library/stringprep.html https://docs.python.org/3/library/struct.html https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/traceback.html https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/turtle.html https://docs.python.org/3/library/uuid.html @R. David Murray: I will try to look at the source code for each module, but I have updated the list to make it possible to work on this subject in the meantime. PS: Is it possible for me to edit my own contributions (comments, files) after uploading? ---------- Added file: http://bugs.python.org/file36817/missing_hint_v2.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: Messages and patches can be unlinked from an issue (and that action recorded in the history list) but not edited. Proofread before clicking. Originally, only a few module docs had a source code link where the source code was thought to be especially useful. When these links were determined to be useful, more have gradually been added as some developer thought of it or as some user requested, as you did in #22528. I don't know if any requests have been denied. I also don't know if there is a written policy anywhere. To me, it might be easier to link all python sources and be done with the issue. ---------- nosy: +terry.reedy _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Ezio Melotti added the comment: The source links have been added where the code proved to be readable, easy to understand, and self documenting, and have been omitted when the code is complicated and not self documenting. This has been done under the assumption that reading the code might help understanding the documentation, but there are other situations where users might want to see the code regardless of how readable it is, so I wouldn't object if you decide to add links to all the modules. ---------- nosy: +ezio.melotti stage: -> needs patch type: -> enhancement versions: +Python 3.4, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Alexander Belopolsky added the comment: +1 "to add links to all the [Python] modules" If the code is not readable, hard to understand or not self-documenting that's the reason to improve the code not to make it harder to find and see. ---------- nosy: +belopolsky _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Guido van Rossum added the comment: Actually uploading a patch for this should be easy, right? ---------- keywords: +easy nosy: +gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: Guido, should we take your comment as approving linking every source, as opposed to only some? This scope issue is the main sticking point for this issue. Otherwise, the standard patch is 3 easy new lines after the :synopsis: (and any 'New in Version x.y' note for the module -- see argparse). The following is Benjamin's patch for #22528. --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -6,6 +6,9 @@ .. module:: pdb :synopsis: The Python debugger for interactive interpreters. +**Source code:** :source:`Lib/pdb.py` + +-------------- .. index:: single: debugging Friedrich, were you planning to write a patch, once the list of modules is decided on? Or should we invite submissions on the core-mentorship list? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Guido van Rossum added the comment: It's fine to add a source link to any module for which there is Python source code. I suppose this adds a slight maintenance burden when a module moves (e.g. when a module is turned into a package, or when the subdirectory structure of the Lib directory changes). I'm a little confused about the "New in x.y" note -- why is that connected to the source code link? Of course, the source tells a different story from the docs -- e.g. undocumented implementation details may change, and sometimes the source is hard to understand (on occasion I've been confused myself :-). But Python is open source, so people can always read the source -- I don't see why we should try to make reading the source harder for people who don't yet have the chops to just read it on their own computer! ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: If there is a New-in note (which follows Synopsis), then (to be consistent) the Source note should follow it. If not (the normal situation), then Source follows Synopsis directly. A point in favor of linking all Python sources from the docs is that default single users installs are much harder to access on Windows than they used to be: /Users/Name/[Appdata - hidden by default]/Roaming?/Python/Pythonxy (I believe) versus the old /Pythonxy. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Terry J. Reedy <tjreedy@udel.edu>: ---------- versions: +Python 3.6 -Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Martin Panter <vadmium+py@gmail.com>: ---------- title: Missing hint to source code - complete -> Missing doc links to source code _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Yoni Lavi <yoni@lavi.fm>: ---------- nosy: +Yoni Lavi _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: I went over all the library rst files and added a link to the source for each of the modules. Also, I standardized the structure of the headers in all of the files to be in the following order: """ .. module .. moduleauthor .. sectionauthor .. versionadded **Source code:** .. testsetup .. index -------------- """ This'll be my first Python patch. Please let me know if I chose to go into too large of a scope and should split this, or if there's anything else I can improve. Also, since I'm modifying most of these files anyway, this might be a good opportunity to modify the header structure I mentioned above. Thanks ---------- keywords: +patch Added file: http://bugs.python.org/file42527/mywork.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Friendly ping ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Berker Peksag added the comment: Thanks for the patch, Yoni. Like Guido said in msg257586, we only want to add links for pure Python modules. Can you update your patch to remove C modules please? I'd prefer keep curses docs untouched. Lib/curses/* is basically a wrapper to _cursesmodule.c with some utility functions (sqlite3 is another example of this). Another candidates for removal are distutils.core, wsgiref, ensurepip, formatted (it's deprecated), test (Lib/test is basically the test suite of CPython) modules. ---------- nosy: +berker.peksag stage: needs patch -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Thank you for the review, Berker. I removed source code links from all the modules you specifically mentioned, and all links to C code, including stat.rst which already had such a link before. But I have to say that I don't quite understand the rationale here and (as a user) would personally prefer easy access to all source code, both C and Python. It is my opinion that a handy link to the source would help promote the conversion of library users to contributors. ---------- Added file: http://bugs.python.org/file42933/mywork2.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Ping. Please advise on next step to get this submitted. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Guido van Rossum <guido@python.org>: ---------- nosy: -gvanrossum _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: [David, could this be a sprint issue?] Here is the difference between python and C coded modules. The x.y docs are for all implementations of x.y. The python-coded modules are intended for use by all implementations, perhaps with minor modifications. In any case, the users of all implementation can presumably read Python code. The C implementations used by CPython are specific to CPython, and not necessarily meaningful to user of Jython, IronPython, PyPy, or any other alternatives. In any case, Guido has unambiguiously pronounced on linking to Python source files. Doing that is enough for one issue. I changed the title to be specific. ---------- title: Missing doc links to source code -> Missing doc links to source code for Python-coded modules. _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
R. David Murray added the comment: You mean someone doing a file review at the sprints? I can add it to the list. No guarantees about it getting committed at the sprints, though. If we are linking to the python version of modules that have both python and C versions, IMO the existence of the C version should be noted as a CPython implementation detail, since otherwise someone might think that the python code was what CPython was actually running. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
R. David Murray added the comment: I meant full review, not file review. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: If someone does a full review (looks at each diff in Rietveld to see if it matches the intended format -- see Yoni's 4-19 message, with :source:`Lib/mod.py` (lib2to3 is an exception) following **Source code:** -- and then downloads patch, applies, builds new html doc, and tests a few of the new links, and find nothing wrong, and says 'ready for commit review) then I may take this, re-check, and commit. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Thank you for looking at this. David, concerning modules that have multiple sources, are you referring to modules such as `stat` and `heapq`? In these, it's the Python source that tries to import the C module if it's available, so I'd consider it ok to leave just a link to the Python source. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
R. David Murray added the comment: Yes, those should be OK. I was thinking of _pyio and _pydecimal, but I don't know if you are linking to those. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Ah, thanks for the explanation. I think in this case we're ok too. I linked io.rst to Lib/io.py, which relies on the C version. And in the case of decimal.rst, it was already linked to Lib/decimal.py which very explicitly attempts to load the C version and falls back to _pydecimal. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Nathan Harold <penw81-ml@yahoo.com>: ---------- nosy: +nharold _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Nathan Harold added the comment: I'll give this a go, per Terry's instructions in msg266846. I've noticed up front that, due to other changes near the headers of a couple of files (specifically fcntl.rst and termios.rst), the patch doesn't apply cleanly anymore. Is it permissible for me to fix that myself and upload a new version, or is it preferable for a patch's original author to handle such things? In any case, there are plenty of other files for me to look over for now. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: For me, the issue is willingness, not permission ;-). Reviewing a patch includes somehow indicating changes needed to apply it 'today', and a revised patch is the best way. Please say whether the header changes are in 3.6 only (in which case a separate 3.5 patch is needed) or in both. 'Commit ready' means that you believe a core dev could (and should) download, apply without issue, and commit when satisfies. I should do so within a week if no one else does. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Nathan Harold added the comment: I went through all the diffs in Rietveld, checking it against msg228576 as I went. There were (naturally) more source links added than there are in that list from 2014, and there were also many more files with slight header reorganizations for consistency. Anything in the diffs I don't mention here looked fine to me. I also built the docs to test links in general and to see how some specific things here looked. A number of the new links are to __init__.py files, and others are to folders rather than individual source files. I noticed a precedent for the former in the case of the http module; for the latter I'm less certain, but I can see that it makes sense in some cases, e.g. when __init__.py is empty. I think these are probably good. One thing I particularly noticed was that in imp.rst and optparse.rst, the deprecation notice occurred below the source link, whereas the versionadded notices are always above it. I think it would be more consistent and look more natural to have the deprecation notices above the source links. This could be taken a step farther by suggesting that the source link always be the last visible thing before the horizontal line. The two other cases this would affect here are asyncio.rst (the notice that the module is provisional is between the source link and line) and getopt.rst (the note, now moved above the line, is after the source link). But this is all fairly subjective. Other details for individual files: marshal.rst - Here an index block a bit farther down into the text was moved up into the header. I think these might have been intended to point right where they were, not to the module as a whole; those things occur only in the paragraph (or two, in the case of 'code object') immediately following where the index originally pointed. unittest.mock-examples.rst - A versionadded notice was completely removed from this file. In fact, that was the only change to this file. I think I can see why you did this, since the actual API documentation for the library is in unittest.mock.rst, which has that same version notice. Smaller details: array.rst - -------------- was inserted above .. index: fcntl.rst - Patch appears to apply correctly, but with fuzz. os.path.rst - It might be good to put a space in "WindowsNT" termios.rst - Patch fails to apply because some other text near the header has changed recently. venv.rst - The source code link (to a folder) works, but is missing a trailing slash. Terry, I got the same results when I tried applying it to 3.5, so I think a single patch will do. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: Great report. It appears that the remaining questions are subjective decisions for a core developer: what is the best link, if any, for a package; should the link also be last thing in the section. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Thank you very much for the detailed review, Nathan. Attached is a new patch that resolves the merge issues, and some of the other issues you mentioned. Concerning the index reference in marshal.rst , ah, yes, I missed that it only related to a single paragraph. I reverted that change. Concerning versionadded in unittest.mock-examples.rst, you're right, I saw that this was already available elsewhere and removed it for consistency with other similar intro files. I'll revert if you tell me. I think I fixed all the smaller issues you mentioned, too. I wasn't sure about the fuzz; I didn't see any issue, so I assume it was automatically resolved by `hg update`. I don't have any input to give concerning the 'subjective' issues and will be happy to wait for a decision. ---------- Added file: http://bugs.python.org/file43252/mywork3.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Nathan Harold added the comment: I went over the sections I above called "Other/Smaller details" as addressed in mywork3.patch. I think they look fine. The patch applies seamlessly to both 3.5 and 3.6. For reference, since I noted them when I reviewed mywork2.patch last week, here are the packages at issue in the question about links to packages: Package docs in which the patch in this issue adds a link to __init__.py: dbm.rst, email.rst, importlib.rst, json.rst, msilib.rst, tkinter.rst, unittest.rst, xml.dom.rst, xml.sax.rst Package docs in which the patch in this issue adds a link to a directory: 2to3.rst, email.mime.rst, idle.rst, multiprocessing.rst, sqlite3.rst, urllib.rst, venv.rst, xml.rst Also, I noticed that asyncio.rst provides a precedent for linking to a source directory. So both ways have been done before. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Changes by Terry J. Reedy <tjreedy@udel.edu>: ---------- assignee: docs@python -> terry.reedy stage: patch review -> commit review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Roundup Robot added the comment: New changeset 222c1d461aa8 by Terry Jan Reedy in branch '3.5': Issue #22558: Add remaining doc links to source code for Python-coded modules. https://hg.python.org/cpython/rev/222c1d461aa8 New changeset 2a01d7a488e9 by Terry Jan Reedy in branch 'default': Merge Issue #22558. https://hg.python.org/cpython/rev/2a01d7a488e9 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: I committed the excellent patch. It should appear in the upcoming 3.5 and 3.6 releases, and within a day online. Thanks to both package author and also to reviewers. I would like to see more patches like this. I reverted the deletion of version added in mock examples. It is a separate page, so the reminder is not inappropriate. Minor nit: Yoni, you 'forgot' to run "python tools/scripts/patchcheck.py" on a patched repository. Not too surprisingly, the editing left trailing whitespace that needed to be removed. See the devguide for more. A possible follow-on. Some of the packages that document contained modules do so on the main page rather than in separate pages. If the source link is to the package __init__, rather than to the package directory, then it would be appropriate to add links to the module sections. The particular example I am thinking of is importlib, where we linked to .__init__. There may be another. Multiprocessing is not, because we linked to the directory. ---------- resolution: -> fixed stage: commit review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Thank you Terry, Nathan and David for your work on this. Sorry about the whitespace issue, I indeed forgot to run patchcheck when I prepared the 3rd patch. Tery, would you like me to prepare a patch for the follow-on you suggested? If so, given that this one is now marked as fixed, we should open a new bug, right? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Terry J. Reedy added the comment: Yes, new issue, make me nosy, assign to me if allowed, upload patch. Start with "This is a follow to #22558." ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
Yoni Lavi added the comment: Created issue 27304 and uploaded a patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue22558> _______________________________________
participants (12)
-
Alexander Belopolsky -
Berker Peksag
-
Ezio Melotti
-
Friedrich Spee von Langenfeld
-
Guido van Rossum
-
Martin Panter
-
Nathan Harold
-
R. David Murray
-
Roundup Robot
-
SilentGhost
-
Terry J. Reedy
-
Yoni Lavi