[docs] [issue15629] Add to regrtest the ability to run Lib and Doc doctests

[Ezio Melotti]

Ezio Melotti added the comment:

We discussed about this on IRC, and I'll try to summarize the outcomes.

If we want devs to run doctests, the syntax should be as simple as possible, ideally there shouldn't be any extra syntax -- at least for the common case.

Note that there are two kind of doctests:
  * the ones in the *.rst files in the Doc dir;
  * the ones in the docstrings of the *.py files in the Lib dir;

Some of the use cases are:
  1) being able to run all the doctests at once;
  2) being able to run individual doctests files;
  3) being able to discover/run doctests in a specified dir;

The use case 1) likely needs an extra flag (e.g. --docall), and could be done by default with -uall.

The use case 2) can be already done with `./python -m doctest path/to/doctest.rst`,  but, if doctest support is added to regrtest, it would be nice to have this too.
The following issues were discussed:
  * should regrtest accept them with no extra syntax (e.g. `./python -m test path/to/doctest.rst` -- this would be nice but probably tricky/hackish) or should they marked somehow (e.g. `fdoc:path/to/doctest.rst`, or `--doctests path/to/doctest1.rst path/to/doctest2.rst`)?
  * should we use paths (e.g. Doc/library/textwrap.rst -- easier) or dotted names (e.g. Doc.library.textwrap -- more consistent with unittests)?
  * should we require the full path/name, or just the filename like we do with regular unittests (e.g. `-m test test_textwrap`)?
  * should we use the fact that Doc/Lib or rst/py are in the name to distinguish the doctests, or can we search for the name in Lib/test/ first, and fallback on Doc/ and Lib/ if it's not there?

The use case 3) is less common, it can be provided with an additional flag but it's not strictly necessary IMHO (might be handy when we want to run all the doctests in a package).

Other issues that we haven't discussed are:
  * it would be nice to have something like run_doctests(['path/to/doctest.rst', 'path/to/module.py']) to be added in the test_module and be called automatically by `-m test test_module`;
  * which way should be used to mark the files/snippets as doctest-compatible;
  * how to have a doctest equivalent to setUp/tearDown (this is optional, we can always skip the test if it fails without initialization);

(FTR: I was trying the patch on 3.2 and since there were a few conflicts I uploaded the resulting patch in case anyone needs it -- this doesn't mean it should go on 3.2 though.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15629>
_______________________________________