[issue42109] Use hypothesis for testing the standard library, falling back to stubs

[Zac Hatfield-Dodds]

Zac Hatfield-Dodds <zac.hatfield.dodds at gmail.com> added the comment:

> Okay, well, I'm trying to understand minithesis.py, but I am despairing. The shrinking code (really everything in class TestingState) is too much to grok. ... I suppose I have to read the paper at https://drmaciver.github.io/papers/reduction-via-generation-preview.pdf ... I am still unclear on some basics too, but presumably it's all related.

Shrinking is probably not important to understand, since it's *not* particularly related to anything but implementation details.  The abstract of that paper explains why the design is important - integrated shrinking means we can report minimal (easier-to-understand) failing examples with zero user effort, and guarantee that all shrunk examples could have been generated.

TestingState is, roughly, the `@given()` decorator internals: responsible for deciding how many times to run the wrapped test function, and how.  Note the three phases (generate, target, shrink) - the distribution of examples can depend on the behaviour of the test function on earlier inputs.  This is _more_ complex in Hypothesis and thus challenging to document.

The important concept is that input examples are generated by a Possibility (ie Strategy), which internally makes a sequence of choices; and that sequence is sourced from a maybe-empty known prefix (e.g. replaying failures, mutating previous inputs) with the suffix determined by a PRNG.


I feel like I'm explaining this badly, but there isn't a simple underlying thing to explain - Hypothesis works well because the user API is relatively simple, and then under the hood we have this many-layered stack of implementation details.  The synergies and coverage of otherwise-pathological cases this delivers is excellent, but "multiple years and PhDs worth of detailed synergistic work" is just intrinsically complicated.


> PS. What does "shortlex" mean?

Sort by length, and among equal-length strings sort in the normal (alphabetical) order.  For example: A < XY < ABC, etc.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42109>
_______________________________________