Hi, I had this on my draft for some time... so I am sending what I have, as I don't know when I will have time to write more. On Sat, 27 Aug 2022 at 00:40, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Fri, Aug 26, 2022 at 4:15 PM Glyph <glyph@twistedmatrix.com> wrote:
[snip]
So, in summary: I'd love to see Hypothesis help us discover and fix bugs but if it is a choice between occasionally helping us find those bugs but constantly blocking important work with difficult-to-debug failures in unrelated parsing code that is now covered by hypothesis, or "nothing", I'd go with "nothing".
I have been using something similar (but much more simpler) to hypothesis.strategies for a long time in my project. For example instead of using received_data = b'binary-data' I have something like this, where 'mk' is a singleton with all find of data generators. received_data = mk.binary() The `binary` will make sure to return a value that will fail an UTF-8 decoding. This helped me a lot to detect accidental handling binary data as text.
So, I think the following steps/policies would address all of the concerns described here:
Restrict Hypothesis usage to Twisted's own test suite (i.e., it is an implementation detail of a private part of Twisted, not the basis of any public interface)
+1
Turn off Hypothesis' deadline feature, probably everywhere
Maybe put a timeout of 15 seconds. If it takes more than 15 seconds to generate in test value, then something might be wrong with the generation code. This should help detect instances in which extreme value generation code is implemented.
At least for normal CI, and maybe for development environments, turn off Hypothesis' randomization feature.
Agree.
Perhaps set up a scheduled CI job that runs with randomization enabled.
We can have a scheduled test run on trunk with randomization... but at this point I don't know who will have the time to follow the result of those tests. A scheduled test run on trunk can also help detect test failures that are generated by external dependencies.
At least initially - and maybe indefinitely - avoid the stateful testing functionality.
Agree. We can take it easy at first :) --------------- A
Jean-Paul
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/HIEAR5OXSAJ... Code of Conduct: https://twisted.org/conduct
-- Adi Roiban