pytest, fixture and parametrize
Hi All, I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality. Thoughts? Chuck
Hi all! I think it mainly depends on how likely we are to move away from Pytest. And I’m assuming not very likely. Anything likely to replace it (which won’t happen short-term in any case) will already have something similar, IMO. So my vote is we go ahead and use them. Hameer Abbasi
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on +0.5 for the fancy fixture stuff in pytest (have to concede that this stuff can be really hard to reason about sometimes, especially with custom scoping decorators, but if it helps to mock something like recently-discovered network dependency for some unit tests it may be better than adding a new dependency or reimplementing effectively the same thing) On Wed, 8 Aug 2018 at 09:09, Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
On Wed, Aug 8, 2018 at 9:25 AM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on
+1 from me, too.
I'm -0 on fixtures, but unfortunately there don't seem to be viable alternatives in pytest. I find the way that pytest automatically match *argument names* with the names of fixtures very distasteful and un-Pythonic.
On 08/08/18 17:34, Juan Nunez-Iglesias wrote:
We have these guidelines http://www.numpy.org/devdocs/reference/testing.html, or if you prefer the source https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt. It was updated for pytest in the 1.15 release, but could use some more editing and refinement. Matti
On Aug 8, 2018, at 5:40 PM, Matti Picus <matti.picus@gmail.com> wrote: We have these guidelines http://www.numpy.org/devdocs/reference/testing.html,
Thanks Matti — that’s clearly the place to document best practices.
It was updated for pytest in the 1.15 release, but could use some more editing and refinement.
Giving it s quick read right now, it clearly has not embraced pytest (yet?) Before we try to update the doc, we should have some guidelines as to the goals. At the broad level — do we want to make use of the pytest testing Framework, or just use pytest as a test runner? I would say that it’s a no brainer, except that he numpy testing framework already has a lot of the key features of pytest that we need. Options: 1) Use pytest only for test running 2) Use pytest features that numpy.testing does not have a good replacement for. 3) prefer pytest features for new tests. In all cases, I think we should limit ourselves to the less magical pytest features. BTW, I see this: “Setup and teardown functions to functions and methods are known as “fixtures”, and their use is not encouraged.” First, is that a typo? I don’t understand what it means. Though I think I know what a fixture is. Second — why are fixtures discouraged? Third, if fixtures are discouraged, rather than banned, we should say when it is OK to use one. -CHB
On Wed, Aug 8, 2018 at 9:38 AM, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
Stdlib unittest supports self.assertRaises context manager from python 3.1
but that requires using unittest :-) On Wed, Aug 8, 2018, 7:30 PM Eric Wieser <wieser.eric+numpy@gmail.com>
Perhaps we need a doc that lays out the prefered testing utilities. Worthy of a NEP? Or is just a README or something in the code base. Personally, I think a commitment to pytest is the best way to go -- but there are a lot of legacy tests, so there will be a jumble. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
as to whether we are should use pytest fixtures and parametrized classes and functions.
Absolutely!
the disadvantage is being dependent on pytest as unittest does not support that functionality.
Which is the whole point of using pytest, yes? I’m very opinionated about this, but I really dislike unittest — it’s simply way too Java-y — makes the easy things harder than they should be, and is missing critical should-be-easy features. I moved to pure pytest a few years ago, and have been very happy about it. In fact, I recently converted some d unittest code to pure pytest, and it was literally about 1/4 as much code. The only reason I can see to avoid pytest features is that it’s not in the standard lib — but it’s not a run-time dependency, and it’s easy to install and well supported — so all good. I suppose we may want to avoid some of pytest’s more magical esoteric features, but certainly not core functionality like fixtures and parameterized tests. -CHB
Hi all! I think it mainly depends on how likely we are to move away from Pytest. And I’m assuming not very likely. Anything likely to replace it (which won’t happen short-term in any case) will already have something similar, IMO. So my vote is we go ahead and use them. Hameer Abbasi
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on +0.5 for the fancy fixture stuff in pytest (have to concede that this stuff can be really hard to reason about sometimes, especially with custom scoping decorators, but if it helps to mock something like recently-discovered network dependency for some unit tests it may be better than adding a new dependency or reimplementing effectively the same thing) On Wed, 8 Aug 2018 at 09:09, Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
On Wed, Aug 8, 2018 at 9:25 AM Tyler Reddy <tyler.je.reddy@gmail.com> wrote:
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on
+1 from me, too.
I'm -0 on fixtures, but unfortunately there don't seem to be viable alternatives in pytest. I find the way that pytest automatically match *argument names* with the names of fixtures very distasteful and un-Pythonic.
On 08/08/18 17:34, Juan Nunez-Iglesias wrote:
We have these guidelines http://www.numpy.org/devdocs/reference/testing.html, or if you prefer the source https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt. It was updated for pytest in the 1.15 release, but could use some more editing and refinement. Matti
On Aug 8, 2018, at 5:40 PM, Matti Picus <matti.picus@gmail.com> wrote: We have these guidelines http://www.numpy.org/devdocs/reference/testing.html,
Thanks Matti — that’s clearly the place to document best practices.
It was updated for pytest in the 1.15 release, but could use some more editing and refinement.
Giving it s quick read right now, it clearly has not embraced pytest (yet?) Before we try to update the doc, we should have some guidelines as to the goals. At the broad level — do we want to make use of the pytest testing Framework, or just use pytest as a test runner? I would say that it’s a no brainer, except that he numpy testing framework already has a lot of the key features of pytest that we need. Options: 1) Use pytest only for test running 2) Use pytest features that numpy.testing does not have a good replacement for. 3) prefer pytest features for new tests. In all cases, I think we should limit ourselves to the less magical pytest features. BTW, I see this: “Setup and teardown functions to functions and methods are known as “fixtures”, and their use is not encouraged.” First, is that a typo? I don’t understand what it means. Though I think I know what a fixture is. Second — why are fixtures discouraged? Third, if fixtures are discouraged, rather than banned, we should say when it is OK to use one. -CHB
On Wed, Aug 8, 2018 at 9:38 AM, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
Stdlib unittest supports self.assertRaises context manager from python 3.1
but that requires using unittest :-) On Wed, Aug 8, 2018, 7:30 PM Eric Wieser <wieser.eric+numpy@gmail.com>
Perhaps we need a doc that lays out the prefered testing utilities. Worthy of a NEP? Or is just a README or something in the code base. Personally, I think a commitment to pytest is the best way to go -- but there are a lot of legacy tests, so there will be a jumble. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
as to whether we are should use pytest fixtures and parametrized classes and functions.
Absolutely!
the disadvantage is being dependent on pytest as unittest does not support that functionality.
Which is the whole point of using pytest, yes? I’m very opinionated about this, but I really dislike unittest — it’s simply way too Java-y — makes the easy things harder than they should be, and is missing critical should-be-easy features. I moved to pure pytest a few years ago, and have been very happy about it. In fact, I recently converted some d unittest code to pure pytest, and it was literally about 1/4 as much code. The only reason I can see to avoid pytest features is that it’s not in the standard lib — but it’s not a run-time dependency, and it’s easy to install and well supported — so all good. I suppose we may want to avoid some of pytest’s more magical esoteric features, but certainly not core functionality like fixtures and parameterized tests. -CHB
participants (11)
-
Charles R Harris -
Chris Barker -
Chris Barker - NOAA Federal
-
Eric Wieser -
Evgeni Burovski -
Hameer Abbasi -
Juan Nunez-Iglesias -
Matti Picus -
Ralf Gommers -
Stephan Hoyer -
Tyler Reddy